An energy management system (EMS) is a system designed for owners of generation companies (GenCos), operators of electricity markets and electric utility grids, as well as consumers and aggregators to monitor, control, and optimize the performance of the generation, transmission, and demand systems. This Ph.D. dissertation is devoted to studying the modeling and optimization of energy management problems associated with demand-side and supply-side of the electricity market comprising of a comprehensive comparison amongst several robust self-scheduling models of GenCos using real-world electricity market prices in which different self-scheduling strategies are proposed based on the price data and the generation company's desired robustness level. Moreover, to practically evaluate the performance of various methodologies, a post-optimization procedure has been proposed to determine the actual profit of each method in different real-market-environment cases. The conclusions drawn from the evaluations can help GenCos select and model the most appropriate non-deterministic self-scheduling approach based on the price information and price forecast method that they have adopted, as well as the robustness level that they desire in their solution. In addition, a novel integrated capacity market and demand response model is introduced as an alternative solution to the transmission expansion planning problem. This strategy includes the use of the demand response resources (DRRs) as power supply resources to participate in the capacity market. The proposed model is implemented on the Base Residual Auction (BRA) of 2020/2021 PJM capacity market real data, and the influence of several important parameters are investigated in detail. As a final step, a comprehensive transactive energy system framework is proposed for the integration of the aggregated load reduction demand response (DR) such as load curtailment and load shifting, and plug-in electric vehicle (PEV). It is observed that utilizing other distributed energy resources (DERs) including on-site renewable energy (RE) resources and behind-the-meter energy storage (ES) systems can increase the obtained profits from day-ahead electricity markets, and improve interactions of aggregators with retail customers.