The dissertation explores the challenges and transformations in modernizing the electrical grid, characterized by increased electric power grid interconnectivity, the widespread integration of Distributed Generation (DG), and frequent network reconfigurations. These transformations pose significant challenges to traditional grid technologies and operations, including power quality issues, protection scheme challenges, and complications in energy market dynamics. This study delves into the challenges of power system protection schemes from two perspectives: the misoperation of directional elements and the miscoordination of directional overcurrent elements. The proper operation of the protection system is critical to ensuring grid reliability.The first perspective investigates the misoperation of directional elements; we model and analyze the fault behaviors of various generators, including Synchronous and Inverter-Based Generators (IBDGs) with differing control architecture, to comprehensively understand their fault characteristics. Furthermore, we explore the misoperation of negative sequence directional elements, proposing and validating a prevention strategy using Real-Time Hardware-in-the-Loop (RT-HIL) setups.The second perspective addresses the coordination issues of Directional Overcurrent Relays (DOCRs) and aims to minimize their operating times. The dissertation illustrates the advantages of employing optimization algorithms over numerical iteration methods for relay coordination. It examines the coordination performance using Genetic Algorithms (GA) and Particle Swarm Optimization (PSO), presenting an enhanced variation of PSO that yields improved performance validated through virtual HIL setups.Additionally, the dissertation investigates the issue of DOCR miscoordination due to variations in fault current during fault isolation. It introduces a dynamic Time-Current Characteristic (TCC) formulation adapted to these variations, validated using IEEE test systems. It also investigates DG instability and miscoordination stemming from changes in network topology and generation short circuit capacity. A prevention strategy that combines clustering and optimization algorithms is proposed and validated. Moreover, a co-optimization strategy is presented and validated to prevent DOCR miscoordination while maintaining DG stability, ensuring that the Critical Clearing Time (CCT) associated with a fault is greater than the operating time of the relays assigned to isolate the fault.This work significantly advances the understanding of how grid modernization impacts power system protection and lays the groundwork for future research in this evolving field. It highlights the need for a collaborative approach between inverter manufacturers and protection engineers to facilitate a seamless and reliable grid transformation.