Power systems are continually subjected to wide range of disturbances and they must adjust to these changes and continue to operate satisfactorily. Small disturbance instability and transient instability are major threats to power system security which can cause out of step conditions. The majority of out-of-step protection systems installed throughout the world use local measurements that is voltage and current measurements at one end of transmission line. The disadvantage of this scheme is its inability to have knowledge about the other parts of the complex power network. Even though, the out-of-step tripping system is well designed and appropriately set, it still has a number of disadvantages in minimizing the effects of the disturbance. In this context, there is a need for Special Protection Schemes (SPS) which receive the data from more than one locations and process the decisions in wide area orientation. More recent technological advancements in microprocessor based relays, combined with GPS receivers for synchronization and accurate time stamping, are providing synchronized measurements called synchrophasors. In this thesis an out-of-step detection algorithm is proposed using system total transient energy based on the synchrophasor data obtained from phasor measurement units. Conventional out-of-step detection techniques are discussed and compared with proposed algorithm. The main advantage of the proposed method is that, it detects unstable swings faster than conventional methods. Further, a method for power system stabilizer (PSS) design is discussed and the performance of the performance of out-of-step protection with PSS is studied.