Vision based Activity Detection is concerned with the automatic extraction, analysis and understanding of useful information from a sequence of images. It involves the development of a theoretical and algorithmic basis to achieve automatic visual understanding. As a scientific discipline, Vision based systems is concerned with the theory behind Artificial systems that extract information from images. The image data can take many forms, such as video sequences, or views from multiple cameras. Cameras provide very rich information about persons and environments, and their presence is becoming more important in everyday environments like airports, train and bus stations, malls, elderly care and even streets. Therefore, reliable Vision-based action detection system is required for various application like healthcare assistance system, crime detection and sports monitoring system in real time scenarios.Our Approach takes initial strides at designing and evaluating a Vision-based system for privacy ensured human activity monitoring. The proposed technology utilizing Artificial Intelligence (AI)-empowered proactive systems offering continuous monitoring, behavioral analysis, and modeling of human activities. To this end, We presents Single Run Action Detector (S-RAD) which is a real-time privacy-preserving action detector that performs end-to-end action localization and classification. It is based on Faster-RCNN combined with temporal shift modeling and segment based sampling to capture the human actions. Results on UCF-Sports and UR-Fall dataset present comparable accuracy to State-of-the-Art approaches with significantly lower model size and computation demand and the ability for real-time execution on edge embedded device (e.g. Nvidia Jetson Xavier).