Cloud computing has attracted a lot of research efforts in recent years.More and more companies start to move their data and operations to cloud computing environment. However, the security issues in cloud computing environment have not been studied to a sufficient depth.For cloud computing end users, they do not have enough technology or methods to verify security Service Level Agreement violation.Furthermore, there are weaknesses while malware detection mechanism and malware are running in the same virtual machine.For cloud providers, if they improve the security of virtual machines at the hypervisor level, they could provide more secure cloud computing environment for the end users.In this dissertation, we present a new understanding of security improvement in cloud computing environment.On both hypervisor and guest virtual machines, we propose mechanisms to detect malware or Service Level Agreement (SLA) violation in order to improve security in cloud computing environment.The key idea is to detect abnormal behaviors through memory analysis.We use techniques such as accumulated memory access latency, non-intrusive introspection of virtual machines, memory reconstruction,and cross-verification, to achieve this security improvement in cloud computing environment.In guest virtual machines, our mechanisms could detect unauthorized access to memory pages,violation of the memory deduplication policies, and under-allocation of memory to virtual machines.In hypervisor, we can detect malware through cross-verification among different components ofthe reconstructed execution states of the virtual machine,and the operation system of a virtual machine could be Linux or Windows.We implement our approaches on Xen and VMware, and experiment results show thatour detection mechanisms can effectively detect these abnormal behaviors with small increases in overhead.