VMGuards: A Novel Virtual Machine Based Code Protection System with VM Security as the First Class Design Concern

Process-level virtual machine (PVM) based code obfuscation is a viable means for protecting software against runtime code tampering and unauthorized code reverse engineering. PVM-based approaches rely on a VM to determine how instructions of the protected code region are scheduled and executed. Therefore, it is crucial to protect the VM against runtime code tampering that alters the instructions and behavior of the VM. This paper presents VMGuards, a novel PVM-based code protection system that puts the security of VM as the first class design concern. Our approach advances prior work by promoting security of the VM as the first class design constraint. We achieve this by introducing two new instruction sets to protect the internal implementations of critical code segments and the host runtime environment where the VM runs in. Our new instruction sets not only have an identical code structure as standard virtual instructions, but also provide additional information to allow the VM to check whether the critical internal implementation or the runtime environment is affected. We evaluate our approach by using a set of real-life applications. Experimental results show that our approach provides stronger and more fine-grained protection when compared to the state-of-the-art with little extra overhead.