Advancing Code Obfuscation: Novel Opaque Predicate Techniques to Counter Dynamic Symbolic Execution

Code obfuscation is a crucial technique for protecting software against reverse engineering and security attacks. Among various obfuscation methods, opaque predicates, which are recognized as flexible and promising, are widely used to increase control-flow complexity. However, traditional opaque predicates are increasingly vulnerable to Dynamic Symbolic Execution (DSE) attacks, which can efficiently identify and eliminate them. To address this issue, this paper proposes a novel approach for anti-DSE opaque predicates that effectively resists symbolic execution-based deobfuscation. Our method introduces two key techniques: single-way function opaque predicates, which leverage hash functions and logarithmic transformations to prevent constraint solvers from generating feasible inputs, and path-explosion opaque predicates, which generate an excessive number of execution paths, overwhelming symbolic execution engines. To evaluate the effectiveness of our approach, we implemented a prototype obfuscation tool and tested it against prominent symbolic execution engines. Experimental results demonstrate that our approach significantly increases resilience against symbolic execution attacks while maintaining acceptable performance overhead. This paper provides a robust and scalable obfuscation technique, contributing to the enhancement of software protection strategies in adversarial environments.