Application of x86 extensions for code protection

A new approach is proposed to protect the program code against reverse engineering tools, such as decompilers and symbolic execution tools. The approach is based on the usage of uncommon x86 processor instructions that could be implemented incorrectly in the aforementioned tools. Existing approaches to this problem are also considered, and the relative performance advantage of the proposed approach is noted. A method for numeric constants obfuscation, following this approach, is developed with the usage of AES-NI extension for the x86 architecture and its AESENC instruction in particular. This method is implemented for Clang compiler with the help of LLVM Intermediate Representation and tested against reverse engineering tools, such as IDA and Ghidra decompilers and angr symbolic execution tool.