Defending in-process memory abuse with mitigation and testing

Modern softwares often include large code bases from different origins with different trust levels. This creates a large attack surface and raises security concerns that sensitive information of one component is directly accessible by other (malicious or manipulated) components in memory. In this thesis, I refer to this problem as in-process memory abuse. Despite the prevalence of in-process abuses, the defense mechanisms are not well studied, due to the complex root causes and attack surfaces of such attacks. First of all, a large amount of the existing software is written in type-unsafe languages such as C and C++. Such languages are notorious for being error-prone. These programming errors have incurred countless high-severity security bugs that lead to in-process memory attacks. Secondly, contemporary defenses such as data execution prevention (DEP) and address space layout randomization (ASLR) have little effect on preventing in-process memory attacks. Last but not least, developers are often helpless when they are trying to protect their sensitive data, due to the lack of operating system support to create boundaries within the same process context. As a result, as long as one of the many components is successfully exploited, the whole program's sensitive data and code are subject to abuse. A common belief is that in-process abuse cannot be defended without high-overhead or loss of backward compatibility. To reduce memory corruption bugs, options like formally verifying every software or rewriting the whole software stack with type safe language is impractical, due to the poor scalability of formal verification methods and the immense engineering cost required to rebuild all existing software infrastructures. To prevent exploitations of memory corruption bugs, one may suggest adopting full memory safety by bound checking all the pointers and tracking the liveness of every allocated memory objects. However, this comes with intolerable overheads. Lastly, existing work proposes rewriting established operating system design paradigms to create sub-process isolation, this creates incompatibility and reduces practicability of the solution. Challenging these common beliefs, this thesis presents a series of practical defenses against in-process memory abuse. It includes runtime protections and offline bug detections. Collectively, these new techniques improved the state-of-the-art defense against in-process memory abuse without sacrificing practicability and compatibility. First, I present CCR, a compiler-binary rewriter toolchain to enable fine-grained soft- ware randomization. CCR solves the incompatibility of existing fine-grained randomization approaches by aligning its defense implementation with established software deployment and bug report paradigms. However, fine-grained randomization alone is still vulnerable to just-in-time info-leak aided code reuse attacks. To tighten this loose end, I introduce NORAX, a binary rewriting framework to retrofit execute-only memory (XOM) protection into source-unavailable programs. Then, I design shreds-fine-grained execution units with private memory-as an extra line of defense to in-process abuse. Shreds enable sub-process isolation without relying on nested paging, virtualization or even modified hardware. It incurs negligible overheads and is highly compatible with the existing operating system design paradigm of process/thread based execution units. Lastly, for offline software test generations, I present SAVIOR and MEUZZ, they are advanced hybrid fuzzing frameworks enlightened with bug-driven oracle to quickly find more bugs and machine learning guidance to learn from past fuzzing statistics to tune the fuzzing scheduling strategies. By designing and conducting the large-scale experiments for these proposed defenses on real-world softwares, I demonstrate that in-process memory abuses can be reasonably well-defended and prevented. The insights and knowledge gained during the development of this thesis have raised the community's awareness of in-process abuse and advanced the state-of-the-art defense against such attacks. Each of the included works have yielded at least one practical defense or automated software testing system. Many of them have also been adopted by the industry, blocking malicious in-process abuse attempts and uncovering highly severe security bugs in critical software infrastructure on daily basis, which highlights the broad impact of the thesis.--Author's abstract

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.