Experiences in using Reinforcement Learning for Directed Fuzzing

Abstract

Directed testing is a technique to analyze user-specified target locations in the program. It reduces the time and effort of developers by excluding irrelevant parts of the program from testing and focusing on reaching the target location. Existing tools for directed testing employ either symbolic execution with heavy-weight program analysis or fuzz testing mixed with hand-tuned heuristics. In this thesis, we explore the feasibility of using a data-driven approach for directed testing. We aim to leverage the data generated by fuzz testing tools. We train an agent on the data collected from the fuzzers to learn a better mutation strategy based on the program input. The agent then directs the fuzzer towards the target location by instructing the optimal action for each program input. We use reinforcement learning based algorithms to train the agent. We implemented a prototype of our approach and tested it on synthetic as well as real-world programs. We evaluated and compared different reward functions. In our experiments, we observe that for simple synthetic programs, our approach can reach the target location with fewer mutations compared to AFL and AFLGo that employ random mutations. However, for complex programs, the results are mixed. No one technique can perform consistently for all programs.