Fuzzing with AFL

Requirements

Let's get this out of the way first. There are some additional requirements to using AFL with JQF apart from what is required to build JQF.

OS

JQF's integration with AFL has been tested on MacOS and Ubuntu, though it should work on any other linux-based systems too. If you have a unix-based system where JQF-AFL does not work as intended, please raise an issue.

Installing AFL

Before using JQF to fuzz your Java programs with AFL, makes sure you have afl-fuzz installed. You can either do this in several ways:

• Download the latest tarball from the author's website.
  • Run make in that directory
• Clone the unofficial GitHub repo
  • Run make in that directory
• Install AFL via standard repositories
  • apt-get install afl on Ubuntu/Debian (check version; see below)
  • brew install afl-fuzz on MacOS with Homebrew

JQF requires at least AFL version 2.31b to work, though something more recent is preferred (JQF has been tested with AFL 2.51b at the time of the writing).

Setting AFL_DIR

JQF must be able to find AFL! If you installed afl-fuzz via standard repositories you may skip this step. JQF will either:

• look for the program afl-fuzz in the current PATH (type afl-fuzz in the command-line to see if it works; also check the minimum version as mentioned above), or
• look for the program afl-fuzz in the directory specified by the environment variable AFL_DIR.

The most common way to get started with JQF is to type in bash:

# Clone and build AFL
git clone https://github.com/mcarpenter/afl && (cd afl && make)

# Set AFL directory location
export AFL_DIR=$(pwd)/afl

# Clone and build JQF
git clone https://github.com/rohanpadhye/jqf && jqf/setup.sh

# Run JQF-AFL
jqf/bin/jqf-afl-fuzz

Running jqf-afl-fuzz

The program jqf-afl-fuzz is a wrapper around the program afl-fuzz that comes with AFL. The wrapper allows you to specify the classpath of your Java tests and then just the test class and method instead of having to invoke the JVM yourself. You can always pass additional options to the JVM (e.g. system properties) using the environment variable JVM_OPTIONS.

Usage: jqf-afl-fuzz [options] TEST_CLASS TEST_METHOD
Options: 
  -c JAVA_CLASSPATH  Classpath used to find your test classes (default is '.')
  -i AFL_INPUT_DIR   Seed inputs for AFL (default is a few seeds of random data)
  -o AFL_OUTPUT_DIR  Where AFL should save fuzz results (default is './fuzz-results')
  -x AFL_DICT        Provide a dictionary to AFL (default is no dictionary)
  -T AFL_TITLE       Customize title banner (default is TEST_CLASS#TEST_METHOD)
  -m MEM_LIMIT       Set a memory limit in MB (default is 8192)
  -v                 Enable verbose logging (in file 'jqf.log')

The wrapper program takes arguments similar to afl-fuzz, for specifying the seed directory, results directory and other relevant AFL options.

Let's assume you have written a standalone JQF test (class MyTest with test method fuzzThis) in a directory called tests:

mkdir tests && vi tests/MyTest.java  # See article on 'Writing a JQF test'
javac -sourcepath tests -cp $(jqf/scripts/classpath.sh) tests/MyTest.java

In this example, we use the handy script jqf/scripts/classpath.sh that expands to the classpath containing the JQF classes and its dependencies, so that your test compiles. In most situations, however, you would want to create tests inside your maven or gradle projects and simply add a dependency on JQF in your pom.xml or build.gradle. See Writing a JQF test for more details.

Finally, let's run jqf-afl-fuzz on our test class:

jqf/bin/jqf-afl-fuzz -c tests MyTest fuzzthis
> Performing pilot run...

The wrapper first performs a pilot run without AFL to ensure that you have set-up your classpath correctly, that JQF can load your test class, and that the test method is a public void method with @Fuzz annotations, and that junit-quickcheck can find generators for your arguments. If there was any problem, you should see an error message here. Otherwise, if all goes well, you'll see:

> Pilot run success! Launching AFL now...

And then the AFL status screen as it fuzzes your program. Good luck! AFL will save the inputs that increase code coverage as well as crashes in the directory fuzz-results (unless you specify a different output directory using the -o option).

Troubleshooting

If the pilot run fails, you should see a stacktrace of the exception that caused this on the screen. Typical reasons for the pilot run failing is providing a wrong or misspelt class name or method name, or providing an incorrect classpath, resulting in a ClassNotFoundException.

You can also use the -v option with jqf-afl-fuzz to make JQF dump logs of certain things in the file jqf.log, such as the bytecode instrumentation of class files. You will see any errors due to instrumentation in this file, along with anything your test classes write to STDOUT or STDERR.

If you cannot see any errors or the error does not seem to be your fault, please open a GitHub issue.

Analyzing crashes (and other fuzzed inputs)

You can reproduce one or more inputs found by AFL using the program jqf-repro:

jqf/bin/jqf-afl-repro -c tests MyTest fuzzThis fuzz-results/crashes/*

The repro script will re-execute the test method with the provided inputs, and print exception stacktraces, if any, on the standard error stream.

If you would like to examine the execution in more detail, you can also modify the @Fuzz annotation to specify the input files to repro; this is useful if you want to perform step-through debugging in an IDE such as IntelliJ or Eclipse, as the @Fuzz tests can usually be run at the click of a button (since they are JUnit tests).

@Fuzz(repro=fuzz-results/crashes/id:000001)
public void fuzzSimple(...) {
   ...
}

A note on multi-threading

jqf-afl-fuzz currently does not like multi-threading. Although multi-threaded programs will work just fine, only the coverage in the main thread will be reported back to AFL. Make sure that the functionality you want to fuzz/test is in the main thread. Also, make sure to Thread.join with any newly created threads before returning from the test method, to prevent resource leaks.