Fuzzing with AFL

Requirements

Let's get this out of the way first. There are some additional requirements to using AFL with JQF apart from what is required to build JQF.

OS

JQF's integration with AFL has been tested on MacOS and Ubuntu, though it should work on any other linux-based systems too. If you have a unix-based system where JQF-AFL does not work as intended, please raise an issue.

Installing AFL

Before using JQF to fuzz your Java programs with AFL, makes sure you have afl-fuzz installed. You can either do this in several ways:

• Download the latest tarball from the author's website.
  • Run make in that directory
• Clone the unofficial GitHub repo
  • Run make in that directory
• Install AFL via standard repositories
  • apt-get install afl on Ubuntu/Debian (check version; see below)
  • brew install afl-fuzz on MacOS with Homebrew

JQF requires at least AFL version 2.31b to work, though something more recent is preferred (JQF has been tested with AFL 2.51b at the time of this writing).

Setting AFL_DIR

JQF must be able to find AFL! If you installed afl-fuzz via standard repositories you may skip this step. JQF will either:

• look for the program afl-fuzz in the current PATH (type afl-fuzz in the command-line to see if it works; also check the minimum version as mentioned above), or
• look for the program afl-fuzz in the directory specified by the environment variable AFL_DIR.

The most common way to get started with JQF is to type in bash:

# Clone and build AFL
git clone https://github.com/mcarpenter/afl && (cd afl && make)

# Set AFL directory location
export AFL_DIR=$(pwd)/afl

# Clone and build JQF
git clone https://github.com/rohanpadhye/jqf && jqf/setup.sh

# Run JQF-AFL
jqf/bin/jqf-afl-fuzz

Running jqf-afl-fuzz

The program jqf-afl-fuzz is a wrapper around the program afl-fuzz that comes with AFL. The wrapper allows you to specify the classpath of your Java tests and then just the test class and method instead of having to invoke the JVM yourself. You can always pass additional options to the JVM (e.g. system properties) using the environment variable JVM_OPTS.

Usage: jqf-afl-fuzz [options] TEST_CLASS TEST_METHOD
Options: 
  -c JAVA_CLASSPATH  Classpath used to find your test classes (default is '.')
  -i AFL_INPUT_DIR   Seed inputs for AFL (default is a few seeds of random data)
  -o AFL_OUTPUT_DIR  Where AFL should save fuzz results (default is './fuzz-results')
  -x AFL_DICT        Provide a dictionary to AFL (default is no dictionary)
  -S SLAVE_ID        A unique identifier when running in parallel mode
  -T AFL_TITLE       Customize title banner (default is TEST_CLASS#TEST_METHOD)
  -m MEM_LIMIT       Set a memory limit in MB (default is 8192)
  -t TIMEOUT         Set a single-run timeout in milliseconds (default is 10000)
  -v                 Enable verbose logging (in file 'jqf.log')

The wrapper program takes arguments similar to afl-fuzz, for specifying the seed directory, results directory and other relevant AFL options.

Let's assume you have written a standalone JQF test (class MyTest with test method fuzzThis) in a directory called tests:

ls tests
> MyTest.java

javac -sourcepath tests -cp $(jqf/scripts/classpath.sh) tests/MyTest.java


ls tests
> MyTest.java MyTest.class

In this example, we use the handy script jqf/scripts/classpath.sh that expands to the classpath containing the JQF classes and its dependencies, so that your test compiles. In most situations, however, you would want to create tests inside your maven or gradle projects and simply add a dependency on JQF in your pom.xml or build.gradle. See Writing a JQF test for more details.

Finally, let's run jqf-afl-fuzz on our test class:

jqf/bin/jqf-afl-fuzz -c tests MyTest fuzzthis
> Performing pilot run...

The wrapper first performs a pilot run without AFL to ensure that you have set-up your classpath correctly, that JQF can load your test class, and that the test method is a public void method with @Fuzz annotations, and that junit-quickcheck can find generators for your arguments. If there was any problem, you should see an error message here. Otherwise, if all goes well, you'll see:

> Pilot run success! Launching AFL now...

And then the AFL status screen as it fuzzes your program. Good luck! AFL will save the inputs that increase code coverage as well as crashes in the directory fuzz-results (unless you specify a different output directory using the -o option).

Parallel mode

You can also run multiple instances of JQF-AFL in parallel to utilize multiple CPU cores and speed up fuzzing. Run each sequential instance with the option -S <id>, where <id> is some unique identifier for that instance. For example, the following spaws two parallel instances:

jqf/bin/jqf-afl-fuzz -c tests -S id1 MyTest fuzzthis
jqf/bin/jqf-afl-fuzz -c tests -S id2 MyTest fuzzthis

You might have to run the above commands in separate terminals/screens. The output for instance id1 will be in directory fuzz_results/id1 and so on. Remember that you can change the root output directory with the -o option.

Troubleshooting

If the pilot run fails, you should see a stacktrace of the exception that caused this on the screen. Typical reasons for the pilot run failing is providing a wrong or misspelt class name or method name, or providing an incorrect classpath, resulting in a ClassNotFoundException.

You can also use the -v option with jqf-afl-fuzz to make JQF dump logs of certain things in the file jqf.log, such as the bytecode instrumentation of class files. You will see any errors due to instrumentation in this file, along with anything your test classes write to STDOUT or STDERR.

If you cannot see any errors or the error does not seem to be your fault, please open a GitHub issue.

Analyzing crashes (and other fuzzed inputs)

You can reproduce one or more inputs found by AFL using the program jqf-repro:

jqf/bin/jqf-repro -c tests MyTest fuzzThis fuzz-results/crashes/*

The repro script will re-execute the test method with the provided inputs, and print exception stacktraces, if any, on the standard error stream.

If you would like to examine the execution in more detail, you can also modify the @Fuzz annotation to specify the input files to repro; this is useful if you want to perform step-through debugging in an IDE such as IntelliJ or Eclipse, as the @Fuzz tests can usually be run at the click of a button (since they are JUnit tests).

@Fuzz(repro="fuzz-results/crashes/id:000001")
public void fuzzSimple(...) {
   ...
}

A note on multi-threading

Multi-threaded tests can be difficult to work with. If your test method spawns new threads to do work, make sure to synchronize with these threads before returning (e.g. using Thread.join()). If a worker thread continues to run after the test method returns, then its code coverage can spill onto subsequent trials (with different inputs) and the fuzzing session will become unreliable.