Measure and investigate heap memory allocation of Apache Maven

General setup • Benchmark heap allocation of several Maven releases

Investigate where heap allocation comes from • Perspectives • Contributors • License

This project is a test bench based on QuickPerf to benchmark and understand heap memory allocation caused by mvn validate on a project source code.

Feel free to use this project re-execute measures, play with it on measuring any build, tweak its code and contribute back to it!

Overview

This project contains two types of measures against Maven build execution, each provided as a JUnit unit test (in src/test/java):

• MvnValidateProfilingTest can be used to investigate the origin of memory allocation (using QuickPerf @ProfileJvm) during one build with a given Maven version,
• MvnValidateAllocationByMaven3VersionTest can be used to measure the heap allocation (using QuickPerf @MeasureHeapAllocation) for every Maven release in a version range.

General setup

This general setup part describes configuration common to both tests, done in src/test/resourrces/maven-bench.properties file:

• the testing.project.path represents the path of the project on which mvn validate will be applied,
• the maven.binaries.path property corresponds to the path where the needed Maven distributions will be automatically downloaded by the tests. The other properties are only used by MvnValidateAllocationByMaven3VersionTest.

The measures shown here are done against the Apache Camel project because it contains more than 800 Maven modules: such a huge build is perfect to get significant measures. But you can choose your own target.

Note: currently, every day, a job @TravisCI run org.quickperf.maven.bench.MvnValidateMaxAllocation in order to watch over Apache Maven project and observe if new features on Apache Maven needs more memory then yesterday.

For reproducibility of our measures, a precisely defined commit of this project was chosen:

git clone -n https://github.com/apache/camel.git
git checkout c409ab7aabb971065fc8384a861904d2a2819be5

If you want to apply measures on build done with Maven HEAD, that cannot be downloaded from public releases, you can execute the following commands where {maven-distrib-location} has to be replaced with the url given by the maven.binaries.path property of maven-bench.properties file:

git clone https://github.com/apache/maven.git
cd maven
mvn -DdistributionTargetDir="{maven-distrib-location}/apache-maven-head" package

Heap size is fixed with the help of @HeapSize.

Contributing

Building the project

git clone https://github.com/pcavezzan/maven-test-bench.git
make build

The above commandline will run behind the scene several actions:

• run mvn package -B

During the test phase, we are going to :

• clone Apache Camel project (commit c409ab7aabb971065fc8384a861904d2a2819be5) into test-classes/camel
• install different version of maven into test-classes/maven
  • for Releases version, we are going to :
    • download releases from https://archive.apache.org/dist/maven/maven-3/<version>/binaries/apache-maven-<version>-bin.zip
    • unzip into test-classes/maven/apache-maven-<version>
  • for HEAD version, we are going to :
    • clone Apache Maven mainstream from https://gitbox.apache.org/repos/asf/maven-sources.git
    • build from source Apache Maven latest development version,
    • rename the built maven to test-classes/maven/apache-maven-master,

Note: the build above could be stuck depending on your machine settings. If it is the case, I would suggest you to create a custom build (see below).

Custom Build

If you prefer to override some settings without editing the default configuration (maven-bench.propertiers), you have several options:

• environment variable
• you can create a local configuration files local.maven-bench.properties without any risk to version it because this file is ignored by GIT. In this file, just override the settings you want.

Overriding by setting environment variable

export MAVEN_VERSION_FROM=3.6.1
export MAVEN_VERSION_TO=3.6.2
make build

Overriding by creating local.maven-bench.properties

cat << EOF > src/main/resources/local.maven-bench.properties
maven.version.from=3.6.1
maven.version.to=3.6.2
EOF
make build

Running only some tests

This project contains a Makefile to easily orchestrate how to build or run specific test. To get more info, do not hesitate to run basic make or make help:

$ make 
build                          Build project with running all tests (basically run `mvn package`)
clean                          Cleanup project files (basically run `mvn clean`)
runMeasures                    Running only measures
runValidateMaxAllocation       Running only memory allocation needed for last commit from Maven GIT Repository on master branch

Memory allocation on head maven version

$ make runValidateMaxAllocation

Measures maven version

$ make runMeasureOnHead

Developing

If you want to help us and make some code, you can easily get the project and open it up with your favorite IDE.

NOTE: if you want to debug head version of Maven

First configure the project to use only head version of maven. For example, by creating a local.maven-bench.properties in src/main/resources directory:

cat << EOF > src/main/resources/local.maven-bench.properties
maven.version.from=head
maven.version.to=head
EOF
make build

Then just open your favorite IDE or running your test.

Benchmark heap allocation of several Maven releases

org.quickperf.maven.bench.MvnValidateAllocationByMaven3VersionTest test allows to benchmark the heap allocation level on several Maven 3 distributions.

Heap allocation level is measured with the help of @MeasureHeapAllocation QuickPerf annotation. This annotation measures the heap allocation level of the thread running the method annotated with @Test. Feel free to contribute to QuickPerf by adding a feature allowing to measure the allocation level aggregated across all the threads! With mvn validate, we have checked that Maven code is not multithreaded during this validate phase by profiling the JVM with the help of @ProfileJvm.

Please read General setup to get some of the setup requirements.

You also have to give a value for the following properties contained in the maven-bench.properties file:

• maven.version.from
• maven.version.to
• warmup.number
• measures.number-by-maven-version

The meaning of these properties is given in the maven-bench.properties file.

Measures can be launched with this command line: mvn -Dtest=org.quickperf.maven.bench.MvnValidateAllocationByMaven3VersionTest test. Before doing it, you can close your IDE, web browser or other applications to free memory.

The benchmark results are exported into a maven-memory-allocation-{date-time}.csv file. The execution context (processor, OS, ...) is reported in an execution-context-{date-time}.txt file.

For several Maven versions, the following graph gives the average of ten heap allocations caused by the application of mvn validate on Apache Camel:

For this graph, you can consult:

• the measures
• the execution context

Measures took around one hour and a quarter.

From Maven versions 3.2.5 to 3.6.2, heap allocation level is the highest with Maven 3.2.5 and the smallest with Maven 3.6.2. The heap allocation decreases from ~7 Gb with Maven 3.6.1 to ~3 Gb with Maven 3.6.2.

Control and reduce heap allocation is an important matter for Maven project. Indeed, a part of the heap allocation is going to be garbage collected and the garbage collection activity is succeptible to slow down your build. In addition, less heap allocation means that you may execute Maven with a smaller heap size.

But where the allocation comes from? In the following part we will see how to spot the Java methods allocating a lot.

Investigate where heap allocation comes from

You can use org.quickperf.maven.bench.MvnValidateProfilingTest to understand the origin of heap allocation. Some of the set up requirements can be found in General setup part.

The Maven version under test can be set with the MAVEN_3_VERSION constant:

    public static org.quickperf.maven.bench.projects.Maven3Version MAVEN_3_VERSION = org.quickperf.maven.bench.projects.Maven3Version.V_3_6_2;

A test method is annotated with @ProfileJvm to profile the test method with Java Flight Recorder (JFR).

The JFR file location is going to be displayed in the console:

[QUICK PERF] JVM was profiled with Java File Recorder (JFR).
The recording file can be found here: C:\Users\JEANBI~1\AppData\Local\Temp\QuickPerf-46868616\jvm-profiling.jfr
You can open it with Java Mission Control (JMC).

You can open it with Java Mission Control (JMC) to discover the methods contributing the most to heap allocation.

Below a JFR file for Maven 3.2.5 and opened with JMC 5.5:

By the way, you can also benefit from an automatic performance analysis with @ExpectNoJvmIssue. For example, the following warning is reported with Maven 3.2.5:

Rule: Thrown Exceptions
Severity: WARNING
Score: 97
Message: The program generated 20 482 exceptions per second during 26,722 s starting at 
03/09/19 17:08:31.

Perspectives

We have developed a test bench that is able to compare the heap allocation level between several Maven versions. We also have given a method to understand the origin of heap allocation.

Feel free to play with this bench and QuickPerf, to perform measures (heap allocation, execution time, ...) with different plugins/goals, use different JDK or garbage collectors, ..., suggest new ideas, create new features or share your measures with PR! Some issues are also available here!

You also have QuickPerf issues to build new performance tools!

Contributors

Many thanks to all our contributors!

Jean Bisutti 🤔 💻 📖 👀

Hervé Boutemy 🤔 📖

Alberto Martinelli 💻

Patrice Cavezzan 💻

emoji key

License

Apache License 2.0