Improved GraalVM Support

[JouperCoding]

Hello,

When using this library 7.5.0 and building it as an GraalVM native-image, the current implementation which leverages log4j is really not desired due to the amount of reflection and configuration needed. Is there any plans to change and maybe use slf4j / logback instead for better support?

For example, when creating a project you currently need the following two dependencies

implementation("nl.basjes.parse.useragent:yauaa:7.5.0")
runtimeOnly("org.apache.logging.log4j:log4j-to-slf4j:2.18.0")

After you have scanned and created your normal reflect-config, resource-config etc. You need to --initialize-at-build-time for all log4j classes that are used.

Below example is required configuration with GraalVM 22.2.0

       --initialize-at-build-time=nl.basjes,yggdrasil.api.services.useragent.UserAgentService,\
            org.apache.commons.logging.impl.SLF4JLocationAwareLog,\
            org.apache.logging.log4j.spi.AbstractLogger,\
            org.apache.logging.log4j.Level,\
            org.apache.logging.slf4j.SLF4JLogger,\
            org.apache.logging.slf4j.SLF4JLogger$1,\
            org.apache.logging.log4j.util.PropertiesUtil,\
            org.apache.logging.log4j.status.StatusLogger,\
            org.apache.logging.log4j.message.ReusableMessageFactory,\
            org.apache.logging.log4j.util.PropertySource$Util,\
            org.apache.logging.log4j.util.Strings,\
            org.apache.logging.log4j.util.Constants,\
            org.apache.logging.log4j.message.ParameterFormatter,\
            org.apache.logging.log4j.util.LoaderUtil,\
            org.apache.logging.log4j.spi.Provider,\
            org.apache.logging.log4j.util.ProviderUtil,\
            org.apache.logging.log4j.LogManager,\
            org.apache.logging.slf4j.SLF4JLoggerContextFactory,\
            org.apache.commons.logging.LogFactory,\
            org.apache.logging.log4j.spi.LoggerRegistry,\
            org.apache.commons.logging.impl.SLF4JLogFactory,\
            org.apache.logging.log4j.spi.StandardLevel,\
            org.apache.logging.log4j.util.OsgiServiceLocator \

[nielsbasjes]

Thanks for your input.
Exactly to make sure it works with Quarkus I included this: https://github.com/nielsbasjes/yauaa/tree/main/analyzer/src/it/Examples/quarkus

Also to show how to use a different logging implementation I have added https://github.com/nielsbasjes/yauaa/tree/main/analyzer/src/it/Examples/java-slf4j

I apparently have missed something for Quarkus. I'm going to need to do some experiments to understand your input.

Do you have your test somewhere that I can have a look at it?

[JouperCoding]

Hello @nielsbasjes,

I created an example project using GraalVM 22.2.0 & Micronaut 3.6.1 which can be built on windows. You will need the full GraalVM toolchains setup properly to build and compile as a native-image .exe.

https://github.com/JouperCoding/yauaa-graalvm-example

There is also a swagger with an API GET endpoint to test retrieving the parsed user agent details if you want to test it.

The configuration i'm not super happy about is in native-image.properties, see

--initialize-at-build-time
--initialize-at-run-time

Micronaut for example uses logback by default which is much nicer when it comes to compiling native-images in regards to logging.

[nielsbasjes]

NOTE: My knowledge and experience with GraalVM is extremely limited.

So I installed GraalVM 22.2 for Java 11 on my Linux machine.
Note that I never use gradle so it took be awhile to figure out which build command on gradle to get the artifact you created: gradle assembleShadowDist seems to do that.

Which gives an error about SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder"., probably because SLF4J is completely not backward compatible with anything existing.

cd yauaa-example/build/libs/

Now when I start it with the java -agentlib:native-image-agent=config-output-dir=META-INF/native-image -jar yauaa-example-0.1-all.jar it simply starts, logs the Yauaa version in the console and the swagger ui works. After some tests I do Ctrl-C to stop it.

Now when I try to build the native image

set USE_NATIVE_IMAGE_JAVA_PLATFORM_MODULE_SYSTEM=false
native-image -jar yauaa-example-0.1-all.jar

I get this output:

$ native-image -jar yauaa-example-0.1-all.jar
Warning: Using a deprecated option --allow-incomplete-classpath from 'META-INF/native-image/io.micronaut/micronaut-inject/native-image.properties' in 'file:///home/nbasjes/workspace/TryingTools/yauaa-graalvm-example/yauaa-example/build/libs/yauaa-example-0.1-all.jar'. Allowing an incomplete classpath is now the default. Use --link-at-build-time to report linking errors at image build time for a class or package.
WARNING: Unknown module: org.graalvm.nativeimage.llvm specified to --add-exports
WARNING: Unknown module: org.graalvm.nativeimage.llvm specified to --add-exports
WARNING: Unknown module: org.graalvm.nativeimage.llvm specified to --add-exports
========================================================================================================================
GraalVM Native Image: Generating 'yauaa-example' (executable)...
========================================================================================================================
[ 2022-08-28 13:46:22.674 UTC +0200 ] [ INFO ] [ main ] [ nl.basjes.parse.useragent.utils.YauaaVersion.logVersion 131 ] - 
[ 2022-08-28 13:46:22.676 UTC +0200 ] [ INFO ] [ main ] [ nl.basjes.parse.useragent.utils.YauaaVersion.logVersion 132 ] - /-----------------------------------------------------------\
[ 2022-08-28 13:46:22.676 UTC +0200 ] [ INFO ] [ main ] [ nl.basjes.parse.useragent.utils.YauaaVersion.logLine 159 ] - | Yauaa 7.5.0 (v7.5.0 @ 2022-08-22T13:02:39Z)               |
[ 2022-08-28 13:46:22.676 UTC +0200 ] [ INFO ] [ main ] [ nl.basjes.parse.useragent.utils.YauaaVersion.logVersion 134 ] - +-----------------------------------------------------------+
[ 2022-08-28 13:46:22.676 UTC +0200 ] [ INFO ] [ main ] [ nl.basjes.parse.useragent.utils.YauaaVersion.logLine 159 ] - | For more information: https://yauaa.basjes.nl             |
[ 2022-08-28 13:46:22.676 UTC +0200 ] [ INFO ] [ main ] [ nl.basjes.parse.useragent.utils.YauaaVersion.logLine 159 ] - | Copyright (C) 2013-2022 Niels Basjes - License Apache 2.0 |
[ 2022-08-28 13:46:22.676 UTC +0200 ] [ INFO ] [ main ] [ nl.basjes.parse.useragent.utils.YauaaVersion.logVersion 145 ] - \-----------------------------------------------------------/
[ 2022-08-28 13:46:22.676 UTC +0200 ] [ INFO ] [ main ] [ nl.basjes.parse.useragent.utils.YauaaVersion.logVersion 146 ] - 
[ 2022-08-28 13:46:22.893 UTC +0200 ] [ INFO ] [ main ] [ nl.basjes.parse.useragent.config.ConfigLoader.loadResources 285 ] - - Loaded 109 files in  214 ms using expression: classpath*:UserAgents/**/*.yaml
[ 2022-08-28 13:46:23.801 UTC +0200 ] [ INFO ] [ main ] [ nl.basjes.parse.useragent.analyze.UserAgentStringMatchMaker.finalizeLoadingRules 279 ] - Building all matchers for all possible fields.
[1/7] Initializing...                                                                                    (8.4s @ 0.31GB)
 Version info: 'GraalVM 22.2.0 Java 11 CE'
 Java version info: '11.0.16+8-jvmci-22.2-b06'
 C compiler: gcc (linux, x86_64, 9.4.0)
 Garbage collector: Serial GC
 4 user-specific feature(s)
 - io.micronaut.buffer.netty.NettyFeature
 - io.micronaut.core.graal.ServiceLoaderFeature
 - io.micronaut.http.netty.graal.HttpNettyFeature
 - io.micronaut.jackson.JacksonDatabindFeature

Fatal error: java.lang.IllegalAccessError: class io.micronaut.buffer.netty.NettyFeature (in unnamed module @0x4a9cb9d8) cannot access class com.oracle.svm.core.jdk.SystemPropertiesSupport (in module org.graalvm.nativeimage.builder) because module org.graalvm.nativeimage.builder does not export com.oracle.svm.core.jdk to unnamed module @0x4a9cb9d8
	at io.micronaut.buffer.netty.NettyFeature.beforeAnalysis(NettyFeature.java:66)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$9(NativeImageGenerator.java:722)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:78)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:722)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:564)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:521)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:407)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:585)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:128)
------------------------------------------------------------------------------------------------------------------------
                        0.9s (9.1% of total time) in 12 GCs | Peak RSS: 1.02GB | CPU load: 3.16
========================================================================================================================
Failed generating 'yauaa-example' after 8.6s.
Error: Image build request failed with exit status 1

So on my machine it fails over something I do not control.

[JouperCoding]

Hello @nielsbasjes

Yes the first command you used runs the application in scan mode, i.e this allows native-image to scan your code in runtime for configuration needed when building your app. Usually at this step is where you need to execute your code paths / hints to generate the configuration files
I.e it will create the reflect-config.json files and more for example. So it's good that that works for you.

The "cannot access class com.oracle.svm.core.jdk.SystemPropertiesSupport ..." when you try to build the code as native-image usually happens if you have not set the environment variable properly for your shell when building.

i.e

set USE_NATIVE_IMAGE_JAVA_PLATFORM_MODULE_SYSTEM=false

Needs to be set for you x86_x64 Cross Tools Comand Promt for VS 2017 before running native-image. You can either set the variable for your current terminal session or in your windows environment variables.

[nielsbasjes]

I think I see what went wrong. I use Linux instead of Windows.

[nielsbasjes]

I now have your example running on my Linux machine. Thank you for providing this example.

I found that indeed the result of the scan step is not enough to get it running. The settings in the provided native-image.properties (i.e. the --initialize-at-build-time= list) are not generated automatically. (I'm really curious to learn more about this: What is the way to determine this complex list of settings?)
So the barrier to use this seems very high to me right now.

About the logging framework.
I used to have Slf4j as the logging API. In June 2021 I switched to Log4j2 because of one main reason: At that point in time SLf4J had been dead in the water for more than 18 months. Between late 2019 and June 2021 there was effectively almost no development at all (only minor tweaks).
Recently Slf4j has been moving again. The latest 2.0.0 release turns out to be a true major release as it breaks everything. So slf4j 2.x causes more problems than I expected.

As you can understand I agree with the high barrier related to the Log4j2 if you want to use GraalVM. Yet slf4j 2.x seems to be less than ideal also.

From my perspective I think this relation between Log4j2 and GraalVM should be submitted as an improvement at the Log4j2 end.
Apparently they already have this under discussion:
https://issues.apache.org/jira/browse/LOG4J2-2649
https://issues.apache.org/jira/browse/LOG4J2-2604

From my end I currently think I'm going to

• try to include a basic example/documentation in the project that uses GraalVM (where I'll be looking closely at your example).
• stick to log4j and automatically update to the latest releases "fast". So as soon as this is solved at their end this project will benefit.

[nielsbasjes]

@JouperCoding
FYI: The Quarkus native image example has improved.

https://github.com/nielsbasjes/yauaa/blob/main/analyzer/src/it/Examples/quarkus/

If I run the integration tests (normal build) it puts all the versions in th pom.xml and stores it in the analyzer/target/it/Examples/quarkus directory.

In there I can now do mvn clean package -Pnative and it builds the native image which successfully runs: ./target/yauaa-example-quarkus-0.0.1-runner

It also creates a docker image that runs the same native image.
docker run -ti -p8080:8080 nielsbasjes/yauaa-quarkus:0.0.1

It is not perfect yet because the native integration tests do not yet work (resource loading issue).