How to make it work in streaming application

[cpanpalia]

When we use user agent vs when we don’t use it in streaming mode, the memory usage of application keeps growing. what we have implemented is

` object UaService extends Serializable{
val uaa: UserAgentAnalyzer = UserAgentAnalyzer
.newBuilder()
.hideMatcherLoadStats()
.withCache(10000)
.build();

def getValueFromUAString(uaString: String): UserAgentClass = {
  if (uaString != null){
    val parsedAgent = userAgentAnalyzer.parse(uaString)
    UserAgentClass(
      parsedAgent.getValue("DeviceName"),
      parsedAgent.getValue("DeviceClass"),
      parsedAgent.getValue("DeviceBrand"),
      parsedAgent.getValue("OperatingSystemName"),
      parsedAgent.getValue("OperatingSystemNameVersion"),
      parsedAgent.getValue("AgentName"),
      parsedAgent.getValue("AgentVersion")
    )
  } else null
}

}

val convertUDF: UserDefinedFunction = udf((uaString:String) => {
UaService.getValueFromUAString(uaString)
})

def prepareForSilver(bronzeDF: DataFrame): DataFrame = {
bronzeDF.withColumn("useragent_string", col("value1.context.userAgent"))
}
`

other option I am trying out now is

val userAgentAnalyzer: UserAgentAnalyzer = UserAgentAnalyzer .newBuilder.hideMatcherLoadStats.withField("DeviceName").withField("DeviceClass").withField("DeviceBrand").withField("OperatingSystemName").withField("OperatingSystemNameVersion").withField("AgentName").withField("AgentVersion").build

How to avoid memory leakage, with this user agent parser. (I am almost certain memory leakage is happening here only)

[nielsbasjes]

Does your memory usage keep growing after 10000 uniquely different useragents have passed through?

[nielsbasjes]

Yauaa should have a total memory usage (including caching) of about 150MiB.

At least that is what is reported by this test:
https://github.com/nielsbasjes/yauaa/blob/main/analyzer/src/test/java/nl/basjes/parse/useragent/profile/TestMemoryFootprint.java

These also show no leakage after 1M useragents.

Just to be sure: you are reusing the same instance of the UserAgentAnalyzer?
If you are creating a new instance on every batch then that would explain this perfectly.
Every 9 minutes adding ~150MiB = ~ 1GiB/hour extra

[cpanpalia]

just to be sure, how do I reuse same instance of UserAgentAnalyzer?
In previous version of code it was not written perfectly. But what I am doing now is creating the UDF once and reusing same UDF. AM I in correct direction.

[nielsbasjes]

So I ran a memory footprint test 1 M times, at every 10K I do a GC and check the memory usage.
I remains stable at about 144MiB, no leakage.

After  930000 iterations and GC --> Used memory is  151498856 bytes (  144 MiB), Average time per parse  443621 ns ( ~ 0.444 ms)
After  931000 iterations        --> Used memory is  407356400 bytes (  388 MiB), Average time per parse  364698 ns ( ~ 0.365 ms)
After  932000 iterations        --> Used memory is  346045224 bytes (  330 MiB), Average time per parse  361720 ns ( ~ 0.362 ms)
After  933000 iterations        --> Used memory is  285212456 bytes (  271 MiB), Average time per parse  369510 ns ( ~ 0.370 ms)
After  934000 iterations        --> Used memory is  222572536 bytes (  212 MiB), Average time per parse  360488 ns ( ~ 0.360 ms)
After  935000 iterations        --> Used memory is  161341792 bytes (  153 MiB), Average time per parse  373536 ns ( ~ 0.374 ms)
After  936000 iterations        --> Used memory is  417194376 bytes (  397 MiB), Average time per parse  358136 ns ( ~ 0.358 ms)
After  937000 iterations        --> Used memory is  354256512 bytes (  337 MiB), Average time per parse  359319 ns ( ~ 0.359 ms)
After  938000 iterations        --> Used memory is  293569088 bytes (  279 MiB), Average time per parse  355610 ns ( ~ 0.356 ms)
After  939000 iterations        --> Used memory is  230818952 bytes (  220 MiB), Average time per parse  365660 ns ( ~ 0.366 ms)
After  940000 iterations and GC --> Used memory is  151503664 bytes (  144 MiB), Average time per parse  370379 ns ( ~ 0.370 ms)
After  941000 iterations        --> Used memory is  407361424 bytes (  388 MiB), Average time per parse  365933 ns ( ~ 0.366 ms)
After  942000 iterations        --> Used memory is  346034600 bytes (  330 MiB), Average time per parse  369414 ns ( ~ 0.369 ms)
After  943000 iterations        --> Used memory is  285197360 bytes (  271 MiB), Average time per parse  358244 ns ( ~ 0.358 ms)
After  944000 iterations        --> Used memory is  222436312 bytes (  212 MiB), Average time per parse  390709 ns ( ~ 0.391 ms)
After  945000 iterations        --> Used memory is  478288856 bytes (  456 MiB), Average time per parse  400165 ns ( ~ 0.400 ms)
After  946000 iterations        --> Used memory is  303962080 bytes (  289 MiB), Average time per parse  358142 ns ( ~ 0.358 ms)
After  947000 iterations        --> Used memory is  559818712 bytes (  533 MiB), Average time per parse  355634 ns ( ~ 0.356 ms)
After  948000 iterations        --> Used memory is  383660144 bytes (  365 MiB), Average time per parse  368375 ns ( ~ 0.368 ms)
After  949000 iterations        --> Used memory is  207941200 bytes (  198 MiB), Average time per parse  365319 ns ( ~ 0.365 ms)
After  950000 iterations and GC --> Used memory is  151502792 bytes (  144 MiB), Average time per parse  359600 ns ( ~ 0.360 ms)
After  951000 iterations        --> Used memory is  407360800 bytes (  388 MiB), Average time per parse  369741 ns ( ~ 0.370 ms)
After  952000 iterations        --> Used memory is  345925264 bytes (  329 MiB), Average time per parse  357176 ns ( ~ 0.357 ms)
After  953000 iterations        --> Used memory is  285174256 bytes (  271 MiB), Average time per parse  359420 ns ( ~ 0.359 ms)
After  954000 iterations        --> Used memory is  222456896 bytes (  212 MiB), Average time per parse  364287 ns ( ~ 0.364 ms)
After  955000 iterations        --> Used memory is  159249056 bytes (  151 MiB), Average time per parse  354316 ns ( ~ 0.354 ms)
After  956000 iterations        --> Used memory is  415102288 bytes (  395 MiB), Average time per parse  363510 ns ( ~ 0.364 ms)
After  957000 iterations        --> Used memory is  354301472 bytes (  337 MiB), Average time per parse  372406 ns ( ~ 0.372 ms)
After  958000 iterations        --> Used memory is  293624696 bytes (  280 MiB), Average time per parse  368884 ns ( ~ 0.369 ms)
After  959000 iterations        --> Used memory is  232860864 bytes (  222 MiB), Average time per parse  369313 ns ( ~ 0.369 ms)
After  960000 iterations and GC --> Used memory is  151504320 bytes (  144 MiB), Average time per parse  374238 ns ( ~ 0.374 ms)
After  961000 iterations        --> Used memory is  407361680 bytes (  388 MiB), Average time per parse  377406 ns ( ~ 0.377 ms)
After  962000 iterations        --> Used memory is  345947312 bytes (  329 MiB), Average time per parse  368704 ns ( ~ 0.369 ms)
After  963000 iterations        --> Used memory is  285183680 bytes (  271 MiB), Average time per parse  377030 ns ( ~ 0.377 ms)
After  964000 iterations        --> Used memory is  222504480 bytes (  212 MiB), Average time per parse  405313 ns ( ~ 0.405 ms)
After  965000 iterations        --> Used memory is  478357024 bytes (  456 MiB), Average time per parse  421189 ns ( ~ 0.421 ms)
After  966000 iterations        --> Used memory is  297808784 bytes (  284 MiB), Average time per parse  377259 ns ( ~ 0.377 ms)
After  967000 iterations        --> Used memory is  553663376 bytes (  528 MiB), Average time per parse  376958 ns ( ~ 0.377 ms)
After  968000 iterations        --> Used memory is  373245048 bytes (  355 MiB), Average time per parse  378222 ns ( ~ 0.378 ms)
After  969000 iterations        --> Used memory is  195203776 bytes (  186 MiB), Average time per parse  386620 ns ( ~ 0.387 ms)
After  970000 iterations and GC --> Used memory is  151499528 bytes (  144 MiB), Average time per parse  377953 ns ( ~ 0.378 ms)
After  971000 iterations        --> Used memory is  407356920 bytes (  388 MiB), Average time per parse  370510 ns ( ~ 0.371 ms)
After  972000 iterations        --> Used memory is  345961712 bytes (  329 MiB), Average time per parse  375609 ns ( ~ 0.376 ms)
After  973000 iterations        --> Used memory is  285269448 bytes (  272 MiB), Average time per parse  367829 ns ( ~ 0.368 ms)
After  974000 iterations        --> Used memory is  222450928 bytes (  212 MiB), Average time per parse  380049 ns ( ~ 0.380 ms)
After  975000 iterations        --> Used memory is  159309064 bytes (  151 MiB), Average time per parse  372299 ns ( ~ 0.372 ms)
After  976000 iterations        --> Used memory is  415162368 bytes (  395 MiB), Average time per parse  366421 ns ( ~ 0.366 ms)
After  977000 iterations        --> Used memory is  354161840 bytes (  337 MiB), Average time per parse  372449 ns ( ~ 0.372 ms)
After  978000 iterations        --> Used memory is  293622256 bytes (  280 MiB), Average time per parse  368757 ns ( ~ 0.369 ms)
After  979000 iterations        --> Used memory is  232930232 bytes (  222 MiB), Average time per parse  373888 ns ( ~ 0.374 ms)
After  980000 iterations and GC --> Used memory is  151502144 bytes (  144 MiB), Average time per parse  395108 ns ( ~ 0.395 ms)
After  981000 iterations        --> Used memory is  407359752 bytes (  388 MiB), Average time per parse  375001 ns ( ~ 0.375 ms)
After  982000 iterations        --> Used memory is  346032640 bytes (  330 MiB), Average time per parse  373574 ns ( ~ 0.374 ms)
After  983000 iterations        --> Used memory is  285200320 bytes (  271 MiB), Average time per parse  366749 ns ( ~ 0.367 ms)
After  984000 iterations        --> Used memory is  222569832 bytes (  212 MiB), Average time per parse  406042 ns ( ~ 0.406 ms)
After  985000 iterations        --> Used memory is  478426272 bytes (  456 MiB), Average time per parse  409833 ns ( ~ 0.410 ms)
After  986000 iterations        --> Used memory is  295769744 bytes (  282 MiB), Average time per parse  365156 ns ( ~ 0.365 ms)
After  987000 iterations        --> Used memory is  551622864 bytes (  526 MiB), Average time per parse  376296 ns ( ~ 0.376 ms)
After  988000 iterations        --> Used memory is  366900128 bytes (  349 MiB), Average time per parse  373973 ns ( ~ 0.374 ms)
After  989000 iterations        --> Used memory is  182609272 bytes (  174 MiB), Average time per parse  393422 ns ( ~ 0.393 ms)
After  990000 iterations and GC --> Used memory is  151504912 bytes (  144 MiB), Average time per parse  388881 ns ( ~ 0.389 ms)
After  991000 iterations        --> Used memory is  407362536 bytes (  388 MiB), Average time per parse  364270 ns ( ~ 0.364 ms)
After  992000 iterations        --> Used memory is  346067496 bytes (  330 MiB), Average time per parse  366986 ns ( ~ 0.367 ms)
After  993000 iterations        --> Used memory is  283130256 bytes (  270 MiB), Average time per parse  363988 ns ( ~ 0.364 ms)
After  994000 iterations        --> Used memory is  222558280 bytes (  212 MiB), Average time per parse  363213 ns ( ~ 0.363 ms)
After  995000 iterations        --> Used memory is  161354472 bytes (  153 MiB), Average time per parse  361918 ns ( ~ 0.362 ms)
After  996000 iterations        --> Used memory is  417206096 bytes (  397 MiB), Average time per parse  353877 ns ( ~ 0.354 ms)
After  997000 iterations        --> Used memory is  354196784 bytes (  337 MiB), Average time per parse  375534 ns ( ~ 0.376 ms)
After  998000 iterations        --> Used memory is  293637112 bytes (  280 MiB), Average time per parse  366918 ns ( ~ 0.367 ms)
After  999000 iterations        --> Used memory is  232860112 bytes (  222 MiB), Average time per parse  364685 ns ( ~ 0.365 ms)
After 1000000 iterations and GC --> Used memory is  151500104 bytes (  144 MiB), Average time per parse  387449 ns ( ~ 0.387 ms)

[nielsbasjes]

just to be sure, how do I reuse same instance of UserAgentAnalyzer? In previous version of code it was not written perfectly. But what I am doing now is creating the UDF once and reusing same UDF. AM I in correct direction.

I do not know this language, I do not know your streaming framework, ...
So I do not know how to do that correctly in your context.

[cpanpalia]

okay, I will try out different approaches. Thanks a lot for your support @nielsbasjes