Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] replace mmdb database sometimes results in crash #171

Open
111andre111 opened this issue Nov 23, 2020 · 2 comments
Open

[bug] replace mmdb database sometimes results in crash #171

111andre111 opened this issue Nov 23, 2020 · 2 comments

Comments

@111andre111
Copy link

For all general issues, please provide the following details for fast resolution:
today there is a need of restarting logstash for reloading the Maxmind geoip databases
Referring to this comment:
#63 (comment)
I would like to raise this now as it would be good to have the possibility to reload the database without restarting logstash, and with that you can update the database and logstash would be quite immediately aware of it.

From implementation perspective there are existing mechanisms already that can be taken as a blueprint from the translate plugin:
https://github.com/logstash-plugins/logstash-filter-translate/blob/5cbb57c9a34865856de9ead577c19d42c4b75ade/lib/logstash/filters/dictionary/file.rb#L119
reload_dictionary
https://github.com/logstash-plugins/logstash-filter-translate/blob/5cbb57c9a34865856de9ead577c19d42c4b75ade/lib/logstash/filters/dictionary/file.rb#L128
needs_refresh
and then offer the 2 configurable settings
refresh_interval
and if needed refresh_behaviour
www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html

@111andre111
Copy link
Author

I think I have to revise this one as I did a short test with two custom geoip databases that had a changing field content and during using copying the mmdb and a running logstash the values changed quite immediately, so logstash is getting aware of it. Sorry for the confusion.

@111andre111
Copy link
Author

When I replace the mmdb under the hood of logstash it seems that it sometimes crashes logstash which should even not happen, so let me reopen this one again.

warning: thread "[main]>worker0" terminated with exception (report_on_exception is true):
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
	at com.maxmind.db.Reader.resolveDataPointer(com/maxmind/db/Reader.java:242)
	at com.maxmind.db.Reader.get(com/maxmind/db/Reader.java:150)
	at com.maxmind.geoip2.DatabaseReader.get(com/maxmind/geoip2/DatabaseReader.java:151)
	at com.maxmind.geoip2.DatabaseReader.city(com/maxmind/geoip2/DatabaseReader.java:202)
	at org.logstash.filters.GeoIPFilter.retrieveCityGeoData(org/logstash/filters/GeoIPFilter.java:191)
	at org.logstash.filters.GeoIPFilter.handleEvent(org/logstash/filters/GeoIPFilter.java:144)
	at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(jdk/internal/reflect/GeneratedMethodAccessor8)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:564)
	at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:456)
	at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:317)
	at Users.andreletterer.Downloads.tools.logstash_minus_7_dot_7_dot_0.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_geoip_minus_6_dot_0_dot_3_minus_java.lib.logstash.filters.geoip.filter(Users/andreletterer/Downloads/tools/logstash_minus_7_dot_7_dot_0/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_filter_minus_geoip_minus_6_dot_0_dot_3_minus_java/lib/logstash/filters//Users/andreletterer/Downloads/tools/logstash-7.7.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/lib/logstash/filters/geoip.rb:109)
	at Users.andreletterer.Downloads.tools.logstash_minus_7_dot_7_dot_0.logstash_minus_core.lib.logstash.filters.base.do_filter(Users/andreletterer/Downloads/tools/logstash_minus_7_dot_7_dot_0/logstash_minus_core/lib/logstash/filters//Users/andreletterer/Downloads/tools/logstash-7.7.0/logstash-core/lib/logstash/filters/base.rb:159)
	at Users.andreletterer.Downloads.tools.logstash_minus_7_dot_7_dot_0.logstash_minus_core.lib.logstash.filters.base.multi_filter(Users/andreletterer/Downloads/tools/logstash_minus_7_dot_7_dot_0/logstash_minus_core/lib/logstash/filters//Users/andreletterer/Downloads/tools/logstash-7.7.0/logstash-core/lib/logstash/filters/base.rb:178)
	at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1809)
	at Users.andreletterer.Downloads.tools.logstash_minus_7_dot_7_dot_0.logstash_minus_core.lib.logstash.filters.base.multi_filter(Users/andreletterer/Downloads/tools/logstash_minus_7_dot_7_dot_0/logstash_minus_core/lib/logstash/filters//Users/andreletterer/Downloads/tools/logstash-7.7.0/logstash-core/lib/logstash/filters/base.rb:175)
	at org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(org/logstash/config/ir/compiler/FilterDelegatorExt.java:127)
	at org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134)
	at org.logstash.generated.CompiledDataset1.compute(org/logstash/generated/CompiledDataset1)
	at org.logstash.generated.CompiledDataset2.compute(org/logstash/generated/CompiledDataset2)
	at org.logstash.config.ir.CompiledPipeline$CompiledOrderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:334)
	at org.logstash.config.ir.CompiledPipeline$CompiledOrderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:318)
	at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:82)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:564)
	at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)
	at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:293)
	at Users.andreletterer.Downloads.tools.logstash_minus_7_dot_7_dot_0.logstash_minus_core.lib.logstash.java_pipeline.start_workers(Users/andreletterer/Downloads/tools/logstash_minus_7_dot_7_dot_0/logstash_minus_core/lib/logstash//Users/andreletterer/Downloads/tools/logstash-7.7.0/logstash-core/lib/logstash/java_pipeline.rb:279)
	at org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)
	at java.lang.Thread.run(java/lang/Thread.java:832)

@111andre111 111andre111 changed the title [enhancement] add functionality to refresh database without the need to restart logstash [bug] replace mmdb database sometimes results in crash Nov 27, 2020
@111andre111 111andre111 reopened this Nov 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants