Logstash 5.4.2 breaks MaxMind City Subset databases

[joewreschnig]

After upgrading to 5.4.2 the City-Europe database no longer works. It is not in the new hardcoded list of permitted databases.

Exception in thread "[main]>worker5" java.lang.IllegalStateException: Unsupported database type GeoIP2-City-Europe
at org.logstash.filters.GeoIPFilter.handleEvent(org/logstash/filters/GeoIPFilter.java:133)
at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
at RUBY.filter(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.1.1-java/lib/logstash/filters/geoip.rb:122)
at LogStash::Filters::Base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)
at LogStash::Filters::Base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)
at LogStash::Filters::Base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)
at LogStash::Filters::Base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)
at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
at LogStash::Filters::Base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)
at LogStash::Filters::Base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)
at LogStash::FilterDelegator.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:43)
at LogStash::FilterDelegator.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:43)
at RUBY.filter_func((eval):732)
at LogStash::Pipeline.filter_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:370)
at LogStash::Pipeline.filter_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:370)
at org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)
at LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:224)
at LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:224)
at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1342)
at LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:223)
at LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:223)
at LogStash::Pipeline.filter_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:369)
at LogStash::Pipeline.filter_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:369)
at RUBY.worker_loop(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:350)
at RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:317)
at java.lang.Thread.run(java/lang/Thread.java:748)

It worked fine before. (Well, not really fine, but it could at least look up IPs from European cities.)

[joewreschnig]

I've updated the title to something less useful / searchable but more alarming as #123 suggests this is a useful way to get a quicker developer response...

[jordansissel]

Is there a full list of geoip database names somewhere?

[jordansissel]

Found this: https://www.maxmind.com/en/geoip2-city-database-by-continent

It's by continent, so we'd need to add:

GeoIP2-City-Africa, GeoIP2-City-Asia-Pacific, GeoIP2-City-Europe, GeoIP2-City-North-America, GeoIP2-City-South-America

Assuming I am guessing the correct names for these databases based on the dropdown on Maxmind's site:

[jordansissel]

I have a PR for this (#128) but I don't have access to Maxmind's pay databases at this time so I cannot test it.

[joewreschnig]

I've tested the PR and it can again use the City-Europe database.

Longer-term, I think making requests appropriate for whatever fields the user wants is a better approach than hardcoding a list of databases. (If they request city_name, make the GeoIP request, if they request asn make the ASN request, etc.) Some organizations roll their own DBs (with e.g. a combination of city subsets, or with custom labels), and MaxMind will probably offer some different ones in the future too.

[jordansissel]

@joewreschnig I agree with your proposal.