Output of geoip filter changed in logstash 5.4.2 (no more GeoJSON) #123

matejzero · 2017-06-30T09:57:54Z

Hello,

I'm testing logstash 5.4.2 in my staging env and I've noticed that output of geoip plugin has changed. Instead of GeoJSON output, like I had in 5.4.1

    "location": [
      -122.3042,
      47.913
    ],

I now get a hash of lat / lon.

    "location": {
      "lat": 47.913,
      "lon": -122.3042
    },

My geoip configuration looks like this:

    geoip {
      source => [ "src_ip" ]
      fields => [ "country_code2", "country_name", "latitude", "longitude", "location" ]
    }

Looking at the latest commits, I saw that this was changed when some of the code was rewritten to Java. In old Ruby plugin, location was saved as array, but in new Java code, it's saved as Hash.

Java code: ec18789#diff-0db10ce49673bb86356e5398c93ad8b3R257
Ruby code: ec18789#diff-0ceaeedc6497c6d61fa5ae1d2db1dc58L212

Is this to be expected? Will ES still talk hash as geopoint?

For all general issues, please provide the following details for fast resolution:

Version: 5.4.1 & 5.4.2
Operating System: Linux
Steps to Reproduce: Just use geoip filter and look at location field.

The text was updated successfully, but these errors were encountered:

matejzero · 2017-06-30T10:00:53Z

I looked at ES documentation and saw that it can take hash as geopoint type as well: https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html

maurom · 2017-06-30T14:49:45Z

Hi @matejzero. Thanks for your posting, I'm on the same situation (see above).

I understand that ES works on both types anyway, but I don't think this issue should be closed so easily. As a result of the update, we've now got an index with two mixed mappings. IMHO, there should have been at least a warning about the type change.

matejzero · 2017-06-30T14:56:19Z

Do you have static field type defined for 'location' field or do you leave it to ES to guess? All my indexes have custom templates where i set 'location' field as geopoint, so this change probably wont hit me the same it happened to you. Nonetheless, this seems like a breaking change acording to your post, so an input from developers would be welcome.

jordansissel · 2017-06-30T16:05:21Z

Agreed this is a breaking change. /cc @suyograo

…

On Fri, Jun 30, 2017 at 7:56 AM Matej Zerovnik ***@***.***> wrote: Do you have static field type defined for 'location' field or do you leave it to ES to guess? All my indexes have custom templates where i set 'location' field as geopoint, so this change probably wont hit me the same it happened to you. Nonetheless, this seems like a breaking change acording to your post, so an input from developers would be welcome. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#123 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIC6rcI1fab8X2yXdlj7t5u3-pjIRwYks5sJQyUgaJpZM4OKYp5> .

maurom · 2017-06-30T17:02:54Z

I'm currently using the default template provided by logstash ("version" : 50001). (Edited, no pun intended)

matejzero · 2017-06-30T17:38:16Z

Interesting... location field is set to geo_point in that template.

"geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "half_float" },
            "longitude" : { "type" : "half_float" }
          }
        }

How is location field mapped now that you upgraded? If you look at the latest index, what type is location field? It should be geo_point if you are using default template at least to my knowledge.

I might have to downgrade my logstash instances as well, since my indexes rotate tomorrow.

matejzero · 2017-06-30T20:00:39Z

I just tried this on my cluster and even with the latest logstash, location field still gets mapped as geo_point and I don't get mixed templates.

Are you sure your indexes use latest logstash template?

Can you paste your logstash template from elasticsearch: curl -XGET es_host:9200/_template/logstash

You might have an old template there that doesn't set geoip fields.

maurom · 2017-07-03T11:57:01Z

I think so. I've attached the output of
curl -XGET es_host:9200/_template/logstash?pretty

issue-123-logstash-template.txt

matejzero · 2017-07-03T12:01:40Z

Weird, this template should not produce mixed mappings for location field, since mapping is set in template.

Can you look at your mappings in indexes and check what type is location?

maurom · 2017-07-03T12:23:59Z

Sure, here it is: issue-123-logstash-2017.06.21-mapping.txt

The offending index here is logstash-2017.06.21, the one related to the day when I upgraded to 5.4.2. geoip.location is set to geo_point. Now that I see it, the mapping seems to be fine, but I'm having differences in the format the values are stored (as discussed on the other ticket).

By the way, thanks for your help. I'm new on ES issues and still am a bit lost but slowly starting to grasp it.

matejzero · 2017-07-03T17:53:48Z

I went and read your issue again and now I get it. Your field type stayed the same, but values saved are different (hash instead of array). That's the reason reindexing fails.

I replied to your ticket, I think it's best to continue debate there there, since it's a different problem then what I have.

zaakiy · 2017-07-05T05:12:00Z

The following config works with Logstash 5.4.1 but not with 5.4.2 (which I think is geoip 4.1.1)

filter {
    if [Source][IP] {
        geoip {
            source => "[Source][IP]"
            target => "Source"
        }
    }
    if [Destination][IP] {
        geoip {
            source => "[Destination][IP]"
            target => "Destination"
        }
    }

}

I get the following error with Logstash 5.4.2. Note that there is no elasticsearch output going on, so I know that it's not a template issue.

[
{"exception"=>"Missing Ruby class handling for full class name=java.util.HashMap, simple name=HashMap",
"backtrace"=>["org.logstash.Javafier.deep(org/logstash/Javafier.java:29)",
"org.logstash.Event.getField(org/logstash/Event.java:151)",
"org.logstash.filters.GeoIPFilter.applyGeoData(org/logstash/filters/GeoIPFilter.java:151)",
"org.logstash.filters.GeoIPFilter.handleEvent(org/logstash/filters/GeoIPFilter.java:143)",
"java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", 
"RUBY.filter(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.1.1-java/lib/logstash/filters/geoip.rb:122)",
"LogStash::Filters::Base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)",
"LogStash::Filters::Base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", 
"LogStash::Filters::Base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", 
"LogStash::Filters::Base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", 
"org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)",
"LogStash::Filters::Base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)",
"LogStash::Filters::Base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", 
"LogStash::FilterDelegator.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:43)", 
"LogStash::FilterDelegator.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:43)",
"RUBY.initialize((eval):26185)",
"org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", 
"RUBY.initialize((eval):26181)",
"org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)", 
"RUBY.filter_func((eval):7655)", 
"LogStash::Pipeline.filter_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:370)",
"LogStash::Pipeline.filter_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:370)", 
"org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)", 
"LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:224)", 
"LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:224)",
"org.jruby.RubyHash.each(org/jruby/RubyHash.java:1342)", 
"LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:223)", 
"LogStash::Util::WrappedSynchronousQueue::ReadBatch.each(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:223)", 
"LogStash::Pipeline.filter_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:369)", 
"LogStash::Pipeline.filter_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:369)", 
"RUBY.worker_loop(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:350)", 
"RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:317)",
"java.lang.Thread.run(java/lang/Thread.java:748)"]}
]

I can only suspect that it may be due to 45ac731#commitcomment-22920532

matejzero closed this as completed Jun 30, 2017

maurom mentioned this issue Jun 30, 2017

Datatype difference on geoip.location after updating to 5.4.2 elastic/logstash#7569

Open

matejzero reopened this Jun 30, 2017

zaakiy referenced this issue Jul 5, 2017

Fix packaging to include the default DB

45ac731

joewreschnig mentioned this issue Jul 7, 2017

Logstash 5.4.2 breaks MaxMind City Subset databases #121

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Output of geoip filter changed in logstash 5.4.2 (no more GeoJSON) #123

Output of geoip filter changed in logstash 5.4.2 (no more GeoJSON) #123

matejzero commented Jun 30, 2017 •

edited

Loading

matejzero commented Jun 30, 2017

maurom commented Jun 30, 2017

matejzero commented Jun 30, 2017 via email

jordansissel commented Jun 30, 2017 via email

maurom commented Jun 30, 2017 •

edited

Loading

matejzero commented Jun 30, 2017

matejzero commented Jun 30, 2017

maurom commented Jul 3, 2017

matejzero commented Jul 3, 2017

maurom commented Jul 3, 2017

matejzero commented Jul 3, 2017

zaakiy commented Jul 5, 2017 •

edited

Loading

Output of geoip filter changed in logstash 5.4.2 (no more GeoJSON) #123

Output of geoip filter changed in logstash 5.4.2 (no more GeoJSON) #123

Comments

matejzero commented Jun 30, 2017 • edited Loading

matejzero commented Jun 30, 2017

maurom commented Jun 30, 2017

matejzero commented Jun 30, 2017 via email

jordansissel commented Jun 30, 2017 via email

maurom commented Jun 30, 2017 • edited Loading

matejzero commented Jun 30, 2017

matejzero commented Jun 30, 2017

maurom commented Jul 3, 2017

matejzero commented Jul 3, 2017

maurom commented Jul 3, 2017

matejzero commented Jul 3, 2017

zaakiy commented Jul 5, 2017 • edited Loading

matejzero commented Jun 30, 2017 •

edited

Loading

maurom commented Jun 30, 2017 •

edited

Loading

zaakiy commented Jul 5, 2017 •

edited

Loading