Datatype difference on geoip.location after updating to 5.4.2

[maurom]

Hi, I have a ES database, populated via logstash, which stores web server logs, mail server logs, platform logs and adds them additional information such as GeoIP coordinates when a IP address is found.

While doing a revision over it and trying to perform a reindex on one of the indexes, elasticsearch failed with the following message:

  "failures" : [
    {
      "index" : "reindexed",
      "type" : "apache",
      "id" : "AVzXZrIdayxwu2TPqhb2",
      "cause" : {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse [geoip.location]",
        "caused_by" : {
          "type" : "json_parse_exception",
          "reason" : "Current token (START_OBJECT) not numeric, can not use numeric value accessors\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@72502245; line: 1, column: 531]"
        }
      },
      "status" : 400
    },

while the indexes from previous days were copied without error.

It seems that after updating logstash (the whole stack, actually) to 5.4.2, all the entries are created with the geoip.location field as

    geoip.location: { "lon": DECIMAL, "lat": DECIMAL }

for example

    geoip.location: { "lon": -60.7351, "lat": -31.7351 }

while the previous entries (logstash 5.4.1 and previous) were added as

    geoip.location: DECIMAL, DECIMAL

for example

    geoip.location: -60.7351, -31.7351

As such, I have indexes with slightly different mappings. Unfortunately, I am still new to this platform and not able to define if this difference on the behavior of 5.4.1 and 5.4.2 is due to:

• a bug on logstash (or logstash-plugins);
• a change done on purpose, as the Release Notes for that version show changes on GeoIP filters;
• a misconfiguration on my behalf.

If such, I would gladly follow the advice of any experienced user who could provide a way to convert the fields and/or resolve the issue.

[maurom]

It seems this is related to logstash-plugins/logstash-filter-geoip#123

[matejzero]

I just saw you are doing reindexing... Can you show me how are you doing reindexing?

[maurom]

Saw the comment on your ticket. You're right, this is a related but slightly different problem.
This is the command that I'm using:

curl -XPOST "$ESHOST/_reindex?pretty" -H "Content-Type: application/json" -d "@-" <<EOF
{
  "source": {
    "index": "logstash-2017.06.21"
  },
  "dest": {
    "index": "logstash-2017.06.21-1"
  }
}
EOF