Unable to change index tokenizer

[tinkerpj]

Hello. First of all, great package!

I am attempting to change the index tokenizer and analyzer but am having no luck. I've tried both via the model and via the config, but the changes do not seem to be applied.

Here is a config mapping attempting to change the tokenizer to 'ngram':

        'local.surveys' => [
            'properties' => [
                'title'         => 'text',
                'preview'       => 'text',
                'instructions'  => 'text',
                'organizations' => 'keyword',
                'facilities'    => 'keyword'
            ],
            'settings' => [
                'analysis' => [
                    'analyzer' => [
                        'ngram_lowercase' => [
                            'type' => 'standard',
                            'tokenizer' => 'ngram',
                            'filter' => ['lowercase']
                        ]
                    ]
                ]
            ]
        ]

Looking at kibana, the index settings have the values from the config:

  "settings": {
    "index": {
      "number_of_shards": "1",
      "provided_name": "local.surveys",
      "creation_date": "1651785833647",
      "analysis": {
        "analyzer": {
          "ngram_lowercase": {
            "filter": [
              "lowercase"
            ],
            "type": "custom",
            "tokenizer": "ngram"
          }
        }
      },
      "number_of_replicas": "1",
      "uuid": "v_WR_U5OTBa1p-1nOE1rCA",
      "version": {
        "created": "7090399"
      }
    }

However, upon inspection, the document has standard tokenization. Here are the term vectors for a survey with a title of '3.16.20':

      "field_statistics" : {
        "sum_doc_freq" : 292,
        "doc_count" : 117,
        "sum_ttf" : 296
      },
      "terms" : {
        "3.16.20" : {
          "term_freq" : 1,
          "tokens" : [
            {
              "position" : 0,
              "start_offset" : 0,
              "end_offset" : 7
            }
          ]
        }
      }
    }

And, finally, here is a partial list of what ES reports '3.16.20' should tokenize to via 'ngram':

{
  "tokens" : [
    {
      "token" : "3",
      "start_offset" : 0,
      "end_offset" : 1,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "3.",
      "start_offset" : 0,
      "end_offset" : 2,
      "type" : "word",
      "position" : 1
    },
    {
      "token" : ".",
      "start_offset" : 1,
      "end_offset" : 2,
      "type" : "word",
      "position" : 2
    },
    {
      "token" : ".1",
      "start_offset" : 1,
      "end_offset" : 3,
      "type" : "word",
      "position" : 3
    },
...
  ]
}

Any ideas why the tokenizer is not being set?

[Jeroen-G]

My initial feeling when I see that the settings are visible in Kibana is that this is an issue beyond the package.

Kibana also contains dev tools where you can execute queries, have you tried that?

[tinkerpj]

@Jeroen-G thanks for the quick response! I found the issue and it was... me. I didn't realize I needed to define the analyzer in the field mapping. Once I specified the analyzer, things began to function as expected.