Logstash [v8.3.3] adding "event" field to payload.

[karun-singh]

Logstash pipeline is adding an event field to the payload containing the original payload from RabbitMQ. This is causing Elasticsearch to throw an error as it goes against the strict mapping set for the indices.

Tested using the RMQ test script here

Error from Logstash:

{
	"level": "WARN",
	"loggerName": "logstash.outputs.elasticsearch",
	"timeMillis": 1664274188010,
	"thread": "[resource-group]>worker0",
	"logEvent": {
		"message": "Could not index event to Elasticsearch.",
		"status": 400,
		"action": ["index", {
			"_index": "test-itms"
		}, {
			"id": "test-itms",
			"last_stop_arrival_time": "15:09:58",
			"trip_delay": 948,
			"actual_trip_start_time": "2020-11-03T14:22:30+05:30",
			"license_plate": "10",
			"last_stop_id": "2028",
			"event": {
				"original": "{\"trip_direction\": \"NT\", \"trip_id\": \"24374871\", \"route_id\": \"17AD\", \"trip_delay\": 948, \"last_stop_arrival_time\": \"15:09:58\", \"actual_trip_start_time\": \"2020-11-03T14:22:30+05:30\", \"vehicle_label\": \"A09\", \"observationDateTime\": \"2022-09-27T10:23:07+00:00\", \"speed\": 25.0, \"license_plate\": \"10\", \"last_stop_id\": \"2028\", \"location\": {\"coordinates\": [72.870511, 21.218943], \"type\": \"Point\"}, \"id\": \"test-itms\"}"
			},
			"trip_id": "24374871",
			"vehicle_label": "A09",
			"route_id": "17AD",
			"observationDateTime": "2022-09-27T10:23:07+00:00",
			"location": {
				"coordinates": [72.870511, 21.218943],
				"type": "Point"
			},
			"speed": 25.0,
			"trip_direction": "NT"
		}],
		"response": {
			"index": {
				"_index": "test-itms",
				"_id": "BMN4foMBfBEPN2SVxrea",
				"status": 400,
				"error": {
					"type": "strict_dynamic_mapping_exception",
					"reason": "mapping set to strict, dynamic introduction of [event] within [_doc] is not allowed"
				}
			}
		}
	}
}

Current solution:

This was temporarily fixed in commit- 8ef8f5a by manually removing the event field in the pipeline filter. Need to debug what's causing this and make a permanent fix.