Support for ElasticSearch v8.x API

[gaby]

Which OpenObserve functionalities are relevant/related to the feature request?

ingestion

Description

The current ES ingest endpoint only works when using Elastic v7.x, when using a client in v8.x the requests will return HTTP 401. The v8.x api has been out since early 2022.

Elastic Pipeline API can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/get-pipeline-api.html

Error in Elastic Beats:

{
  "log.level": "error",
  "@timestamp": "2024-05-08T02:03:58.635Z",
  "log.logger": "publisher_pipeline_output",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run",
    "file.name": "pipeline/client_worker.go",
    "file.line": 148
  },
  "message": "Failed to connect to backoff(elasticsearch(http://localhost:5080/api/default/)): Connection marked as failed because the onConnect callback failed: 1 error: error loading pipeline packetbeat-8.13.3-amqp: couldn't load pipeline: couldn't load json. Error: 401 Unauthorized: Unauthorized Access. Additionally, error decoding response body: Unauthorized Access",
  "service.name": "packetbeat",
  "ecs.version": "1.6.0"
}

OpenObserve Logs:

2024-05-08T02:07:51.838910454+00:00 INFO actix_web::middleware::logger: 127.0.0.1 "GET /api/default/ HTTP/1.1" 200 222 "-" "-" "Elastic-packetbeat/8.13.3 (linux; amd64; 79b1528b7bfbf5152041db8f4ab497af6afa06e2; 2024-04-29 19:50:16 +0000 UTC)" 0.000577    
2024-05-08T02:07:51.839525732+00:00 INFO actix_web::middleware::logger: 127.0.0.1 "GET /api/default/_license?human=false HTTP/1.1" 200 39 "-" "-" "Elastic-packetbeat/8.13.3 (linux; amd64; 79b1528b7bfbf5152041db8f4ab497af6afa06e2; 2024-04-29 19:50:16 +0000 UTC)" 0.000149    
2024-05-08T02:07:51.839816072+00:00 INFO actix_web::middleware::logger: 127.0.0.1 "GET /api/default/_license?human=false HTTP/1.1" 200 39 "-" "-" "Elastic-packetbeat/8.13.3 (linux; amd64; 79b1528b7bfbf5152041db8f4ab497af6afa06e2; 2024-04-29 19:50:16 +0000 UTC)" 0.000031    
2024-05-08T02:07:51.840156604+00:00 INFO actix_web::middleware::logger: 127.0.0.1 "HEAD /api/default/_index_template/packetbeat HTTP/1.1" 200 275 "-" "-" "Elastic-packetbeat/8.13.3 (linux; amd64; 79b1528b7bfbf5152041db8f4ab497af6afa06e2; 2024-04-29 19:50:16 +0000 UTC)" 0.000045    
2024-05-08T02:07:51.902561990+00:00 INFO actix_web::middleware::logger: 127.0.0.1 "PUT /api/default/_index_template/packetbeat HTTP/1.1" 200 54 "13063" "-" "Elastic-packetbeat/8.13.3 (linux; amd64; 79b1528b7bfbf5152041db8f4ab497af6afa06e2; 2024-04-29 19:50:16 +0000 UTC)" 0.000729    
2024-05-08T02:07:51.902887745+00:00 INFO actix_web::middleware::logger: 127.0.0.1 "GET /api/default/_data_stream/packetbeat HTTP/1.1" 200 81 "-" "-" "Elastic-packetbeat/8.13.3 (linux; amd64; 79b1528b7bfbf5152041db8f4ab497af6afa06e2; 2024-04-29 19:50:16 +0000 UTC)" 0.000069    
2024-05-08T02:07:51.903511553+00:00 INFO actix_web::middleware::logger: 127.0.0.1 "GET /api/default/ HTTP/1.1" 200 222 "-" "-" "Elastic-packetbeat/8.13.3 (linux; amd64; 79b1528b7bfbf5152041db8f4ab497af6afa06e2; 2024-04-29 19:50:16 +0000 UTC)" 0.000433    
2024-05-08T02:07:51.910492367+00:00 INFO actix_web::middleware::logger: 127.0.0.1 "GET /api/default/_ingest/pipeline/packetbeat-8.13.3-amqp HTTP/1.1" 401 39 "-" "-" "Elastic-packetbeat/8.13.3 (linux; amd64; 79b1528b7bfbf5152041db8f4ab497af6afa06e2; 2024-04-29 19:50:16 +0000 UTC)" 0.002782    
2024-05-08T02:07:51.910842927+00:00 INFO actix_web::middleware::logger: 127.0.0.1 "PUT /api/default/_ingest/pipeline/packetbeat-8.13.3-amqp HTTP/1.1" 401 39 "570" "-" "Elastic-packetbeat/8.13.3 (linux; amd64; 79b1528b7bfbf5152041db8f4ab497af6afa06e2; 2024-04-29 19:50:16 +0000 UTC)" 0.000041

Proposed solution

Add support for pipelines api in ElasticSearch v8

Alternatives considered

Send data to a middleware before forwarding to O2. Not ideal but should work.