Investigate: cleanup listeners

[adilhafeez]

We should have two listeners one for prompt and other for llm routing.

Here is a list of listeners we have today,

• 10000 prompt (to create traceparent)
• 10001 prompt - hosts prompt gatreway wasm filter code
• 11000 internal api - to communicate with internal hosts like embeddings, nli, archfc etc.
• 12000 llm routing (to create traceparent)
• 12001 llm routing - hosts llm gateway filter code

[adilhafeez]

review async client and come up with recommendation

[junr03]

The two main issues that made the need for new listeners are:
a) Creating new spans for the inline http calls
b) Getting access logs for the inline http calls.

About tracing

Under the hood Proxy Wasm's dispatch_http_call uses Envoy's AsyncClient. Which internally uses RequestOptions to manage desired options for the http call. Crucially this is the code that manages creation of a child span during an HTTP call in the async client.

Therefore: the changes needed to support creating new spans in wasm do not need to reach into the AsyncClient code. They solely need to affect the WASM layers in Envoy, all the way to WASM's interaction with the async client..

Access Log

As it turns out, the Async Client does not have Access Logging infrastructure in it. It seems from reading comments (for instance this one) that there was an intention to hook on Access Logging onto the Async Client, but it hasn't materialized. IMO the lift is considerable given that Access Logging is a publicly configurable entity in Envoy, but Async Clients are not. i.e, I am not sure what exposing the concept of an AsyncClient client would mean for Envoy. My initial glimmer of possibility would be to expose the Access Logging configuration under the Cluster configuration given that internally the "Cluster" is what (through some indirection) spawns an AsyncClient.

[adilhafeez]

Therefore: the changes needed to support creating new spans in wasm do not need to reach into the AsyncClient code. They solely need to affect the WASM layers in Envoy, all the way to WASM's interaction with the async client..

This will be great. That means we can get parent activeSpan() and set it to RequestOptions. We can take additional input when making http_call from rust filter and attach parent_span based on passed in option from wasm filter.