Protocol aware toxics

[sirupsen]

We could build protocol aware toxics, e.g.

• HTTP. Send extremely large headers, slow headers, slow body but fast headers, 5xx, infinite redirects..
• Redis. Slow writes and fast reads, out of memory conditions (reads but no writes), ..

[dwradcliffe]

This would be neat. I'd like to be able to return HTTP 404 and test that.

[sirupsen]

@dwradcliffe it's completely possible to build already BTW if you really need it. We just haven't. Thanks for the input on a use-case :-)

[dwradcliffe]

My specific use-case is for elasticsearch. ES often returns non-200 responses when things are not quite right or when the request was routed to the wrong place.

[xthexder]

I wrote a simple HTTP toxic while writing the docs for 2.0
You can find it here: https://github.com/xthexder/toxic-example/blob/master/http.go

If you're up for it, you could modify it for ES, and compile it in to the 2.0-dev branch.
Hopefully I'll have some time in the next week or so to release 2.0, but I've got a fair amount of other work to do at the moment.

You may also be interested in looking at https://github.com/Shopify/toxiproxy/blob/2.0-dev/CREATING_TOXICS.md

[sirupsen]

@xthexder how about taking the HTTP toxic into core?

[xthexder]

The toxic needs some work to generalize the usage, since it needs to be controllable via json, but it's definitely something we can put in core.
It will also need a matching request http toxic, since this one is only for http responses.

[xthexder]

This might be interesting to bring up again. I'm not sure of all the use-cases for this, but a couple things I can think of to put as options are:

• Change status code
• Add/remove headers
• Set response body
• Toxicity per-request vs per-connection

I can work on implementing some of these, but I'd like to hear some others opinions.

For some of the things I'm thinking of, there's the problem that downstream toxics can't read any information from the upstream / request, so things can't be filtered based on URL for example.

[sirupsen]

The status code change one would be interesting for sure, adding removing random headers, etc., would all be awesome.

there's the problem that downstream toxics can't read any information from the upstream / request

Ugh, this is a great point. How hard would it be to add this information though?

[xthexder]

I'm currently looking at it, and I think it's possible to add, it just might be a little messy with matching corresponding upstream/downstream toxics together.
The question is mostly if it will be transparent to the user/API or not.

[aebicalho]

Is there any update to this functionality? It's the only thing missing for us to be able to fully use toxiproxy.

[jpittis]

@aebicalho: We have two open PRs that add this functionality. There's no one currently trying to get them out the door. They need a champion to make it happen.

I'd love to get to it at some point but I have higher priorities at this moment.

[worldtiki]

I created an http toxic, based on the work from @xthexder, so that I could modify the host header.
I can open a pr with this if you'd like, as this seems a pretty common use case (using reverse proxies). The changes were: https://github.com/worldtiki/toxiproxy/pull/2/files

[maxgalbu]

@worldtiki how do you use this toxic? I compiled your version of toxiproxy but i'm not able to add a http toxic neither from the CLI nor from a client.

$ ./toxiproxy-cli toxic add api_core_slow -t http_request_headers -a headers=Host:host.net
Toxic attributes was expected to be an integer.

[worldtiki]

Hi @maxgalbu!

If you forked my repo then make sure you use the branch http_toxic and not master, as the master has other changes that have not been tested (some experiments I was doing).

I was able to use the http toxic successfully with these configs:

Initial proxies config:
[ { "name": "example", "listen": "0.0.0.0:9091", "upstream": "www.example.com:80" } ]

If I curl 127.0.0.1:8474/proxies I get this:
{ "example": { "name": "example", "listen": "[::]:9091", "upstream": "www.example.com:80", "enabled": true, "toxics": [] } }

And if I add the following toxic with curl -X POST 127.0.0.1:8474/proxies/example/toxics -d @example.json:
{ "attributes":{ "headers": { "Host": "www.example.com" } }, "name": "example", "type": "http_request_headers", "stream": "upstream", "toxicity": 1 }

I can curl toxiproxy again and confirm it was added successfully:

{ "example": { "name": "example", "listen": "[::]:9091", "upstream": "www.example.com:80", "enabled": true, "toxics": [ { "attributes": { "headers": { "Host": "www.example.com" } }, "name": "example", "type": "http_request_headers", "stream": "upstream", "toxicity": 1 } ] } }

And if I curl 127.0.0.1:9091 I will go to www.example.com

You're probably missing some params (stream: upstream, ...). Try doing a post instead of using the client to confirm it works and then go from there.

ps: I'll update this pr with an example: #215

[maxgalbu]

@worldtiki thanks for your example

$ curl 127.0.0.1:8474/proxies
{"api_core_slow":{"name":"api_core_slow","listen":"127.0.0.1:8080","upstream":"host.net:80","enabled":true,"toxics":[]}}
$ curl -X POST 127.0.0.1:8474/proxies/api_core_slow/toxics -d '{ "attributes":{ "headers": { "Host": "host.net" } }, "name": "example", "type": "http_request_headers", "stream": "upstream", "toxicity": 1 }'
{"error":"invalid toxic type","status":400}

Please also share how to build your toxiproxy :)

[worldtiki]

@maxgalbu just clone my repo, checkout the branch http_toxic and run
make clean && make build && make linux && make docker

If you want you can pull this docker image: worldtiki/toxiproxy:1.7

[maxgalbu]

Ok i was probably doing something wrong, the docker image works fine. Thanks