Slowdown server throughput #101
Comments
|
I thought nbthread had replaced nbproc ?!?
…On Sat, 3 Oct 2020 at 01:45, pakit84 ***@***.***> wrote:
Hello,
we are doing some stress test of our webserver in order to fine tune our
system.
We've notice that putting nuster in front of our tomcat server the
throughput reduce by 1/5.
In order to test we have a simple endpoint: (GET) /test that does not
return any content, just an http code 200.
We've used apache ab tool to stress test the server in the following way:
*ab -k -n 100000 -c 1000 xxxxx/test*
so 100K request with 1000 concurrent clients.
Without using nuster we'have following results:
Concurrency Level: 1000
Time taken for tests: 19.463 seconds
Complete requests: 100000
Failed requests: 0
Non-2xx responses: 100000
Keep-Alive requests: 99453
Total transferred: 31184137 bytes
HTML transferred: 12500000 bytes
*Requests per second: 5138.08 [#/sec] (mean)*
Time per request: 194.625 [ms] (mean)
Time per request: 0.195 [ms] (mean, across all concurrent requests)
Transfer rate: 1564.71 [Kbytes/sec] received
With nuster in front of our server:
Concurrency Level: 1000
Time taken for tests: 98.257 seconds
Complete requests: 100000
Failed requests: 0
Non-2xx responses: 100000
Keep-Alive requests: 100000
Total transferred: 28800000 bytes
HTML transferred: 12500000 bytes
*Requests per second: 1017.74 [#/sec] (mean)*
Time per request: 982.572 [ms] (mean)
Time per request: 0.983 [ms] (mean, across all concurrent requests)
Transfer rate: 286.24 [Kbytes/sec] received
We're using image nuster/nuster:5.2 with following configuration:
global
maxconn 4000
user root
group root
nbproc 2
cpu-map 1 0
cpu-map 2 1
daemon
log 127.0.0.1 local0 debug
tune.ssl.default-dh-param 2048
nuster cache on data-size 5m dir /tmp
#tune.ssl.cachesize 1000000
# nuster cache on data-size 5m uri /nuster
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
option http-server-close
option forwardfor
maxconn 4000
timeout connect 5s
timeout client 15min
timeout server 15min
stats enable
stats hide-version
stats refresh 30s
stats show-node
Is there something wrong in this configuration?
Thank you for your help.
Regards,
Pasquale
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#101>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKG6CCKUOBLLTYOMNMLPILSIXYQXANCNFSM4SB3E3CQ>
.
|
|
@pakit84 Can you remove |
|
hello @jiangwenyuan , It doesn't make so much difference, but I understand that is better to remove this option. What indeed is making a lot of difference, 10X faster is when I do the test using directly the ip address in the url ab -k -n 100000 -c 1000 <ip_address>/users/test I can get up to 10K req/s |
|
@pakit84 Seems there's no default backend when you use ip_address. Does |
|
@jiangwenyuan you're right there's no default backend to handle this request. So please ignore it. |
|
@pakit84 You mentioned that /test only returns 200 without any content. but the ab shows: Non-2xx responses: 100000 which is not 200. can you double check that? |
|
Hi @jiangwenyuan , Unfortunately many requests go into timeout after 20 seconds front side. On the server side we use Apache Tomcat and monitor every incoming request and no request is served with more than 200ms. So this means that requests that go into timeout are never actually transferred to our API by nuster. We have checked the CPU / memory status of the server and database and the values remain very low. They never exceed 10/15 %. Do you have any idea what it might be? Here is our configuration: |
|
@dario30186 So how about the /test throughput? As I suspect the setup is incorrect(non 200) so nuster terminates requests each time, which means keep alive does not work here. And for this timeout issue, can you enable http log ? |
The /test throughput is a special case where the API returns 204 No Content. We already have the http log enabled I think. but we don't see any nuster log when we execute command: We only see haproxy logs |
No, happens when the setup is incorrect, such as no backend server, etc.
I don't think so.. Is there any logs like: |
|
Hello @jiangwenyuan
Hello @jiangwenyuan , no we do not see any logs like this one. About /test , sorry for the confusion. It was just test that has nothing to do with the case we've in prod exposed by Dario and yes the endpoint was replying 400 because of an issue on server side, so you can ignore this point. |
Exact these logs, first one is served by backend server and second one is by nuster. So I would suggest look into these bold fields which you can find definitions here |
|
Hello, Is there something configuration that allows this behavior? |
|
Yes, you can use |
|
thank you , this could be really useful for us. I'm using this nuster image : nuster/nuster:5.2 |
|
@pakit84 Explained here: #91 (comment) |
Hello,
we are doing some stress test of our webserver in order to fine tune our system.
We've notice that putting nuster in front of our tomcat server the throughput reduce by 1/5.
In order to test we have a simple endpoint: (GET) /test that does not return any content, just an http code 200 and it's not cached.
We're performing the test on a machine with 2 cpu cores and 8GB ram
We've used apache ab tool (from remote machine) to stress test the server in the following way:
ab -k -n 100000 -c 1000 xxxxx/test
so 100K request with 1000 concurrent clients.
Without using nuster we'have following results:
Concurrency Level: 1000
Time taken for tests: 19.463 seconds
Complete requests: 100000
Failed requests: 0
Non-2xx responses: 100000
Keep-Alive requests: 99453
Total transferred: 31184137 bytes
HTML transferred: 12500000 bytes
Requests per second: 5138.08 [#/sec] (mean)
Time per request: 194.625 [ms] (mean)
Time per request: 0.195 [ms] (mean, across all concurrent requests)
Transfer rate: 1564.71 [Kbytes/sec] received
With nuster in front of our server:
Concurrency Level: 1000
Time taken for tests: 98.257 seconds
Complete requests: 100000
Failed requests: 0
Non-2xx responses: 100000
Keep-Alive requests: 100000
Total transferred: 28800000 bytes
HTML transferred: 12500000 bytes
Requests per second: 1017.74 [#/sec] (mean)
Time per request: 982.572 [ms] (mean)
Time per request: 0.983 [ms] (mean, across all concurrent requests)
Transfer rate: 286.24 [Kbytes/sec] received
We're using image nuster/nuster:5.2 with following configuration:
Is there something wrong in this configuration?
Thank you for your help.
Regards,
Pasquale
The text was updated successfully, but these errors were encountered: