Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

5.1.6,使用RedisPool高并发出错 ERROR php_swoole_server_rshutdown() (ERRNO 503) #5630

Open
mdys opened this issue Dec 20, 2024 · 9 comments
Open

5.1.6,使用RedisPool高并发出错 ERROR php_swoole_server_rshutdown() (ERRNO 503) #5630

mdys opened this issue Dec 20, 2024 · 9 comments

Comments

@mdys
Copy link

mdys commented Dec 20, 2024 •
edited
Loading

Please answer these questions before submitting your issue.

  1. What did you do? If possible, provide a simple script for reproducing the error.
    server.php
$redis_pool = new RedisPool((new RedisConfig)
		        ->withHost('127.0.0.1')
		        ->withPort(6379)
		        ->withAuth('')
		        ->withDbIndex(0)
		        ->withTimeout(1)
		        ,100000
		  );
$this->redis_pool = $redis_pool;
public function onOpen( $serv, $request ) {
    $redis_pool=$this->redis_pool; 
    $redis = $redis_pool->get(); 
    $redis->get('fd_'.$request->fd);
    $redis_pool->put($redis);
}

在websocket onpen事件时，如果有高并发,大约1s 1000个连接，会有一定几率产生错误，1000个连接失败几十个，产生如下错误：

[2024-12-20 03:53:02 *2291984.253]	
ERROR	php_swoole_server_rshutdown() (ERRNO 503): 
Fatal error: Uncaught RedisException: Connection timed out in @swoole/library/core/Database/RedisPool.php:43
Stack trace:
#0 @swoole/library/core/Database/RedisPool.php(43): Redis->connect()
#1 @swoole/library/core/ConnectionPool.php(88): Swoole\Database\RedisPool->Swoole\Database\{closure}()
#2 @swoole/library/core/ConnectionPool.php(54): Swoole\ConnectionPool->make()
#3 /home/wwwroot/html/server.php(876): Swoole\ConnectionPool->get()
#4 /home/wwwroot/html/server.php(1530): serv_s->onOpen()
#5 [internal function]: serv_s->{closure}()
#6 {main}
  thrown in @swoole/library/core/Database/RedisPool.php on line 43
[2024-12-20 03:53:02 #2291153.0]	WARNING	Server::check_worker_exit_status(): worker(pid=2291967, id=236) abnormal exit, status=0, signal=11
A bug occurred in Swoole-v5.1.6, please report it.
The Swoole developers probably don't know about it,
and unless you report it, chances are it won't be fixed.
You can read How to report a bug doc before submitting any bug reports:
>> https://github.com/swoole/swoole-src/blob/master/.github/ISSUE.md 
Please do not send bug reports in the mailing list or personal letters.
The issue page is also suitable to submit feature requests.

OS: Linux 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64
GCC_VERSION: 8.3.1 20190311 (Red Hat 8.3.1-3)
OPENSSL_VERSION: OpenSSL 1.0.2k-fips  26 Jan 2017
PHP_VERSION : 8.3.8

  1. What did you expect to see?

  2. What did you see instead?

  3. What version of Swoole are you using (show your php --ri swoole)?

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 5.1.6
Built => Dec  2 2024 14:14:42
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
openssl => OpenSSL 1.0.2k-fips  26 Jan 2017
http2 => enabled
json => enabled
curl-native => enabled
pcre => enabled
c-ares => 1.10.0
zlib => 1.2.7
brotli => E16777225/D16777225
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
mysqlnd => enabled
async_redis => enabled
Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_fiber_mock => Off => Off
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608
  1. What is your machine environment used (show your uname -a & php -v & gcc -v) ?
PHP 8.3.8 (cli) (built: Jun 4 2024 14:53:17) (NTS gcc x86_64)
Copyright (c) The PHP Group
Zend Engine v4.3.8, Copyright (c) Zend Technologies
with Zend OPcache v8.3.8, Copyright (c), by Zend Technologies`
`COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/opt/rh/devtoolset-8/root/usr --mandir=/opt/rh/devtoolset-8/root/usr/share/man --infodir=/opt/rh/devtoolset-8/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --with-default-libstdcxx-abi=gcc4-compatible --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-8.3.1-20190311/obj-x86_64-redhat-linux/isl-install --disable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC)
@mdys
Copy link
Author

mdys commented Dec 20, 2024

补充下报错信息：


Dec 20 00:34:47 localhost Swoole_process_server_worker_155: *** Error in `Swoole_process_server_worker_155': corrupted size vs. prev_size: 0x0000562e5f02b780 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_132: *** Error in `Swoole_process_server_worker_132': corrupted size vs. prev_size: 0x0000562e5f0274b0 ***
Dec 20 00:34:47 localhost kernel: php[1769282]: segfault at 7f70148001e0 ip 00007f682ab8da5d sp 00007fff39abb788 error 6 in libc-2.17.so[7f682aafe000+1c4000]
Dec 20 00:34:47 localhost kernel: php[1769501]: segfault at 2148 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769232]: segfault at 48 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769411]: segfault at 588 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769269]: segfault at 3e8 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769306]: segfault at 138 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769316]: segfault at 48 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769308]: segfault at 48 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769335]: segfault at 2148 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769278]: segfault at 48 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost Swoole_process_server_worker_93: *** Error in `Swoole_process_server_worker_93': corrupted size vs. prev_size: 0x0000562e5f032850 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_117: *** Error in `Swoole_process_server_worker_117': corrupted size vs. prev_size: 0x0000562e5f0274b0 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_8: *** Error in `Swoole_process_server_worker_8': corrupted size vs. prev_size: 0x0000562e5f054d80 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_200: *** Error in `Swoole_process_server_worker_200': double free or corruption (!prev): 0x0000562e5f232f20 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_194: *** Error in `Swoole_process_server_worker_194': corrupted size vs. prev_size: 0x0000562e5f02b780 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_345: *** Error in `Swoole_process_server_worker_345': corrupted size vs. prev_size: 0x0000562e5f054620 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_110: *** Error in `Swoole_process_server_worker_110': corrupted size vs. prev_size: 0x0000562e5f0274b0 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_202: *** Error in `Swoole_process_server_worker_202': double free or corruption (!prev): 0x0000562e5f0456a0 ***
Dec 20 03:28:11 localhost kernel: php             D ffff9b505fb1acc0     0 2218768 2194582 0x00000080
Dec 20 03:28:11 localhost kernel: Call Trace:
Dec 20 03:28:11 localhost kernel: [<ffffffffa4588089>] schedule_preempt_disabled+0x29/0x70
Dec 20 03:28:11 localhost kernel: [<ffffffffa4585ff7>] __mutex_lock_slowpath+0xc7/0x1d0
Dec 20 03:28:11 localhost kernel: [<ffffffffa45853cf>] mutex_lock+0x1f/0x2f
Dec 20 03:28:11 localhost kernel: [<ffffffffa412d206>] ima_file_check+0xa6/0x1b0
Dec 20 03:28:11 localhost kernel: [<ffffffffa405ddda>] do_last+0x59a/0x1340
Dec 20 03:28:11 localhost kernel: [<ffffffffa405ec4d>] path_openat+0xcd/0x5a0
Dec 20 03:28:11 localhost kernel: [<ffffffffa3f11db8>] ? get_futex_key+0x1c8/0x2c0
Dec 20 03:28:11 localhost kernel: [<ffffffffa3eda830>] ? try_to_wake_up+0x190/0x390
Dec 20 03:28:11 localhost kernel: [<ffffffffa4060e9d>] do_filp_open+0x4d/0xb0
Dec 20 03:28:11 localhost kernel: [<ffffffffa406ef97>] ? __alloc_fd+0x47/0x170
Dec 20 03:28:11 localhost kernel: [<ffffffffa404c9e4>] do_sys_open+0x124/0x220
Dec 20 03:28:11 localhost kernel: [<ffffffffa404cafe>] SyS_open+0x1e/0x20
Dec 20 03:28:11 localhost kernel: [<ffffffffa4593f92>] system_call_fastpath+0x25/0x2a
Dec 20 03:28:11 localhost kernel: INFO: task php:2218773 blocked for more than 120 seconds.
Dec 20 03:28:11 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

sysctl.conf

vm.swappiness = 0 
fs.aio-max-nr=1048576
fs.file-max = 9999990
vm.min_free_kbytes = 2097152
vm.max_map_count = 655360
kernel.threads-max=2061206
kernel.pid_max=4194303
kernel.perf_cpu_time_max_percent=3
kernel.perf_event_max_sample_rate=1000
net.ipv4.ip_forward = 1
kernel.msgmnb = 8203520
kernel.msgmni = 10240
kernel.msgmax = 18192
net.ipv4.tcp_mem  = 379008        505344        758016
net.ipv4.tcp_wmem = 4096        16384        4194304
net.ipv4.tcp_rmem = 4096        87380        4194304
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.unix.max_dgram_qlen = 8024 
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_max_syn_backlog=81920
net.ipv4.tcp_synack_retries=3
net.ipv4.tcp_syn_retries=3
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 20000 65000
net.ipv4.tcp_max_tw_buckets = 200000
net.ipv4.route.max_size = 5242880
net.ipv4.tcp_tw_recycle=0
net.ipv4.tcp_timestamps=0
net.core.somaxconn=12590 
kernel.core_pattern = /core_data/core-%e-%p-%t
vm.overcommit_memory=1

@NathanFreeman
Copy link
Member

看看服务器或者redis服务器的ulimit -n是多少

@NathanFreeman
Copy link
Member

检查一下redis客户端的最大连接数，这里段错误是因为连接超时，抛异常了没有捕获

@mdys
Copy link
Author

mdys commented Dec 21, 2024

ulimit -n
1024000

@mdys
Copy link
Author

mdys commented Dec 21, 2024

  • soft nofile 1024000
  • hard nofile 1024000
    root soft nofile 1024000
    root hard nofile 1024000
  • soft nproc unlimited
  • hard nproc unlimited

@mdys
Copy link
Author

mdys commented Dec 21, 2024

redis配置也没问题 ，就是高并发时候 会出现链接问题。 目前我用的 Table 替代的 redis，但有些功能Table不能满足. 所以还是希望能redis.
我的场景是websocket 收发二进制声音媒体文件，二进制文件比较大一直发送的话 会一直触发 onopen onmessage，然后redis就撑不住了

@NathanFreeman
Copy link
Member

执行redis-cli，输入config get maxclients看看是多少
或者在实例化完$redis_pool之后，执行$redis_pool->fill()提前创建redis

@NathanFreeman
Copy link
Member

你这个10万个连接有点多了，控制一下进程可以创建的redis数量，最好所有进程可以创建的redis连接数加起来小于redis允许的客户端最大连接数

@mdys
Copy link
Author

mdys commented Dec 21, 2024

执行redis-cli，输入config get maxclients看看是多少 或者在实例化完$redis_pool之后，执行$redis_pool->fill()提前创建redis

OK 我试试提前创建

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mdys @NathanFreeman