Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

关于1.6.1版本docker-compose部署 提交代码卡Judging的问题 #480

Open
wellwei opened this issue Oct 18, 2024 · 4 comments
Open

关于1.6.1版本docker-compose部署 提交代码卡Judging的问题 #480

wellwei opened this issue Oct 18, 2024 · 4 comments

Comments

@wellwei
Copy link

wellwei commented Oct 18, 2024

按照官网的docker-compose一键部署命令部署：

git clone -b v1.6.1 https://github.com/QingdaoU/OnlineJudgeDeploy.git

部署后未更改任何设置，提交非CE代码时卡Judging（CE代码正常）
以下是 ~/data/backend/log/dramatiq.log的日志

[2024-10-18 04:06:50] - [DEBUG] - [dramatiq.worker.ConsumerThread(default):326]  - Pushing message '2e93045a-f01f-468b-9056-2e16a154173b' onto work queue.
[2024-10-18 04:06:50] - [DEBUG] - [dramatiq.worker.WorkerThread:479]  - Received message judge_task('f16d911230421eb40a133701b81a6750', 1) with id '2e93045a-f01f-468b-9056-2e16a154173b'.
[2024-10-18 04:06:50] - [ERROR] - [dramatiq.worker.WorkerThread:513]  - Failed to process message judge_task('f16d911230421eb40a133701b81a6750', 1) with unhandled exception.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/dramatiq/worker.py", line 485, in process_message
    res = actor(*message.args, **message.kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/dramatiq/actor.py", line 182, in __call__
    return self.fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/judge/tasks.py", line 14, in judge_task
    JudgeDispatcher(submission_id, problem_id).judge()
  File "/app/judge/dispatcher.py", line 174, in judge
    self._compute_statistic_info(resp["data"])
  File "/app/judge/dispatcher.py", line 106, in _compute_statistic_info
    self.submission.statistic_info["time_cost"] = max([x["cpu_time"] for x in resp_data])
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: max() iterable argument is empty
[2024-10-18 04:06:50] - [WARNING] - [dramatiq.middleware.retries.Retries:103]  - Retries exceeded for message '2e93045a-f01f-468b-9056-2e16a154173b'.
[2024-10-18 04:06:50] - [DEBUG] - [dramatiq.worker.ConsumerThread(default):344]  - Rejecting message '2e93045a-f01f-468b-9056-2e16a154173b'.
[2024-10-18 04:06:50] - [ERROR] - [sentry.errors:686]  - Sentry responded with an error: module 'ssl' has no attribute 'wrap_socket' (url: https://sentry.io/api/263057/store/)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/raven/transport/threaded.py", line 165, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)
  File "/usr/local/lib/python3.12/site-packages/raven/transport/http.py", line 38, in send
    response = urlopen(
               ^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 66, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 515, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 532, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 46, in https_open
    return self.do_open(ValidHTTPSConnection, req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 1344, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/lib/python3.12/http/client.py", line 1331, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1377, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1326, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1085, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.12/http/client.py", line 1029, in send
    self.connect()
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 37, in connect
    self.sock = ssl.wrap_socket(
                ^^^^^^^^^^^^^^^
AttributeError: module 'ssl' has no attribute 'wrap_socket'
b"Sentry responded with an error: module 'ssl' has no attribute 'wrap_socket' (url: https://sentry.io/api/263057/store/)"
[2024-10-18 04:06:50] - [ERROR] - [sentry.errors.uncaught:712]  - ["Failed to process message judge_task('f16d911230421eb40a133701b81a6750', 1) with unhandled exception.", '  File "dramatiq/worker.py", line 485, in process_message', '  File "dramatiq/actor.py", line 182, in __call__', '  File "judge/tasks.py", line 14, in judge_task', '  File "judge/dispatcher.py", line 174, in judge', '  File "judge/dispatcher.py", line 106, in _compute_statistic_info']
b'["Failed to process message judge_task(\'f16d911230421eb40a133701b81a6750\', 1) with unhandled exception.", \'  File "dramatiq/worker.py", line 485, in process_message\', \'  File "dramatiq/actor.py", line 182, in __call__\', \'  File "judge/tasks.py", line 14, in judge_task\', \'  File "judge/dispatcher.py", line 174, in judge\', \'  File "judge/dispatcher.py", line 106, in _compute_statistic_info\']'

询问AI可能是docker容器Python版本太高不兼容，不知如何解决
image
@wellwei
Copy link
Author

wellwei commented Oct 18, 2024

以下日志是judge_server启动时，~/data/judge_server/log/judge_server.log中的报错信息

2024-10-18 12:00:19,853 ERROR HTTPConnectionPool(host='oj-backend', port=8000): Max retries exceeded with url: /api/judge_server_heartbeat/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x79c571698410>: Failed to establish a new connection: [Errno 111] Connection refused'))
Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/app/.venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/usr/lib/python3.12/http/client.py", line 1326, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.12/http/client.py", line 1085, in _send_output
    self.send(msg)
  File "/usr/lib/python3.12/http/client.py", line 1029, in send
    self.connect()
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 238, in connect
    self.sock = self._new_conn()
                ^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 213, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x79c571698410>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='oj-backend', port=8000): Max retries exceeded with url: /api/judge_server_heartbeat/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x79c571698410>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/service.py", line 17, in _request
    resp = requests.post(self.backend_url, json=data,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='oj-backend', port=8000): Max retries exceeded with url: /api/judge_server_heartbeat/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x79c571698410>: Failed to establish a new connection: [Errno 111] Connection refused'))
2024-10-18 12:00:19,857 ERROR 
Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/app/.venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/usr/lib/python3.12/http/client.py", line 1326, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.12/http/client.py", line 1085, in _send_output
    self.send(msg)
  File "/usr/lib/python3.12/http/client.py", line 1029, in send
    self.connect()
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 238, in connect
    self.sock = self._new_conn()
                ^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 213, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x79c571698410>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='oj-backend', port=8000): Max retries exceeded with url: /api/judge_server_heartbeat/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x79c571698410>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/service.py", line 17, in _request
    resp = requests.post(self.backend_url, json=data,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='oj-backend', port=8000): Max retries exceeded with url: /api/judge_server_heartbeat/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x79c571698410>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/service.py", line 42, in <module>
    service.heartbeat()
  File "/app/service.py", line 35, in heartbeat
    self._request(data)
  File "/app/service.py", line 22, in _request
    raise JudgeServiceError("Heartbeat request failed")
exception.JudgeServiceError

说未链接到后端，但是管理面板可以看到judge的状态正常

@mikucat0309
Copy link
Member

mikucat0309 commented Oct 21, 2024 •
edited
Loading

後台的 judge server 狀態是透過檢測最後一次 judge server 往 web backend 送的 healthcheck 時間來判斷，從您提供的 log 來看很像是 judge server 沒法往 web backend 送 health check，應該是網路設定方面的問題

想確認一下您是用 Docker 與 Docker Compose 架設的嗎？有無添加自訂義的 network？
剛剛在 AWS 起了台 EC2 測試，建立一套 judge 的網路環境是會通的

@wellwei
Copy link
Author

wellwei commented Oct 27, 2024

问题已解决，是因为导入fps格式题目时，测试数据没导入，导致评测结果返回空异常。

@kindle1126
Copy link

你是怎麼解決的?我一直卡Pending

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mikucat0309 @wellwei @kindle1126