Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable the run_async=False option for the InferenceClient. #17

Merged
merged 2 commits into from
Dec 15, 2023
Merged

Enable the run_async=False option for the InferenceClient. #17

merged 2 commits into from
Dec 15, 2023

Conversation

SangwonSUH
Copy link
Contributor

Intro

The InferenceClient did not have functionality implemented for run_async=False. This PR addresses that by implementing the feature and updating unit tests to ensure it.
If there were no plans for implementing this feature yet, I hope this PR would be helpful 😀

Changes

  • InferenceClient._call_request
    • This is a new pipeline corresponding to the _call_async.
    • I'm not sure it is structurally identical, but it has been implemented to follow the same logic of _call_async.
  • tests/*.py
    • Updated port numbers to match those in bin/run_triton_tritony_sample.sh.
    • Added a new test parameter run_async.

Tests

pytest -s --cov-report term-missing --cov=tritony tests
My test outputs
=================================================================================== test session starts ====================================================================================
platform darwin -- Python 3.10.13, pytest-7.4.3, pluggy-1.3.0
Matplotlib: 3.8.2
Freetype: 2.6.1
rootdir: /Users/Sangwon/git/tritony
plugins: cov-4.1.0, mpl-0.16.1, xdist-3.5.0
collected 24 items                                                                                                                                                                         

tests/test_connect.py Testing http with run_async=True
.Testing grpc with run_async=True
.Testing grpc with run_async=False
.tests.test_connect, Testing http with run_async=True
.tests.test_connect, Testing grpc with run_async=True
.tests.test_connect, Testing grpc with run_async=False
.tests.test_connect, Testing http with run_async=True
<class 'RuntimeError'>
.tests.test_connect, Testing grpc with run_async=True
.tests.test_connect, Testing grpc with run_async=False
.
tests/test_model_call.py Testing http with run_async=True
.Testing grpc with run_async=True
.Testing grpc with run_async=False
.Testing http with run_async=True
.Testing grpc with run_async=True
.Testing grpc with run_async=False
.Testing http with run_async=True
.Testing grpc with run_async=True
.Testing grpc with run_async=False
.Testing http with run_async=True
.Testing grpc with run_async=True
.Testing grpc with run_async=False
.Testing http with run_async=True
.Testing grpc with run_async=True
.Testing grpc with run_async=False
.

===================================================================================== warnings summary =====================================================================================
tritony/tools.py:23
  /Users/Sangwon/git/tritony/tritony/tools.py:23: UserWarning: tritonclient[all]>=2.34.0
    warnings.warn(UserWarning("tritonclient[all]>=2.34.0"))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

--------- coverage: platform darwin, python 3.10.13-final-0 ----------
Name                  Stmts   Miss  Cover   Missing
---------------------------------------------------
tritony/__init__.py       5      0   100%
tritony/helpers.py       71      0   100%
tritony/tools.py        243     12    95%   118-119, 135-137, 152, 175-177, 434-435, 495
tritony/version.py        1      0   100%
---------------------------------------------------
TOTAL                   320     12    96%

======================================================================== 24 passed, 1 warning in 377.57s (0:06:17) =========================================================================

Copy link
Member

@kimdwkimdw kimdwkimdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing.

Added small fixes

@kimdwkimdw kimdwkimdw merged commit da0998d into rtzr:main Dec 15, 2023
6 checks passed
@SangwonSUH SangwonSUH deleted the feat/disable_run_async branch December 15, 2023 04:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants