Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Improve networking code pt 1 #10219

Merged
merged 14 commits into from May 9, 2024
Merged

Conversation

ajbt200128
Copy link
Contributor

@ajbt200128 ajbt200128 commented May 7, 2024

This PR is part 1 of 2 of improvements I plan to make to the networking library. The next PR will introduce a custom client that will (optionally) use proxies when establishing connections directly, instead of the hacky way we're doing rn that abuses Cohttp.

Changes:

  • remove lwt platform functor
    • this was ugly, and we use the async functions pretty much everywhere already, so let's just commit
    • consumers of http_helpers can use lwt_platform.run if they need sync
  • Bump cohttp to 6.0.0 beta 2
    • @akuhlens has a PR that will come out in the release version of the library, and there's been a bunch of improvements anyways, so let's just use it
    • The proxy changes I'm going to make I want to upstream, so might as well do them in the version they might be released in instead of writing 2 different versions
  • Better error handling/typing
    • Before we were assuming cohttp doesn't explode. But it does sometimes. So let's have http helpers return a result on whether or not the network request actually happened, and within that another result on what the response from the server was, if it was ok or not

Semgrep Pro PR

https://github.com/semgrep/semgrep-proprietary/pull/1488

Test plan

Tests pass

pretty soon we will want to upstream our proxy changes or at least fork
cohttp. Either way we need to use the new version
Before we were assuming that cohttp doesn't blow up over random stuff.
But it does. So lets improve our http helpers api to wrap these errors
Copy link
Contributor

github-actions bot commented May 7, 2024

PR checklist:

  • Purpose of the code is evident to future readers
  • Tests included or PR comment includes a reproducible test plan
  • Documentation is up-to-date
  • A changelog entry was added to changelog.d for any user-facing change
  • Change has no security implications (otherwise, ping security team)

If you're unsure about any of this, please see:

@ajbt200128
Copy link
Contributor Author

ajbt200128 commented May 7, 2024

It looks like the new version of cohttp relies on opam 2.1, which windows doesn't have :/ so not sure if the cohttp bump part of this PR is possible. I may just bump cohttp back down a version

Dockerfile Outdated Show resolved Hide resolved
@r2c-argo
Copy link
Contributor

r2c-argo bot commented May 8, 2024

semgrep-compare-github-qsndg results

Ran benchmark on 38 repositories

The number of findings differs for 3 repos

Whole benchmark is 1.3% faster (a bit of noise is expected)

Relative speed improvement is 1.01 on average

Relative memory improvement is 0.99 on average

Copy link
Contributor

@kopecs kopecs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall the API updates look good. Some various questions/other minor improvements suggested.

A couple spots I would want to see more docs before merging but other stuff is mostly informative/nits.

dune-project Outdated Show resolved Hide resolved
libs/networking/http_helpers/http_helpers.ml Show resolved Hide resolved
libs/networking/http_helpers/http_helpers.ml Outdated Show resolved Hide resolved
libs/networking/http_helpers/http_helpers.ml Outdated Show resolved Hide resolved
libs/networking/http_helpers/http_helpers.ml Outdated Show resolved Hide resolved
src/osemgrep/networking/Semgrep_App.ml Outdated Show resolved Hide resolved
src/osemgrep/networking/Unit_Networking.ml Show resolved Hide resolved
src/osemgrep/networking/Unit_Networking.ml Outdated Show resolved Hide resolved
src/osemgrep/networking/Unit_Networking.ml Outdated Show resolved Hide resolved
src/osemgrep/networking/Semgrep_App.ml Show resolved Hide resolved
@r2c-argo
Copy link
Contributor

r2c-argo bot commented May 8, 2024

semgrep-compare-github-e40oh results

Ran benchmark on 38 repositories

The number of findings differs for 4 repos

Whole benchmark is 4.0% faster (a bit of noise is expected)

Relative speed improvement is 1.01 on average

  • 3% of scans are significantly faster
  • 3% of scans are significantly slower

Relative memory improvement is 0.99 on average

@r2c-argo
Copy link
Contributor

r2c-argo bot commented May 8, 2024

semgrep-compare-github-p9179 results

Ran benchmark on 38 repositories

The number of files checked differs for 1 repos

The number of findings differs for 3 repos

Whole benchmark is 0.3% slower (a bit of noise is expected)

Relative speed improvement is 1.01 on average

Relative memory improvement is 0.99 on average

@r2c-argo
Copy link
Contributor

r2c-argo bot commented May 8, 2024

semgrep-compare-github-vxc5f results

Ran benchmark on 38 repositories

The number of findings differs for 2 repos

Whole benchmark is 2.9% slower (a bit of noise is expected)

Relative speed improvement is 0.98 on average

  • 3% of scans are significantly slower

Relative memory improvement is 0.99 on average

@r2c-argo
Copy link
Contributor

r2c-argo bot commented May 8, 2024

semgrep-compare-github-4y5o0 results

Ran benchmark on 38 repositories

The number of findings differs for 3 repos

Whole benchmark is 1.3% faster (a bit of noise is expected)

Relative speed improvement is 1.01 on average

  • 3% of scans are significantly faster

Relative memory improvement is 0.99 on average

@r2c-argo
Copy link
Contributor

r2c-argo bot commented May 8, 2024

semgrep-compare-github-7zrkd results

Ran benchmark on 38 repositories

The number of findings differs for 2 repos

Whole benchmark is 0.6% slower (a bit of noise is expected)

Relative speed improvement is 0.98 on average

  • 3% of scans are significantly slower

Relative memory improvement is 0.99 on average

@ajbt200128 ajbt200128 merged commit 1e46fb5 into develop May 9, 2024
47 of 49 checks passed
@ajbt200128 ajbt200128 deleted the austin/ocaml-proxy-improvements branch May 9, 2024 22:43
Copy link
Collaborator

@aryx aryx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Glad we removed a functor! I hate functors!

dune-project Show resolved Hide resolved
Http_helpers.post ~body ~headers caps#network url
match%lwt Http_helpers.post ~body ~headers caps#network url with
| Ok { body = Ok body; _ } -> Lwt.return_ok body
| Ok { body = Error msg; code; _ } ->
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the error management seems a bit heavier now ... it's not possible to factorize
a bit so we just handle Ok body = Ok body, and then whatever else as a general error?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants