Parse DNS over TCP messages #6935

errorxyz · 2024-06-17T18:02:34Z

Description

Changes made in this PR allow parsing of DNS over TCP messages. This is done by extracting the length label from the raw data and waiting until we receive data of that size.
Also handles query pipelining

Continuation of #6912

Checklist

I have updated tests where applicable.
I have added an entry to the CHANGELOG.

mhils

This looks great so far, but definitely needs tests and fuzz tests! :)

mitmproxy/addons/next_layer.py

mitmproxy/proxy/layers/dns.py

mhils

This looks super great % nits!

mitmproxy/proxy/layers/dns.py

test/mitmproxy/proxy/test_mode_servers.py

errorxyz · 2024-06-23T16:53:58Z

Do we need to update these tests to use TCP as well?

mitmproxy/test/mitmproxy/addons/test_proxyserver.py

Line 281 in b8fd7eb

async def test_dns(caplog_async) -> None:

mitmproxy/test/mitmproxy/proxy/layers/test_modes.py

Line 176 in b8fd7eb

def test_reverse_dns(tctx):

errorxyz · 2024-06-23T18:45:52Z

How do we fix the failing test?

mhils · 2024-06-24T10:41:31Z

How do we fix the failing test?

We specify dependency version ranges where possible so that downstream packagers have an easier time packaging mitmproxy. If we realize we're incompatible with an older version, we just bump the minimum in pyproject.toml! I try to make a bit of an effort to keep the range as wide as possible, but if it's not trivial to figure out I just raise it to the latest release.

mitmproxy/proxy/layers/dns.py

* Update CHANGELOG.md * Autofix generated JS files and do not patch them in tests (#6910) * autofix generated JS files and do not patch them in tests * autofix: setup python * [autofix.ci] apply automated fixes * autofix: setup node * add missing newline * fixup --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Update package-lock.json, bump esbuild (#6915) update node version, bump esbuild * Fix clipboard handling in safari (#6917) * fix clipboard handling in safari closes #6911, #6909 Co-authored-by: Can Yesilyurt <[email protected]> * [autofix.ci] apply automated fixes * update dependencies --------- Co-authored-by: Can Yesilyurt <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * mitmproxy 10.3.1 * reopen main for development * release script: add one less newline * fix zstd decompression (#6921) * fix zstd decompression (issue #6914) * add our fix to CHANGELOG * add explicit read_across_frames=True + move zstd test to test_encoding.py --------- Co-authored-by: Maximilian Hils <[email protected]> * Update CHANGELOG.md * web: Upgrade Redux (#6926) * update redux and fix resulting test and type failures * update prettier * Use upstream urwid again (#6929) use upstream urwid again * Add `HttpConnectedHook` and `HttpConnectErrorHook` (#6930) * Add HttpConnectedHook and HttpConnectErrorHook * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * make DNS mode listen for both UDP and TCP (#6912) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * release ci: strip "v" prefix from sigstore file * fix: OSError raised when ipv6 is disabled (#6942) the function should return None in this case * Update autofix action (#6946) update autofix action * Fix issue #6944: Optimize response/request_body_buf by using list of byte chunks to avoid concatenation overhead (#6952) * Fix the issue #6944: non-linear growth in processing time with mitmproxy regarding packet size. Replace the string (bytes) concatenation implementation of request_body_buf and response_body_buf to a list of chunk of bytes. * Update the CHANGELOG.md regarding issue #6944 * [autofix.ci] apply automated fixes * Update CHANGELOG.md * [autofix.ci] apply automated fixes * move receivebuffer into dedicated class * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Maximilian Hils <[email protected]> * Use existing API to unpack/pack domain names in HTTPS records (#6949) * Use existing API to unpack/pack domain names in HTTPS records * [autofix.ci] apply automated fixes * tangentially related grammar fix --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Maximilian Hils <[email protected]> * Add support for DNS over TCP (#6935) * Parse DNS over TCP messages * [autofix.ci] apply automated fixes * Update _next_layer to detect DNS over TCP messages * [autofix.ci] apply automated fixes * Fixup * allow query pipelining * [autofix.ci] apply automated fixes * Restructure pack/unpack mechanism and update tests * Add tests * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes * Update next_layer test * Add suggested changes * [autofix.ci] apply automated fixes * bump minimum hypothesis version * bump hypothesis for good * Update CHANGELOG.md --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Maximilian Hils <[email protected]> * web: fix "show more" button (#6958) fix #6951 --------- Co-authored-by: Maximilian Hils <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Can Yesilyurt <[email protected]> Co-authored-by: mitmproxy release bot <[email protected]> Co-authored-by: Andras Spitzer <[email protected]> Co-authored-by: Gaurav Jain <[email protected]> Co-authored-by: Walt Chen <[email protected]> Co-authored-by: jackfromeast <[email protected]> Co-authored-by: Maximilian Hils <[email protected]> Co-authored-by: Alexander Prinzhorn <[email protected]>

commit 5ca998d Author: Matteo Luppi <[email protected]> Date: Wed Jun 26 17:27:47 2024 +0200 Sync capture tab to main (#6963) * Update CHANGELOG.md * Autofix generated JS files and do not patch them in tests (#6910) * autofix generated JS files and do not patch them in tests * autofix: setup python * [autofix.ci] apply automated fixes * autofix: setup node * add missing newline * fixup --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Update package-lock.json, bump esbuild (#6915) update node version, bump esbuild * Fix clipboard handling in safari (#6917) * fix clipboard handling in safari closes #6911, #6909 Co-authored-by: Can Yesilyurt <[email protected]> * [autofix.ci] apply automated fixes * update dependencies --------- Co-authored-by: Can Yesilyurt <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * mitmproxy 10.3.1 * reopen main for development * release script: add one less newline * fix zstd decompression (#6921) * fix zstd decompression (issue #6914) * add our fix to CHANGELOG * add explicit read_across_frames=True + move zstd test to test_encoding.py --------- Co-authored-by: Maximilian Hils <[email protected]> * Update CHANGELOG.md * web: Upgrade Redux (#6926) * update redux and fix resulting test and type failures * update prettier * Use upstream urwid again (#6929) use upstream urwid again * Add `HttpConnectedHook` and `HttpConnectErrorHook` (#6930) * Add HttpConnectedHook and HttpConnectErrorHook * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * make DNS mode listen for both UDP and TCP (#6912) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * release ci: strip "v" prefix from sigstore file * fix: OSError raised when ipv6 is disabled (#6942) the function should return None in this case * Update autofix action (#6946) update autofix action * Fix issue #6944: Optimize response/request_body_buf by using list of byte chunks to avoid concatenation overhead (#6952) * Fix the issue #6944: non-linear growth in processing time with mitmproxy regarding packet size. Replace the string (bytes) concatenation implementation of request_body_buf and response_body_buf to a list of chunk of bytes. * Update the CHANGELOG.md regarding issue #6944 * [autofix.ci] apply automated fixes * Update CHANGELOG.md * [autofix.ci] apply automated fixes * move receivebuffer into dedicated class * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Maximilian Hils <[email protected]> * Use existing API to unpack/pack domain names in HTTPS records (#6949) * Use existing API to unpack/pack domain names in HTTPS records * [autofix.ci] apply automated fixes * tangentially related grammar fix --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Maximilian Hils <[email protected]> * Add support for DNS over TCP (#6935) * Parse DNS over TCP messages * [autofix.ci] apply automated fixes * Update _next_layer to detect DNS over TCP messages * [autofix.ci] apply automated fixes * Fixup * allow query pipelining * [autofix.ci] apply automated fixes * Restructure pack/unpack mechanism and update tests * Add tests * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes * Update next_layer test * Add suggested changes * [autofix.ci] apply automated fixes * bump minimum hypothesis version * bump hypothesis for good * Update CHANGELOG.md --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Maximilian Hils <[email protected]> * web: fix "show more" button (#6958) fix #6951 --------- Co-authored-by: Maximilian Hils <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Can Yesilyurt <[email protected]> Co-authored-by: mitmproxy release bot <[email protected]> Co-authored-by: Andras Spitzer <[email protected]> Co-authored-by: Gaurav Jain <[email protected]> Co-authored-by: Walt Chen <[email protected]> Co-authored-by: jackfromeast <[email protected]> Co-authored-by: Maximilian Hils <[email protected]> Co-authored-by: Alexander Prinzhorn <[email protected]> commit e047e76 Author: Matteo Luppi <[email protected]> Date: Wed Jun 26 13:07:27 2024 +0200 Feature/wireguard mode (#6959) * define first components * Update CHANGELOG.md * Autofix generated JS files and do not patch them in tests (#6910) * autofix generated JS files and do not patch them in tests * autofix: setup python * [autofix.ci] apply automated fixes * autofix: setup node * add missing newline * fixup --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Update package-lock.json, bump esbuild (#6915) update node version, bump esbuild * Fix clipboard handling in safari (#6917) * fix clipboard handling in safari closes #6911, #6909 Co-authored-by: Can Yesilyurt <[email protected]> * [autofix.ci] apply automated fixes * update dependencies --------- Co-authored-by: Can Yesilyurt <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * define first modes components * adapt code to reuse checkbox component * mitmproxy 10.3.1 * reopen main for development * [autofix.ci] apply automated fixes * improve mode toggle design * remove inline style * release script: add one less newline * fix zstd decompression (#6921) * fix zstd decompression (issue #6914) * add our fix to CHANGELOG * add explicit read_across_frames=True + move zstd test to test_encoding.py --------- Co-authored-by: Maximilian Hils <[email protected]> * first attempt to make the modes functional * [autofix.ci] apply automated fixes * Update CHANGELOG.md * web: Upgrade Redux (#6926) * update redux and fix resulting test and type failures * update prettier * refactor code according to review * [autofix.ci] apply automated fixes * first prototype regular mode * [autofix.ci] apply automated fixes * Use upstream urwid again (#6929) use upstream urwid again * change name regular duck * Add `HttpConnectedHook` and `HttpConnectErrorHook` (#6930) * Add HttpConnectedHook and HttpConnectErrorHook * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * rewrite updateMode function and error handling regular mode * remove label component * add input field to local mode and fix onKeyDown issue * add dropdown to reverse mode * [autofix.ci] apply automated fixes * change defualt string reverse dropdown * add new logic to handle modes in the duck * add local mode with no applications * [autofix.ci] apply automated fixes * regular mode is now persistent * forgot recieve event in regular mode * join receive and update event * make local mode functional * make wireguard mode functional * [autofix.ci] apply automated fixes * make reverse mode functional * [autofix.ci] apply automated fixes * fix bug reverse mode when refreshing * fix bug local mode * [autofix.ci] apply automated fixes * fix old test * implement first tests * [autofix.ci] apply automated fixes * make DNS mode listen for both UDP and TCP (#6912) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * remove wireguard and reverse modes * change name function to set local applications * remove error handling * remove left over * add some other review changes * [autofix.ci] apply automated fixes * release ci: strip "v" prefix from sigstore file * fix addListen Addr function * adjust reverse protocol attribute * create proper method to parse the mode * adjust passing applications to local mode * fix onKeyDown listener and add new input field to local mode * [autofix.ci] apply automated fixes * remove leftover * update tests * [autofix.ci] apply automated fixes * remove wireguard and reverse files * remove additional state local mode * adjust tests local mode * update tests local mode * update last tests modes * [autofix.ci] apply automated fixes * fix: OSError raised when ipv6 is disabled (#6942) the function should return None in this case * use fetchMock instead of jest.mock this is slightly nicer because we are testing at the application boundary * duck tests should use action creators and not manually construct actions * move toggleLocal to fetchMock * move updateMode and ModeState into utils this avoids circular imports. utils may not be the perfect place, but much better than the circular imports * nits * [autofix.ci] apply automated fixes * update tests regular and local mode * [autofix.ci] apply automated fixes * review changes tests * [autofix.ci] apply automated fixes * adjust parseMode issue * [autofix.ci] apply automated fixes * add first components for wireguard mode * fix CI * fix typo * add tests to wireguard mode * [autofix.ci] apply automated fixes --------- Co-authored-by: Maximilian Hils <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Can Yesilyurt <[email protected]> Co-authored-by: mitmproxy release bot <[email protected]> Co-authored-by: Andras Spitzer <[email protected]> Co-authored-by: Gaurav Jain <[email protected]> Co-authored-by: Walt Chen <[email protected]> commit d9770ad Author: Matteo Luppi <[email protected]> Date: Mon Jun 24 19:40:47 2024 +0200 Feature/implement mvp new tab page (#6923) commit afb22c2 Author: Maximilian Hils <[email protected]> Date: Fri Jun 21 23:09:31 2024 +0200 Sync `capture-tab` with `main` (#6943) * Update CHANGELOG.md * Autofix generated JS files and do not patch them in tests (#6910) * autofix generated JS files and do not patch them in tests * autofix: setup python * [autofix.ci] apply automated fixes * autofix: setup node * add missing newline * fixup --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Update package-lock.json, bump esbuild (#6915) update node version, bump esbuild * Fix clipboard handling in safari (#6917) * fix clipboard handling in safari closes #6911, #6909 Co-authored-by: Can Yesilyurt <[email protected]> * [autofix.ci] apply automated fixes * update dependencies --------- Co-authored-by: Can Yesilyurt <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * mitmproxy 10.3.1 * reopen main for development * release script: add one less newline * fix zstd decompression (#6921) * fix zstd decompression (issue #6914) * add our fix to CHANGELOG * add explicit read_across_frames=True + move zstd test to test_encoding.py --------- Co-authored-by: Maximilian Hils <[email protected]> * Update CHANGELOG.md * web: Upgrade Redux (#6926) * update redux and fix resulting test and type failures * update prettier * Use upstream urwid again (#6929) use upstream urwid again * Add `HttpConnectedHook` and `HttpConnectErrorHook` (#6930) * Add HttpConnectedHook and HttpConnectErrorHook * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * make DNS mode listen for both UDP and TCP (#6912) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * release ci: strip "v" prefix from sigstore file * fix: OSError raised when ipv6 is disabled (#6942) the function should return None in this case * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Can Yesilyurt <[email protected]> Co-authored-by: mitmproxy release bot <[email protected]> Co-authored-by: Andras Spitzer <[email protected]> Co-authored-by: Gaurav Jain <[email protected]> Co-authored-by: Walt Chen <[email protected]> commit d0867f6 Author: Matteo Luppi <[email protected]> Date: Mon Jun 10 11:27:54 2024 +0200 Feature/add new capture tab (#6887) * add just the new tab to the header * remove header menu from capture tab * create context to handle the current active menu tab * update tests * remove useless comments * update tests to reach higher coverage * remove context, use props to achieve same result * update tests * [autofix.ci] apply automated fixes * add test for capture menu component * adjust changelog format * [autofix.ci] apply automated fixes * fix CHANGELOG --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Maximilian Hils <[email protected]> Co-authored-by: Maximilian Hils <[email protected]>

errorxyz and others added 2 commits June 17, 2024 23:27

Parse DNS over TCP messages

e6e996a

[autofix.ci] apply automated fixes

7ed4259

errorxyz requested review from mhils and removed request for mhils June 17, 2024 18:05

Update _next_layer to detect DNS over TCP messages

f095101

errorxyz force-pushed the DoT branch from 713fb69 to f095101 Compare June 18, 2024 09:12

autofix-ci bot and others added 2 commits June 18, 2024 09:13

[autofix.ci] apply automated fixes

25d40f0

Fixup

2dcaee6

mhils reviewed Jun 18, 2024

View reviewed changes

mitmproxy/addons/next_layer.py Show resolved Hide resolved

mitmproxy/proxy/layers/dns.py Outdated Show resolved Hide resolved

mitmproxy/proxy/layers/dns.py Outdated Show resolved Hide resolved

errorxyz and others added 2 commits June 21, 2024 21:59

allow query pipelining

bf55d5c

[autofix.ci] apply automated fixes

115e3ab

errorxyz commented Jun 21, 2024

View reviewed changes

mitmproxy/proxy/layers/dns.py Outdated Show resolved Hide resolved

errorxyz added 2 commits June 22, 2024 20:41

Restructure pack/unpack mechanism and update tests

aae8be3

Add tests

8a5fc39

errorxyz force-pushed the DoT branch from b198c54 to 8a5fc39 Compare June 22, 2024 18:45

[autofix.ci] apply automated fixes

dce2d3a

errorxyz force-pushed the DoT branch 2 times, most recently from 1dad59d to dce2d3a Compare June 22, 2024 19:00

[autofix.ci] apply automated fixes (attempt 2/3)

49907c2

errorxyz marked this pull request as ready for review June 22, 2024 19:04

errorxyz requested a review from mhils June 22, 2024 19:08

autofix-ci bot and others added 4 commits June 23, 2024 00:40

[autofix.ci] apply automated fixes

29da32f

Update CHANGELOG.md

f9bcae9

[autofix.ci] apply automated fixes

4aba5e1

Update next_layer test

43d0171

errorxyz mentioned this pull request Jun 23, 2024

Add support for DNS over TCP #6912

Merged

2 tasks

mhils reviewed Jun 23, 2024

View reviewed changes

mitmproxy/proxy/layers/dns.py Outdated Show resolved Hide resolved

mitmproxy/proxy/layers/dns.py Outdated Show resolved Hide resolved

test/mitmproxy/proxy/test_mode_servers.py Show resolved Hide resolved

Add suggested changes

6ffdf12

[autofix.ci] apply automated fixes

24291f3

mhils and others added 3 commits June 24, 2024 12:36

bump minimum hypothesis version

6df2c1b

Merge branch 'main' into DoT

7fa54e9

bump hypothesis for good

6f7bb73

mhils reviewed Jun 24, 2024

View reviewed changes

mitmproxy/proxy/layers/dns.py Show resolved Hide resolved

Update CHANGELOG.md

fa2722a

mhils approved these changes Jun 24, 2024

View reviewed changes

mhils enabled auto-merge (squash) June 24, 2024 10:46

mhils disabled auto-merge June 24, 2024 10:46

mhils enabled auto-merge (squash) June 24, 2024 10:46

mhils merged commit 7f7cb28 into mitmproxy:main Jun 24, 2024
23 checks passed

errorxyz deleted the DoT branch June 24, 2024 11:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parse DNS over TCP messages #6935

Parse DNS over TCP messages #6935

errorxyz commented Jun 17, 2024 •

edited

Loading

mhils left a comment

mhils left a comment

errorxyz commented Jun 23, 2024 •

edited

Loading

errorxyz commented Jun 23, 2024

mhils commented Jun 24, 2024

Parse DNS over TCP messages #6935

Parse DNS over TCP messages #6935

Conversation

errorxyz commented Jun 17, 2024 • edited Loading

Description

Checklist

mhils left a comment

Choose a reason for hiding this comment

mhils left a comment

Choose a reason for hiding this comment

errorxyz commented Jun 23, 2024 • edited Loading

errorxyz commented Jun 23, 2024

mhils commented Jun 24, 2024

errorxyz commented Jun 17, 2024 •

edited

Loading

errorxyz commented Jun 23, 2024 •

edited

Loading