Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve nanosecond precision support for pcap #1368

Merged
merged 26 commits into from
May 24, 2024
Merged

Conversation

egecetin
Copy link
Collaborator

@egecetin egecetin commented Apr 18, 2024

Instead of compile time constant definitions use pcap_get_tstamp_precision() to detect timestamp precision which introduced with libpcap 1.5.1

This should also fix #906

  • Update pcap open functions to control precision
  • Add arguments to specify desired precision
  • Keep backward compatibility since original Winpcap does not support nanosecond precision
  • Manually (or with CI) test different precisions for pcap/pcapng files
    • Read Microseconds -> Write Microseconds
    • Read Microseconds -> Write Nanoseconds
    • Read Nanoseconds -> Write Microseconds
    • Read Nanoseconds -> Write Nanoseconds

@egecetin egecetin changed the title Runtime nanosecond precision support for pcap Improve nanosecond precision support for pcap Apr 18, 2024
@@ -61,6 +61,7 @@ Tests/Pcap++Test/cUrl/curl_output.txt
*.VC.db
*.vcxproj.user
**/.vs
CMakeSettings.json
Copy link
Collaborator Author

@egecetin egecetin May 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is Visual Studio generated file to hold configuration when remote compilation (ssh/wsl etc) requested or modify build mode from UI

@egecetin egecetin marked this pull request as ready for review May 11, 2024 11:56
@egecetin egecetin requested a review from seladb as a code owner May 11, 2024 11:56
@egecetin
Copy link
Collaborator Author

Manually tested, it looks like it works for pcap files. Can you also test before merging for different precisions?

@seladb
Copy link
Owner

seladb commented May 12, 2024

@egecetin why can't we write tests for it? 🤔

@egecetin
Copy link
Collaborator Author

@seladb If i can figure out a test flow yes, it is possible 😀. One way I'm thinking.

  • Craft two RawPackets with one microsecond and one nanosecond precision
  • Write both of them to two pcap files with one opened with microsecond and the other nanosecond
  • Read these files
  • Compare the timestamp headers of read packets with crafted ones. Microsecond precision should truncate, nanosecond should persist.

If I'm right only missed part with this flow, if somehow, I write mathematically inverse operations in write and read only the pcap file itself will be wrong but all other cases should be covered right? 🤔

@seladb
Copy link
Owner

seladb commented May 14, 2024

@seladb If i can figure out a test flow yes, it is possible 😀. One way I'm thinking.

* Craft two `RawPacket`s with one microsecond and one nanosecond precision

* Write both of them to two `pcap` files with one opened with microsecond and the other nanosecond

* Read these files

* Compare the timestamp headers of read packets with crafted ones. Microsecond precision should truncate, nanosecond should persist.

If I'm right only missed part with this flow, if somehow, I write mathematically inverse operations in write and read only the pcap file itself will be wrong but all other cases should be covered right? 🤔

Yes, I think it's a valid test that mostly covers what is mentioned in the ticket: #906 (comment)
You can even extend the test a bit more:

  • Craft two RawPackets with one microsecond and one nanosecond precision
  • Write both of them to two pcap files with one opened with microsecond and the other nanosecond
  • Read both files in 2 ways:
    1. Read microsecond file with microsecond precision
    2. Read microsecond file with nanosecond precision
    3. Read nanosecond file with microsecond precision
    4. Read nanosecond file with nanosecond precision
  • Make sure you get the expected result in each case:
    1. crafted packet timestamp == read packet timestamp
    2. crafted packet timestamp * 1000 == read packet timestamp
    3. crafted packet timestamp / 1000 == read packet timestamp
    4. crafted packet timestamp == read packet timestamp

@egecetin
Copy link
Collaborator Author

Actually, for the reading side there is no option implemented. It always uses as much high as precision always. So only WinPcap will use microsecond, and all other systems/tests will use nanosecond for reading mode. So, I added a function isTimestampPrecisionNanoSupported to check compiled binary supports the nano precision.

@egecetin
Copy link
Collaborator Author

@seladb It is ready to review again

Pcap++/header/PcapFileDevice.h Outdated Show resolved Hide resolved
Pcap++/header/PcapFileDevice.h Outdated Show resolved Hide resolved
Pcap++/header/PcapFileDevice.h Outdated Show resolved Hide resolved
Pcap++/header/PcapFileDevice.h Outdated Show resolved Hide resolved
Pcap++/src/PcapFileDevice.cpp Outdated Show resolved Hide resolved
Pcap++/src/PcapFileDevice.cpp Outdated Show resolved Hide resolved
@egecetin
Copy link
Collaborator Author

@seladb Probably I addressed all your comments. Can you review it again when you have time?

Pcap++/header/PcapFileDevice.h Outdated Show resolved Hide resolved
Pcap++/header/PcapFileDevice.h Outdated Show resolved Hide resolved
/**
* @return The precision of the timestamps in the file.
*/
FileTimestampPrecision getTimestampPrecision() const { return m_Precision; }
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For better test coverage, maybe we can add tests for this method?

It'll actually be interesting to test it before opening the file and after opening it

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You only added tests for PcapFileWriterDevice::getTimestampPrecision() but not for PcapFileReaderDevice::getTimestampPrecision()...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seladb libpcap scales precision because of pcap_open_offline_with_tstamp_precision behaviour. So, I added a test and updated comments of reader function. Can you take a look again before merging for anything i missed?

https://www.tcpdump.org/manpages/pcap_open_offline.3pcap.html

pcap_open_offline_with_tstamp_precision() takes an additional precision argument specifying the time stamp precision desired; if PCAP_TSTAMP_PRECISION_MICRO is specified, packet time stamps will be supplied in seconds and microseconds, and if PCAP_TSTAMP_PRECISION_NANO is specified, packet time stamps will be supplied in seconds and nanoseconds. If the time stamps in the file do not have the same precision as the requested precision, they will be scaled up or down as necessary before being supplied.

Pcap++/header/PcapFileDevice.h Outdated Show resolved Hide resolved
Pcap++/src/PcapFileDevice.cpp Outdated Show resolved Hide resolved
@egecetin egecetin requested a review from seladb May 23, 2024 09:26
Copy link
Owner

@seladb seladb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see this last comment: #1368 (comment)

Can you fix it before merging?

@egecetin egecetin merged commit ed6db69 into seladb:dev May 24, 2024
39 checks passed
@egecetin egecetin deleted the timestamp branch May 24, 2024 08:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants