Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve streaming data rate #66

Open
aregtech opened this issue Nov 13, 2022 · 0 comments
Open

Improve streaming data rate #66

aregtech opened this issue Nov 13, 2022 · 0 comments
Labels
C++ challenge enhancement New feature or request help wanted Extra attention is needed optimization

Comments

@aregtech
Copy link
Owner

aregtech commented Nov 13, 2022

Goal
The example 20_pubdatarate (currently available in candidate branch) shows that the communication data rate is about 100-120 MB/s on the PC. This value should be increased to achieve at least 300 MB/s on the PC.

Motivation
AREG SDK should handle high data rate without a problem (the minimum data rate on PC should be 300 MB/s). In most of the cases, the data rate should be dependent on CPU / HW, and less dependent on the code implementation. Currently nearly half of delay is because of the code. In the current implementation we serialize data in streaming object then copy data in the message. The creating event object for each target, copying data in the message, applying CRC cause delays.

Background information
The events are instantiated for each client. Currently the data is serialized in the shared buffer, but then they are copied to each message.
Analyze the possibility:

  • check presence of the event listeners before starting serialization. Escape serialization if none is present.
  • create an event only once for all targets. Instead, set the list of targets sorted by channels.
  • the list should be handled by mcrouter, so that the source sends an event one time.

The current business logic serializes data in the shared buffer then copied in the message.

  • analyze the possibility to unify messages (unify protocol) for IPC and local targets.
  • make message generic and independent on target location / channel.
  • Pay attention on mechanism of applying CRC. Maybe apply the CRC only on data and escape target / source.

Test changes using 20_pubdatarate

@aregtech aregtech added enhancement New feature or request C++ optimization labels Nov 13, 2022
@aregtech aregtech added the help wanted Extra attention is needed label Jun 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C++ challenge enhancement New feature or request help wanted Extra attention is needed optimization
Projects
Status: No status
Development

No branches or pull requests

1 participant