Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Variable rate, on-demand transaction ingest #1

Open
myawnhc opened this issue Aug 6, 2019 · 2 comments
Open

Variable rate, on-demand transaction ingest #1

myawnhc opened this issue Aug 6, 2019 · 2 comments

Comments

@myawnhc
Copy link
Owner

myawnhc commented Aug 6, 2019

Current design: Transaction generator runs concurrently with the processing, generating random transactions that are written to the preAuth map. So transaction generation is competing for the same CPU cores that the processing needs.

Preferred design: Pregenerate transactions and store them (database or whatever); have an ingest mechanism that can vary in rate to drive different loads for testing. Rate should be variable to simulate spikes as expected in real-world workloads.

@myawnhc myawnhc added the hipri Hi Priority bug or feature label Aug 7, 2019
@myawnhc
Copy link
Owner Author

myawnhc commented Sep 18, 2019

This has been mostly completed - transactions are now pre-generated and stored into the database. Merchant and Account info is eagerly loaded at start-up, and the transaction info is then loaded once everything else has completed loading.

The variable rate for the ingest mechanism is not yet done, so leaving this open until I can circle back to that.

@myawnhc myawnhc changed the title Rework front-end Variable rate, on-demand transaction ingest Sep 18, 2019
@myawnhc myawnhc removed the hipri Hi Priority bug or feature label Sep 18, 2019
@myawnhc
Copy link
Owner Author

myawnhc commented Nov 2, 2020

Implemented, and rate is adjustable through some throttling behavior that somehow got reverted off main - the throttling controls are implemented in jet-engine branch, might merge changes separately or wait until that branch merges. (I think changes are limited to LazyPreAuthLoader and Launcher classes).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant