test: fuzz test for foyer-memory #337

xiaguan · 2024-04-14T11:07:50Z

What's changed and what's your intention?

Please explain IN DETAIL what the changes are in this PR and why they are needed. :D

As title

Checklist

I have written the necessary rustdoc comments
I have added the necessary unit tests and integration tests
I have passed make all (or make fast instead if the old tests are not modified) in my local environment.

Related issues or PRs (optional)

* chore: add license checker Signed-off-by: MrCroxx <[email protected]> * fix license checker config Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

…x#15) Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

* feat: introduce instrusive indexers and collections Signed-off-by: MrCroxx <[email protected]>

* feat: introduce FTL-like storage engine Signed-off-by: MrCroxx <[email protected]> * update ci Signed-off-by: MrCroxx <[email protected]> * sort cargo file Signed-off-by: MrCroxx <[email protected]> * fix memory leak Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

* feat: enable direct i/o on linux target - enable direct i/o on linux target - refine flusher and reclaimer Signed-off-by: MrCroxx <[email protected]> * fix unit test Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

) * chore: remove unused old storage engien and other components Signed-off-by: MrCroxx <[email protected]> * update ci Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

…xx#31) Signed-off-by: MrCroxx <[email protected]>

* feat: impl storage recovery Signed-off-by: MrCroxx <[email protected]> * chore Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

* feat: add segment fifo eviction policy Signed-off-by: MrCroxx <[email protected]> * export fifo fs store Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

* feat: export mods Signed-off-by: MrCroxx <[email protected]> * make cargo sort happy Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

This reverts commit a8a56cc.

Signed-off-by: MrCroxx <[email protected]>

* fix: deadlock fix deadlocks: 1. exclusive lock was not `Send` but unsafe impled. 2. submit flush task await blocks `set_region_evictable` 3. no new reclamation task created after `set_region_evctable` if all write process are waiting Signed-off-by: MrCroxx <[email protected]> * update ci Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

* fix: gracefullly shutdown runners Use `Store::shutdown_runners()` to gracefully shutdown runners.` Signed-off-by: MrCroxx <[email protected]> * log warn if send failed instead of panic Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

- fix region advance - fix recovery slice destroy - seal last dirty region when shutdown Signed-off-by: MrCroxx <[email protected]>

* feat: introduce prometheus metrics Signed-off-by: MrCroxx <[email protected]> * calc metrics Signed-off-by: MrCroxx <[email protected]> * fix registry metrics Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

…rCroxx#56) - support multiple admission or reinsertion policies - loose flush exclusive lock Signed-off-by: MrCroxx <[email protected]>

* feat: introduce rated random admission policy Signed-off-by: MrCroxx <[email protected]> * make clippy happy Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

) * chore: bump foyer-memory to 0.1.3 Signed-off-by: MrCroxx <[email protected]> * chore: bump foyer-intrusive to 0.3.1 Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

…roxx#314) Signed-off-by: MrCroxx <[email protected]>

…x#315) Signed-off-by: MrCroxx <[email protected]>

* refactor: remove usage of unstable features Signed-off-by: MrCroxx <[email protected]> * refactor: some more Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

…roxx#317) * refactor: use crate allocator_api2 to make foyer build on stable Signed-off-by: MrCroxx <[email protected]> * fix: use stable pipeline for ci tests, use nightly for sanitizer Signed-off-by: MrCroxx <[email protected]> * fix: update ci prefix key Signed-off-by: MrCroxx <[email protected]> * fix: try fix ci Signed-off-by: MrCroxx <[email protected]> * fix: update ci nightly version Signed-off-by: MrCroxx <[email protected]> * chore: try resolve github issue Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

* test: run CI on both linux and macos Signed-off-by: MrCroxx <[email protected]> * fix: fix build on macos Signed-off-by: MrCroxx <[email protected]> * fix: try fix ci run on targets Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

* doc: update README and add a simple example Signed-off-by: MrCroxx <[email protected]> * chore: update Cargo.toml Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

* chore: fix ci coverage Signed-off-by: MrCroxx <[email protected]> * chore: fix check Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

* refactor: remove Handle bound from Eviction Signed-off-by: MrCroxx <[email protected]> * refactor: loose bound of in-memory eviction related traits Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

* test: upload binaries when asan/lsan test fails Signed-off-by: MrCroxx <[email protected]> * fix: fix if condition Signed-off-by: MrCroxx <[email protected]> * fix: use different name for different artifacts Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: xiaguan <[email protected]>

MrCroxx · 2024-04-16T08:57:42Z

Sorry for the delay. I've been working on some refactors these days.

What's the size of the fuzzy test dataset? Should it also be uploaded?

xiaguan · 2024-04-16T09:16:17Z

foyer-memory/fuzz/fuzz_targets/cache_fuzz.rs

+#[derive(Debug, Arbitrary)]
+struct Input {
+ capacity: usize,
+ cache_type: CacheType,
+ operations: Vec<Op>,
+}


The cargo-fuzz just random generate the Input to max the code coverage with sanitizer.
The afl fuzz seems better one, it need some input to start,and it is hard to use and maintain.

MrCroxx added 30 commits May 26, 2023 16:35

chore: add license checker (MrCroxx#14)

5c09e60

* chore: add license checker Signed-off-by: MrCroxx <[email protected]> * fix license checker config Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

chore: wrapper cache type, reorg utils, add metrics framework (MrCrox…

e38ad6a

…x#15) Signed-off-by: MrCroxx <[email protected]>

fix: fix read only store recovery, impl random drop (MrCroxx#16)

f44aec1

Signed-off-by: MrCroxx <[email protected]>

feat: add metrics (MrCroxx#17)

8e80c6c

Signed-off-by: MrCroxx <[email protected]>

feat: add log support (MrCroxx#19)

79610aa

Signed-off-by: MrCroxx <[email protected]>

feat: add write stall config for read only file store (MrCroxx#20)

23e2612

Signed-off-by: MrCroxx <[email protected]>

feat: introduce instrusive indexers and collections (MrCroxx#21)

506f3ef

* feat: introduce instrusive indexers and collections Signed-off-by: MrCroxx <[email protected]>

chore: remove unused deps (MrCroxx#28)

64b72b4

Signed-off-by: MrCroxx <[email protected]>

feat: foyer storage bench support flexible writers and readers (MrCro…

85d8341

…xx#31) Signed-off-by: MrCroxx <[email protected]>

feat: impl storage recovery (MrCroxx#32)

7c81272

* feat: impl storage recovery Signed-off-by: MrCroxx <[email protected]> * chore Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

feat: introduce segment fifo eviction policy (MrCroxx#35)

ad3f125

* feat: add segment fifo eviction policy Signed-off-by: MrCroxx <[email protected]> * export fifo fs store Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

feat: export mods (MrCroxx#37)

98a0af7

* feat: export mods Signed-off-by: MrCroxx <[email protected]> * make cargo sort happy Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

fix: export extern crate (MrCroxx#38)

a8a56cc

Signed-off-by: MrCroxx <[email protected]>

Revert "fix: export extern crate (MrCroxx#38)" (MrCroxx#39)

09bf57a

This reverts commit a8a56cc.

fix: disable O_DIRECT on non-linux targets (MrCroxx#40)

691054a

Signed-off-by: MrCroxx <[email protected]>

chore: ignore Cargo.lock and bump toolchain (MrCroxx#46)

7fc8542

Signed-off-by: MrCroxx <[email protected]>

chore: export storage config (MrCroxx#50)

1018bca

Signed-off-by: MrCroxx <[email protected]>

fix: AdmitAll & ReinsertNone no longer requires KV default (MrCroxx#51)

011a401

Signed-off-by: MrCroxx <[email protected]>

fix: region advance & recovery & seal (MrCroxx#53)

e61eb5d

- fix region advance - fix recovery slice destroy - seal last dirty region when shutdown Signed-off-by: MrCroxx <[email protected]>

fix: recovery test (MrCroxx#55)

3c0a0ca

Signed-off-by: MrCroxx <[email protected]>

feat: support multiple admission or reinsertion policies, loose lock (M…

809ca83

…rCroxx#56) - support multiple admission or reinsertion policies - loose flush exclusive lock Signed-off-by: MrCroxx <[email protected]>

feat: introduce rated random admission policy (MrCroxx#57)

32ec880

* feat: introduce rated random admission policy Signed-off-by: MrCroxx <[email protected]> * make clippy happy Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

feat: add rated random option for bench args (MrCroxx#58)

d037322

fix: recovery run flushers and reclaimers first (MrCroxx#59)

e502b30

Signed-off-by: MrCroxx <[email protected]>

MrCroxx and others added 22 commits April 8, 2024 07:53

chore: bump foyer-storage to 0.5.1 (MrCroxx#309)

5899ee4

Signed-off-by: MrCroxx <[email protected]>

fix: fix state() for s3fifo (MrCroxx#312)

b50813b

Signed-off-by: MrCroxx <[email protected]>

chore: bump foyer-memory to 0.1.4 (MrCroxx#313)

5a410c8

Signed-off-by: MrCroxx <[email protected]>

refactor: use generic type for cursor instead of associated type (MrC…

823a879

…roxx#314) Signed-off-by: MrCroxx <[email protected]>

refactor: remove unstable feature let_chains and lint_reasons (MrCrox…

3aa689d

…x#315) Signed-off-by: MrCroxx <[email protected]>

refactor: remove usage of unstable features (MrCroxx#316)

2693669

* refactor: remove usage of unstable features Signed-off-by: MrCroxx <[email protected]> * refactor: some more Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

doc: update README and add a simple example (MrCroxx#319)

475aee0

* doc: update README and add a simple example Signed-off-by: MrCroxx <[email protected]> * chore: update Cargo.toml Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

chore: add msrv for foyer, update bager in readme (MrCroxx#321)

2abe3df

Signed-off-by: MrCroxx <[email protected]>

fix: do not truncate when open wal (MrCroxx#322)

cb2616e

Signed-off-by: MrCroxx <[email protected]>

chore: bump foyer to 0.7.0 (MrCroxx#323)

48c3990

Signed-off-by: MrCroxx <[email protected]>

refactor: loose query key bound for in-memory cache (MrCroxx#324)

cc64fc9

Signed-off-by: MrCroxx <[email protected]>

feat: impl contains for in-memory cache (MrCroxx#325)

e47c090

Signed-off-by: MrCroxx <[email protected]>

perf: add dynamic dispatch bench (MrCroxx#326)

818e3f9

Signed-off-by: MrCroxx <[email protected]>

chore: fix ci coverage (MrCroxx#331)

3fda8cc

* chore: fix ci coverage Signed-off-by: MrCroxx <[email protected]> * chore: fix check Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

refactor: refine generic handle type (MrCroxx#330)

6b820fa

Signed-off-by: MrCroxx <[email protected]>

test: fuzz test for foyer-memory

541a329

more readable fuzz test

1f7f231

Signed-off-by: xiaguan <[email protected]>

xiaguan force-pushed the memory_fuzz_test branch from 7943b4a to 1f7f231 Compare April 14, 2024 14:18

Merge branch 'main' into memory_fuzz_test

3f48bf4

xiaguan force-pushed the memory_fuzz_test branch from a33b1d0 to 348d135 Compare April 14, 2024 14:30

add license

75c8a51

Signed-off-by: xiaguan <[email protected]>

xiaguan force-pushed the memory_fuzz_test branch from 348d135 to 75c8a51 Compare April 14, 2024 14:57

xiaguan commented Apr 16, 2024

View reviewed changes

MrCroxx force-pushed the main branch from 04d5ac3 to a2b91e3 Compare April 17, 2024 01:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: fuzz test for foyer-memory #337

test: fuzz test for foyer-memory #337

xiaguan commented Apr 14, 2024 •

edited

MrCroxx commented Apr 16, 2024

xiaguan Apr 16, 2024

test: fuzz test for foyer-memory #337

Are you sure you want to change the base?

test: fuzz test for foyer-memory #337

Conversation

xiaguan commented Apr 14, 2024 • edited

What's changed and what's your intention?

Checklist

Related issues or PRs (optional)

MrCroxx commented Apr 16, 2024

xiaguan Apr 16, 2024

Choose a reason for hiding this comment

xiaguan commented Apr 14, 2024 •

edited