Integrate `util/groupcommit` package with the coordinator table commit #1728

komamitsu · 2024-05-14T10:21:29Z

Description

We've already added util/groupcommit package, but not used yet. This PR integrate the group commit with the coordinator table writes.

Related issues and/or PRs

Add util/groupcommit package in core subproject #1607

Changes made

Add CoordinatorGroupCommitter that inherits GroupCommitter and CommitHandlerWithGroupCommit that inherits CommitHandler
- They write a record to coordinator.state table as Emitter.emit()
- There are 2 types of coordinator.state table record
  - In the case of normal group commit, the columns of a record are as follows:
    - tx_id: $parent_id (a key of the normal group contains multiple transactions)
    - tx_child_ids: $child_id_1,$child_id_2,...,$child_id_n (values of the normal group contains multiple transactions)
  - In the case of delayed group commit which contains a single transaction isolated from a normal group due to delayed transaction, the columns of a record are as follows:
    - tx_id: $parent_id:$child_id (a key and a value of the delayed group contains a single transaction)
    - tx_child_ids: Empty
Modify ConsensusCommitConfig, ConsensusCommit, ConsensusCommitManager, TwoPhaseConsensusCommit, and TwoPhaseConsensusCommitManager to use CommitHandlerWithGroupCommit
Modify util/groupcommit package for fixing some issues occurred in this integration
- Sorry, this includes an improvement changing oldGroupAbortTimeoutSeconds to oldGroupAbortTimeoutMillis that can be handled in another PR...
Add some unit tests and modify some existing tests
Modify some existing integration tests
- ConsensusJdbcEnv is introduced to inject the group commit related properties.

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes.
Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
Tests (unit, integration, etc.) have been added for the changes.
My changes generate no new warnings.
Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

Transaction ID conventions

Parent ID
- A key of normal group that contains multiple transactions
- coordinator.state.tx_id will be set to this ID when storing transactions in a normal group
- This consists of 24 length 62 type characters [0-9A-Za-z]
Child ID
- A key to identify each transaction in a group
- coordinator.state.tx_child_ids will contain this type of ID when storing transactions in a normal group
- This will be UUID v4 when automatically generating on ScalarDB side or an arbitrary string passed by users via begin(String txId)
Full ID
- A concatenated string of Parent ID and Child ID for a transaction
- coordinator.state.tx_id will be set to this ID when storing transactions in a delayed group that contains only a single transaction (coordinator.state.tx_child_ids will be empty in this case)
- The format of this ID is $ParentID:$ChildID whose total length is 61 (24 + 36 + 1). So we don't need to extend the maximum length of coordinator.state.tx_id

Limitations

Release notes

Added Group Commit feature for Coordinator Table

komamitsu · 2024-05-15T08:29:19Z

...scalar/db/storage/jdbc/ConsensusCommitCrossPartitionScanIntegrationTestWithJdbcDatabase.java

@@ -8,6 +8,6 @@ public class ConsensusCommitCrossPartitionScanIntegrationTestWithJdbcDatabase

 @Override
 protected Properties getProps(String testName) {
- return JdbcEnv.getProperties(testName);
+ return ConsensusCommitJdbcEnv.getProperties(testName);


ConsensusCommitJdbcEnv is introduced to inject GroupCommit related properties.

Sorry, we might have already discussed it, but do we add integration tests for GroupCommit only for JDBC, right?

IIRC, this is a kind of trade-off topic between the cost and completeness, based on the memory of the discussion we had before. The reason why I picked JDBC is the current main target of the group commit feature is YugabyteDB.

But, on second thought, it might be better to add the group commit integration tests with other storages. Maybe scheduled weekly integration test with all the storages is a good trade-off? If it's okay, I'll take care of it as another PR later. What do you think?

Yeah, I also think it is better to add the group commit integration tests with other storages. But I'm wondering if we should add them to the regular integration tests. You assume they will impact the integration test time, right?

Yes. In the case of https://github.com/scalar-labs/scalardb/actions/runs/9202687096, the group commit integration test using PostgreSQL which executed only **ConsensusCommit** cases took 12 minutes, while the full integration test using PostgreSQL without the group commit feature took only 2 minutes. (It was basically because of frequent open/close CoordinatorGroupCommitter and garbage ongoing transactions that were not explicitly rollbacked, though.)

So, this is a very rough estimate, but it may take 6 times larger computation resources of the GitHub Actions in total. I think the combination of 1) the group commit integration test only using PostgreSQL in the regular workflow, and 2) testing using all the storages with the group commit feature enabled in a weekly or daily workflow. 1 is already implemented in this PR, and 2 will be added in another PR later. What do you think?

Okay, that sounds good. I thought we should run CI for the group commit feature with other storages as much as possible before pushing changes, because we should detect bugs early. However, it seems like this significantly impacts the regular CI time.

Let's add item 2 in another PR. Also, maybe we should add the workflow_dispatch trigger to the workflow so that we can manually run the CI for the group commit feature with other storages when we make changes related to the group commit feature.

komamitsu · 2024-05-15T08:31:19Z

core/src/main/java/com/scalar/db/transaction/consensuscommit/CommitHandler.java

 }

 commitState(snapshot);
 commitRecords(snapshot);
 }

+ protected void handleCommitConflict(Snapshot snapshot, Exception cause)


Extracted this from commitState() so that CommitHandlerWithGroupCommit can reuse the impl

komamitsu · 2024-05-15T10:04:50Z

core/src/main/java/com/scalar/db/util/groupcommit/DelayedGroup.java

@@ -17,7 +17,7 @@ class DelayedGroup<PARENT_KEY, CHILD_KEY, FULL_KEY, EMIT_KEY, V>
 FULL_KEY fullKey,
 Emittable<EMIT_KEY, V> emitter,
 KeyManipulator<PARENT_KEY, CHILD_KEY, FULL_KEY, EMIT_KEY> keyManipulator) {
- super(emitter, keyManipulator, 1, config.oldGroupAbortTimeoutSeconds());
+ super(emitter, keyManipulator, 1, config.oldGroupAbortTimeoutMillis());


Strictly speaking, this refactoring isn't directly related to this PR.

komamitsu · 2024-05-15T10:06:58Z

core/src/test/java/com/scalar/db/transaction/consensuscommit/CommitHandlerTest.java

- @Mock private Coordinator coordinator;
- @Mock private TransactionTableMetadataManager tableMetadataManager;
- @Mock private ConsensusCommitConfig config;
+ @Mock protected DistributedStorage storage;


This class is inherited by CommitHandlerWithGroupCommitTest and these variables are used by it.

josh-wong

Looking good! I've added some comments and suggestions. PTAL!

core/src/main/java/com/scalar/db/util/groupcommit/GroupManager.java

docs/configurations.md

docs/api-guide.md

Co-authored-by: Josh Wong <[email protected]>

josh-wong

I've left some minor suggestions. Other than that, LGTM. Thank you!🙇🏻‍♂️

docs/api-guide.md

brfrn169

Overall LGTM, but I'll review this again later. Left several comments. Please take a look when you have time!

brfrn169 · 2024-05-20T04:28:16Z

...scalar/db/storage/jdbc/ConsensusCommitCrossPartitionScanIntegrationTestWithJdbcDatabase.java

@@ -8,6 +8,6 @@ public class ConsensusCommitCrossPartitionScanIntegrationTestWithJdbcDatabase

 @Override
 protected Properties getProps(String testName) {
- return JdbcEnv.getProperties(testName);
+ return ConsensusCommitJdbcEnv.getProperties(testName);


Sorry, we might have already discussed it, but do we add integration tests for GroupCommit only for JDBC, right?

brfrn169 · 2024-05-20T08:37:12Z

.../src/main/java/com/scalar/db/transaction/consensuscommit/TwoPhaseConsensusCommitManager.java

@@ -137,19 +157,34 @@ TwoPhaseCommitTransaction join(String txId, Isolation isolation, SerializableStr
 return resume(txId);
 }

- return createNewTransaction(txId, isolation, strategy);
+ // Participant services don't use the group commit feature even if it's enabled. They simply use


Could you please elaborate why participants don't use the group commit feature even if it's enabled?

Sure thing!

How GroupCommitter places transactions over multiple groups depends on various timings, including some types of timeouts (e.g., moving delayed transactions to a DelayedGroup) and events (e.g., GroupCommitter.ready(SnapShot) is called after the prepare and validate phases are done). So, it's unlikely that the Coordinator service and all the Participant services will have the same the groups with the same transaction distributions in-memory. So, if the participant services use the group commit feature, they may insert transaction groups different from other nodes, including the Coordinator service.

See
https://github.com/scalar-labs/scalardb/blob/ebbb0274bb01fd52dc553a333aff5183434c44b6/docs/api-guide.md#commit-in-two-phase-commit-interface also.

Understood. Thank you for your explanation!

So the way to use the two-phase interface varies depending on whether group commit is enabled or not, which might be confusing for users. One option is to always disable group commit when using the two-phase interface. What do you think?

So the way to use the two-phase interface varies depending on whether group commit is enabled or not, which might be confusing for users.

Yes. Users need to change the commit order in their applications as a part of the limitation, and it may be confusing.

One option is to always disable group commit when using the two-phase interface

You mean to throw an exception or something if TwoPhaseCommitTransactionManager is used with the group commit feature enabled, right? Indeed, the idea sounds simpler and less confusing. On the other hand, I'm wondering what if a user wants to use the two-phase commit interface on a high-latency underlying storage where the coordinator table is placed 🤔

You mean to throw an exception or something if TwoPhaseCommitTransactionManager is used with the group commit feature enabled, right?

Yes, I meant that.

On the other hand, I'm wondering what if a user wants to use the two-phase commit interface on a high-latency underlying storage where the coordinator table is placed 🤔

Correct. But, actually, there are also some other optimizations and features we have to give up for the two-phase commit interface.

I see. That sounds a reasonable concern.

I guess it's related to the product specification. @feeblefakie Do you think we can/should prohibit the group commit feature in the two-phase commit interface?

@brfrn169 I had a chance to talk with @feeblefakie about that and he agreed with you. ~~I'll revise this PR (or create another PR to disable that?)~~.

This PR already contains many changes and comments, and got some approvals. So, I think it would be great if I can disable the combination of the group commit and two-phase commit interface in a separate PR.

Users need to change the commit order in their applications as a part of the limitation

BTW, although this is a kind of memo for me about future development when we'll enable the group commit with two-phase commit interface, we should add a check whether the transaction is committed in the coordinator table, before committing records on participant services.

feeblefakie

Overall, looking good!
Left some comments and suggestions. PTAL!

feeblefakie · 2024-05-21T07:21:08Z

core/src/main/java/com/scalar/db/transaction/consensuscommit/Coordinator.java

 }

 public Optional<Coordinator.State> getState(String id) throws CoordinatorException {
+ if (keyManipulator.isFullKey(id)) {
+ // The following read operation order is important. Reading a coordinator state is likely to


These sentences are a bit hard to understand, especially the if a transaction scanned by another transaction executing lazy recovery is delayed part.
Let's chat about it and rephase them.

Thanks for the feedback! I agree with you. I moved these logics for the group commit to getStateForGroupCommit() and revised the comment. What you think?

Of course, chatting on the comment is welcome!

feeblefakie · 2024-05-21T07:31:59Z

core/src/main/java/com/scalar/db/transaction/consensuscommit/Coordinator.java

+ // transaction ID. Therefore, looking up with the full transaction ID should be tried first
+ // to minimize read operations as much as possible.
+
+ // Scan with the full ID for a delayed group that contains only a single transaction.


I feel this method is also used in normal cases without group commit, so the wording is probably too specific to group commit. So, we need to describe two cases: one for the normal case without group commit and the other for the group commit case?

Also, regarding the Scan with the full ID, should it be something like Regard the ID as a full ID and scan with the ID since we don't know if the given ID is a full ID or not.

I moved this logic for the group commit to getStateForGroupCommit() to make things clearer.

regarding the Scan with the full ID, should it be something like Regard the ID as a full ID and scan with the ID since we don't know if the given ID is a full ID or not.

Ah, whether the ID is a full ID is already checked. But I changed the argument name to a more clear one in the commit.

feeblefakie · 2024-05-21T07:38:45Z

core/src/main/java/com/scalar/db/transaction/consensuscommit/Coordinator.java

+ return state;
+ }
+
+ // Scan with the parent ID for a normal group that contains multiple transactions with


A parent ID is explicitly extracted from a full ID here. So, we can go with the current comment as is?

feeblefakie · 2024-05-21T07:47:33Z

core/src/main/java/com/scalar/db/transaction/consensuscommit/Coordinator.java

- .withValue(Attribute.toStateValue(state.getState()))
+ Put put = new Put(new Key(Attribute.toIdValue(state.getId())));
+ if (!state.getChildIds().isEmpty()) {
+ put.withValue(Attribute.toChildIdsValue(Joiner.on(',').join(state.getChildIds())));


"" is hard-coded, so should CHILD_IDS_DELIMITER be defined at the top level and used here, or should a helper method be defined in the State inner class?

Good point! Yeah, I overlooked it. State class already knows how to handle the column including what's the delimiter. So I took care of it in State class.

feeblefakie · 2024-05-21T08:04:36Z

core/src/main/java/com/scalar/db/transaction/consensuscommit/CoordinatorGroupCommitter.java

+ private final Random random = new Random();
+
+ @Override
+ public String generateParentKey() {


Random string generation might affect the performance, so it would be great if you could do a simple benchmark with multiple threads.

I previously encountered crazily slow random string generation when using an Apache Commons library, so I used a method taken from the benchbase one instead.
https://github.com/scalar-labs/scalardb-benchmarks/blob/master/src/main/java/com/scalar/db/benchmarks/ycsb/YcsbCommon.java#L109

So, it would be great if you could compare this with the one above.

Thanks for sharing the concern. That makes sense.

I benchmarked the following combinations with this commit.

a. Simple impl (the original generateParentKey() impl) + Random (as same as the current impl)
b. Faster impl (ported from scalardb-benchmark) + Random
c. Simple impl (the original generateParentKey() impl) + ThreadLocalRandom
d. Faster impl (ported from scalardb-benchmark) + ThreadLocalRandom

[komamitsu@pop-os ~]$ lscpu | grep '^CPU(s)' CPU(s): 16 [komamitsu@pop-os ~]$ uname -a Linux pop-os 6.6.10-76060610-generic #202401051437~1704728131~22.04~24d69e2 SMP PREEMPT_DYNAMIC Mon J x86_64 x86_64 x86_64 GNU/Linux

What I noticed from the result are:

The implementation ported from scalardb-benchmark is about 10-15% faster than the original simple generateParentKey() impl when using ThreadLocalRandom

Which to use Random or ThreadLocalRandom significantly affects the performance along with the number of threads. The average duration was 0.015 ms in case a with 16 threads which is the same number of the Linux machine's CPUs. The duration would increase when using a larger number of CPU machine

I think we should use ThreadLocalRandom just in case. When I benchmarked the group commit feature with ThreadLocalRandom before, I observed a lot of parent key conflicts at the beginning of benchmarking. I'll come up with a way to mitigate the issue and update this PR.

@feeblefakie BTW, I checked the distribution of the random character values when using the implementation ported from scalardb-benchmark. It seems skewed compared to the original simple impl. I also observed a similar distribution when directly testing https://github.com/scalar-labs/scalardb-benchmarks/blob/master/src/main/java/com/scalar/db/benchmarks/ycsb/YcsbCommon.java#L109.

I think it's okay to generate just a random value, but the simple one would be better when generating unique identifiers.

The implementation ported from scalardb-benchmark, where the number in the square brackets is the index of the valid char sequence 0,1,2,...,A,B,..Z,a,b,..z and the next number indicates how many the target character is used

[000] 12170 [001] 21654 [002] 13412 [003] 23733 [004] 15577 [005] 26786 [006] 15973 [007] 56956 [008] 14013 [009] 26212 [010] 14705 [011] 107740 [012] 13730 [013] 28530 [014] 19633 [015] 106851 [016] 10670 [017] 29521 [018] 19581 [019] 61491 [020] 17591 [021] 46623 [022] 22265 [023] 48223 [024] 16174 [025] 58574 [026] 17782 [027] 44353 [028] 16499 [029] 69603 [030] 18314 [031] 40725 [032] 8898 [033] 73426 [034] 17933 [035] 47698 [036] 12817 [037] 81226 [038] 18393 [039] 30052 [040] 12386 [041] 64607 [042] 14837 [043] 53717 [044] 15901 [045] 73947 [046] 17029 [047] 30590 [048] 11894 [049] 34586 [050] 22686 [051] 52105 [052] 16454 [053] 43558 [054] 22372 [055] 33789 [056] 15510 [057] 24684 [058] 17335 [059] 28752 [060] 18036 [061] 29118

The original simple implementation

[000] 39054 [001] 38329 [002] 38803 [003] 39094 [004] 38654 [005] 38724 [006] 38692 [007] 38661 [008] 38667 [009] 38810 [010] 38764 [011] 38945 [012] 38641 [013] 38578 [014] 38603 [015] 38749 [016] 38753 [017] 38709 [018] 38565 [019] 38706 [020] 38555 [021] 38993 [022] 38613 [023] 38477 [024] 38939 [025] 38521 [026] 38813 [027] 38752 [028] 38829 [029] 38728 [030] 38606 [031] 38984 [032] 38803 [033] 38611 [034] 39012 [035] 38577 [036] 38516 [037] 38485 [038] 38580 [039] 38982 [040] 38378 [041] 38650 [042] 38782 [043] 38494 [044] 38518 [045] 38574 [046] 38892 [047] 38815 [048] 38335 [049] 38681 [050] 38510 [051] 38572 [052] 38581 [053] 38473 [054] 38734 [055] 38822 [056] 39052 [057] 39031 [058] 38996 [059] 38902 [060] 38711 [061] 38620

When I benchmarked the group commit feature with ThreadLocalRandom before, I observed a lot of parent key conflicts at the beginning of benchmarking

This was my silly mistake. I stored the return value from ThreadLocalRandom.current(), but I should've not done that...

Usages of this class should typically be of the form: ThreadLocalRandom.current().nextX(...) (where X is Int, Long, etc). When all usages are of this form, it is never possible to accidently share a ThreadLocalRandom across multiple threads.

https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadLocalRandom.html

Fixed it in 755b911

komamitsu · 2024-05-22T06:52:05Z

core/src/main/java/com/scalar/db/transaction/consensuscommit/CommitHandlerWithGroupCommit.java

@@ -84,7 +84,10 @@ public void commitState(Snapshot snapshot)
 commitStateViaGroupCommit(snapshot);
 }

- private void groupCommitState(String parentId, List<Snapshot> snapshots)
+ // TODO: Emitter interface should have `emitNormalGroup()` and `emitDelayedGroup()` separately and


Found there was room to improve this method and related ones to make which type of ID is passed clearer, so added this TODO.

Torch3333

LGTM, thank you!

docs/configurations.md

to improve the performance

Co-authored-by: Josh Wong <[email protected]>

feeblefakie

LGTM! Thank you!

josh-wong

I've left some minor suggestions. Other than that, LGTM. Thank you!🙇🏻‍♂️

docs/api-guide.md

docs/configurations.md

Co-authored-by: Josh Wong <[email protected]>

brfrn169

LGTM! Thank you!

#1728) Co-authored-by: Josh Wong <[email protected]>

komamitsu added 18 commits May 14, 2024 14:07

[skip ci] Integrate group commit with coordinator state

73e228b

[skip ci] Add GroupCommitConflictException

4bc9245

[skip ci] Change the unit of old_group_abort_to

ba94819

[skip ci] Fix a silly bug of logging

02e4412

[skip ci] Support other GC props in CCJdbcEnv

dd31336

[skip ci] Update group-commit related docs

c51b248

Remove unnecessary changes

d2a629f

Extract CommitHandlerWithGroupCommit

f72d31b

Update documents about group commit

d19de06

[skip ci] Update comments and so on

2e83f28

Fix warnings

d45cd54

[skip ci] Clean up Coordinator

69353b4

Update documents

ebbb027

[skip ci] Revert unnecessary change

6e39d76

[skip ci] Fix wrong config name

ee15fd6

[skip ci] Minor improvement for TwoPhaseConsensusCommit*

6dac62d

Minor improvement of CommitHandlerTest

ad33354

[skip ci] Improve TwoPhaseConsensusCommitManagerTest

248cea6

komamitsu commented May 15, 2024

View reviewed changes

komamitsu marked this pull request as ready for review May 15, 2024 10:31

komamitsu self-assigned this May 15, 2024

komamitsu added the enhancement New feature or request label May 15, 2024

komamitsu requested review from brfrn169, feeblefakie, Torch3333 and josh-wong May 15, 2024 10:31

Add a few assertions to TwoPhaseConsensusCommitManagerTest

a7342b7

josh-wong requested changes May 16, 2024

View reviewed changes

komamitsu and others added 2 commits May 16, 2024 18:03

Apply suggestions from code review

46793f6

Co-authored-by: Josh Wong <[email protected]>

Update docs/configurations.md

320e79b

Co-authored-by: Josh Wong <[email protected]>

komamitsu requested a review from josh-wong May 17, 2024 00:35

josh-wong approved these changes May 20, 2024

View reviewed changes

docs/api-guide.md Outdated Show resolved Hide resolved

docs/api-guide.md Outdated Show resolved Hide resolved

docs/api-guide.md Outdated Show resolved Hide resolved

brfrn169 reviewed May 20, 2024

View reviewed changes

feeblefakie reviewed May 21, 2024

View reviewed changes

komamitsu added 2 commits May 22, 2024 14:25

Move a logic about child_ids to State class

e12a3d8

Refactoring on Coordinator and so on

dc53956

komamitsu commented May 22, 2024

View reviewed changes

Torch3333 approved these changes May 22, 2024

View reviewed changes

docs/configurations.md Outdated Show resolved Hide resolved

komamitsu added 2 commits May 22, 2024 16:53

Fix unit test

d6e9e37

Use ThreadLocalRandom in a proper way

755b911

to improve the performance

komamitsu requested a review from feeblefakie May 23, 2024 05:40

komamitsu and others added 3 commits May 23, 2024 14:51

[skip ci] Apply suggestions from code review

bdad881

Co-authored-by: Josh Wong <[email protected]>

[skip ci] Add a footnote about group-commit params

905457d

[skip ci] Move the footnote

e466a16

komamitsu requested review from brfrn169 and josh-wong May 23, 2024 08:28

feeblefakie approved these changes May 27, 2024

View reviewed changes

josh-wong approved these changes May 27, 2024

View reviewed changes

docs/api-guide.md Outdated Show resolved Hide resolved

docs/configurations.md Outdated Show resolved Hide resolved

docs/configurations.md Outdated Show resolved Hide resolved

docs/configurations.md Outdated Show resolved Hide resolved

docs/configurations.md Outdated Show resolved Hide resolved

[skip ci] Apply suggestions from code review

477c5d5

Co-authored-by: Josh Wong <[email protected]>

brfrn169 approved these changes May 28, 2024

View reviewed changes

Merge branch 'master' into integrate-group-commit-with-coordinator

e563118

komamitsu merged commit 03f7321 into master May 28, 2024
25 checks passed

komamitsu deleted the integrate-group-commit-with-coordinator branch May 28, 2024 03:39

feeblefakie mentioned this pull request May 28, 2024

Backport to branch(3) failed: Integrate util/groupcommit package with the coordinator table commit #1794

Closed

komamitsu added a commit that referenced this pull request May 28, 2024

Integrate util/groupcommit package with the coordinator table commit (

7bdb46f

#1728) Co-authored-by: Josh Wong <[email protected]>

This was referenced May 28, 2024

Backport to branch(3) : Integrate util/groupcommit package with the coordinator table commit #1795

Merged

Prohibit the group commit feature with 2PC interface #1797

Open

Split Group Commit Emitter method for normal group emit and delayed group emit #1805

Open

Integrate util/groupcommit package with the coordinator table commit #1728

Integrate util/groupcommit package with the coordinator table commit #1728

Conversation

komamitsu commented May 14, 2024 • edited

Description

Related issues and/or PRs

Changes made

Checklist

Additional notes (optional)

Transaction ID conventions

Limitations

Release notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

komamitsu May 21, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josh-wong left a comment

Choose a reason for hiding this comment

josh-wong left a comment

Choose a reason for hiding this comment

brfrn169 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brfrn169 May 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

komamitsu May 27, 2024 • edited

Choose a reason for hiding this comment

feeblefakie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

komamitsu May 22, 2024 • edited

Choose a reason for hiding this comment

komamitsu May 23, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Torch3333 left a comment

Choose a reason for hiding this comment

feeblefakie left a comment

Choose a reason for hiding this comment

josh-wong left a comment

Choose a reason for hiding this comment

brfrn169 left a comment

Choose a reason for hiding this comment

Integrate `util/groupcommit` package with the coordinator table commit #1728

Integrate `util/groupcommit` package with the coordinator table commit #1728

komamitsu commented May 14, 2024 •

edited

komamitsu May 21, 2024 •

edited

brfrn169 May 20, 2024 •

edited

komamitsu May 27, 2024 •

edited

komamitsu May 22, 2024 •

edited

komamitsu May 23, 2024 •

edited