Make System Test V2 pass

[julienrbrt]

One of the last thing to do for v2 is make the system tests be green.

How to?

Run the following command and fix all issues until it passes.
This tracking issue can be closed once CI is green on this job.
PRs can be split to fix issue by issue.

COSMOS_BUILD_OPTIONS=v2 make test-system

Following an exhaustive list:

• TestAuthzGRPCQueries (@julienrbrt) -> test(systemtests): fix gRPC tests for v1 & v2 #22774
• TestBankGRPCQueries (@julienrbrt) -> test(systemtests): fix gRPC tests for v1 & v2 #22774
• cometbft_client_test.go (@hieuvubk / @julienrbrt) related: Fix non module gRPC gateway queries #22785 -> refactor(server/v2/cometbft): Handle non-module service queries #22803 / test(systemtest): fix cometbft client #22835
• gov_test.go (@julienrbrt) Works fine.
• tx_test.go (@hieuvubk, @julienrbrt) related: Fix non module gRPC gateway queries #22785 -> -> refactor(server/v2/cometbft): Handle non-module service queries #22803, fix(stf,server/v2/cometbft): fix default events + improve codec handling #22837
• TestMintQueries (@julienrbrt) -> test(systemtests): fix mint v2 tests #22789
• TestStakeUnstake (test(systemtests): fix query block-results in TestStakeUnstake for v2 #22782)
• TestExportCmd_WithHeight / TestExportCmd_WithFileFlag (@julienrbrt) -> test(systemtests): fix export v2 #22799
• IAVL flakyness: Make System Test V2 pass #22753 (comment) -> test(systemtests): fix iavl nil #22849

[julienrbrt]

I am seeing some system tests being flaky and failing with:
It should be investigated as well.

7.0.0.1:58516
8< chain err -----------------------------------------
err> panic: runtime error: invalid memory address or nil pointer dereference
err> [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x133942c]
err> 
err> goroutine 159 [running]:
err> github.com/cosmos/iavl.(*MutableTree).set(0x4001076d20, {0x4002cbb168?, 0x1?, 0x14b4d88?}, {0x5a85540?, 0x4001122e80?, 0x4000e4bb08?})
err>    github.com/cosmos/[email protected]/mutable_tree.go:262 +0x2c
err> github.com/cosmos/iavl.(*MutableTree).Set(...)
err>    github.com/cosmos/[email protected]/mutable_tree.go:172
err> cosmossdk.io/store/v2/commitment/iavl.(*IavlTree).Set(0x2d585e0?, {0x4002cbb168?, 0x4001f76ad8?, 0x6?}, {0x5a85540?, 0x0?, 0x0?})
err>    cosmossdk.io/store/[email protected]/commitment/iavl/tree.go:45 +0x28
err> cosmossdk.io/store/v2/commitment.(*CommitStore).WriteChangeset(0x4001212360, 0x0?)
err>    cosmossdk.io/store/[email protected]/commitment/store.go:76 +0x1a0
err> cosmossdk.io/store/v2/root.(*Store).Commit.func1()
err>    cosmossdk.io/store/[email protected]/root/store.go:313 +0x40
err> golang.org/x/sync/errgroup.(*Group).Go.func1()
err>    golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x54
err> created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 118
err>    golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x98

  __      _ _          _ 
 / _|    (_) |        | |
| |_ __ _ _| | ___  __| |
|  _/ _| | | |/ _ \/ _| |
| || (_| | | |  __/ (_| |
|_| \__,_|_|_|\___|\__,_|
FAIL    github.com/cosmos/cosmos-sdk/tests/systemtests  8.422s
FAIL
make[1]: *** [Makefile:8: test] Error 1
make[1]: Leaving directory '/home/julien/projects/cosmos/cosmos-sdk/tests/systemtests'
make: *** [scripts/build/testing.mk:27: test-system] Error 2

[mmsqe]

I am seeing some system tests being flaky and failing with: It should be investigated as well.

The iavl one seems triggered after this Close, not sure it's related to stopping servers

[technicallyty]

looks like the addition of a telemetry value set is now causing a panic. opened: #23021

[julienrbrt]

This one is still free to take: #22849 (comment), or I'll investigate again in two weeks.