My validator reward is gone

[omergoktas]

I am an experienced Linux Systems Admin operating an Avalanche validator node. I have been operating the same node for the last few months without an issue, but I recently experienced an issue with the staking rewards. I usually stake monthly and somehow lost my staking reward for the last month. I am having trouble understanding why and am seeking your help.

Here are my node ID, avalanche-go logs, and other details:

My Node ID: NodeID-AThE3ad1aiPLKsj1XfmhCsNbai8sMyPc

AddPermissionlessValidatorTx hash (for the reward-wise unsuccessful staking period): XCrPhRmnRKxzcAfsnVNePngc7A86Hiv4Rzc74moXtx8kbZnwA

Avalanche-Go Logs (starting from the last system restart at 22 June 2024):

27 June 2024 15:13
[06-27|15:13:08.124] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-A6onFGyJjA37EZ7kYHANMR1PFRT8NmXrF", "peerVersion": "avalanchego/1.11.8"}
[06-27|15:12:25.927] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-A6onFGyJjA37EZ7kYHANMR1PFRT8NmXrF", "peerVersion": "avalanchego/1.11.8"}

26 June 2024
[06-26|04:01:10.693] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-Fv3t2shrpkmvLnvNzcv1rqRKbDAYFnUor", "peerVersion": "avalanchego/1.11.8"}

24 June 2024
[06-24|01:34:35.331] INFO health/worker.go:255 check started passing {"namespace": "health", "name": "P", "tags": ["11111111111111111111111111111111LpoYY"]}
[06-24|01:34:27.116] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-9T7NXBFpp8LWCyc58YdKNoowDipdVKAWz", "peerVersion": "avalanchego/1.11.8"}
[06-24|01:34:24.560] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-Hr78Fy8uDYiRYocRYHXp4eLCYeb8x5UuM", "peerVersion": "avalanchego/1.11.8"}
[06-24|01:34:24.098] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-HCw7S2TVbFPDWNBo1GnFWqJ47f9rDJtt1", "peerVersion": "avalanchego/1.11.8"}
[06-24|01:33:35.331] WARN health/worker.go:246 check started failing {"namespace": "health", "name": "P", "tags": ["11111111111111111111111111111111LpoYY"], "error": "not connected to enough stake: connected to 78.391378%; required at least 80.000000%"}

22 June 2024
[06-22|02:15:29.528] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-Hr78Fy8uDYiRYocRYHXp4eLCYeb8x5UuM", "peerVersion": "avalanchego/1.11.8"}
[06-22|02:15:29.397] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-HCw7S2TVbFPDWNBo1GnFWqJ47f9rDJtt1", "peerVersion": "avalanchego/1.11.8"}
[06-22|02:15:23.534] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-9T7NXBFpp8LWCyc58YdKNoowDipdVKAWz", "peerVersion": "avalanchego/1.11.8"}
[06-22|01:23:05.706] INFO health/worker.go:255 check started passing {"namespace": "health", "name": "P", "tags": ["11111111111111111111111111111111LpoYY"]}
[06-22|01:22:35.334] INFO health/worker.go:255 check started passing {"namespace": "health", "name": "network", "tags": ["application"]}
[06-22|01:22:19.755] INFO <P Chain> bootstrap/bootstrapper.go:372 starting to fetch blocks {"numKnownBlocks": 99, "numAcceptedBlocks": 2, "numMissingBlocks": 105}
[06-22|01:22:14.426] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-A6onFGyJjA37EZ7kYHANMR1PFRT8NmXrF", "peerVersion": "avalanchego/1.11.8"}
[06-22|01:22:14.312] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-Fv3t2shrpkmvLnvNzcv1rqRKbDAYFnUor", "peerVersion": "avalanchego/1.11.8"}
[06-22|01:22:14.172] INFO network/ip_tracker.go:346 reset validator tracker bloom filter {"currentCount": 184}
[06-22|01:22:13.905] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-9T7NXBFpp8LWCyc58YdKNoowDipdVKAWz", "peerVersion": "avalanchego/1.11.8"}
[06-22|01:22:13.875] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-HCw7S2TVbFPDWNBo1GnFWqJ47f9rDJtt1", "peerVersion": "avalanchego/1.11.8"}
[06-22|01:22:13.845] INFO peer/peer.go:947 peer attempting to connect with newer version. You may want to update your client {"nodeID": "NodeID-Hr78Fy8uDYiRYocRYHXp4eLCYeb8x5UuM", "peerVersion": "avalanchego/1.11.8"}
[06-22|01:22:13.680] INFO node/node.go:657 API server listening {"uri": "http://127.0.0.1:9650"}
[06-22|01:22:13.676] INFO node/node.go:630 writing process context {"path": "/home/avalanche/.avalanchego/process.json"}
[06-22|01:22:13.676] INFO chains/manager.go:1324 starting chain creator
[06-22|01:22:13.662] INFO <P Chain> bootstrap/bootstrapper.go:167 starting bootstrapper {"lastAcceptedID": "2uwUNagwm2QD9UWM2nn9r8D6ikrcMY6bJyCRtCJ5nc5jkWLTdF", "lastAcceptedHeight": 13834271}
[06-22|01:22:13.661] INFO health/worker.go:109 registered new check and initialized its state to failing {"namespace": "health", "name": "bootstrapped", "tags": ["application"]}
[06-22|01:22:13.661] INFO health/worker.go:109 registered new check and initialized its state to failing {"namespace": "readiness", "name": "bootstrapped", "tags": ["application"]}
[06-22|01:22:13.661] INFO server/server.go:211 adding route {"url": "/ext/bc/11111111111111111111111111111111LpoYY", "endpoint": ""}
[06-22|01:22:13.660] INFO health/worker.go:109 registered new check and initialized its state to failing {"namespace": "health", "name": "P", "tags": ["11111111111111111111111111111111LpoYY"]}
[06-22|01:22:13.660] INFO <P Chain> snowman/transitive.go:97 initializing consensus engine
[06-22|01:22:13.659] INFO <P Chain> proposervm/vm.go:216 initialized proposervm {"state": "after fork", "forkHeight": 805732, "lastAcceptedHeight": 13834271}
[06-22|01:22:13.655] INFO <P Chain> state/state.go:2323 blocks already reindexed
[06-22|01:22:13.654] INFO <P Chain> platformvm/vm.go:224 initializing last accepted {"blkID": "2WYQfiato1ssD6M5kmigVTu7oQNWtUo1LUed3yb4zxfyiPEd7"}
[06-22|01:21:35.334] INFO <P Chain> platformvm/vm.go:114 using VM execution config {"config":     {"network":{"max-validator-set-staleness":60000000000,"target-gossip-size":20480,"push-gossip-percent-stake":0.9,"push-gossip-num-validators":100,"push-gossip-num-peers":0,"push-regossip-num-validators":10,"push-regossip-num-peers":0,"push-gossip-discarde    d-cache-size":16384,"push-gossip-max-regossip-frequency":30000000000,"push-gossip-frequency":500000000,"pull-gossip-poll-size":1,"pull-gossip-frequency":1500000000,"pull-gossip-throttling-period":10000000000,"pull-gossip-throttling-limit":2,"expected-bloo    m-filter-elements":8192,"expected-bloom-filter-false-positive-probability":0.01,"max-bloom-filter-false-positive-probability":0.05},"block-cache-size":67108864,"tx-cache-size":134217728,"transformed-subnet-tx-cache-size":4194304,"reward-utxos-cache-size":    2048,"chain-cache-size":2048,"chain-db-cache-size":2048,"block-id-cache-size":8192,"fx-owner-cache-size":4194304,"checksums-enabled":false,"mempool-prune-frequency":1800000000000}}
[06-22|01:21:35.333] INFO chains/manager.go:1045 creating proposervm wrapper {"activationTime": "[09-22|21:00:00.000]", "minPChainHeight": 793005, "minBlockDelay": "1s", "numHistoricalBlocks": 0}
[06-22|01:21:35.331] INFO chains/manager.go:304 creating chain {"subnetID": "11111111111111111111111111111111LpoYY", "chainID": "11111111111111111111111111111111LpoYY", "vmID": "rWhpuQPF1kb72esV2momhMuTYGkEb1oL29pt2EBXWmSy4kxnT"}
[06-22|01:21:35.331] INFO health/worker.go:255 check started passing {"namespace": "health", "name": "router", "tags": ["application"]}
[06-22|01:21:35.331] INFO health/worker.go:255 check started passing {"namespace": "health", "name": "database", "tags": ["application"]}
[06-22|01:21:35.331] INFO health/worker.go:255 check started passing {"namespace": "health", "name": "diskspace", "tags": ["application"]}
[06-22|01:21:35.331] INFO node/node.go:865 initializing chains
[06-22|01:21:35.331] INFO node/node.go:1289 skipping profiler initialization because it has been disabled
[06-22|01:21:35.002] INFO node/node.go:1479 initializing API aliases
[06-22|01:21:34.703] INFO node/node.go:1452 initializing chain aliases
[06-22|01:21:34.703] INFO server/server.go:236 adding route {"url": "/ext/info", "endpoint": ""}
[06-22|01:21:34.702] INFO node/node.go:1316 initializing info API
[06-22|01:21:34.702] INFO node/node.go:1259 skipping admin API initialization because it has been disabled
[06-22|01:21:34.701] INFO node/node.go:1114 initializing VMs
[06-22|01:21:34.301] INFO node/node.go:989 adding the default VM aliases
[06-22|01:21:34.301] INFO server/server.go:236 adding route {"url": "/ext/health", "endpoint": "/liveness"}
[06-22|01:21:34.301] INFO server/server.go:236 adding route {"url": "/ext/health", "endpoint": "/health"}
[06-22|01:21:34.300] INFO server/server.go:236 adding route {"url": "/ext/health", "endpoint": "/readiness"}
[06-22|01:21:34.300] INFO server/server.go:236 adding route {"url": "/ext/health", "endpoint": ""}
[06-22|01:21:34.300] INFO health/worker.go:109 registered new check and initialized its state to failing {"namespace": "health", "name": "diskspace", "tags": ["application"]}
[06-22|01:21:34.300] INFO health/worker.go:109 registered new check and initialized its state to failing {"namespace": "health", "name": "database", "tags": ["application"]}
[06-22|01:21:34.300] INFO health/worker.go:109 registered new check and initialized its state to failing {"namespace": "health", "name": "router", "tags": ["application"]}
[06-22|01:21:34.300] INFO health/worker.go:109 registered new check and initialized its state to failing {"namespace": "health", "name": "network", "tags": ["application"]}
[06-22|01:21:34.300] INFO node/node.go:1367 initializing Health API
[06-22|01:21:34.299] INFO node/node.go:474 initializing networking {"ip": "45.143.99.140:9651"}
[06-22|01:21:34.296] INFO node/node.go:1197 initializing SharedMemory
[06-22|01:21:34.296] INFO node/node.go:1212 skipping keystore API initialization because it has been disabled
[06-22|01:21:34.295] INFO node/node.go:1205 initializing keystore
[06-22|01:21:34.295] INFO node/node.go:787 initializing database {"genesisHash": "UUvXi6j7QhVvgpbKM89MP5HdrxKm9CaJeHc187TsDNf8nZdLk"}
[06-22|01:21:33.520] INFO leveldb/db.go:205 creating leveldb {"config":     {"blockCacheCapacity":12582912,"blockSize":0,"compactionExpandLimitFactor":0,"compactionGPOverlapsFactor":0,"compactionL0Trigger":0,"compactionSourceLimitFactor":0,"compactionTableSize":0,"compactionTableSizeMultiplier":0,"compactionTableSizeMultiplierPer    Level":null,"compactionTotalSize":0,"compactionTotalSizeMultiplier":0,"disableSeeksCompaction":true,"openFilesCacheCapacity":1024,"writeBuffer":6291456,"filterBitsPerKey":10,"maxManifestFileSize":9223372036854775807,"metricUpdateFrequency":10000000000}}
[06-22|01:21:33.520] INFO server/server.go:236 adding route {"url": "/ext/metrics", "endpoint": ""}
[06-22|01:21:33.520] INFO node/node.go:1243 initializing metrics API
[06-22|01:21:33.519] INFO server/server.go:150 API created {"allowedOrigins": ["*"]}
[06-22|01:21:33.518] INFO node/node.go:903 initializing API server
[06-22|01:21:33.517] INFO node/node.go:885 initializing NAT
[no data]
Started AvalancheGo systemd service.

I am already subscribed to the Avalanche Notify email newsletter and have been getting email notifications for when my node is offline. However, I didn't get any notifications this time.

If my node was doing poorly:

1. Why did I not get notified by [email protected] like before (checked spam forlder, too)?
2. Why can I not see any meaningful warning in the avalanche-go logs (or am I missing it?)
3. Why did every node health monitor site that I occasionally checked over the month show over %99 health¹ (except apparently, Avascan did, but I guess it missed it)?
4. Is there a bug in the avalanche-go program? Because my system has 26,4 GiB empty RAM (32 GiB in total), 16 CPU cores (%90 load free), and over 160 GiB empty disk space. Plus, I just performed an internet speed test over speed test-cli on my gigabit internet network, and the results seem satisfactory (Download: 328.09 Mbit/s, Upload: 269.98 Mbit/s (despite testing in the busy hours of the location of my server))
5. Does the Avalanche Foundation provide funds to cover financial loses of validators for such situations (i.e., ~15.32 AVAX in my case)?

I think the avalanche-go team should investigate the results thoroughly. I would gladly help. The team's response is significant for me because it will determine my trust in the team's competency and the project's future.

Thanks in advance.

Footnotes

1. Sites that showed perfect score when checked for validator health:
   https://check.allnodes.com/avax/NodeID-AThE3ad1aiPLKsj1XfmhCsNbai8sMyPc
   https://stats.avax.network/dashboard/validator-health-check/
   https://vscout.io/validator/NodeID-AThE3ad1aiPLKsj1XfmhCsNbai8sMyPc ↩

[omergoktas]

@StephenButtolph, @danlaine, @aaronbuchwald you devs don't care about community but scammers do. Very well.

[aaronbuchwald]

Hey @omergoktas , I see that the RewardValidatorTx here: https://subnets.avax.network/p-chain/tx/2wPDDTN1z2oNHQwaciF1SPQDvoetdtPswtKxVcoqbUSm84dhtD confirms there was no reward for the validation transaction you linked: https://subnets.avax.network/p-chain/tx/XCrPhRmnRKxzcAfsnVNePngc7A86Hiv4Rzc74moXtx8kbZnwA.

Validators should receive a reward when a majority of the network determines that they are eligible. Each node marks a validator as eligible for their reward if they maintain >= 80% uptime during their validation period (ref: https://docs.avax.network/nodes/validate/how-to-stake#how-is-it-determined-whether-a-validator-receives-a-staking-reward).

It looks like you've started staking again under the same nodeID, but I can still see under AllNodes that your node seems to have 77% uptime (less than the required threshold to earn a reward): https://check.allnodes.com/avax/NodeID-AThE3ad1aiPLKsj1XfmhCsNbai8sMyPc. This likely indicates that your node is not consistently connected to 100% of the network.

The best way to determine this is via the info.uptime API, could you try running this on your node: https://docs.avax.network/reference/avalanchego/info-api#infouptime

curl -X POST --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"info.uptime"
}' -H 'content-type:application/json;' 127.0.0.1:9650/ext/info

This should tell you what the network's view of your uptime is. If your node is healthy and connected, it should return something similar to what's given in the example documentation:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "rewardingStakePercentage": "100.0000",
    "weightedAveragePercentage": "99.0000"
  }
}

Since it looks like your node may be having issues, could you also run the info.peers API call to get more granular detail on which nodes are observing your uptime as lower than expected: https://docs.avax.network/reference/avalanchego/info-api#infopeers

curl -X POST --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"info.peers",
    "params": {
        "nodeIDs": []
    }
}' -H 'content-type:application/json;' 127.0.0.1:9650/ext/info

Could you also provide a zip of your log directory and metrics output from your node as described in the bug report template here: https://github.com/ava-labs/avalanchego/issues/new?assignees=&labels=bug&projects=&template=bug_report.md&title=?

[omergoktas]

After 2 weeks of silience here, I have decided to delete everything, including logs, update avalanchego client to the latest version and start staking/validating over again. So the logs I provided above are the only logs I have.

Results of the commands you provided:

curl -X POST --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"info.uptime"
}' -H 'content-type:application/json;' 127.0.0.1:9650/ext/info

.

{"jsonrpc":"2.0","result":{"rewardingStakePercentage":"93.0794","weightedAveragePercentage":"84.9034"},"id":1}

.

curl -X POST --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"info.peers",
    "params": {
        "nodeIDs": []
    }
}' -H 'content-type:application/json;' 127.0.0.1:9650/ext/info

.

[{
  "ip": "54.71.94.235:10420",
  "publicIP": "10.68.112.134:9651",
  "nodeID": "NodeID-E2Cit72yJYZUdo8Ppk6E9CksCcvFc9Q9y",
  "version": "avalanchego/1.11.5",
  "lastSent": "2024-07-15T17:38:18+03:00",
  "lastReceived": "2024-07-15T17:38:18+03:00",
  "observedUptime": "78",
  "observedSubnetUptimes": {},
  "trackedSubnets": [
    "11111111111111111111111111111111LpoYY",
    "2kyZRnEHakKriNufphojbeFR5hi247bJeTjFDgQgo7VRRjoqyq"
  ],
  "supportedACPs": [],
  "objectedACPs": [],
  "benched": []
},
{
  "ip": "18.237.55.130:19920",
  "publicIP": "10.68.180.239:9651",
  "nodeID": "NodeID-B8rBFq98rKV4MnKfmcdzMM6SqJxKKhBwD",
  "version": "avalanchego/1.11.5",
  "lastSent": "2024-07-15T17:38:18+03:00",
  "lastReceived": "2024-07-15T17:38:18+03:00",
  "observedUptime": "79",
  "observedSubnetUptimes": {},
  "trackedSubnets": [
    "11111111111111111111111111111111LpoYY",
    "eYwmVU67LmSfZb1RwqCMhBYkFyG8ftxn6jAwqzFmxC9STBWLC"
  ],
  "supportedACPs": [],
  "objectedACPs": [],
  "benched": []
},
{
  "ip": "35.85.48.105:58075",
  "publicIP": "10.68.104.161:9651",
  "nodeID": "NodeID-NqXGHnQmXgW1Zmg8KQqVqr1ZLTYUWQ1Qz",
  "version": "avalanchego/1.11.5",
  "lastSent": "2024-07-15T17:38:05+03:00",
  "lastReceived": "2024-07-15T17:38:06+03:00",
  "observedUptime": "79",
  "observedSubnetUptimes": {},
  "trackedSubnets": [
    "11111111111111111111111111111111LpoYY",
    "2hoHzxcZktxrTNoqCZhR9bEctc5oCuBqSp6SpRn6uvwxzLkvnD"
  ],
  "supportedACPs": [],
  "objectedACPs": [],
  "benched": []
},
{
  "ip": "54.92.94.21:62063",
  "publicIP": "10.89.100.88:9651",
  "nodeID": "NodeID-8rzLkQkWWzGbCxUmECvEX4TaxrNJJHEgo",
  "version": "avalanchego/1.11.5",
  "lastSent": "2024-07-15T17:38:07+03:00",
  "lastReceived": "2024-07-15T17:38:08+03:00",
  "observedUptime": "78",
  "observedSubnetUptimes": {},
  "trackedSubnets": [
    "11111111111111111111111111111111LpoYY",
    "2mBbfPEFbdEqeaBvRLgvmQQA3cG5mbpqbC8fcXCTZaycXZrXDq"
  ],
  "supportedACPs": [],
  "objectedACPs": [],
  "benched": []
},
{
  "ip": "54.216.18.73:20220",
  "publicIP": "10.80.106.212:9651",
  "nodeID": "NodeID-GjmR2jJS7tczTwh87dyQE4eh9jaC3BZ85",
  "version": "avalanchego/1.11.9",
  "lastSent": "2024-07-15T17:38:02+03:00",
  "lastReceived": "2024-07-15T17:38:02+03:00",
  "observedUptime": "80",
  "observedSubnetUptimes": {},
  "trackedSubnets": [
    "11111111111111111111111111111111LpoYY"
  ],
  "supportedACPs": [],
  "objectedACPs": [],
  "benched": []
},
{
  "ip": "209.249.216.180:34296",
  "publicIP": "[2001:438:8000:101:8888:8888:39:51]:9651",
  "nodeID": "NodeID-GMnczqcYS2mAnKgjfxXRGc5UtrsahGiz4",
  "version": "avalanchego/1.11.9",
  "lastSent": "2024-07-15T17:38:06+03:00",
  "lastReceived": "2024-07-15T17:38:06+03:00",
  "observedUptime": "99",
  "observedSubnetUptimes": {},
  "trackedSubnets": [
    "11111111111111111111111111111111LpoYY"
  ],
  "supportedACPs": [],
  "objectedACPs": [],
  "benched": []
},
{
  "ip": "163.172.132.184:40126",
  "publicIP": "163.172.132.184:9651",
  "nodeID": "NodeID-WH2mrP9QuEyXZFdqMYivgchEZhyon7Bz",
  "version": "avalanchego/1.11.9",
  "lastSent": "2024-07-15T17:38:07+03:00",
  "lastReceived": "2024-07-15T17:38:07+03:00",
  "observedUptime": "91",
  "observedSubnetUptimes": {},
  "trackedSubnets": [
    "11111111111111111111111111111111LpoYY"
  ],
  "supportedACPs": [],
  "objectedACPs": [],
  "benched": []
},
{
  "ip": "52.2.138.32:7208",
  "publicIP": "10.86.94.170:9651",
  "nodeID": "NodeID-252gW378La2tpG2HXKTWFL5j6rVsWNqUe",
  "version": "avalanchego/1.11.2",
  "lastSent": "2024-07-15T17:38:18+03:00",
  "lastReceived": "2024-07-15T17:38:18+03:00",
  "observedUptime": "79",
  "observedSubnetUptimes": {},
  "trackedSubnets": [
    "11111111111111111111111111111111LpoYY"
  ],
  "supportedACPs": [],
  "objectedACPs": [],
  "benched": []
},
{
  "ip": "51.79.229.74:59358",
  "publicIP": "51.79.229.74:9651",
  "nodeID": "NodeID-38iwavUWTDP1avR4PpbhR1sLLVpw2PYPP",
  "version": "avalanchego/1.11.2",
  "lastSent": "2024-07-15T17:38:06+03:00",
  "lastReceived": "2024-07-15T17:38:17+03:00",
  "observedUptime": "79",
  "observedSubnetUptimes": {},
  "trackedSubnets": [
    "11111111111111111111111111111111LpoYY",
    "Vn3aX6hNRstj5VHHm63TCgPNaeGnRSqCYXQqemSqDd2TQH4qJ"
  ],
  "supportedACPs": [],
  "objectedACPs": [],
  "benched": []
},
{
  "ip": "45.77.38.132:39628",
  "publicIP": "45.77.38.132:9651",
  "nodeID": "NodeID-F88MNto2Ly74K5swtdfmbKAaPTnV1pZNL",
  "version": "avalanchego/1.11.8",
  "lastSent": "2024-07-15T17:38:13+03:00",
  "lastReceived": "2024-07-15T17:38:13+03:00",
  "observedUptime": "0",
  "observedSubnetUptimes": {},
  "trackedSubnets": [
    "11111111111111111111111111111111LpoYY"
  ],
  "supportedACPs": [],
  "objectedACPs": [],
  "benched": []
}]

I guess my node started to function normally again after the reset, and the uptimes are gradually increasing. Thank you for providing these APIs; I'll use them to check my node's health from now on. If the logs I provided are not sufficient enough for you to further inspect any potential problems in the avalanchego client, I guess we are left with no other option than to close this discussion (feel free to do so). But I was really hoping that the logs I provided above could help you inspect the problem (because, despite providing all the conditions the client needs to operate, it stopped working suddenly without giving me any warnings in the systemd logs -I did not check other logs in the logs directory, though. My bad.).

Thank you for assisting.

[omergoktas]

In addition, I think it would have been nice if the Avalanche Notify email newsletter could notify people when their node is doing poorly, too (because it successfully does so when the node is completely offline). Just so you know.

[5twelve]

I'm not quite sure I'm having issues on my end also Rlopez

…

On Mon, Jul 15, 2024, 10:08 AM Ömer Göktaş ***@***.***> wrote: In addition, I think it would have been nice if the Avalanche Notify <https://docs.avax.network/nodes/maintain/avalanche-notify> email newsletter could notify people when their node is doing poorly, too (because it successfully does so when the node is completely offline). Just so you know. — Reply to this email directly, view it on GitHub <#3159 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXEJGIGJBDLCF47FZRKLVYDZMPQXZAVCNFSM6AAAAABKELWW52VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBVGI2DQMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[omergoktas]

I have written a simple C++ program to check the node health, If you are familar with coding, here is the source code and installation on Linux: https://gist.github.com/omergoktas/8be8db8d3347b278f98f817ee16ce5e0

The code is pretty short and self explanatory. It just connects to your node on 127.0.0.1:9650, queries the node health statistics broadcasted by other nodes, and returns a calculated average health point in %. I occasionally check my node's health with it, and it helped me a lot in detecting issues.