Extent log message about conflicted shards

[sergey-safarov]

Description

In my installation, in CouchDB logs messages I can see messages like

1 conflicted shard in cluster

I have tried to delete old databases and in the server logs can see

[critical] 2024-03-26T17:28:24.081510Z [email protected] <0.16743.70> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:28:29.187245Z [email protected] <0.18085.70> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:28:34.311446Z [email protected] <0.21253.69> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:28:39.541320Z [email protected] <0.31584.67> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:28:44.649152Z [email protected] <0.20660.67> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:28:49.717562Z [email protected] <0.23910.70> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:28:54.921189Z [email protected] <0.9403.70> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:29:00.033066Z [email protected] <0.19084.68> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:29:05.174271Z [email protected] <0.25777.70> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:29:10.262848Z [email protected] <0.30249.69> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:29:15.402787Z [email protected] <0.27844.70> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:29:20.721704Z [email protected] <0.20999.70> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:29:25.680320Z [email protected] <0.100.70> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:29:30.777084Z [email protected] <0.243.71> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:29:35.905120Z [email protected] <0.23983.70> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:29:41.078100Z [email protected] <0.4260.71> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:29:46.252992Z [email protected] <0.5119.71> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:29:51.305345Z [email protected] <0.6351.71> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:29:56.444482Z [email protected] <0.6549.71> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:30:01.605705Z [email protected] <0.5639.61> -------- 1 conflicted shard in cluster
[critical] 2024-03-26T17:30:06.766324Z [email protected] <0.9687.69> -------- 1 conflicted shard in cluster

These log messages do not provide information about:

1. database with conflicts;
2. shard file;
3. conflicted revision id.

Could you extent logs message?

Steps to Reproduce

I do not know

Expected Behaviour

Log messages should allow identifying conflicted database, shard, and revision id.

Your Environment

• CouchDB version used:

{
  "couchdb": "Welcome",
  "version": "3.3.3",
  "git_sha": "40afbcfc7",
  "uuid": "385419103339-esrp",
  "features": [
    "access-ready",
    "partitioned",
    "pluggable-storage-engines",
    "reshard",
    "scheduler"
  ],
  "vendor": {
    "name": "The Apache Software Foundation"
  }
}

• Browser name and version:
  Not used

• Operating system and version:

NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

Additional Context

[nickva]

Conflicted shards means that the database metadata doc in the _dbs has a conflicts. That database is replicated in a circle (n1->n2->n3->n1) and since any node can receive database creation requests it's possible to get conflicts in those documents. We only use the winning (latest) version of the db but warn the user via custodian (a helper background watcher service) if there are conflicts present. I believe you can delete the conflicted revision one of the nodes (via /_node/_local/_dbs/$dbname) and then custodian will stop reporting it.

The code that detects is somewhere here: https://github.com/apache/couchdb/blob/main/src/custodian/src/custodian_util.erl#L206-L208

[sergey-safarov]

Could you update the log message to

1 conflicted shard in the cluster, please check document revisions for "_dbs" REST endpoint on all cluster nodes

[nickva]

I think it may work as a documentation string better as opposed a longer log message. Maybe a paragraph in the docs indicating what the error means, with a few examples, so it's easy to search, and how to react to it.