Context aware error types based on ClickHouse error codes

[srikanthccv]

Not every error is retryable. Some errors are permanent and retrying them makes no sense. The sync mode today retries all the errors forever and blocks everything else from getting ingested. There should be an error type-based retry mechanism and the exporters should return context aware errors to the caller instead of just returning err as is from the clickhouse-go client.

[srikanthccv]

2024-04-12T17:18:22.026Z error clickhousetracesexporter/writer.go:385 Could not append span to batch: {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "span": {"traceId": "", "spanId": "", "parentSpanId": "", "name": "TestingSpan", "durationNano": 0, "startTimeUnixNano": 11651379494838206464, "serviceName": "<nil-service-name>", "kind": 0, "statusCode": 0, "externalHttpMethod": "", "httpUrl": "", "httpMethod": "", "httpHost": "", "httpRoute": "", "httpCode": "", "msgSystem": "", "msgOperation": "", "externalHttpUrl": "", "component": "", "dbSystem": "", "dbName": "", "dbOperation": "", "peerService": "", "gRPCCode": "", "gRPCMethod": "", "rpcSystem": "", "rpcService": "", "rpcMethod": "", "responseStatusCode": "", "hasError": false, "errorID": "", "errorGroupID": "", "errorEvent": {"name": "", "timeUnixNano": 0, "isError": false, "attributeMap": "map[]"}, "traceModel": {"traceId": "", "spanId": "", "name": "TestingSpan", "durationNano": 0, "startTimeUnixNano": 11651379494838206464, "serviceName": "<nil-service-name>", "kind": 0, "statusCode": 0, "hasError": false, "references": [{"traceId": "", "spanId": "", "refType": "CHILD_OF"}], "tagMap": "map[k8s.pod.ip:10.5.78.213]", "event": "[]"}, "event": "[]", "tagMap": "map[k8s.pod.ip:10.5.78.213]"}, "error": "clickhouse: dateTime overflow. timestamp must be between 1900-01-01 00:00:00 and 2262-04-11 23:47:16"}

Without such context-aware error types, one garbage span can block the whole trace ingestion.