Slow Data ingestion in influxdb

[Satish2007]

Hi all,
Facing peculiar slow data ingestion issue.
H/w and s/w Info:
16vCPU
32GB RAM
100GB
Default configuration of config file
Engine type: TSM
Version: 1.8.10 OSS Community
OS : Ubuntu 22.04

Issue can be reproduce, we tried to test performance in test setup. We are just writing data points in batch size of 250/sec. Only one datastore, 1200 measurements, 6000 series cardinality.

We are generating Load thru c# application. When we start c# application to ingest data , for first few batches it will be able to ingest 250 batch size in mili seconds and then gradually write time increases to minutes.

Everytime same data is being inserted.

No high CPU , RAM or Disk IO being observed while testing the scenario. No write failed or timeout errors observed in log file.

It has become road blocker for us.

Please help/advice to solve the issue. Let me know if you need any details.

Thanks in advance @davidby-influx , @sanderson .

Regards