Thanos compactor: block missing index file

[chrisghill]

Thanos, Prometheus and Golang version used
improbable/thanos:v0.4.0-rc.1

What happened
Thanos compactor continues to crash due to a block missing it's index file. Obviously I could just delete the block, but if there is a way to salvage the data without deleting the block I'd prefer to do that. Logs are below.

What you expected to happen
The block to have an index file so the compactor could run.

How to reproduce it (as minimally and precisely as possible):
Remove index file from a block

Full logs to relevant components

Logs

$ kc logs thanos-compactor-0
level=info ts=2019-05-30T21:49:47.251138485Z caller=factory.go:39 msg="loading bucket configuration"
level=info ts=2019-05-30T21:49:47.892961149Z caller=compact.go:319 msg="starting compact node"
level=info ts=2019-05-30T21:49:47.893064515Z caller=main.go:309 msg="Listening for metrics" address=0.0.0.0:10902
level=info ts=2019-05-30T21:49:47.896321692Z caller=compact.go:968 msg="start sync of metas"
level=debug ts=2019-05-30T21:49:47.977516704Z caller=compact.go:249 msg="download meta" block=01D5BAXYS1QM2XBA64NKSFC7QK
level=debug ts=2019-05-30T21:49:47.977554087Z caller=compact.go:249 msg="download meta" block=01D6F53CSR6579K7WB0G9DH5FC
...
level=debug ts=2019-05-30T21:49:48.099322501Z caller=compact.go:249 msg="download meta" block=01DC3FVJPBYR64STZK2MYM5WQ4
level=debug ts=2019-05-30T21:49:48.099758202Z caller=compact.go:249 msg="download meta" block=01DC3GHYTYW1H7CY73GFY9EGK3
level=debug ts=2019-05-30T21:49:48.099832903Z caller=compact.go:249 msg="download meta" block=01DC3H7NMWKFQWB291NDPGF16P
level=debug ts=2019-05-30T21:49:48.10016137Z caller=compact.go:249 msg="download meta" block=01DC487ZM0EFY6RHAS1G1PFNQA
level=debug ts=2019-05-30T21:49:48.102051224Z caller=compact.go:249 msg="download meta" block=01DC488DKAH0V4TYZ7FPPBTJE5
level=debug ts=2019-05-30T21:49:48.103366583Z caller=compact.go:249 msg="download meta" block=01DC51SZ1HVBJJ8JA2JKF4PR5H
level=debug ts=2019-05-30T21:49:48.104125187Z caller=compact.go:249 msg="download meta" block=01DC51SZA3C3SB0MV4JSQ6XBTJ
level=debug ts=2019-05-30T21:49:48.107107368Z caller=compact.go:249 msg="download meta" block=01DC53T71WP5BCP3H3ZG2SB6GQ
level=debug ts=2019-05-30T21:49:48.107371969Z caller=compact.go:249 msg="download meta" block=01DC53TN2JG5ATNA96J11A9ZH5
level=debug ts=2019-05-30T21:49:48.109276743Z caller=compact.go:249 msg="download meta" block=01DC58NP9HR4W6VQ68AXSA7S5V
level=debug ts=2019-05-30T21:49:48.111920585Z caller=compact.go:249 msg="download meta" block=01DC58NPJ39KP1S9ATEEJ5B83N
level=info ts=2019-05-30T21:49:48.184969165Z caller=compact.go:974 msg="start of GC"
level=info ts=2019-05-30T21:49:48.187189817Z caller=compact.go:980 msg="start of compaction"
level=info ts=2019-05-30T21:49:48.254589753Z caller=compact.go:238 msg="compaction iterations done"
level=info ts=2019-05-30T21:49:48.254618508Z caller=compact.go:245 msg="start first pass of downsampling"
level=info ts=2019-05-30T21:50:56.063686184Z caller=downsample.go:212 msg="downloaded block" id=01DC3GHYTYW1H7CY73GFY9EGK3 duration=1m7.18090834s
level=error ts=2019-05-30T21:50:56.063800144Z caller=main.go:182 msg="running command failed" err="error executing compaction: first pass of downsampling failed: downsampling to 60 min: input block index not valid: open index file: try lock file: open /var/thanos/store/downsample/01DC3GHYTYW1H7CY73GFY9EGK3/index: no such file or directory"

Anything else we need to know
The compactor has been running fine for weeks. This is just one bad block and I didn't see any other comments or issues about how to recover from this situation so I figured I'd make one. Like I said I'm sure I could just delete the block but I'd prefer to salvage it if possible.

Here is a picture of the block contents in S3:

You can see it has an index.cache.json and a meta.json but no index. Here is a healthy block:

[bwplotka]

Thanks for this. Just to check you removed index manually or compactor/source upload process made such block in this way? (this is unexpected).

As for how to mitigate the situations like this, there is literally zero we can do. We could reproduce meta.json from index and chunks, but not index itself. This is because index stores all the information about chunks (so encoded samples) labels and what series are in what bytes of chunk files. That's why Thanos as well as TSDB do all they can do make sure we don't drop or corrupt those crucial files.

[chrisghill]

I did not remove the index file. The compactor/source upload process made it this way. I should have clarified that in the original post.

Sounds like my only course of action is to delete that block in S3? We're running a HA prometheus w/ 2 instances (each with a thanos sidecar) so hopefully this just means that we'll only have 1 source of data during that block time window?

[chrisghill]

Also, in case you're interested in the logs from the event, here are 2 excerpts. The first is from filtering for just that block name (01DC3GHYTYW1H7CY73GFY9EGK3), and the 2nd is all logs across that time period. It appears it was a compactor issue, as the first time it tried to downsample it worked on the first pass. The 2nd pass is where it failed. At least thats how I read the logs:

**Filtering for block**

level=info ts=2019-05-30T04:47:51.614790337Z caller=downsample.go:239 msg="downsampled block" from=01DC3DZZPWEBWD2BJP3JJ1ZCTY to=01DC3GHYTYW1H7CY73GFY9EGK3 duration=8m36.27489592s
level=info ts=2019-05-30T04:47:51.614717644Z caller=streamed_block_writer.go:218 msg="finalized downsampled block" mint=1557964800000 maxt=1559174400000 ulid=01DC3GHYTYW1H7CY73GFY9EGK3 resolution=300000
level=info ts=2019-05-30T04:49:02.462978023Z caller=downsample.go:253 msg="uploaded block" id=01DC3GHYTYW1H7CY73GFY9EGK3 duration=52.074245974s
level=info ts=2019-05-30T04:59:55.030602997Z caller=downsample.go:212 msg="downloaded block" id=01DC3GHYTYW1H7CY73GFY9EGK3 duration=1m0.757689487s
level=error ts=2019-05-30T04:59:55.229735382Z caller=main.go:182 msg="running command failed" err="error executing compaction: second pass of downsampling failed: downsampling to 60 min: input block index not valid: open index file: try lock file: open /var/thanos/store/downsample/01DC3GHYTYW1H7CY73GFY9EGK3/index: no such file or directory"
level=debug ts=2019-05-30T04:59:57.440203564Z caller=compact.go:249 msg="download meta" block=01DC3GHYTYW1H7CY73GFY9EGK3
level=error ts=2019-05-30T05:01:01.978600532Z caller=main.go:182 msg="running command failed" err="error executing compaction: first pass of downsampling failed: downsampling to 60 min: input block index not valid: open index file: try lock file: open /var/thanos/store/downsample/01DC3GHYTYW1H7CY73GFY9EGK3/index: no such file or directory"
level=info ts=2019-05-30T05:01:01.978487747Z caller=downsample.go:212 msg="downloaded block" id=01DC3GHYTYW1H7CY73GFY9EGK3 duration=1m0.911601864s
level=debug ts=2019-05-30T05:01:16.125616373Z caller=compact.go:249 msg="download meta" block=01DC3GHYTYW1H7CY73GFY9EGK3
level=info ts=2019-05-30T05:02:21.106342594Z caller=downsample.go:212 msg="downloaded block" id=01DC3GHYTYW1H7CY73GFY9EGK3 duration=1m4.007754557s
level=error ts=2019-05-30T05:02:21.10647458Z caller=main.go:182 msg="running command failed" err="error executing compaction: first pass of downsampling failed: downsampling to 60 min: input block index not valid: open index file: try lock file: open /var/thanos/store/downsample/01DC3GHYTYW1H7CY73GFY9EGK3/index: no such file or directory"
level=debug ts=2019-05-30T05:02:49.119970978Z caller=compact.go:249 msg="download meta" block=01DC3GHYTYW1H7CY73GFY9EGK3
level=error ts=2019-05-30T05:03:45.528042658Z caller=main.go:182 msg="running command failed" err="error executing compaction: first pass of downsampling failed: downsampling to 60 min: input block index not valid: open index file: try lock file: open /var/thanos/store/downsample/01DC3GHYTYW1H7CY73GFY9EGK3/index: no such file or directory"
level=info ts=2019-05-30T05:03:45.527927627Z caller=downsample.go:212 msg="downloaded block" id=01DC3GHYTYW1H7CY73GFY9EGK3 duration=55.722935372s
level=debug ts=2019-05-30T05:04:31.13264687Z caller=compact.go:249 msg="download meta" block=01DC3GHYTYW1H7CY73GFY9EGK3
level=error ts=2019-05-30T05:05:25.145144221Z caller=main.go:182 msg="running command failed" err="error executing compaction: first pass of downsampling failed: downsampling to 60 min: input block index not valid: open index file: try lock file: open /var/thanos/store/downsample/01DC3GHYTYW1H7CY73GFY9EGK3/index: no such file or directory"

**All logs**

level=info ts=2019-05-30T04:35:48.694095414Z caller=streamed_block_writer.go:218 msg="finalized downsampled block" mint=1557964800000 maxt=1559174400000 ulid=01DC3FVJPBYR64STZK2MYM5WQ4 resolution=300000
level=info ts=2019-05-30T04:35:48.694954598Z caller=downsample.go:239 msg="downsampled block" from=01DC3D3Z5Y0ET1QQ7D4967CZ88 to=01DC3FVJPBYR64STZK2MYM5WQ4 duration=8m46.649659441s
level=info ts=2019-05-30T04:36:51.991469204Z caller=downsample.go:253 msg="uploaded block" id=01DC3FVJPBYR64STZK2MYM5WQ4 duration=44.27935803s
level=info ts=2019-05-30T04:38:55.887579685Z caller=downsample.go:212 msg="downloaded block" id=01DC3DZZPWEBWD2BJP3JJ1ZCTY duration=2m2.315804148s
level=warn ts=2019-05-30T04:42:13.896309772Z caller=streamed_block_writer.go:138 emptychunkshappened,skipseries="{__name__=\"kube_configmap_metadata_resource_version\",configmap=\"cluster-autoscaler-status\",endpoint=\"https-main\",instance=\"100.125.0.23:8443\",job=\"kube-state-metrics\",namespace=\"kube-system\",pod=\"kube-state-metrics-654c55bb59-dsccm\",resource_version=\"95784901\",service=\"kube-state-metrics\"}"
level=info ts=2019-05-30T04:47:51.614790337Z caller=downsample.go:239 msg="downsampled block" from=01DC3DZZPWEBWD2BJP3JJ1ZCTY to=01DC3GHYTYW1H7CY73GFY9EGK3 duration=8m36.27489592s
level=info ts=2019-05-30T04:47:51.614717644Z caller=streamed_block_writer.go:218 msg="finalized downsampled block" mint=1557964800000 maxt=1559174400000 ulid=01DC3GHYTYW1H7CY73GFY9EGK3 resolution=300000
level=info ts=2019-05-30T04:49:02.462978023Z caller=downsample.go:253 msg="uploaded block" id=01DC3GHYTYW1H7CY73GFY9EGK3 duration=52.074245974s
level=info ts=2019-05-30T04:49:04.117573247Z caller=compact.go:251 msg="start second pass of downsampling"
level=info ts=2019-05-30T04:50:48.358192611Z caller=downsample.go:212 msg="downloaded block" id=01DC3FVJPBYR64STZK2MYM5WQ4 duration=1m42.515075356s
level=info ts=2019-05-30T04:58:04.520026438Z caller=streamed_block_writer.go:218 msg="finalized downsampled block" mint=1557964800000 maxt=1559174400000 ulid=01DC3H7NMWKFQWB291NDPGF16P resolution=3600000
level=info ts=2019-05-30T04:58:04.520086579Z caller=downsample.go:239 msg="downsampled block" from=01DC3FVJPBYR64STZK2MYM5WQ4 to=01DC3H7NMWKFQWB291NDPGF16P duration=6m57.692852558s
level=info ts=2019-05-30T04:58:52.476765709Z caller=downsample.go:253 msg="uploaded block" id=01DC3H7NMWKFQWB291NDPGF16P duration=29.324094568s
level=info ts=2019-05-30T04:59:55.030602997Z caller=downsample.go:212 msg="downloaded block" id=01DC3GHYTYW1H7CY73GFY9EGK3 duration=1m0.757689487s
level=error ts=2019-05-30T04:59:55.229735382Z caller=main.go:182 msg="running command failed" err="error executing compaction: second pass of downsampling failed: downsampling to 60 min: input block index not valid: open index file: try lock file: open /var/thanos/store/downsample/01DC3GHYTYW1H7CY73GFY9EGK3/index: no such file or directory"
level=info ts=2019-05-30T04:59:56.469703182Z caller=factory.go:39 msg="loading bucket configuration"
level=info ts=2019-05-30T04:59:57.162382209Z caller=main.go:309 msg="Listening for metrics" address=0.0.0.0:10902
level=info ts=2019-05-30T04:59:57.162292436Z caller=compact.go:319 msg="starting compact node"
level=info ts=2019-05-30T04:59:57.166162247Z caller=compact.go:968 msg="start sync of metas"
level=debug ts=2019-05-30T04:59:57.295276271Z caller=compact.go:249 msg="download meta" block=01D5BAT9XF6XDATRWNGYNQ4MCB
level=debug ts=2019-05-30T04:59:57.295391353Z caller=compact.go:249 msg="download meta" block=01D6F53CSR6579K7WB0G9DH5FC
level=debug ts=2019-05-30T04:59:57.295539137Z caller=compact.go:249 msg="download meta" block=01D6F63DM439NMRFZEXADFQZDZ
...
level=info ts=2019-05-30T04:59:59.651093822Z caller=compact.go:420 msg="deleting outdated block" block=01DBESWYTA5MYYRH6YRJVS3T4Y
level=info ts=2019-05-30T04:59:59.920331704Z caller=compact.go:420 msg="deleting outdated block" block=01DBY87G7JGFG2G7TTM4K9NKP1
level=info ts=2019-05-30T05:00:00.091383101Z caller=compact.go:420 msg="deleting outdated block" block=01DBS3E9FJBYB909EYTZNKDTMK
level=info ts=2019-05-30T04:59:57.544517206Z caller=compact.go:974 msg="start of GC"
level=info ts=2019-05-30T05:00:00.37374314Z caller=compact.go:980 msg="start of compaction"
level=info ts=2019-05-30T05:00:00.433758546Z caller=compact.go:238 msg="compaction iterations done"
level=info ts=2019-05-30T05:00:00.433787386Z caller=compact.go:245 msg="start first pass of downsampling"
level=error ts=2019-05-30T05:01:01.978600532Z caller=main.go:182 msg="running command failed" err="error executing compaction: first pass of downsampling failed: downsampling to 60 min: input block index not valid: open index file: try lock file: open /var/thanos/store/downsample/01DC3GHYTYW1H7CY73GFY9EGK3/index: no such file or directory"
level=info ts=2019-05-30T05:01:01.978487747Z caller=downsample.go:212 msg="downloaded block" id=01DC3GHYTYW1H7CY73GFY9EGK3 duration=1m0.911601864s
level=info ts=2019-05-30T05:01:15.252989759Z caller=factory.go:39 msg="loading bucket configuration"
level=info ts=2019-05-30T05:01:15.901897927Z caller=main.go:309 msg="Listening for metrics" address=0.0.0.0:10902
level=info ts=2019-05-30T05:01:15.90181567Z caller=compact.go:319 msg="starting compact node"
level=info ts=2019-05-30T05:01:15.904865373Z caller=compact.go:968 msg="start sync of metas"
level=debug ts=2019-05-30T05:01:16.011532922Z caller=compact.go:249 msg="download meta" block=01D6F7FEXMP2PGQB2MGRPWYADX
level=debug ts=2019-05-30T05:01:16.011082834Z caller=compact.go:249 msg="download meta" block=01D6F63DM439NMRFZEXADFQZDZ
level=debug ts=2019-05-30T05:01:16.011026247Z caller=compact.go:249 msg="download meta" block=01D5BAH4EQKTQ2GP94FT1J3TTN
...
level=debug ts=2019-05-30T05:01:16.131758593Z caller=compact.go:249 msg="download meta" block=01DC3H7NMWKFQWB291NDPGF16P
level=debug ts=2019-05-30T05:01:16.133500997Z caller=compact.go:249 msg="download meta" block=01DC3HQWJ22D10DQ83PR22C29K
level=debug ts=2019-05-30T05:01:16.135321678Z caller=compact.go:249 msg="download meta" block=01DC3HQWT13R47FQ7GZG9BR6N8
level=debug ts=2019-05-30T05:01:16.141583381Z caller=compact.go:267 msg="block is too fresh for now" block=01DC3HQWJ22D10DQ83PR22C29K
level=debug ts=2019-05-30T05:01:16.141784128Z caller=compact.go:267 msg="block is too fresh for now" block=01DC3HQWT13R47FQ7GZG9BR6N8
level=info ts=2019-05-30T05:01:16.185622735Z caller=compact.go:974 msg="start of GC"
level=info ts=2019-05-30T05:01:16.187661508Z caller=compact.go:980 msg="start of compaction"
level=info ts=2019-05-30T05:01:16.244680588Z caller=compact.go:245 msg="start first pass of downsampling"
level=info ts=2019-05-30T05:01:16.244656089Z caller=compact.go:238 msg="compaction iterations done"
level=info ts=2019-05-30T05:02:21.106342594Z caller=downsample.go:212 msg="downloaded block" id=01DC3GHYTYW1H7CY73GFY9EGK3 duration=1m4.007754557s
level=error ts=2019-05-30T05:02:21.10647458Z caller=main.go:182 msg="running command failed" err="error executing compaction: first pass of downsampling failed: downsampling to 60 min: input block index not valid: open index file: try lock file: open /var/thanos/store/downsample/01DC3GHYTYW1H7CY73GFY9EGK3/index: no such file or directory"
level=info ts=2019-05-30T05:02:48.273495409Z caller=factory.go:39 msg="loading bucket configuration"
level=info ts=2019-05-30T05:02:48.912743671Z caller=compact.go:319 msg="starting compact node"
level=info ts=2019-05-30T05:02:48.91284444Z caller=main.go:309 msg="Listening for metrics" address=0.0.0.0:10902
level=info ts=2019-05-30T05:02:48.916133509Z caller=compact.go:968 msg="start sync of metas"
level=debug ts=2019-05-30T05:02:48.997602616Z caller=compact.go:249 msg="download meta" block=01D7K7051X97DABMCA9S480AZ9
level=debug ts=2019-05-30T05:02:48.997029396Z caller=compact.go:249 msg="download meta" block=01D6F53CSR6579K7WB0G9DH5FC
level=debug ts=2019-05-30T05:02:48.997625164Z caller=compact.go:249 msg="download meta" block=01D5BB1P7J5H357WJWD28RJ3E4
...
level=debug ts=2019-05-30T05:02:49.119970978Z caller=compact.go:249 msg="download meta" block=01DC3GHYTYW1H7CY73GFY9EGK3
level=debug ts=2019-05-30T05:02:49.121109062Z caller=compact.go:249 msg="download meta" block=01DC3HQWJ22D10DQ83PR22C29K
level=debug ts=2019-05-30T05:02:49.121808823Z caller=compact.go:249 msg="download meta" block=01DC3HQWT13R47FQ7GZG9BR6N8
level=debug ts=2019-05-30T05:02:49.135020097Z caller=compact.go:267 msg="block is too fresh for now" block=01DC3HQWJ22D10DQ83PR22C29K
level=debug ts=2019-05-30T05:02:49.135026279Z caller=compact.go:267 msg="block is too fresh for now" block=01DC3HQWT13R47FQ7GZG9BR6N8
level=info ts=2019-05-30T05:02:49.162801887Z caller=compact.go:974 msg="start of GC"
level=info ts=2019-05-30T05:02:49.164348335Z caller=compact.go:980 msg="start of compaction"
level=info ts=2019-05-30T05:02:49.22216161Z caller=compact.go:238 msg="compaction iterations done"
level=info ts=2019-05-30T05:02:49.222189701Z caller=compact.go:245 msg="start first pass of downsampling"
level=error ts=2019-05-30T05:03:45.528042658Z caller=main.go:182 msg="running command failed" err="error executing compaction: first pass of downsampling failed: downsampling to 60 min: input block index not valid: open index file: try lock file: open /var/thanos/store/downsample/01DC3GHYTYW1H7CY73GFY9EGK3/index: no such file or directory"
level=info ts=2019-05-30T05:03:45.527927627Z caller=downsample.go:212 msg="downloaded block" id=01DC3GHYTYW1H7CY73GFY9EGK3 duration=55.722935372s
level=info ts=2019-05-30T05:04:30.281912522Z caller=factory.go:39 msg="loading bucket configuration"
level=info ts=2019-05-30T05:04:30.937669544Z caller=compact.go:319 msg="starting compact node"
level=info ts=2019-05-30T05:04:30.937834854Z caller=main.go:309 msg="Listening for metrics" address=0.0.0.0:10902
level=info ts=2019-05-30T05:04:30.940609107Z caller=compact.go:968 msg="start sync of metas"
level=debug ts=2019-05-30T05:04:31.063597427Z caller=compact.go:249 msg="download meta" block=01D5BAT9XF6XDATRWNGYNQ4MCB
level=debug ts=2019-05-30T05:04:31.063558605Z caller=compact.go:249 msg="download meta" block=01D5BAXYS1QM2XBA64NKSFC7QK
...
level=debug ts=2019-05-30T05:04:31.132007232Z caller=compact.go:249 msg="download meta" block=01DC3FVJPBYR64STZK2MYM5WQ4
level=debug ts=2019-05-30T05:04:31.133554999Z caller=compact.go:249 msg="download meta" block=01DC3HQWJ22D10DQ83PR22C29K
level=debug ts=2019-05-30T05:04:31.134085846Z caller=compact.go:249 msg="download meta" block=01DC3HQWT13R47FQ7GZG9BR6N8
level=debug ts=2019-05-30T05:04:31.139816213Z caller=compact.go:267 msg="block is too fresh for now" block=01DC3HQWT13R47FQ7GZG9BR6N8
level=debug ts=2019-05-30T05:04:31.141313836Z caller=compact.go:267 msg="block is too fresh for now" block=01DC3HQWJ22D10DQ83PR22C29K
level=info ts=2019-05-30T05:04:31.281645947Z caller=compact.go:974 msg="start of GC"
level=info ts=2019-05-30T05:04:31.284467016Z caller=compact.go:980 msg="start of compaction"
level=info ts=2019-05-30T05:04:31.354913163Z caller=compact.go:245 msg="start first pass of downsampling"
level=info ts=2019-05-30T05:04:31.354872977Z caller=compact.go:238 msg="compaction iterations done"

From the logs it appears it succeeded on the first pass of downsampling, but failed on the 2nd pass. Then all future attempts are failing on the first pass.

[FUSAKLA]

Hi, sorry for the inactivity. Does the issue still persist? Possibly did you try upgrading to recent 0.5.0? There should be some compactor fixes maybe it would solve your issue.

[chrisghill]

@FUSAKLA I ended up just deleting the bad block in S3. It's unclear what the source of the issue was, but this issue can probably be closed as the compactor is working fine now.

[FUSAKLA]

Ok, thanks for the info. Sorry to hear you had to delete the affected block.

Please feel free to reopen this issue or create a new one if it shows up again.

[BenManifold]

We're seeing a similar issue at the moment, no real clue yet how the block got into this state, but the compactor is in a crash loop as a result.

[bobykus31]

This stuff also happened to me

#7234

I was able to reproduce it by trying to download block with s3cmd

s3cmd ls   s3://ocsysinfra-prometheus-metrics/$chunk/
s3cmd get --recursive  s3://ocsysinfra-prometheus-metrics/$chunk/ data/$chunk/

The result was a missing index file in random situation. It seems it is correlated with Obj storage load (f.x. StorageGrid). And this is not just index file, in my case file /chunks/000001 was missing. Thats weird...