Make persist interval configurable via environment variable

[alexott]

By default, the remote state backend persists state to the remote storage every 20 seconds—this value is hardcoded. But when the state is big (> 100Mb), this operation takes a long time and consumes a significant portion of these 20 seconds. This PR allows to make the persist interval configurable via the TF_BACKEND_PERSIST_INTERVAL_SECONDS environment variable, which allows to specify how often persistence should happen.

Please guide what part of the documentation needs to be updated to document it...

P.S. I thought about adding the ability to completely disable this persist interval like it's done when the remote execution backend is used - but this could be a topic for discussion.

Fixes #35115

Target Release

1.8.x

Draft CHANGELOG entry

NEW FEATURES | UPGRADE NOTES | ENHANCEMENTS | BUG FIXES | EXPERIMENTS

• Added TF_BACKEND_PERSIST_INTERVAL_SECONDS environment variable to control persist interval in the remote state backend.

[brandonc]

@alexott Hey, Alex, thanks for this improvement. We'd like to enable this capability, but we need to account for allowing HCP Terraform to set a minimum interval in order to protect itself from state snapshot load. What I've found with your version is that I can override any interval our platform sets due to the way that backends are structured and used.

What I'm thinking is that this env variable that you've recommended can define the maximum interval to allow you to store state less frequently than what is set by HCP Terraform, which will now define the minimum interval. I'll incorporate your code in an upcoming PR of my own.

[alexott]

@brandonc hmmm, as I saw in the code, this change doesn't affect the behavior of HCP Terraform, etc. It just allows to specify persist interval via environment variable instead of using the hardcoded one... No other places are affected. Of course, if the new behavior exists, then it's just a way a previous version was implemented...

But anyway, I'm happy if you implement something else that will allow to reach the desired behavior - do you have any ETA on that?

[brandonc]

@alexott It's a little roundabout, but the gist is that HCP Terraform ends up running terraform using a local backend, and users may set environment variables in that context. ETA probably next week or two, as it's prioritized in with some other stuff.

[alexott]

@brandonc do you need any help from me to speedup this implementation?

[brandonc]

@alexott not at this time. The work is prioritized and narrowly scoped so it shouldn't be too much of a wait

[sebasslash]

Hey Alex! Thanks again for your contribution here. Going to close this PR in favor of #35176 which includes some minor refactoring. All credit goes to you! 👍

[brandonc]

@alexott We discovered that the maximum interval was already implemented but opened a new PR to move ENV evaluation. Thanks again