Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to pull topics using Terraform #40

Open
maxverbosity opened this issue Feb 15, 2024 · 7 comments
Open

Unable to pull topics using Terraform #40

maxverbosity opened this issue Feb 15, 2024 · 7 comments

Comments

@maxverbosity
Copy link

Summary

When running terraform plan, the refresh fails and errors out on topics, claiming too may connections.

kafka.tf snippet

resource "upstash_kafka_topic" "topic" {
    for_each = { for topic in var.cluster_topics : topic.topic_name => topic }

    topic_name       = "${ each.key }"
    partitions       = "${ each.value.partitions }"
    retention_time   = "${ each.value.retention_time }"
    retention_size   = "${ each.value.retention_size }"
    max_message_size = "${ each.value.max_message_size }"
    cleanup_policy   = "${ each.value.cleanup_policy }"
    cluster_id       = upstash_kafka_cluster.cluster.cluster_id
}

Terraform output snippet

<snip>
Planning failed. OpenTofu encountered an error while generating this plan.

╷
│ Error: Get Kafka Topic failed, status code: 429 response: <html>
│ <head><title>429 Too Many Requests</title></head>
│ <body>
│ <center><h1>429 Too Many Requests</h1></center>
│ <hr><center>nginx</center>
│ </body>
│ </html>
│
│
│   with module.v1.module.prod_cluster.upstash_kafka_topic.topic["ccm.lateralled"],
│   on ../../../modules/digitalocean/kafka/1.0.0/main.tf line 17, in resource "upstash_kafka_topic" "topic":
│   17: resource "upstash_kafka_topic" "topic" {
│
╵
╷
│ Error: Get Kafka Topic failed, status code: 429 response: <html>
│ <head><title>429 Too Many Requests</title></head>
│ <body>
│ <center><h1>429 Too Many Requests</h1></center>
│ <hr><center>nginx</center>
│ </body>
│ </html>
│
│
│   with module.v1.module.prod_cluster.upstash_kafka_topic.topic["registrant.class"],
│   on ../../../modules/digitalocean/kafka/1.0.0/main.tf line 17, in resource "upstash_kafka_topic" "topic":
│   17: resource "upstash_kafka_topic" "topic" {
│
╵
╷
│ Error: Get Kafka Topic failed, status code: 429 response: <html>
│ <head><title>429 Too Many Requests</title></head>
│ <body>
│ <center><h1>429 Too Many Requests</h1></center>
│ <hr><center>nginx</center>
│ </body>
│ </html>
│
│
│   with module.v1.module.prod_cluster.upstash_kafka_topic.topic["event.updated"],
│   on ../../../modules/digitalocean/kafka/1.0.0/main.tf line 17, in resource "upstash_kafka_topic" "topic":
│   17: resource "upstash_kafka_topic" "topic" {
│
╵
</snip>

Expected outcome

The plan should succeed.
@maxverbosity
Copy link
Author

I've figured out the cause of the problem.

For background, we have 68 topics we are creating in this loop. Terraform is running those resource creations in parallel which is being flagged on your end and throttled.

Adding a flag to the execution fixes the problem. Removing it causes the problem to occur again.

# This works
$ terraform plan -parallelism=1

# This does NOT work and I get the 429 errors listed above
$ terraform plan

I'm not sure what else we can do on our end other than use the flag. We can work around this with the parallelism flag but it doesn't seem like a viable permanent solution. I don't think our number of topics is large and we don't have insight into your systems to know what's happening when the request hits.

@maxverbosity
Copy link
Author

Is any one even monitoring this? I submitted this ticket 4 days ago and haven't received anything- not even an acknowledgement that the issue has been received or read.

@fahreddinozcan
Copy link
Collaborator

Hey,

sorry for the latency here. I will be investigating this issue, and see what I can come up with

@fahreddinozcan fahreddinozcan transferred this issue from upstash/issues Feb 19, 2024
@maxverbosity
Copy link
Author

No worries- thanks for looking into it.

@fahreddinozcan
Copy link
Collaborator

Hey, I wanted to update you on the progress here. I've looked into several possible solutions, however none were feasible in this case. Unfortunately, Terraform doesn't support batch requests other than minimal internal optimizations. So we can't really intercept and batch the requests as far as I see. One other solution could be to utilize caching and reduce the API call count, however Terraform doesn't have support for that either. For both of these topics, there's an ongoing issue on the sdk plugin repo.

We've also considered increasing the request limit per second on the API, however it doesn't really seem possible to scale it to these numbers, and we'd still need to update the limit whenever someone hits this limit on the Terraform. So we should be able to solve this problem on the TF Provider side.

I will keep looking into this, and update you as soon as I find a feasible solution.

@binnsr
Copy link

binnsr commented Jun 7, 2024

Is there an update on this issue?

@binnsr binnsr mentioned this issue Jun 7, 2024
Closed
@fahreddinozcan
Copy link
Collaborator

@binnsr unfortunately not. I will check with the Terraform SDK maintainers, to see if they have any work related to parallelism.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@maxverbosity @binnsr @fahreddinozcan