Prefect Block Resource: Type error in docs #255

scondo-prof · 2024-09-16T18:56:12Z

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Attribute in Prefect Block Resource

Data Type Difference

In the documentation it states that the type for Attribute Data -> String, Sensitive

This simply is not true, when I built it, it was of type map(string).

Godspeed and happy developing

mitchnielsen · 2024-09-16T21:49:17Z

Hey @scondo-prof, thanks for reaching out.

This field is a bit unique - it's technically a schema.StringAttribute, but it has a CustomType of jsontypes.NormalizedType. This is what helps us unmarshal user-provided content into a format that the API recognizes. Under the hood, we unmarshal it into map[string]interface.

This simply is not true, when I built it, it was of type map(string).

Could you show an example of the configuration you were using so I can see what specific example you have in mind? This field was a bit weird to work with since it had to be so flexible, so I'm curious to see if we could use map here (I don't think so, because that means you have to be able to predict every potential key in the Terraform provider code - which is just not very reasonable for Blocks since there are so many different kinds of them).

scondo-prof · 2024-09-19T14:32:13Z

Thank you for the response mitchell:

`main.tf`

terraform {
  required_providers {
    prefect = {
      source = "prefecthq/prefect"
    }
  }
}

provider "prefect" {
  account_id   = var.prefectAccountId
  workspace_id = var.prefectWorkspaceId
  api_key      = var.prefectApiKey
}

resource "prefect_block" "prefectBlock" {
  data      = jsonencode(var.prefectBlockData)
  name      = "${var.resourceName}-prefect-block"
  type_slug = var.prefectBlockTypeSlug
}

`variables.tf`

variable "prefectAccountId" {
  type    = string
  default = "9759ff6b-6f30-404f-92f3-02c593f6f7aa"
}

variable "prefectWorkspaceId" {
  type    = string
  default = "1733fd7b-18ad-4093-a0a1-4f128be45747"
}

variable "prefectApiKey" {
  type      = string
  sensitive = true
}

#Dependent on prefectBlockTypeSlug, use "prefect block type inspect <slug>" to see the required format
variable "prefectBlockData" {
  type = map(string)
}

variable "resourceName" {
  type = string
}

variable "prefectBlockTypeSlug" {
  type = string
  validation {
    condition     = contains(["aws-credentials", "aws-secret", "azure", "azure-blob-storage-container", "azure-blob-storage-credentials", "azure-container-instance-credentials", "azure-container-instance-job", "azure-cosmos-db-credentials", "azureml-credentials", "bigquery-warehouse", "bitbucket-credentials", "bitbucket-repository", "database-credentials", "databricks-credentials", "date-time", "discord-webhook", "docker-container", "docker-host", "docker-registry", "docker-registry-credentials", "ecs-task", "email", "email-server-credentials", "cloud-run-job", "gcp-credentials", "gcs", "gcs-bucket", "gcpsecret", "github", "github-credentials", "github-repository", "gitlab-credentials", "gitlab-repository", "json", "kubernetes-cluster-config", "kubernetes-credentials", "kubernetes-job", "lambda-function", "local-file-system", "mattermost-webhook", "ms-teams-webhook", "minio-credentials", "modal-credentials", "opsgenie-webhook", "pager-duty-webhook", "process", "remote-file-system", "s3", "s3-bucket", "smb", "sqlalchemy-connector", "secret", "sendgrid-email", "shell-operation", "slack-credentials", "slack-webhook", "snowflake-connector", "snowflake-credentials", "string", "twilio-sms", "vertex-ai-custom-training-job", "webhook", "dbt-cli-bigquery-target-configs", "dbt-cli-global-configs", "dbt-cli-postgres-target-configs", "dbt-cli-profile", "dbt-cli-snowflake-target-configs", "dbt-cli-target-configs", "dbt-cloud-credentials", "dbt-core-operation"], var.prefectBlockTypeSlug)
    error_message = "Variable prefectBlockTypeSlug must be one of the following values: : aws-credentials, aws-secret, azure, azure-blob-storage-container, azure-blob-storage-credentials, azure-container-instance-credentials, azure-container-instance-job, azure-cosmos-db-credentials, azureml-credentials, bigquery-warehouse, bitbucket-credentials, bitbucket-repository, database-credentials, databricks-credentials, date-time, discord-webhook, docker-container, docker-host, docker-registry, docker-registry-credentials, ecs-task, email, email-server-credentials, cloud-run-job, gcp-credentials, gcs, gcs-bucket, gcpsecret, github, github-credentials, github-repository, gitlab-credentials, gitlab-repository, json, kubernetes-cluster-config, kubernetes-credentials, kubernetes-job, lambda-function, local-file-system, mattermost-webhook, ms-teams-webhook, minio-credentials, modal-credentials, opsgenie-webhook, pager-duty-webhook, process, remote-file-system, s3, s3-bucket, smb, sqlalchemy-connector, secret, sendgrid-email, shell-operation, slack-credentials, slack-webhook, snowflake-connector, snowflake-credentials, string, twilio-sms, vertex-ai-custom-training-job, webhook, dbt-cli-bigquery-target-configs, dbt-cli-global-configs, dbt-cli-postgres-target-configs, dbt-cli-profile, dbt-cli-snowflake-target-configs, dbt-cli-target-configs, dbt-cloud-credentials, dbt-core-operation"
  }
}

`outputs.tf`

output "prefectBlockId" {
  value = prefect_block.prefectBlock.id
}

output "prefectBlockAccountId" {
  value = prefect_block.prefectBlock.account_id
}

output "prefectBlockWorkspaceId" {
  value = prefect_block.prefectBlock.workspace_id
}

`test.tfvars`

prefectApiKey = "<api-key>"

prefectBlockData = {
    value = "test-data"
}

resourceName = "test"

prefectBlockTypeSlug = "secret"

This results in the image:

Below is the example of variables.tf being modified and .tfvars being modified for the data attribute to be of type string:

`variables.tf`

variable "prefectAccountId" {
  type    = string
  default = "9759ff6b-6f30-404f-92f3-02c593f6f7aa"
}

variable "prefectWorkspaceId" {
  type    = string
  default = "1733fd7b-18ad-4093-a0a1-4f128be45747"
}

variable "prefectApiKey" {
  type      = string
  sensitive = true
}

#Dependent on prefectBlockTypeSlug, use "prefect block type inspect <slug>" to see the required format
variable "prefectBlockData" {
  type = string
}

variable "resourceName" {
  type = string
}

variable "prefectBlockTypeSlug" {
  type = string
  validation {
    condition     = contains(["aws-credentials", "aws-secret", "azure", "azure-blob-storage-container", "azure-blob-storage-credentials", "azure-container-instance-credentials", "azure-container-instance-job", "azure-cosmos-db-credentials", "azureml-credentials", "bigquery-warehouse", "bitbucket-credentials", "bitbucket-repository", "database-credentials", "databricks-credentials", "date-time", "discord-webhook", "docker-container", "docker-host", "docker-registry", "docker-registry-credentials", "ecs-task", "email", "email-server-credentials", "cloud-run-job", "gcp-credentials", "gcs", "gcs-bucket", "gcpsecret", "github", "github-credentials", "github-repository", "gitlab-credentials", "gitlab-repository", "json", "kubernetes-cluster-config", "kubernetes-credentials", "kubernetes-job", "lambda-function", "local-file-system", "mattermost-webhook", "ms-teams-webhook", "minio-credentials", "modal-credentials", "opsgenie-webhook", "pager-duty-webhook", "process", "remote-file-system", "s3", "s3-bucket", "smb", "sqlalchemy-connector", "secret", "sendgrid-email", "shell-operation", "slack-credentials", "slack-webhook", "snowflake-connector", "snowflake-credentials", "string", "twilio-sms", "vertex-ai-custom-training-job", "webhook", "dbt-cli-bigquery-target-configs", "dbt-cli-global-configs", "dbt-cli-postgres-target-configs", "dbt-cli-profile", "dbt-cli-snowflake-target-configs", "dbt-cli-target-configs", "dbt-cloud-credentials", "dbt-core-operation"], var.prefectBlockTypeSlug)
    error_message = "Variable prefectBlockTypeSlug must be one of the following values: : aws-credentials, aws-secret, azure, azure-blob-storage-container, azure-blob-storage-credentials, azure-container-instance-credentials, azure-container-instance-job, azure-cosmos-db-credentials, azureml-credentials, bigquery-warehouse, bitbucket-credentials, bitbucket-repository, database-credentials, databricks-credentials, date-time, discord-webhook, docker-container, docker-host, docker-registry, docker-registry-credentials, ecs-task, email, email-server-credentials, cloud-run-job, gcp-credentials, gcs, gcs-bucket, gcpsecret, github, github-credentials, github-repository, gitlab-credentials, gitlab-repository, json, kubernetes-cluster-config, kubernetes-credentials, kubernetes-job, lambda-function, local-file-system, mattermost-webhook, ms-teams-webhook, minio-credentials, modal-credentials, opsgenie-webhook, pager-duty-webhook, process, remote-file-system, s3, s3-bucket, smb, sqlalchemy-connector, secret, sendgrid-email, shell-operation, slack-credentials, slack-webhook, snowflake-connector, snowflake-credentials, string, twilio-sms, vertex-ai-custom-training-job, webhook, dbt-cli-bigquery-target-configs, dbt-cli-global-configs, dbt-cli-postgres-target-configs, dbt-cli-profile, dbt-cli-snowflake-target-configs, dbt-cli-target-configs, dbt-cloud-credentials, dbt-core-operation"
  }
}

`test.tfvars`

prefectApiKey = "<api-key>"

prefectBlockData = <<-EOF
    {
    value = "test-data"
}
EOF

resourceName = "test"

prefectBlockTypeSlug = "secret"

Resulted in this error:

I also tried a non multiline for the .tfvars:

`test.tfvars`

prefectApiKey = "<api-key>"

prefectBlockData = "{value = \"test-data\"}"


resourceName = "test"

prefectBlockTypeSlug = "secret"

which resulted in this error:

I look forward to your response.

Best,

Scott

Clarifies the expected data format for the "data" argument in the Block resource. The specific type is `schema.stringAttribute`, but we also set a CustomType of `jsontypes.NormalizedType`, which is an RFC 7159 JSON string. The Terraform Documentation tooling does not consider the CustomType field, so in the generated markdown it only say `String, Sensitive`. This can be confusing for users who expect to just pass any string here. So this change: - Clarifies the format expectations in the `data` argument description - Updates the examples to ensure we use `jsonencode` consistently Related to #255

mitchnielsen · 2024-09-19T16:22:39Z

Thanks for the extra context @scondo-prof, that helps a lot. I opened #257 to make this clearer. Could you take a look and see if that helps?

scondo-prof · 2024-09-19T17:31:02Z

Mitch,

First, I would like to apologize for calling you Mitchell in my response.

Second, that is incredibly helpful!

I have another question Mitch unrelated to my confusion in the documentation,

Will there be a Terraform Resource for Account API Keys developed?

I know that there is one for ServiceAccount which in turn gives an API key with a permission set, but that requires to be on a paid version of Prefect. As a developer whom cant afford that at the current moment, I was wondering if you and your fellow incredible devs were thinking about creating that resource? For people whom are lower on the totem pole and are utilizing the lowest tier of Prefect, this Terraform Resource would truly help with credential rotation to keep an environment as secure as possible.

Best,

Scott

mitchnielsen · 2024-09-19T23:23:07Z

No worries at all! I go by either name.

We'll get that PR merged, just need to fix a CI configuration drift that's making acceptance tests fail.

To your other question: this sounds very reasonable, and we definitely plan to (and already do) support APIs that are not specific to Prefect Cloud (see https://github.com/PrefectHQ/terraform-provider-prefect?tab=readme-ov-file#supported-objects).

Let me open a separate issue and we can get on the same page there.
Edit: opened #264

* Block "data" argument: Clarify expected data format Clarifies the expected data format for the "data" argument in the Block resource. The specific type is `schema.stringAttribute`, but we also set a CustomType of `jsontypes.NormalizedType`, which is an RFC 7159 JSON string. The Terraform Documentation tooling does not consider the CustomType field, so in the generated markdown it only say `String, Sensitive`. This can be confusing for users who expect to just pass any string here. So this change: - Clarifies the format expectations in the `data` argument description - Updates the examples to ensure we use `jsonencode` consistently Related to #255 * Fix Marvin account email

mitchnielsen · 2024-09-23T19:13:42Z

The data field description is now updated at https://registry.terraform.io/providers/PrefectHQ/prefect/latest/docs/resources/block#required.

scondo-prof added the bug Something isn't working label Sep 16, 2024

mitchnielsen changed the title ~~Type error~~ Prefect Block Resource: Type error in docs Sep 16, 2024

jamiezieziula added customer-requested open-source development labels Sep 17, 2024 — with Linear

jamiezieziula assigned mitchnielsen Sep 17, 2024

mitchnielsen mentioned this issue Sep 19, 2024

Block "data" argument: Clarify expected data format #257

Merged

mitchnielsen mentioned this issue Sep 19, 2024

Add support for API keys #264

Open

mitchnielsen closed this as completed Sep 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prefect Block Resource: Type error in docs #255

Prefect Block Resource: Type error in docs #255

scondo-prof commented Sep 16, 2024

mitchnielsen commented Sep 16, 2024 •

edited

Loading

scondo-prof commented Sep 19, 2024 •

edited

Loading

mitchnielsen commented Sep 19, 2024

scondo-prof commented Sep 19, 2024 •

edited

Loading

mitchnielsen commented Sep 19, 2024 •

edited

Loading

mitchnielsen commented Sep 23, 2024

Prefect Block Resource: Type error in docs #255

Prefect Block Resource: Type error in docs #255

Comments

scondo-prof commented Sep 16, 2024

Community Note

Data Type Difference

mitchnielsen commented Sep 16, 2024 • edited Loading

scondo-prof commented Sep 19, 2024 • edited Loading

main.tf

variables.tf

outputs.tf

test.tfvars

variables.tf

test.tfvars

test.tfvars

mitchnielsen commented Sep 19, 2024

scondo-prof commented Sep 19, 2024 • edited Loading

mitchnielsen commented Sep 19, 2024 • edited Loading

mitchnielsen commented Sep 23, 2024

mitchnielsen commented Sep 16, 2024 •

edited

Loading

scondo-prof commented Sep 19, 2024 •

edited

Loading

`main.tf`

`variables.tf`

`outputs.tf`

`test.tfvars`

`variables.tf`

`test.tfvars`

`test.tfvars`

scondo-prof commented Sep 19, 2024 •

edited

Loading

mitchnielsen commented Sep 19, 2024 •

edited

Loading