Mimir on AWS ECS Fargate.

[inugravi]

Grafana Mimir on AWS ECS Fargate.

Hello Team,

Since we are thinking to leverage Grafana Mimir to our PROD infrastructure, We chose Monolith deployment with 3 mimir fargate containers as below. (We are still not into Kubernetes, as of today for Microservice type deployment)

Thanks to the documentation which has everything, but we are a bit confused while authoring the mimir.yml .
We are not sure about the mandatory/non-mandatory sections of the config file.
As we know common block for AWS s3 storage in mimir.yml for monolith deployment is mandatory, but sections like compactor, distributor, ingester etc, are they mandatory to declare in mimir.yml ?

So, my final question can be put it the below way.
Is there any document/template of mimir.yml available? (specifically, to host it in ECS FARGATE or docker swarm)

Describe the solution that you’d like or the expected outcome

The expected implementation is that.

1. There will be 3 ECS Fargate Containers which are service discovery enabled (each should be able to discover each other) and should sit behind a load balancer.
2. The mentioned 3 containers should persist data to respective s3 buckets.
3. The above-mentioned load balancer end point should be consumed by grafana-agent running on windows/linux target for remote writing.

Here our only requirement is that we just need the precise mimir.yml (or at least it's structure) to get it embedded within all the 3 mimir ecs fargate containers.

Sorry if it is too long but here is our requirement. Please help.

#GrafanaMimir_HelpingHands

[pracucci]

👋 Hi and welcome to Mimir!

As we know common block for AWS s3 storage in mimir.yml for monolith deployment is mandatory, but sections like compactor, distributor, ingester etc, are they mandatory to declare in mimir.yml ?

They're not mandatory. We actually recommend to run Mimir mainly with the default configuration except for the object storage and configuring memberlist to let Mimir replicas to communicate eachother and share the ring (which is the data structure Mimir uses extensively for sharding and replication).

Is there any document/template of mimir.yml available? (specifically, to host it in ECS FARGATE or docker swarm)

We don't have a document or template for Fargate or Docker swarm, but there's a simple example setup here:
https://github.com/grafana/mimir/blob/main/docs/sources/mimir/tutorials/play-with-grafana-mimir/config/mimir.yaml#L23

[aledeulo]

The above path does not exist anymore.
The current url is: https://github.com/grafana/mimir/blob/main/docs/sources/mimir/get-started/play-with-grafana-mimir/config/mimir.yaml

[pracucci]

There will be 3 ECS Fargate Containers which are service discovery enabled (each should be able to discover each other) and should sit behind a load balancer.

I'm not familiar with ECS Fargate so I'm not sure what services discovery is in this context. Keep in mind that Mimir needs to build a cluster too. If you use memberlist (recommended), then you need some "seed nodes" configured as follows:

memberlist:
  join_members: [ "address-of-seed-node-1", "address-of-seed-node-2" ]

Basically, a seed node is a Mimir replica (one of your 3 replicas) with a well know address. If address is dynamic, you can configure your infrastructure to have 1 DNS record which replies with the addresses of (some of) your Mimir replicas. Mimir will periodically resolve such DNS record and contact the new addresses when discovered through it.

The mentioned 3 containers should persist data to respective s3 buckets.

You need persistence at two levels:

• The replica level: you will need a persistent disk for the WAL
• The object storage (bucket): you need 1 bucket for blocks storage, and 1 for ruler and alertmanager configs. These buckets must be shared across all replicas (so don't create 1 bucket for each replica).

[inugravi]

Thanks @pracucci for the swift response.

I got my doubts cleared for now.

Happy Christmas 🎄 and New Year ✨️

Regarding the WAL which persists data to Disk storage.
Even from this disk, the data will get dumped/pushed to aws s3 storage after the specified time period right ?

Correct me if I'm wrong.

[pracucci]

Regarding the WAL which persists data to Disk storage.
Even from this disk, the data will get dumped/pushed to aws s3 storage after the specified time period right ?

The WAL is not persisted to S3 (just on local disk) but the metrics data is uploaded to the AWS S3 storage every 2 hours. After data is uploaded to the AWS S3 storage (which is technically when a TSDB block is cut from the in-memory TSDB head every 2 hours) then the WAL is truncated too.

[KaladinAlThor]

@inugravi I'm finding myself in the same situation with fargate and mimir.yml with the monolithic mode. Would you mind posting what you came up with? Hoping it would save me a couple of wheels, especially if you made any tweaks for ECS/Fargate!

[inugravi]

Hi @KaladinAlThor ,

Apologies for the late response, I'm on vacation and will be back tomorrow. So, once I get something will definitely help you out with everything.

Thanks,
inugravi

[pracucci]

Would be great if you could contribute a documentation page on how to run Mimir on AWS Fargate. We (Mimir maintainers) have no experience with it.

[inugravi]

Definitely @pracucci
I have started the work today and will keep you posted. Helpful if you could keep this thread open for discussion.
@KaladinAlThor Will let you know the progress.

[pracucci]

Sure, thanks!

[a0s]

Starting Mimir on ECS is not the rocket science. You just need to use the service discovery. Some copy paste from my terraform (cdktf typescript) config:

readonly portHTTP = 8080
readonly portGRPC = 9095
readonly portMemberlist = 7946

const ecsDiscovery = new ServiceDiscoveryPrivateDnsNamespace(this, "private_dns_service_discrovery", <ServiceDiscoveryPrivateDnsNamespaceConfig>{
      provider: providers.awsPrimary,
      name: "ecs.local",
      vpc: network.vpcModule.vpcIdOutput
    })

this.discoveryService = new ServiceDiscoveryService(this, "discovery_service", <ServiceDiscoveryServiceConfig>{
      provider: config.provider,
      name: "grafana-mimir",
      dnsConfig: <ServiceDiscoveryServiceDnsConfig>{
        namespaceId: ecsDiscovery.id,
        dnsRecords: [
          <ServiceDiscoveryServiceDnsConfigDnsRecords>{
            ttl: 10,
            type: "A"
          }
        ],
        routingPolicy: "MULTIVALUE"
      },
      healthCheckCustomConfig: <ServiceDiscoveryServiceHealthCheckCustomConfig>{
        failureThreshold: 1
      }
    })

const task = new EcsTaskDefinition(this, "task", <EcsTaskDefinitionConfig>{
      provider: config.provider,
      lifecycle: <TerraformResourceLifecycle>{
        createBeforeDestroy: true
      },
      executionRoleArn: taskExecutionRole.arn,
      taskRoleArn: taskRole.arn,
      family: "grafana-mimir",
      networkMode: "awsvpc",
      requiresCompatibilities: ["FARGATE"],
      cpu: "512",
      memory: "1024",
      runtimePlatform: <EcsTaskDefinitionRuntimePlatform>{
        operatingSystemFamily: "LINUX",
        cpuArchitecture: this.cpuArchitecture
      },
      volume: [],
      containerDefinitions: Fn.jsonencode([
        {
          cpu: 0,
          name: "main",
          image: "grafana/mimir:2.5.0",
          command: [
            "-server.http-listen-port", `${this.portHTTP}`,
            "-server.grpc-listen-port", `${this.portGRPC}`,
// !!!!! THE MOST IMPORTAND PART - dns+ resolving !!!!!!!
            "-memberlist.join", `dns+${this.discoveryService.name}.${ecsDiscovery.name}:${this.portMemberlist}`,
            "-common.storage.backend", "s3",
            "-common.storage.s3.endpoint", `s3.${config.provider.region}.amazonaws.com`,
            "-common.storage.s3.region", config.provider.region,
            "-blocks-storage.s3.bucket-name", blocksStorage.bucket,
            "-alertmanager-storage.s3.bucket-name", alertmanagerStorage.bucket,
            "-ruler-storage.s3.bucket-name", rulerStorage.bucket,
            "-auth.multitenancy-enabled", `${config.multitenancyEnabled}`,
            "-distributor.ingestion-rate-limit", "1000000",
            "-distributor.ingestion-burst-size", "10000000"
          ],
          healthCheck: {
            command: ["CMD", "/bin/true"],
            interval: 10,
            retries: 1,
            timeout: 5
          },
          portMappings: [
            {
              containerPort: this.portHTTP,
              hostPort: this.portHTTP,
              protocol: "tcp"
            },
            {
              containerPort: this.portGRPC,
              hostPort: this.portGRPC,
              protocol: "tcp"
            },
          ],
          essential: true,
          environment: [],
          mountPoints: [],
          volumesFrom: [],
          startTimeout: 15,
          logConfiguration: {
            logDriver: "awslogs",
            options: {
              "awslogs-group": logGroup.name,
              "awslogs-region": config.provider.region,
              "awslogs-stream-prefix": "ecs",
              "awslogs-datetime-format": "%Y-%m-%d %H:%M:%S,%f"
            }
          }
        }])
    })

new EcsService(this, "service", <EcsServiceConfig>{
      provider: config.provider,
      name: "grafana-mimir",
      cluster: cluster.name,
      deploymentController: <EcsServiceDeploymentController>{
        type: "ECS"
      },
      deploymentMinimumHealthyPercent: 50,
      deploymentMaximumPercent: 100,
      desiredCount: config.instanceCount || 1,
      enableEcsManagedTags: true,
      healthCheckGracePeriodSeconds: 0,
      networkConfiguration: <EcsServiceNetworkConfiguration>{
        assignPublicIp: false,
        subnets: config.subnetIds,
        securityGroups: [this.securityGroup.id]
      },
      capacityProviderStrategy: [
        <EcsServiceCapacityProviderStrategy>{
          capacityProvider: this.capacityProvider,
          weight: 100
        }
      ],
      platformVersion: "LATEST",
      schedulingStrategy: "REPLICA",
      taskDefinition: `${task.family}:${task.revision}`,
      serviceRegistries: <EcsServiceServiceRegistries>{  // <= DONT FORGET THIS TOO
        registryArn: this.discoveryService.arn
      }
    })

[a0s]

@mirnujAtom, yes, i stop using Mimir :) I switched to JuiceFS as storage.

[a0s]

@mirnujAtom i believe that new "service connect for ecs" from Amazon should works good, but i didn't test it yet

[KaladinAlThor]

@a0s @mirnujAtom
Service connect won't work. I raised it with their support. It doesn't create any r53 records, and is only meant for service to service communication, not for tasks within a service

[a0s]

@KaladinAlThor thank you for info. So, fallback plan: write a lambda that reacts to the task start event and updates r53 record..

[udomsak]

confirm that @KaladinAlThor said that AWS ECS service-connect does not work as expect. I run with single instance of Mimir.

// configuration run 
port_mappings = [
        local.mimir_ports
      ]
      command = [
        "-server.http-listen-port=${local.mimir_port}",
        "-common.storage.backend=s3",
        "-common.storage.s3.endpoint=s3.${data.aws_region.current.name}.amazonaws.com",
        "-common.storage.s3.region=${data.aws_region.current.name}",
        "-blocks-storage.s3.bucket-name=${aws_s3_bucket.mimir_storage.id}",
        "-ruler-storage.s3.region=ap-southeast-1",
        "-server.grpc-listen-address=0.0.0.0",
        "-server.grpc-listen-port=9095",
        "-server.http-listen-address=0.0.0.0",
        "-ruler-storage.s3.bucket-name=${aws_s3_bucket.mimir_ruler_storage.id}",
        "-ruler-storage.s3.endpoint=s3.${data.aws_region.current.name}.amazonaws.com"
      ]

January 13, 2024 at 24:07 (UTC+7:00) | {"@level":"warn","@message":"Failed to get prometheus heuristics","@timestamp":"2024-01-12T17:07:38.913300Z","err":"failed to get buildinfo: error querying resource: Get \"http://mimir:9009/prometheus/api/v1/status/buildinfo\": read tcp 127.0.0.1:51180-\u003e127.255.0.2:9009: read: connection reset by peer","logger":"tsdb.prometheus"} 

[a0s]

@inugravi managing config files (like mimir.yml) with terraform on ECS is a real asspain. Thats why i am using command cli parameters.
If we were talking about EKS, we could collect a config file from a template, put it in a configmap, and then mount it to the container would be an elementary task. But in the case of ECS, we only have EFS, and we do not have the ability to put the file on EFS using terraform.

[udomsak]

better If Mimir support to get configuration from AWS SSM parameter like ADOT does.

[radevkal]

@KaladinAlThor @a0s @mirnujAtom Hi Guys, I am also facing issues making Loki Simple scalable work on ECS fargate which follows similar architecture and memberlist.
The Loki Reader and Writer are separated in different services in same Fargate cluster and We are using AWS CloudMap Service discovery to make the tasks communicate. The tasks do come online but there are constant UDP timeout and memberlist related errors.
I would like to know if this is a supported architecture with Loki or could it be a Configuration issue? Kindly help.

[a0s]

@radevkal We threw out Mimir, now we use JuiceFS to store Prometheus metrics. We are going to try OpenObserve as a remote-write storage for Prometheus, because we already use O2 for traces and logs.

[uginroot]

@inugravi I want to configure machines in AWS ECS forgate, but I can't quite figure out if I can use one EFS storage that will be mounted to all instances?

Or maybe I should use a separate EFS for each instance?

If it is a separate EFS, can I be sure that if the instances are terminated correctly during scaling, all the data from the EFS will go to s3?