A way to create 1 instance for every n pods

[pocesar]

We are currently scaling up and down using spot instances (1000's) and the overhead of 2GB is really showing, almost defeating the purpose of using spot instances.
Ideally the operator should see how many pods are currently there and create a rule for 1 proxy pod for every n pods (that you can set), say, 40 connections per pod and distribute accordingly.

[enocom]

Typically we recommend customers run the Proxy on the same pod as their workload. Otherwise, there's a risk of cross VM traffic that isn't encrypted by the Proxy.

What if we added support for configuring resource limits instead? 2GB is really just a ballpark starting point and not necessarily the right value for every workload.

[hessjcg]

You can already configure the resource limits on the proxy container. This is a snippet from docs/examples/deployment-postgres-tcp.yaml showing how to reduce the resource limits on the proxy container pod.

apiVersion: cloudsql.cloud.google.com/v1
kind: AuthProxyWorkload
metadata:
  name: authproxyworkload-sample
spec:
  authProxyContainer:
    # Resource configuration depends on an application's requirements. You
    # should adjust the following values based on what your application
    # needs. For details, see https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
    resources:
      requests:
        # The proxy's CPU use scales linearly with the amount of IO between
        # the database and the application. Adjust this value based on your
        # application's requirements.
        cpu: "0.25"
        # The proxy's memory use scales linearly with the number of active
        # connections. Fewer open connections will use less memory. Adjust
        # this value based on your application's requirements.
        memory: "512Mi"
  workloadSelector:
    kind: "Deployment" # Applies to a "Deployment"
    name: "gke-cloud-sql-app" # named 'gke-cloud-sql-app'
  instances:
    - connectionString: "my-project:us-central1:instance" # from your Cloud SQL Database instance
      portEnvName: "DB_PORT" # Will set an env var named 'DB_PORT' to the database port
      hostEnvName: "DB_HOST" # Will set an env var named 'DB_HOST' to the proxy's host, 127.0.0.1

Alternatively, you could run the proxy as a deployment with a service. This is described in the k8s service example in the cloud-sql-proxy repo. This would allow you to scale the proxy pods independently from the application pods.

Here's how you would adapt that example to let the operator apply the proxy pod to the PGBouncer deployment.

• Remove the cloud-sql-proxy container from the pgbouncer_deployment.yaml.
• Add an AuthProxyWorkload resource that adds the proxy to your pgbouncer_deployment

[pocesar]

Typically we recommend customers run the Proxy on the same pod as their workload. Otherwise, there's a risk of cross VM traffic that isn't encrypted by the Proxy.

What if we added support for configuring resource limits instead? 2GB is really just a ballpark starting point and not necessarily the right value for every workload.

from my observations, it's barely using 1% of the memory and CPU allocated, but each container in the pod is connecting once to the proxy, so it's pretty light in that sense, mostly write queries

[enocom]

In that case, I'd definitely try tuning the resource limits down as @hessjcg suggested.

[pocesar]

but for an operator point-of-view, you should be able to ask it to set it up for you and load balance between the n maximum connections per pod set automatically with an HPA, service + hpa + deployment and be able to connect to said service name exported as a environment variable that could be interpolated with $()

[enocom]

We could explore adding another resource that is a proxy-as-a-service where multiple pods could all connect, provided we have proper security in place. Is that what you're suggesting here?

Otherwise, we think of the Operator as an opinionated deployment of the Proxy where best practices are built-in: e.g., Proxy always as a sidecar.

[pocesar]

@enocom yes, exactly a proxy-as-a-service that does the heavy-lifting for you, including setting it properly to the amount of expected connections from other pods

[enocom]

Got it. Thanks.

One of the biggest reasons we don't already do this is that it encourages an anti-pattern of having multiple applications all go through one Proxy. So we encourage people who explore this option to ensure they're using only one application (with N replicas) per Proxy service. Also, the Proxy doesn't support incoming encryption, so we instead suggest people put it behind a connection pooler (e.g., pgbouncer) and then configure some sort of authentication such that only authorized workloads can connect.

Maybe we should be showing people how to use the Operator with a pgbouncer deployment for maximum extensibility.