This repo is a basic demo of a type of "systemd" for a container which seeks to simply bootstrap mTLS certificates for the 'background' application to use.
Essentially, this is either a go (or nodejs) application running in a container that on start, reads mTLS certificates from GCP Secret Manager.
Once it read the certificates, it will then provide them to background applications to use.
Basically its a way to acquire mTLS certificates for applications that may not have the capability to specifically read them from GCP.
this repo is not supported by google
- Its not good container practice to run multiple applications in one container...anyway why do all this and not just
systemd
?
Good question!....the intent for all this is to use with GCP Confidential Space instances. These VMs just run containers with a specific profile for very secure workloads.
One characteristic is that the not even the VM operator can access the runtime (eg ssh in) and the runtime container can demonstrate that it is running a specific workload prior to receiving sensitive data or decryption keys.
Each container is treated as a Trusted Execution Environment (TEE)
is initially startsup with no credentials (no certificates, no decryption keys). It uses the TEE's Attestation Service to 'prove' to a remote system the application is in a trusted environment. Once that step is done, it will receive decryption keys or credentials.
For TEE --> TEE
traffic, one such credential would be mTLS keypairs issued by the remote party. This ensures each TEE is actually communicating with another authorized TEE (and vice versa).
Its fairly well know how an application can retrieve or decrypt data within a GCP TEE:
- Constructing Trusted Execution Environment (TEE) with GCP Confidential Space
- Multiparty Consent Based Networks (MCBN)
but the real issue this repo addresses is what happens if the "application" (redis, mysql, etc) has no built in mechanism to acquire mTLS keypairs thorough Confidential Space?
In this case, we needed a 'bootstrap' application which understands how to interact with GCP Confidential Space, acquire the credentials, and then furbish it to the backend application.
The bootstrap application in this case is a simple golang app that uses the attestation token to retrieve mTLS secrets from a collaborators Secret Manager.
Once it acquires the mTLS keys, it will os/exec the background application and await its completion.
Basically, is a container which starts a go application that gets keys, provides those keys to a background application and launches the background app.
See
-
As for why not use
systemd
?
Well, running systemd requires a lot of additional software which can compromise the TEE itself. Its far better to use minimal container surface. In most of the examples, i just "copied" the static-compiled binary over into a distroless/base.
I recognize not all background services can run on distroless images...
Anyway, there are four variations described here with different background applications and key requirements.
-
Redis
In ths mode the foreground application acquires keys and provides those keys to redis as its own startup arguments for mTLS
-
Postgres
In ths mode the foreground application acquires keys and provides those keys to postgres as its own startup arguments for mTLS
-
Envoy (HTTPFilter)
The foreground application acquires keys and writes the keys to the filesystem. Envoy HTTPFilter is started which uses those keys for mTLS
-
Envoy (NetworkFilter)
The foreground application acquires keys and writes the keys to the filesystem. Envoy NetworkFilter is started which uses those keys for mTLS
-
Multiparty Consent Based Networks (MCBN)
This mode describes a very rare situation where multiple collaborators each provide a partial key to use for TLS between TEEs. In this, mTLS is not used in the traditional RSA keypair mode but the actual
TEE->TEE
traffic is only allowed if each collaborator provides their key share.
We will primarily use GCP Secret Manager to store the mTLS secrets so upload those
gcloud secrets create ca --replication-policy=automatic --data-file=certs/ca.pem
gcloud secrets create server-cert --replication-policy=automatic --data-file=certs/server.crt
gcloud secrets create server-key --replication-policy=automatic --data-file=certs/server.key
gcloud auth application-default login
Then edit the following in ths repo and replace the PROJECT_ID
value you use
# edit the following
- server_envoy_http_proxy/main.go
- server_envoy_network_proxy/main.go
- server_redis/main.go
- server_postgres/main.go
- psk/main.js
For Redis, build the container and test:
cd server_redis/
docker build -t server_redis .
docker run -ti -v "$HOME"/.config/gcloud:/root/.config/gcloud -p 16379:16379 server_redis
At this point the redis container listens on :16379
for mTLS traffic where the keys were downloaded from Secret Manger
You can verify using the test redis client:
cd client
go run main.go
For Postgres, we use the default docker image and entrypoint after bootstrapping the certificates.
We also setup postgres cert based auth (which IMO, i s basic, it uses the CN field as the username; i suspect you can do more but my knowledge is basic).
main.go
sets up the startup parameters:
cmd := exec.Command("/usr/local/bin/docker-entrypoint.sh", "postgres", "--port=5432", "--ssl=on", "--ssl_cert_file=/config/server.crt", "--ssl_key_file=/config/server.key", "--ssl_ca_file=/config/ca.pem", "--hba_file=/config/pg_hba.conf")
//cmd.Env = os.Environ()
//cmd.Env = append(cmd.Env, "POSTGRES_PASSWORD=mysecretpassword")
the pg_hba.conf
file enforces certs:
# TYPE DATABASE USER ADDRESS METHOD
local postgres all peer
hostssl all all 0.0.0.0/0 cert
to use
cd server_postgres/
docker build -t server_postgres .
docker run -ti -v "$HOME"/.config/gcloud:/root/.config/gcloud -p 5432:5432 server_postgres
At this point the postgres container listens on :5432
for mTLS traffic where the keys were downloaded from Secret Manger
You can verify using the a postgres client with client certs
# change vm's public address to /etc/hosts
127.0.0.1 server.domain.com
# POSTGRES_PASSWORD=mysecretpassword
$ psql "host=server.domain.com port=5432 user=postgres sslmode=verify-full sslcert=certs/postgres.crt sslkey=certs/postgres.key sslrootcert=certs/ca.pem"
postgres=# SHOW SERVER_VERSION;
server_version
--------------------------------
15.2 (Debian 15.2-1.pgdg110+1)
(1 row)
note the postgres.crt
has the CN field thats the username....like i said, its a real basic way to do auth...
C=US, O=Google, OU=Enterprise, CN=postgres
For envoy HTTP backend, the envoy config simply sends a static echo response back
cd server_envoy_http_proxy/
docker build -t server_envoy_http_proxy .
docker run -ti -v "$HOME"/.config/gcloud:/root/.config/gcloud -p 8081:8081 server_envoy_http_proxy
Now that the envoy background process is running, send mTLS traffic
curl -v --connect-to server.domain.com:443:127.0.0.1:8081 --cacert certs/ca.pem \
--cert certs/client.crt \
--key certs/client.key https://server.domain.com/
For envoy Network backend, we will launch two processes in the background: envoy in TCP mode which handles mTLS and a background TCP application which just happens to be an http server.
cd server_envoy_network_proxy/
cd backend_app/
GOOS=linux GOARCH=amd64 go build -o server
cp server ../
rm server
docker build -t server_envoy_network_proxy .
docker run -ti -v "$HOME"/.config/gcloud:/root/.config/gcloud -p 8081:8081 server_envoy_network_proxy
Now that the envoy background process is running, send mTLS traffic (again, we just happen to be running an http TCP server so curl works here)
curl -v --connect-to server.domain.com:443:127.0.0.1:8081 --cacert certs/ca.pem \
--cert certs/client.crt \
--key certs/client.key https://server.domain.com/
This demonstrates the multiparty consent network described earlier.
For this, we need to seed secret manager with each collaborators partal TLS-PSK
keys
cd psk/
echo -n "2c6f63f8c0f53a565db041b91c0a95add8913fc102670589db3982228dbfed90" > alice.psk
echo -n "b15244faf36e5e4b178d3891701d245f2e45e881d913b6c35c0ea0ac14224cc2" > bob.psk
gcloud secrets create alice_psk --replication-policy=automatic --data-file=alice.psk
gcloud secrets create bob_psk --replication-policy=automatic --data-file=bob.psk
Then build and run the node application. The node application simulates the go application above but we are only using node as the "only application" since very few systems support defining TLS-PSK keys (see golang#6379, envoy#13237). Ideally you could use the node app as a TCP proxy for a background application you would run. This is not demonstrated in thsi repo
docker build -t psk_nodejs .
docker run -ti -v "$HOME"/.config/gcloud:/root/.config/gcloud -p 8081:8081 psk_nodejs
Now that the node app has all the keys, run the client
cd client/
node main.js
TODO: write certificates to any /tmpfs
that that is mapped to the container