Unable to write to GCS bucket

[trojocerqlar]

Summary

After following the documentation from https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#provision-static it is possible for my pod to read from my GCS bucket but not writing files to it.
I always get the errors like:
touch: close failed: Permission denied
mkdir: can't create directory 'tmp/dir1': Permission denied

I would highly appreciate if anyone can give me a hint on how to resolve this.

Details

Here are my files with my PersistentVolume, PersistentVolumeClaim and Pod configuration

# pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: gcs-fuse-csi-pv
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 5Gi
  storageClassName: example-storage-class
  mountOptions:
    - implicit-dirs   
    - uid=65534       # Changing it to a different number does not work either
    - gid=65534       # Changing it to a different number does not work either
    - dir-mode=777
    - file-mode=777
    - o=allow_other
  csi:
    driver: gcsfuse.csi.storage.gke.io
    volumeHandle: test-bucket-cqlr
    volumeAttributes:
      gcsfuseLoggingSeverity: debug
  claimRef:
    name: gcs-fuse-csi-static-pvc
    namespace: default

# pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gcs-fuse-csi-static-pvc
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: example-storage-class

# pod.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
 name: tomas-sa
 namespace: default
 annotations:
    iam.gke.io/gcp-service-account: << Google_Service_Account >>
---
apiVersion: v1
kind: Pod
metadata:
  name: gcs-fuse-csi-example-static-pvc
  namespace: default
  annotations:
    gke-gcsfuse/volumes: "true"
spec:
  containers:
  - image: busybox
    name: server
    command: ["sleep"]
    args: ["infinity"]
    volumeMounts:
    - name: gcs-fuse-csi-static
      mountPath: /tmp
  serviceAccountName: tomas-sa
  volumes:
  - name: gcs-fuse-csi-static
    persistentVolumeClaim:
      claimName: gcs-fuse-csi-static-pvc
      readOnly: false

Cloud Storage Fuse CSI driver is enabled on my cluster and I've also checked that the bucket has full permissions under my service account

gcloud storage buckets get-iam-policy gs://test-bucket-cqlr 

bindings:
- members:
  - serviceAccount:<< Google_Service_Account >>
  role: roles/storage.admin

Environment

GKE version: 1.30.5-gke.1443001

[hime]

We have updated our documentation. Please make sure you follow the guide below, and make sure you have workload identity enabled on all of the GKE Nodes.

Guide: https://cloud.google.com/kubernetes-engine/docs/how-to/cloud-storage-fuse-csi-driver-setup#authentication