Download a file securely from GCS on an untrusted system

The Problem

We publish some of our build artifacts to Google Cloud Storage, and users need to download these to the target installation system. But, this target system is not always trusted and can have shared local users, so we don't want to store long-lived credentials.

As a user, I can download the artifact on my (secure) laptop and transfer it to the target system. But, the artifact can be large (several GBs). So, downloading and uploading again makes it cumbersome and slow.

Option 1: use gcloud CLI on the target system

Log in to the target system, install gcloud CLI, authenticate, and then download the file:

gcloud storage cp gs://$BUCKET/$FILE ./

This has two problems:

1. The user must install (and maybe update) gcloud CLI on the target system.
2. The user needs to store their credentials on the target system. These credentials have full access to whatever resources the user has. So, it's a huge security risk, especially if we don't trust the target system.

To mitigate (2), the user can log out of gcloud CLI after downloading. But, this is a manual step they might miss.

Option 2: use gcloud CLI with a service account

This is a variation of the above solution - we log in using a service account instead of the user account. This service account can have restricted access to only the resources needed.

gcloud iam service-accounts create $SA_NAME \
    --description="Service Account for downloading artifacts"
gsutil iam ch \
    serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com:roles/storage.objectViewer \
    gs://$BUCKET

This partially mitigates problem (2) above. If the user forgets to log out of gcloud CLI, the damage will be restricted to the resources accessible by the service account.

Option 3: Short-lived access token

Gcloud CLI supports creating short-lived credentials for the end-user account or any service account.

This credential can be used to download the artifact using wget with an authorization header - no need to install gcloud CLI.

Here's a small script that asks for the auth token as input, parses various GCS bucket URL formats, and downloads the requested artifact directly using wget:

#!/bin/bash
# Download artifact from GCS bucket

set -e

echo -e "====> Run \`gcloud auth print-access-token\` on a system where you've setup gcloud to get access token\n"
read -r -p "Enter access token: " StorageAccessToken
read -r -p "Enter GCS artifact URL: " ArtifactURL

if [[ "${ArtifactURL:0:33}" == "https://console.cloud.google.com/" ]]; then
    BucketAndFile="${ArtifactURL#*https://console.cloud.google.com/storage/browser/_details/}"
elif [[ "${ArtifactURL:0:33}" == "https://storage.cloud.google.com/" ]]; then
    BucketAndFile="${ArtifactURL#*https://storage.cloud.google.com/}"
elif [[ "${ArtifactURL:0:5}" == "gs://" ]]; then
    BucketAndFile="${ArtifactURL#*gs://}"
else
    echo "Invalid GCS artifact URL"
    exit 1
fi

StorageBucket="${BucketAndFile%%/*}"
StorageFile="${BucketAndFile#*/}"
StorageFileEscaped=$(echo "${StorageFile}" | sed 's/\//%2F/g')
OutputFileName="${StorageFile##*/}"

echo -e "\n====> Downloading gs://${StorageBucket}/${StorageFile} to ${OutputFileName}\n"

wget -O "${OutputFileName}" --header="Authorization: Bearer ${StorageAccessToken}" \
    "https://storage.googleapis.com/storage/v1/b/${StorageBucket}/o/${StorageFileEscaped}?alt=media"

Option 4: Signed URLs

Google Cloud Storage also supports signed URLs - which give time-limited access to a specific Cloud Storage resource. Anyone possessing the signed URL can use it while it's active without any further credentials. This fits our use case brilliantly.

To do this, first we need to give ourselves the iam.serviceAccountTokenCreator role so that we can impersonate a service account.

gcloud iam service-accounts add-iam-policy-binding \
	$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com \
    --member=$MY_EMAIL \
    --role=roles/iam.serviceAccountTokenCreator

Then, we can generate a signed URL:

gcloud config set auth/impersonate_service_account \
    $SA_NAME@$PROJECT_ID.iam.gserviceaccount.com

gsutil signurl -u -r $REGION -d 10m gs://$BUCKET/$FILE

gcloud config unset auth/impersonate_service_account

And we can use wget to download the artifact from this URL without any further authentication.