Upload/Download files from a browser with GCS Signed URLs and Signed Policy Documents
Small javascript application showing how to upload/download files with GCS Signed URLs and Signed Policy Documents. This article will not cover in detail what those two mechanisms are but rather demonstrate a basic application that exercises both on the browser. This is a simple client-server app that uploads files using these two mechanisms from a user's browser. SignedURLs w/ javascript has been done many times before (see references); this article describes SignedURLs and Policy document differences and implementations.
Briefly, SignedURLs and Policy Document based operations are similar: a URL with a signature that an unauthenticated user can perform certain GCS operations. There are a couple of differences to note:
- GCS SignedURLs do not support multiple file uploads with one URL.
- A single Policy document can be used for multiple uploads and also define a prefix/path for all uploads.
- Policy document currently support uploads only. GCS Signed URLS can perform both uploads and downloads.
- Policy documents offer several additional conditions and constraints over SignedURLs (see usage and examples section of the Policy Document)
The code snippet provided here is Flask application that provides signedURLS and Policy documents to a javascript browser. Snippet also shows how to configure CORS access for SignedURLs
Note: this is just a basic sample with some awkward javascript, nothing more. The sample here allows you to generate URLs to upload+download any file in the bucket by specifying its name.
Setup
Create Service Account
PROJECT_NAME=$(gcloud config list --format="value(core.project)")
$ gcloud iam service-accounts create urlsigner --display-name="GCS URL Signer" --project=${PROJECT_NAME}
$ gcloud iam service-accounts keys create service_account.json --iam-account=urlsigner@${PROJECT_NAME}.iam.gserviceaccount.com
Create GCS Bucket
gsutil mb gs://$PROJECT_NAME-urlsigner
Assign IAM permissions on bucket to Service Account
gsutil iam ch serviceAccount:urlsigner@${PROJECT_NAME}.iam.gserviceaccount.com:roles/storage.admin gs://$PROJECT_NAME-urlsigner
GCS SignedURL
Set /etc/hosts override
Add to /etc/hosts
.
127.0.0.1 gcs.somedomain.com
Set CORS Policy
Set CORS policy to allow request from a test domain:
-
https://gcs.somedomain.com:8081
-
define
cors.txt
:
[
{
"origin": ["https://gcs.somedomain.com:8081"],
"responseHeader": ["Content-Type", "Authorization", "Content-Length", "User-Agent", "x-goog-resumable"],
"method": ["GET", "POST", "PUT", "DELETE", "OPTIONS"],
"maxAgeSeconds": 3600
}
]
- set cors policy on the bucket
gsutil cors set cors.txt gs://$PROJECT_NAME-urlsigner
- Verify:
curl -v -X OPTIONS -H "Host: storage.googleapis.com" -H "Access-Control-Request-Method: PUT" \
-H "Origin: https://gcs.somedomain.com:8081" \
"https://storage.googleapis.com/$PROJECT_NAME-urlsigner/cors.txt"
< HTTP/2 200
< access-control-allow-origin: https://gcs.somedomain.com:8081
< access-control-max-age: 3600
< access-control-allow-methods: GET,POST,PUT,DELETE,OPTIONS
< access-control-allow-headers: Content-Type,Authorization,Content-Length,User-Agent,x-goog-resumable
< vary: Origin
Set bucket name to sign for
- Edit
main.py
, set
bucketName = '$PROJECT_NAME-urlsigner'
(ofcourse use the value, not literal)
Start Server
virtualenv env
source env/bin/activate
pip install -r requirements.txt
python main.py
Access UI to upload/download:
https://gcs.somedomain.com:8081/signedurl
(you may want to enable Developer Mode
in Chrome; that will show you the CORS requests too)
- Upload
-
Select the
upload
radio button. -
Define a filename to generate the URL for (default:
README.md
) -
Click
Generate Signed URL
. This will make a request to/getSignedURL
endpoint which inturn returns a signedURL for the filename specified. -
Click "Choose File" to select a file to upload.
-
Select "Upload SignedURL"
-
Verify
$ gsutil ls gs://$PROJECT_NAME-urlsigner/
gs://$PROJECT_NAME-urlsigner/README.md
- Download
-
Select the
download
radio button. -
Click
Generate Signed URL
. This will make a request to/getSignedURL
endpoint which inturn returns a signedURL for the filename specified. -
Click the link provided on the browser. The link will download the file requested
Using gsutil
You can also genereate upload/download urls with gsutil or any google-cloud-storage
library
gsutil signurl -m GET service_account.json gs://$PROJECT_NAME-urlsigner/README.md
gsutil signurl -m PUT service_account.json gs://$PROJECT_NAME-urlsigner/README.md
Policy Document
Policy Document allows for uploads only but also provides several configurations and conditions canonical SignedURLs do not: you can define a prefix/path with which a user can upload multiple files with one URL. In the example below, one URL can upload N files to the bucket and each file must have the prefix /myfolder/
and be of Content-Type: text/plain
- Upload
-
Access
https://gcs.somedomain.com:8081/policydocument
-
Define a filename to generate the URL for (default:
README.md
) -
Click
Generate Policy Document Signed URL
. This will make a request to/getSignedPolicyDocument
endpoint which inturn returns a signed policy document for the filename specified. The code snippet on the server which generates this is from thegoogle-cloud-storage
python library bucket.generate_upload_policy() -
Click "Choose File" to select a file to upload.
-
Select "Upload SignedURL"
-
Verify
$ gsutil ls gs://$PROJECT_NAME-urlsigner/myfolder/
gs://$PROJECT_NAME-urlsigner/myfolder/README.md
Policy documents can also get embedded into a form as described in the links above and in the snippet below
As an HTML form, you can post the same generated signature, file and policy document. For example:
<html>
<form action="http://storage.googleapis.com/$PROJECT_NAME-urlsigner" method="post" enctype="multipart/form-data">
<input type="hidden" name="key" value="myfolder/myfile.txt">
<input type="hidden" name="bucket" value="$PROJECT_NAME-urlsigner">
<input type="hidden" name="Content-Type" value="text/plain">
<input type="hidden" name="GoogleAccessId" value="urlsigner@${PROJECT_NAME}.iam.gserviceaccount.com">
<input type="hidden" name="acl" value="bucket-owner-read">
<input type="hidden" name="policy" value="b64encodedPolicy">
<input type="hidden" name="signature" value="b64encodedSignature">
<input name="file" type="file">
<input type="submit" value="Upload">
</form>
</html>
References
-
How to SignURL on GCE|GKE|anywhere without a key (local key, that is)
-
Javascript with GCS SignedURLs: