areviana / direct-upload-example

This repository contains a tutorial to make direct upload to S3 with an Ruby on Rails API

Ruby on Rails Direct Upload example

In this example we learn how to implement Direct uploads with AWS S3 in our API project.

Things you may want to cover to run this project:

• Ruby version - 2.7.0

• Database - postgresql

• Database initialization

  Set up database:

  rails db:create
  rails db:migrate

---

Preview

Before to start you should cover those basic concepts

Direct Upload

This term is usually used in conjunction with cloud storage services (e.g., Amazon S3) and means the following: instead of uploading a file using the API server, the client uploads it directly to the cloud storage using credentials generated by the API server

First thing's first!

Configure the amazon stuffs

1. Sign in to your AWS account
2. Go to Service S3 Console
3. Choose one Region according to your preferences
4. Create a Bucket
5. Set bucket permissions, create a policy, IAM User

For more help, you can take this Cloud Academy - Free Course!

Make from scratch

API Project set up

# Create new api rails project
rails new direct-upload-example --api --database=postgresql

# Add Active Storage
rails active_storage:install

# Create models
## User model
rails generate model user full_name:string email:string:uniq

# Run migrations
rails db:migrate

# Create Direct Upload Controller
rails generate controller api/direct_upload

# Create User Controler
rails generate controller api/users

Gems added

gem 'dotenv-rails'
gem 'aws-sdk-s3', require: false

# Run
bundle install

Amazon S3 configuration

Add in storage.yml

amazon:
  service: S3
  access_key_id: <%= ENV['S3_KEY_ID'] %>
  secret_access_key: <%= ENV['S3_SECRET_KEY'] %>
  region: <%= ENV['S3_REGION'] %>
  bucket: <%= ENV['S3_BUCKET'] %>

NOTE: If you want to use environment variables, standard SDK configuration files, profiles, IAM instance profiles or task roles, you can omit the access_key_id, secret_access_key, and region keys in the example above. -- Reference here!

Or create an AWS initializer in /config/initializers/aws.rb

require 'aws-sdk-s3'

Aws.config.update(
  {
    region: ENV.fetch('S3_REGION'),
    credentials: Aws::Credentials.new(
      ENV.fetch('S3_KEY_ID'),
      ENV.fetch('S3_SECRET_KEY')
    )
  }
)

# Extra(optional): Create a Bucket global instance
S3_BUCKET = Aws::S3::Resource.new.bucket(ENV.fetch('S3_BUCKET'))

Change configuration in development.rb or production.rb

  config.active_storage.service = :amazon

Model + Active Storage

Add attach references in your models

class User < ApplicationRecord
  has_one_attached :avatar
end

Signed URL

Create new service to handle signed url. To do this feature you have different options:

Use Active Storage methods

The easiest way!

create_before_direct_upload!
service_url_for_direct_upload
service_headers_for_direct_upload

Rails documentation: Blob methods

Use SDK methods

Advantages:

• Not need active storage
• With POST request you can keep the original filename

S3_BUCKET.presigned_post
S3_OBJECT.presigned_url

SDK documentation:

• presigned_post
• presigned_url
• More options - Presigner class

See the examples in /app/services/signed_url.rb

Request

Endpoint

• Method: POST
• Path: /api/presigned_url
• Headers: Content-Type = application/json

Example Body request

The "url" node is optional!

{
  "file": {
    "byte_size": 78324,
    "checksum": "RfczTs94io0MRZlXjzEm/w==",
    "filename": "custon_name",
    "content_type": "image/png",
	"metadata": {
		"message": "active_storage_test"
	}
  },	   
  "url": {
    "expiration_time": "300",
    "folder": "development/users"
  }
}

Example JSON response

{
  "direct_upload": {
    "url": "https://your-bucket.s3.amazonaws.com/development/users/blob-key?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=S3_KEY_ID%2F20200429%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200429T025334Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost&X-Amz-Signature=CUSTOM_SIGNATURE",
    "headers": {
      "Content-Type": "image/png",
      "Content-MD5": "RfczTs94io0MRZlXjzEm/w=="
    }
  },
  "blob_signed_id": "eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBDZz09IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--4397fc4a06ba581d521819faaafd4c230a073a7e"
}

Final part!

Add the uploaded file to your model

The JSON response above provide you the blob_signed_id parameter, this ID must be sent as the reference of your uploaded file in the endpoint where you create or update your model.

IMPORTANT NOTE: Make sure to upload the file on your bucket first, because Active Storage has some problems if you don't respect this flow.

Request

Endpoint

• Method: POST
• Path: /api/users
• Headers: Content-Type = application/json

Example Body request

{
  "user": {
    "full_name": "Arely Viana",
    "email": "[email protected]",
    "linkedin": "linkedin.com/in/areviana",
    "avatar": "eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBDZz09IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--4397fc4a06ba581d521819faaafd4c230a073a7e"
   }
}

Example JSON response

{
  "id": 1,
  "full_name": "Arely Viana",
  "linkedin": "linkedin.com/in/areviana",
  "email": "[email protected]",
  "created_at": "2020-04-29T03:53:41.653Z",
  "avatar_url": "https://your-bucket.s3.amazonaws.com/development/users/jhvbrzt2m81ukpsvtsaaqzpnb5i6"
}

An that's it! The integration of direct upload and Active Storage is complete!

direct-upload-example's Issues

Cannot find signed_url.rb file in app/services.

You have mentioned that for SDK methods like presigned_post, Active Storage is not required. So can we remove the active_storage_attachments and blobs tables and create a custom table to store the url to the files that is returned from AWS s3 upload. Need more clarity on this please.