Store Your Files on S3 Using the Ruby Shrine Gem

iRonin IT Team•2017-12-15

In this article, experts from a top Ruby on Rails development company show you how to directly upload files from your Ruby web app to Amazon S3 storage using the Shrine gem. It covers all the code and configuration files you need to upload efficiently as a background process.

Clever uploading to Amazon S3 from Ruby requires using a specific Ruby gem - Shrine.

In Part 1 - Setup & Configuration of our Store Your Files on S3 Using the Ruby Shrine Gem series, we installed and configured the gem in preparation for uploads.
In today's Part 2 - Direct File Uploads, we detail how to upload files directly from your web app, first to temp storage and then to a persistent location.

Later, follow on for Part 3, Uploading Files from a Remote URL.

Direct file upload using a presigned URL

Our most desirable type of file uploading involves directly uploading files to AWS S3 from the frontend (FE) application (based on the URL provided - presigned - by the API), with the intermediate backend (BE) application taking care of finding an upload location on S3 and transferring the file from temporary to persistent storage.

A pre-signed URL is a location where the front-end app should upload the file; the back-end app provides the URL and contains the necessary authentication info so that the FE can upload the file directly to a temporary bucket without the back-end being an intermediary.

The workflow for a direct file upload should look like this:

1. The FE application sends a request to the BE for a presign endpoint (provided by Shrine), which will be used to upload the file directly from the FE application to the S3.

2. The BE application responds with JSON, which contains the data required to authenticate with S3 and upload a file.

3. The FE application uploads the file using the data obtained from the previous step.

4. FE generates JSON with the file cached on the S3 and sends it to the BE application.

5. BE application either creates or updates a record in the database and moves files between "cache" storage to "store" on S3.

Configuration for our Ruby application uploading files to Amazon S3 directly

To complete this workflow, Shrine comes with the `presign_endpoint` plugin. This plugin should be enabled in your `config/initializers/shrine.rb` by adding the following line of code:

# config/initializers/shrine.rb
# ...
Shrine.plugin :presign_endpoint

The `presign_endpoint` plugin provides a Rack endpoint that generates presigned JSON data. Let's mount it in the routes:

# config/routes.rb
Rails.application.routes.draw do
  mount Shrine.presign_endpoint(:cache) => "/presign"
end

The above will make a `GET /presign` endpoint available, which will return, in response, JSON as below:

{
  "url": "https://bucket-name.s3.eu-central-1.amazonaws.com",
  "fields": {
    "key": "cache/2084783353bf6e5fdfe3420b8bff3a8c",
    "policy": "eyJleHBpcmF0aW9uIjoiMjAxNy0xMC0xOVQwODoxMDowMFoiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJpcm9uaW4tdG1zLXVwbG9hZP3hjRBnaW5nIn0seyJrZXkiOiJjYWNoZS8yMDg0NzgzMzUzYmY2ZTVmZGZlMzQyMGI4YmZmM2E4YyJ9LHsieC1hbXotY3JlZGVudGlhbCI6IkFLSUFKWk5BTFlBUUJBRE8zTzVBLzIwMTcxMDE5L2V1LWNlbnRyYWwtMS9zMy9hd3M0X3JlcXVlc3QifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1VZy1TSEEyNTYifSx7IngtYW26LWRhdGUiOiIyMDE3MTAxOVQwNzEwMDBaIn1dfQ==",
    "x-amz-credential": "AKIAJZNAKLVM1ADO124A/20171019/eu-central-1/s3/aws4_request",
    "x-amz-algorithm": "AWS4-HMAC-SHA256",
    "x-amz-date": "20171019T071000Z",
    "x-amz-signature": "19c31571fb1031d64bed4c31eecc2bf5fe5737855z4378a640326d6061114714"
  },
  "headers": {}
}

The above data allows us to upload a file directly to S3. However, there are a few problems with this.

Configuration for sending directly to development and testing storage

So, how can we create a mechanism to upload a file to the development environment, consistent with sending directly to S3 for staging and production? The solution for that is to emulate the presign data, where the URL will point to our backend server.

To do that, two things are needed:

1. Enable the `upload_endpoint` plugin

2. Create a `presign` endpoint for getting the upload URL

We should enable the upload_endpoint plugin conditionally in our uploader so that it will be available only for the specific uploader instead of globally for all of them. It should be active only in development and testing. Here's the additional code to do just that:

# app/uploaders/attachment.rb
class AttachmentUploader < Shrine
  plugin :determine_mime_type
  plugin :upload_endpoint if Rails.env.development? || Rails.env.test?
end

The upload endpoint is similar to the presign endpoint in that it provides a Rack endpoint. It accepts file uploads and forwards them to specified storage, so specific routing needs to be mounted in the routes configuration.

We also create a custom `/presign` endpoint for `development` and `test` environments using `Shrine.presign_endpoint` which simply returns the URL of the upload endpoint.

# config/routes.rb
Rails.application.routes.draw do
  if Rails.env.development? || Rails.env.test?
    require 'ostruct'
    presign_endpoint = Shrine.presign_endpoint(:cache, lambda do |id, _opts, req|
      OpenStruct.new(url: "#{req.base_url}/attachments", key: "cache/#{id}")
    end)
    mount presign_endpoint => '/presign'
    mount AttachmentUploader.upload_endpoint(:cache) => '/attachments'
  else
    mount Shrine.presign_endpoint(:cache) => '/presign'
  end
end

The `POST` `attachments` endpoint is now available only for development and test purposes.

Directly uploaded files are stored in a temporary location. When we assign the cached file to a database record, Shrine will copy it to a permanent location.

file_data = {
  id: '2084783353bf6e5fdfe3420b8bff3a8c',
  storage: 'cache',
  metadata: {
    filename: 'example.pdf',
    size: 1024,
    mime_type: 'application/pdf'
  }
}.to_json
Attachment.create(file: file_data)

Background file processing

Shrine, by default, copies files from temporary to permanent storage in the foreground - but it also comes with the `backgrounding` plugin, which moves this process to the background. To use this functionality, just enable the `backgrounding` plugin, create background workers, and then show Shrine which workers to use.

Of course, a `backgrounding` library needs to be configured before all this. For our purposes, we use Sidekiq as an adapter for ActiveJob.

# config/application.rb
# ...
class Application < Rails::Application
  config.active_job.queue_adapter = :sidekiq
  # ...
end
# config/initializers/shrine.rb
# ...
Shrine.plugin :backgrounding
# app/jobs/promote_worker.rb
class PromoteWorker < ActiveJob::Base
  def perform(data)
    Shrine::Attacher.promote(data)
  end
end
# app/jobs/delete_worker.rb
class DeleteWorker < ActiveJob::Base
  def perform(data)
    Shrine::Attacher.delete(data)
  end
end
# app/uploaders/attachment.rb
class AttachmentUploader < Shrine
  # ...
  Attacher.promote { |data| PromoteWorker.perform_later(data) }
  Attacher.delete { |data| DeleteWorker.perform_later(data) }
end

And that's it! Now copying files between the cache and store, and deleting previously stored files will be queued and completed in the background.

Read on with Part 3, Uploading Files from a Remote URL to learn how to upload files to S3 from a remote URL, in the final installment of our Amazon S3 integration with Ruby series.

At iRonin, we strive to always configure the most efficient data flows and systems for our clients. If you would like us to take a look over your current project to see where we can make it more efficient, or have a new project you'd like developed, then make sure to get in touch. Whether it's Ruby development using Amazon S3, web application development using Amazon S3, or something else entirely, we'd love to hear from you!

Let’s get in touch

Read Similar Articles

Ruby on Rails vs Django in-depth comparison. Picking the Best Web Development Framework for 2025

2018 at iRonin - reflecting on an exciting year

Smart Contracts on Bitcoin

Using Node.js in Enterprise Environments