GithubHelp home page GithubHelp logo

shrine-url's Introduction

Shrine::Storage::Url

Provides a "storage" for Shrine for attaching uploaded files defined by a custom URL.

Installation

gem "shrine-url", "~> 2.4"

Usage

require "shrine/storage/url"

Shrine.storages[:cache] = Shrine::Storage::Url.new

The custom URL should be assigned to the id field in the Shrine uploaded file JSON representation:

{
  "id": "http://example.com/image.jpg",
  "storage": "cache",
  "metadata": {
    # ...
  }
}

Now you can assign this data as the cached attachment:

photo = Photo.new(image: data)
photo.image #=> #<Shrine::UploadedFile>
photo.image.url           #=> "http://example.com/image.jpg"

No HTTP requests are made when file is assigned (but you can load the restore_cached_data Shrine plugin if you want metadata to be extracted on assignment). When this "cached file" is about to be uploaded to a permanent storage, shrine-url will download the file from the given URL using Down.

uploaded_file.download      # Sends a GET request and streams body to Tempfile
uploaded_file.open { |io| } # Sends a GET request and yields `Down::ChunkedIO` ready for reading
uploaded_file.exists?       # Sends a HEAD request and returns true if response status is 2xx
uploaded_file.delete        # Sends a DELETE request if :delete is set to true

By default the Down::Http backend will be used for downloading, which is implemented using HTTP.rb. You can change the Down backend via the :downloader option:

Shrine::Storage::Url.new(downloader: :wget)
# or
require "down/http"
Shrine::Storage::Url.new(downloader: Down::Http)
# or
require "down/net_http"
Shrine::Storage::Url.new(downloader: Down::NetHttp.new("User-Agent" => "MyApp/1.0.0"))

Note that if you're using permanent storage that supports uploading from a remote URL (like shrine-cloudinary or shrine-uploadcare), downloading will be completely skipped as the permanent storage will use only the URL for uploading the file.

Deleting

Calling Shrine::UploadedFile#delete will call Shrine::Storage::Url#delete, which for safety doesn't do anything by default. If you want it to make a DELETE request to the URL, you can set :delete to true on initialization:

Shrine::Storage::Url.new(delete: true)

Advantages and Use Cases

The main advantage of using shrine-url over the remote_url Shrine plugin is that you can put downloading from the URL into a background job by loading the backgrounding Shrine plugin. Another advantage is that you can assign multiple remote URLs as multiple versions.

This storage can be used with shrine-transloadit for direct uploads, where a temporary URL of the uploaded file is returned, and we want to use that URL for further background processing, eventually replacing the attachment with processed files.

It is also used in shrine-tus-demo, where the files are uploaded to a separate endpoint, and then its file URL is attached to a database record and promoted to permanent storage.

Contributing

$ rake test

The test suite pulls and runs kennethreitz/httpbin as a Docker container, so you'll need to have Docker installed and running.

License

MIT

shrine-url's People

Contributors

janko avatar jrochkind avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

Forkers

bkdruby

shrine-url's Issues

Support for redirects

Hey @janko-m ,
Thanks for this gem - it's super useful but I have a one question - is it possible to support redirects? It would especially useful for links from dropbox like: https://www.dropbox.com/s/992urij4brdzckd/test_pdf.pdf?dl=1

Unfortunately those links redirect to single use URL so basically every call to exists?, download, etc. would require to generate a new link by following its redirect. I think it needs to remember how many redirect it should perform for download - otherwise it can invalidate the link by checking the response code, i.e.:

https://www.dropbox.com/s/992urij4brdzckd/test_pdf.pdf?dl=1 -> redirects to A

if you made a request to A (i.e. to check for additional redirect), you invalidate the A link so you can not use it to download the file.

breaks webmock?

I like using WebMock in my tests.

When trying to write my own test against a shrine uploader using shrine-url, and using WebMock to mock out the HTTP request that shrine-url would be doing...

I get a mysterious error. I haven't yet been able to isolate this error into something that only uses down and not shrine-url, is why I'm reporting it here. It's really quite mysterious and I'm not sure what's going on, when I try to isolate it into a simple reproduction case, i end up not being able to reproduce.

Here's the error/stack trace as reported by my rspec, in case it gives you any ideas

     Failure/Error: response.connection.close unless @client.persistent?

     NoMethodError:
       undefined method `close' for #<HTTP::Response::Streamer:0x00007ff14aa89380>
       Did you mean?  clone
     # /Users/jrochkind/.gem/ruby/2.5.3/gems/down-4.7.0/lib/down/http.rb:109:in `stream_body'
     # /Users/jrochkind/.gem/ruby/2.5.3/gems/down-4.7.0/lib/down/chunked_io.rb:282:in `each'
     # /Users/jrochkind/.gem/ruby/2.5.3/gems/down-4.7.0/lib/down/chunked_io.rb:282:in `block in chunks_fiber'

(Does this also suggest it's using http_rb instead of net_http? Ah, I see, even though down says net-http is the default, shrine-url makes http_rb it's default. Any particular rationale?)

breaks on long filenames

i'm trying to use shrine-url with a signed S3 URL. Which is a pretty convenient usecase, importing from S3 (And I def want to do it in backgrounding, thus shrine-url).

However, it fails on too-long URLs, and signed S3 URLs are quite long (with query params).

The stack trace doesn't even involve shrine-url, I think it's maybe failing before it even gets to calling shrine url. Based on trying to make a filename based on id, which is the long URL.

Error performing Kithe::AssetPromoteJob (Job ID: 0979ac75-3611-4e99-a63a-be6f5db54957) from Async(default) in 5.67ms: Errno::ENAMETOOLONG (File name too long @ rb_sysopen - /Users/jrochkind/code/scihist_digicoll/tmp/shrine_storage_testing/store/cc65f8b0ebe79f859195410a8a71b025.tif?x-amz-algorithm=aws4-hmac-sha256&x-amz-credential=akiajorcl6txivhsdrhq%2f20181119%2fus-east-1%2fs3%2faws4_request&x-amz-date=20181119t182037z&x-amz-expires=14400&x-amz-signedheaders=host&x-amz-signature=fakefake45db944222adfadfadfadf7d20eacce3e6d062979f993db0ffa1e6e):
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine/storage/file_system.rb:129:in `initialize'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine/storage/file_system.rb:129:in `copy_stream'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine/storage/file_system.rb:129:in `upload'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:348:in `copy'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:337:in `put'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:321:in `_store'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:248:in `store'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine/plugins/rack_file.rb:72:in `store'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:226:in `upload'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine/plugins/rack_file.rb:66:in `upload'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:658:in `store!'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:588:in `promote'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine/plugins/backgrounding.rb:152:in `promote'

I would try to add a failing test... but I notice now shrine-url doesn't even have any tests, so not sure where to add it. Also no idea how to fix this, since it happens before it even gets to shrine-url. I guess it would have to be fixed in shrine itself, to not try to make local files based on id... but it's not expecting id to be super-long like this, the way shrine-url uses it. I wonder if the whole design of shrine-url should be rethought?

Or wait, maybe it looks like the part of shrine that's extracting the filename suffix (to preserve it as it gets promoted), needs to be smart enough to ignore query parameter at the end, in order to be compatible with shrine-url? That might be it?

If you give me feedback and advice, I can try to send some PRs!

Lack of support for ssl

When url to the remote file is specified by https protocol then EOFError: end of file reached is raised from the Net::HTTP

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.