Carrierwave storage adapter that supports Google Cloud Storage and is based on gcloud library from Google

License: MIT License

Ruby 99.39% Shell 0.61%

carrierwave-google-storage's People

Contributors

Stargazers

Watchers

carrierwave-google-storage's Issues

How to Upload certain files as public and other files as private in same bucket.

@jasdeepsingh ,
If there is no solution for this as stated by you in #5 ,

Then can you please provide more details how the initializer would be to achieve this?

How can we initialize two buckets in CarrierWave?

Ruby 3 support

I got the following issue after upgrading to Ruby 3:

ArgumentError (wrong number of arguments (given 4, expected 2..3))

google-cloud-storage only added support for Ruby 3 in 1.31.0. In the gemspec ~> 1.18 is defined, but in the last release it's still ~> 1.18.2 and I cannot update.

Could you release the current master and possibly bump google-cloud-storage?

Problem using hash version of keyfile

google-cloud-core supports having the keyfile (credentials) passed as a hash (https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-core/lib/google/cloud.rb#L45).

There is a slight problem caused by this library and ruby, because how the object is created (https://github.com/metaware/carrierwave-google-storage/blob/master/lib/carrierwave/storage/gcloud.rb#L42), specifically when the second argument of the ::Google::Cloud.new is a hash ruby tries to map it onto keyword arguments which causes an ArgumentError: unknown keywords: type, project_id, private_key_id,.... Explicitly adding a third argument to the new call (like an empty hash) solves the problem.

I can cook up a small PR to add an empty hash (or even support the 2 options Google::Cloud.new currently has, e.g. retries and timeout) if you agree with the above.

How to implement with CarrierWave Uploader

First of my apologies if im being stupid on this, but I am struggling to see how to integrate this into my application. I have previously been integrating with google storage via carrierwave and fog but I am getting performance issues so I thought I would try this instead.

I have gone through all the steps and have carrierwave configured to use the cloud settings but what do I need to do with my CarrierWave Uploader? Previously I was using :fog as the storage setting but obviously that is no longer the case. Do I need to use a particular value to get this to work? I have tried guessing a few options but with no joy.

Many thanks in advance

All the best

Security issue

Github dependant bot is raising:
Dependabot cannot update carrierwave to a non-vulnerable version
The latest possible version that can be installed is 1.2.3 because of the following conflicting dependency:

carrierwave-google-storage (0.9.0) requires carrierwave (~> 1.2.0)
The earliest fixed version is 1.3.2.

Handle buckets with retention policy

The bucket that I want to upload the file to has a retention policy, ie. files in it cannot be deleted.

After the file is copied to the target folder, the module attempts to delete the tmp folder that was created - which returns a 403 Object 'xxxx/uploads/tmp/1666619955-603810510513191-0001-1813/xxxxxx.jpg' is subject to bucket's retention policy and cannot be deleted, overwritten or archived until 2027-10-24T13:01:49.373695-07:00

Is there something we can do to avoid using the bucket for the tmp directory - such as copy the file directly to the target folder?

(I noticed in the store method there is a flag delete_tmp_file_after_storage that presumably could be used as a workaround to prevent attempting to delete tmp file, but that isn't ideal as we would then be storing doubles for each file ie double the storage cost etc)

Thanks

Gems:

carrierwave 2.2.2
carrierwave-google-storage-1.0.0

'mount_uploaders' not supported.

Your gem depended on Carierwave ~> 0.11.2 which not support multiple file uploading, so 'mount_uploaders' on model raise an error - Undefined method.

When calling `remove_avatar!` I get delete for nil:nil class method error not found.

#<Hurley::Response GET https://www.googleapis.com/storage/v1/b/my-testing-documents/o/filestore%2Freport%2FCuRv9xloZS8kiUbVW_ibLw. == 404 (165 bytes) 122ms>
Caught error notFound: Not Found
Error - #<Google::Apis::ClientError: notFound: Not Found>

   (0.4ms)  ROLLBACK
NoMethodError: undefined method `delete' for nil:NilClass
from /home/raman/.rvm/gems/ruby-2.3.1/gems/carrierwave-google-storage-0.7.0/lib/carrierwave/storage/gcloud_file.rb:46:in `delete'

The file was successfully deleted from the google cloud bucket but this error is thrown.
Ruby version: 2.3.1
Rails version 5.1.5

Thanks for working on this gem, I prefer the use of service accounts to access my cloud storage bucket.

Content-Type is not being set correctly on gcloud storage

ConfigurationError naming conflict

need

as noted, carrierwave-google-storage is heavily based on carrierwave-aws (thanks!)
but I need both carrierware-aws and carrierwave-google-storage to allow uploads to either depending on an app setting

This causes a warning because of the duplicate extending/monkeypatching of carrierwave in both gems... I assume renaming ConfigurationError would help. Thanks!

ConfigurationError = Class.new(StandardError)

warning:

/home/vagrant/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/carrierwave-aws-1.5.0/lib/carrierwave-aws.rb:22: warning: already initialized constant CarrierWave::Uploader::Base::ConfigurationError

/home/vagrant/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/bundler/gems/carrierwave-google-storage-9e225cafad8a/lib/carrierwave-google-storage.rb:13: warning: previous definition of ConfigurationError was here

Only depend on google-cloud-storage gem

We are now including the complete google cloud gem with everything.

In light of killing dependencies could this gem only depend on the actual storage gem?
https://github.com/GoogleCloudPlatform/google-cloud-ruby#storage

gem install google-cloud-storage

Different versions not saved in google storage

Hi, thanks for the great gem!

I am having an issue, though, with saving thumbnails of my files in storage. It all works well in development, where I use file system but when I try to save page thumbnails through gcloud, it does not seem to be working.

Here's my uploader code:

class PageUploader < CarrierWave::Uploader::Base
  # Include RMagick or MiniMagick support:
  # include CarrierWave::RMagick
  include CarrierWave::MiniMagick

  def store_dir
    if Rails.env.production?
      "uploads/pages/#{model.parent_file_id}"
    elsif Rails.env.demo?
      "demo/uploads/pages/#{model.parent_file_id}"
    elsif Rails.env.test?
      "tmp/uploads/pages/#{model.parent_file_id}"
    else
      "development/uploads/pages/#{model.parent_file_id}"
    end
  end

  def extension_white_list
    ['pdf']
  end

  version :thumb do
    begin
      unless Rails.env.test?
        process convert: 'jpg' 
        process resize_to_fit: [595, 842]
        process :set_content_type
      end
      def full_filename(for_file = model.attachment.file)
        "#{model.id}-thumb.jpg"
      end
    rescue Exception => e
      puts e.inspect
      true
    end
  end

  def set_content_type(*args)
    self.file.instance_variable_set(:@content_type, 'image/jepg')
  end

  def thumb_url
    thumb.try(:authenticated_url) || thumb.url
  end
end

Any help would be appreciated :)

CarrierWave::ProcessingError unable to open image `/tmp/magick-6No4015N4HWij': No such file or directory @ error/blob.c/OpenBlob/2712

I'm suffering from this error. This occurs sometimes, not always. What is the cause of this?

Magick::ImageMagickError: unable to open image `/tmp/magick-6No4015N4HWij': No such file or directory @ error/blob.c/OpenBlob/2712
  from carrierwave/processing/rmagick.rb:352:in `read'
  from carrierwave/processing/rmagick.rb:352:in `manipulate!'
  from carrierwave/processing/rmagick.rb:202:in `resize_to_fill'
  from carrierwave/uploader/processing.rb:83:in `block (2 levels) in process!'
  from carrierwave/uploader/processing.rb:75:in `each'
  from carrierwave/uploader/processing.rb:75:in `block in process!'
  from carrierwave/uploader/callbacks.rb:15:in `with_callbacks'
  from carrierwave/uploader/processing.rb:74:in `process!'
  from backgrounder/delay.rb:14:in `process!'
  from carrierwave/uploader/callbacks.rb:14:in `block in with_callbacks'
  from carrierwave/uploader/callbacks.rb:14:in `each'
  from carrierwave/uploader/callbacks.rb:14:in `with_callbacks'
  from carrierwave/uploader/cache.rb:137:in `cache!'
  from carrierwave/uploader/versions.rb:275:in `block in cache_versions!'
  from carrierwave/uploader/versions.rb:273:in `each'
  from carrierwave/uploader/versions.rb:273:in `cache_versions!'
  from backgrounder/delay.rb:6:in `cache_versions!'
  from carrierwave/uploader/callbacks.rb:16:in `block in with_callbacks'
  from carrierwave/uploader/callbacks.rb:16:in `each'
  from carrierwave/uploader/callbacks.rb:16:in `with_callbacks'
  from carrierwave/uploader/cache.rb:137:in `cache!'
  from carrierwave/uploader/versions.rb:234:in `recreate_versions!'
  from backgrounder/workers/process_asset_mixin.rb:17:in `perform'
  from sidekiq/processor.rb:187:in `execute_job'
  from sidekiq/processor.rb:169:in `block (2 levels) in process'
  from sidekiq/middleware/chain.rb:128:in `block in invoke'
  from raven/integrations/sidekiq.rb:9:in `call'
  from sidekiq/middleware/chain.rb:130:in `block in invoke'
  from sidekiq/middleware/chain.rb:133:in `invoke'
  from sidekiq/processor.rb:168:in `block in process'
  from sidekiq/processor.rb:139:in `block (6 levels) in dispatch'
  from sidekiq/job_retry.rb:98:in `local'
  from sidekiq/processor.rb:138:in `block (5 levels) in dispatch'
  from sidekiq/rails.rb:42:in `block in call'
  from active_support/execution_wrapper.rb:85:in `wrap'
  from active_support/reloader.rb:68:in `block in wrap'
  from active_support/execution_wrapper.rb:85:in `wrap'
  from active_support/reloader.rb:67:in `wrap'
  from sidekiq/rails.rb:41:in `call'
  from sidekiq/processor.rb:134:in `block (4 levels) in dispatch'
  from sidekiq/processor.rb:199:in `stats'
  from sidekiq/processor.rb:129:in `block (3 levels) in dispatch'
  from sidekiq/job_logger.rb:8:in `call'
  from sidekiq/processor.rb:128:in `block (2 levels) in dispatch'
  from sidekiq/job_retry.rb:73:in `global'
  from sidekiq/processor.rb:127:in `block in dispatch'
  from sidekiq/logging.rb:48:in `with_context'
  from sidekiq/logging.rb:42:in `with_job_hash_context'
  from sidekiq/processor.rb:126:in `dispatch'
  from sidekiq/processor.rb:167:in `process'
  from sidekiq/processor.rb:85:in `process_one'
  from sidekiq/processor.rb:73:in `run'
  from sidekiq/util.rb:16:in `watchdog'
  from sidekiq/util.rb:25:in `block in safe_thread'
CarrierWave::ProcessingError: unable to open image `/tmp/magick-6No4015N4HWij': No such file or directory @ error/blob.c/OpenBlob/2712
  from carrierwave/processing/rmagick.rb:374:in `rescue in manipulate!'
  from carrierwave/processing/rmagick.rb:349:in `manipulate!'
  from carrierwave/processing/rmagick.rb:202:in `resize_to_fill'
  from carrierwave/uploader/processing.rb:83:in `block (2 levels) in process!'
  from carrierwave/uploader/processing.rb:75:in `each'
  from carrierwave/uploader/processing.rb:75:in `block in process!'
  from carrierwave/uploader/callbacks.rb:15:in `with_callbacks'
  from carrierwave/uploader/processing.rb:74:in `process!'
  from backgrounder/delay.rb:14:in `process!'
  from carrierwave/uploader/callbacks.rb:14:in `block in with_callbacks'
  from carrierwave/uploader/callbacks.rb:14:in `each'
  from carrierwave/uploader/callbacks.rb:14:in `with_callbacks'
  from carrierwave/uploader/cache.rb:137:in `cache!'
  from carrierwave/uploader/versions.rb:275:in `block in cache_versions!'
  from carrierwave/uploader/versions.rb:273:in `each'
  from carrierwave/uploader/versions.rb:273:in `cache_versions!'
  from backgrounder/delay.rb:6:in `cache_versions!'
  from carrierwave/uploader/callbacks.rb:16:in `block in with_callbacks'
  from carrierwave/uploader/callbacks.rb:16:in `each'
  from carrierwave/uploader/callbacks.rb:16:in `with_callbacks'
  from carrierwave/uploader/cache.rb:137:in `cache!'
  from carrierwave/uploader/versions.rb:234:in `recreate_versions!'
  from backgrounder/workers/process_asset_mixin.rb:17:in `perform'
  from sidekiq/processor.rb:187:in `execute_job'
  from sidekiq/processor.rb:169:in `block (2 levels) in process'
  from sidekiq/middleware/chain.rb:128:in `block in invoke'
  from raven/integrations/sidekiq.rb:9:in `call'
  from sidekiq/middleware/chain.rb:130:in `block in invoke'
  from sidekiq/middleware/chain.rb:133:in `invoke'
  from sidekiq/processor.rb:168:in `block in process'
  from sidekiq/processor.rb:139:in `block (6 levels) in dispatch'
  from sidekiq/job_retry.rb:98:in `local'
  from sidekiq/processor.rb:138:in `block (5 levels) in dispatch'
  from sidekiq/rails.rb:42:in `block in call'
  from active_support/execution_wrapper.rb:85:in `wrap'
  from active_support/reloader.rb:68:in `block in wrap'
  from active_support/execution_wrapper.rb:85:in `wrap'
  from active_support/reloader.rb:67:in `wrap'
  from sidekiq/rails.rb:41:in `call'
  from sidekiq/processor.rb:134:in `block (4 levels) in dispatch'
  from sidekiq/processor.rb:199:in `stats'
  from sidekiq/processor.rb:129:in `block (3 levels) in dispatch'
  from sidekiq/job_logger.rb:8:in `call'
  from sidekiq/processor.rb:128:in `block (2 levels) in dispatch'
  from sidekiq/job_retry.rb:73:in `global'
  from sidekiq/processor.rb:127:in `block in dispatch'
  from sidekiq/logging.rb:48:in `with_context'
  from sidekiq/logging.rb:42:in `with_job_hash_context'
  from sidekiq/processor.rb:126:in `dispatch'
  from sidekiq/processor.rb:167:in `process'
  from sidekiq/processor.rb:85:in `process_one'
  from sidekiq/processor.rb:73:in `run'
  from sidekiq/util.rb:16:in `watchdog'
  from sidekiq/util.rb:25:in `block in safe_thread'

My settings are like below.

CarrierWave.configure do |config|
    config.gcloud_bucket                       = ENV.fetch("GCP_STORAGE_BUCKET_NAME", "")
    config.gcloud_bucket_is_public             = true
    config.gcloud_authenticated_url_expiration = 600
    config.gcloud_content_disposition          = "attachment"

    config.gcloud_attributes = {
      expires: 600
    }

    config.gcloud_credentials = {
      gcloud_project: "my_project_id",
      gcloud_keyfile: "credentials.json"
    }
  end

Which role should the service account have?

Cloud Datastore User, perhaps? :)

Options params is ignored on url method.

Hi people.
I'm trying set options params on file.url(query: { "response-content-disposition": "attachment; filename=#{filename(output_file)}"}) method, but it's not work.

Created a PR to fix this method #40

Tks

Low maintenance activity?

@jasdeepsingh and other authors/contributors

Hello guys, I'm @mshibuya a maintainer of the CarrierWave project.
Looks like this project carrierwave-google-storage is not so actively maintained. It's sad for me, because I think this project deserves more attention.
As the CarrierWave project, currently I'm thinking of the possibility to move away from the Fog gem. Fog comes with a lot of features that CarrierWave doesn't need, and the maintenance status is not so ideal (varies among cloud providers). So the idea is to encourage users to use CarrierWave integration libraries, which depends on not Fog but the cloud provider's official libraries, just like carrierwave-google-storage does.

Hence here's my question, is there anything I can help to maintain this project? If you're fine with it, I'm willing to become a maintainer to support managing dependencies and releasing. Or in case you don't have enough bandwidth to keep maintaining this, it may be worth considering transfer of the maintenance ownership to the carrierwaveuploader organization, just like what was done to carrierwave-aws.

Thank you for your attention, looking forward to hearing back from you!

GcloudFile DelegationError

Thanks for a great gem.

When remote files are not present, some GcloudFile methods that are delegated to the file fail...

Module::DelegationError: CarrierWave::Storage::GcloudFile#size delegated to file.size, but file is nil:

Suggest allowing nil in the delegation in gcloud_file.rb line 11

delegate :content_disposition, :content_type, :size, to: :file, allow_nil: true

Would you suggest that we handle the exception in our application rather than allow nil on the delegation? I am happy to offer a PR if you would accept the allow_nil: true option.

Cannot find file tmp/...

I am switching from AWS storage to Google and while setup is quite simple and seems to work (the files are uploaded to my bucket), I am getting an error for the tmp files created by carrierwave, "cannot find file uploads/tmp/...". I can also see files that are already referenced in the database without error.

The file is uploaded and the error seems to stem back from the google-cloud-storage gem.

CarrierWave.configure do |config|
  config.storage                             = :gcloud
  config.cache_storage                       = :gcloud
  config.gcloud_bucket                       = [my bucket]
  config.gcloud_bucket_is_public             = true
  config.gcloud_authenticated_url_expiration = 600

  config.gcloud_attributes = {
    expires: 600
  }
  
  config.gcloud_credentials = {
    gcloud_project: [my project],
    gcloud_keyfile: [my keyfile]
  }
end

and my uploader

class ImageUploader < CarrierWave::Uploader::Base
  include CarrierWave::MiniMagick
  include Piet::CarrierWaveExtension

  storage :gcloud

  def store_dir
    "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end
end

Allow newly uploaded assets from an active record mount automatically be set to publicly available if bucket is set to public

1.0 is missing on Rubygems.org

As titled, the latest version on Rubygems.org is still 0.9.0:
https://rubygems.org/gems/carrierwave-google-storage/

Maybe @jasdeepsingh forgot to publish it?

Bump dependency on carrierwave '~> 0.7.0' if possible. Latest version is around '1.1.0'

by_verifying_existance will not work if gcloud doesn't throw an exception on a not found file

https://github.com/metaware/carrierwave-google-storage/blob/master/lib/carrierwave/storage/gcloud_file.rb#L26

Explore option to avoid expensive gcloud API calls when ActiveRecord resource loads

I need to be able to use environment variables instead of a .json file

My rails app is hosted on Heroku. I do not believe I can have a file on heroku that exists outside the repository and I would very much like to not include my keyfile.json in my commit history. I do this everywhere else by using environment variables: ENV['KEYFILE_CONTENTS']

That way I can have a dev version and a production version and leave them out of commit history.

Is there anyway to do something like this:

  config.gcloud_credentials = {
    gcloud_project: ENV['PROJECT_NAME'],
    gcloud_keyfile_contents: ENV['KEYFILE_CONTENTS']
  }

Thanks!

How to upload public and private files ?

I need to upload some files publicly, some other ones privately

Good security practise to commit credentials to service accounts?

Hi,

Is there a way to avoid adding service account credentials to the source?
Ref

CarrierWave.configure do |config|
  # [cut..]
  config.gcloud_credentials = {
    gcloud_keyfile: 'path-to-gcp-keyfile.json'
  }
end

How about using Kubernetes secrets instead?

Bump dependency on google-cloud-storage '~> 0.21.0' if possible. Latest version is around '1.4.0'

So going from 0.21.0 -> is 'supposed' to have a breaking change so we may want to be a little careful tackling this one.

Expiration time signed url

I am trying to configure the expiration time, I have tried with several values but always the expiration time of the url is 5 minutes, I have the following configuration of carrierwave

CarrierWave.configure do |config|

  config.storage                             = :gcloud
  config.gcloud_bucket                       = ENV['GCLOUD_BUCKET']
  config.gcloud_bucket_is_public             = false
  config.gcloud_authenticated_url_expiration = 604800

  config.gcloud_attributes = {
    expires: 604800
  }

   config.gcloud_credentials = {
    gcloud_project: ENV['GCLOUD_PROJECT_ID'],
    gcloud_keyfile: ENV['GCLOUD_KEY_FILE']
  }
end

Request for updated RubyGems release of carrierwave-google-storage to support carrierwave 3.0.7

I want to use carrierwave-google-storage with the latest version of carrierwave (3.0.7), but the version on RubyGems has a dependency on carrierwave (< 3). The master branch has updated the dependency to carrierwave (< 4), so I currently bypass the issue by adding the following to my Gemfile:

gem 'carrierwave-google-storage', github: 'carrierwaveuploader/carrierwave-google-storage', branch: 'master'

instead of

gem 'carrierwave-google-storage'

Could you please update the version on RubyGems to reflect these changes?

carrierwaveuploader / carrierwave-google-storage Goto Github PK

carrierwave-google-storage's People

Contributors

Stargazers

Watchers

Forkers

carrierwave-google-storage's Issues

need

Recommend Projects

Recommend Topics

Recommend Org

Jobs