Comments (4)
That looks great. Perhaps https://github.com/rlmcpherson/s3gof3r https://github.com/rlmcpherson/s3gof3r is a better client to use?
On 9 Nov 2016, at 19:04, Michael Gebetsroither [email protected] wrote:
eg. as implemented in https://github.com/rlmcpherson/s3gof3r https://github.com/rlmcpherson/s3gof3r
The other feature that isnβt available in most other S3 clients is pipeline support, which is made easy with Goβs reader and writer interfaces. This allows usage like
$ tar -czf - <my_dir/> | gof3r put -b <s3_bucket> -k <s3_object>
$ gof3r get -b <s3_bucket> -k <s3_object> | tar -zx
We use the command line tool at CodeGuard to transfer many terabytes into and out of S3 every day, tarring directories in parallel with the uploads and downloads.Beside parallel upload in general, streaming upload is a really handy feature for big file transfer.
This project also has the added benefit of much robuster error handling and parallel upload/download.I'm posting this here in the spirit of providing feedback as eg. parallel multipart uploading might influence the API and possible required configuration if added later on
β
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub #105, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGNG_XF1V3nM9YUkLvdS_e2rUsb3G9Pks5q8hkzgaJpZM4Kt3zg.
from stow.
s3gof3r is only for object transfer, it eg. has no implementation for listing
from stow.
This would be pretty useful for my use case -- I'm currently maintaining a little abstraction layer for writing database backups to s3, google cloud storage, http or local disk, and was looking at moving to Stow. However we have hundreds of ~100MB files so it'd be rough to try to reserve gigs of RAM to buffer them for upload on a running production DB server (where that RAM is usually otherwise in use), so streaming is pretty important to us.
from stow.
from stow.
Related Issues (20)
- Support location encryption
- Support for object ACL
- Stow makes too many API, also fails to list buckets in other regions
- Out of date documentation
- Is it possible to work with non-Amazon S3 service HOT 1
- Infinite loop in `stow.Walk` HOT 3
- Support for Google Drive, Dropbox, OneDrive, etc.
- Directly use bucket instead of Walking through containers HOT 3
- RemoveContainer for` local` seems incorrect
- b2 - bad_request: accountId invalid HOT 1
- GCS requires permissions not available in standard role
- RequestBodyTooLarge while uploading data into azure storage HOT 1
- S3 upload is failing when data is read from stdin
- s3: specifying bucket region causes Container() to return non-existent containers HOT 4
- Access Azure blobstore container using container_uri and sas_token HOT 1
- How can I obtain the Region from the Location
- Support for Digital Ocean - Spaces
- Item(id) panics in local implementation if the id is not a correct path
- Legacy S3 path being used
- Example of how to connect to Google Cloud Storage
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stow.