dreamscapes / skipper-better-s3 Goto Github PK

Would a copy/move method fall outside of the intended scope for this adapter?

When upload from Windows, path module build remote path in Windows style instead of posix.
const fd = path.format(parts) // eslint-disable-line one-var

The function onProgress is not being called.

onProgess is not working. I don't get any callbacks.... for the progress.
I tried debugging the skipper code and saw that there are progress events internally, but for some reason I couldn't find, it is not calling the function defined in onProgress.
req.file('videoFile').upload({
adapter: require('skipper-better-s3'),
key: process.env.S3_KEY || sails.config.nexcvConfig.S3Settings.KEY,
secret: process.env.S3_SECRET || sails.config.nexcvConfig.S3Settings.SECRET,
bucket: process.env.S3_BUCKET || sails.config.nexcvConfig.S3Settings.BUCKET,
onProgress: progressUpdate,
s3params:
{ ACL: 'public-read'
},

I'll be happy to help debug and fix this, but I will need your guidlines/help please. Please contact me
at [email protected]

Hi team, thanks so much for this adapter I've had a lot of success with it except for one issue. I'm having trouble catching an error triggered by the receiver.write function. When an error occurs it completely stops my Sails app. I've tried wrapping the method in try/catch blocks and setting up promises but I can't figure it out.

My code is:

const adapter = require('skipper-better-s3')(sails.config.uploads.s3);
const receiver = adapter.receive(receiverParams);

receiver.write(localFile, (err) => {
    if (err) {
        sails.log.error('Error uploading to Amazon S3:', err);
    } else if ( localFile.extra && localFile.extra.key ) {
        sails.log.info('Uploaded file to Amazon S3: ' + localFile.extra.key);
    }
});

You can reliably get an error firing by setting the s3 region to a random string.

I did have a similar problem with the adapter.read function causing my app to crash if there was no matching key on S3 matching the file descriptor, however I was able to resolve this by stringing on an "on('error')" function. However this doesn't work to resolve the issue I'm having with receiver.write :(

adapter.read(fd).on('error', (err) => {
  console.log(err);
});

I'm a bit of a beginner so it's probably really obvious what I'm doing wrong. Is anyone able to help?

Thanks so much,
Madison

Hello, I'm using adapter.ls to get a directory listing from s3 but I need to paginate this request to avoid timeout with a gigantic list of files but I can't see any option for that.

Any idea how I could make the request with something similar to the use of LIMIT and OFFSET within a Mysql query?

Thanks in advance.

Getting below error only on server ```
Error details: ImplementationError: Could not find compile-statement, one of the modules in `pkg.machinepack.machines`.
Maybe it was deleted or mistyped?

hello, I am facing a problem with skipper-better-s3, when I upload a file over 5 MB, it crash sailsjs server and give me an error like:
t IncomingMessage.onReqAborted (/home/duan/projects/ds_back_end/node_modules/multiparty/index.js:183:17)
at emitNone (events.js:106:13)
at IncomingMessage.emit (events.js:208:7)
at abortIncoming (_http_server.js:412:9)
at socketOnClose (_http_server.js:406:3)
at emitOne (events.js:121:20)
at Socket.emit (events.js:211:7)
at TCP._handle.close [as _onclose] (net.js:554:12)

I have my upload code as following:

`upload: (req, res) => {
const {fileName, previewName, coverName} = req.body;

async.waterfall([callback => {
  async.parallel([
    cb => {
      if (!fileName) {
        return cb();
      }

      req.file(fileName).upload(disAWS.uploadConfig(fileName, 'MUSIC'), (err1) => {
        if (err1) {
          return cb(err1);
        }

        cb();
      });
    },
    cb => {
      if (!previewName) {
        return cb();
      }

      req.file(previewName).upload(disAWS.uploadConfig(previewName, 'PREVIEW'), (err1) => {
        if (err1) {
          return cb(err1);
        }

        cb();
      });
    },
    cb => {
      if (!coverName) {
        return cb();
      }

      req.file(coverName).upload(disAWS.uploadConfig(coverName, 'COVER'), (err1) => {
        if (err1) {
          return cb(err1);
        }

        cb();
      });
    }
  ], callback);
}, (result, cb) => music.createMusic(req.body, req.ctx.payload.id, cb)], (err) => {
  if (err) {
    return res.negotiate(err);
  }

  res.created();
});`

and disAWS.uploadConfig is a function to generate skipper s3 config:

module.exports.uploadConfig = (fileName, key) => {
return {
adapter,
key: accessKeyId,
region,
maxBytes: Number.MAX_VALUE,
secret: secretAccessKey,
bucket: configPath.bucket,
dirname: folderMap[key],
saveAs: (__newFileStream, next) => {
return next(undefined, fileName);
},
onProgress: progress => console.log('Upload progress:', progress)
};
};

any help is appreciated, thanks in advance

When I do a file upload, the uploaded file doesn't keep the original name. How can I force it?

Example:

I upload file "MYFILE1.BMP" and the file uploaded gets the name "0d05a338-80c9-4683-b040-9caf9e387240.BMP"

Thanks in advance

MaxBytes seems to have no effect.

If using roles in AWS EC2 instance, there is no need to pass down AWS key id nor secret to AWS tooling so it shouldn't be required in those cases and in fact the current functionality prevents use of roles with this package.

How I can set bucket directory to upload file, or replace file after success upload?

in uploader.js line 56

const params = merge({ Key: fd, Body: outStream, ContentType: type }).and(scope.opts.request) .excluding('onProgress') .into({})

Key is "fd" and it always overrides any other value. Key should be something like

Key: scope.opts.request.Key || fd

When setting s3 ACL rule via s3params, it seems it has no effect on the file uploaded to the s3. (Also the onProgress is not working - but this is less important for now....)
sample code I am using:
req.file('videoFile').upload({
adapter: require('skipper-s3'),
key: process.env.S3_KEY || sails.config.nexcvConfig.S3Settings.KEY,
secret: process.env.S3_SECRET || sails.config.nexcvConfig.S3Settings.SECRET,
bucket: process.env.S3_BUCKET || sails.config.nexcvConfig.S3Settings.BUCKET,
s3params: {ACL:"public-read"},
onProgress: progress => sails.log.info('Upload progress:', progress)
}, async function (err, filesUploaded) {

const options =
{
key: 'XX'
, secret: 'YY'
, bucket: 'stacktapimages'
, region: 'eu-central-1'
// Let's use the custom s3params to upload this file as publicly readable by anyone
, s3params:
{
ContentType: 'image/jpeg',
ACL: 'public-read'
}
}
sails.s3 = require('skipper-better-s3')(options);

The file does upload to S3, but if I check the uploaded file it doesn't have the correct mime type nor does it have the permission for everyone to open/download.

the following is the part that uploads to the file to s3:

const receiver = sails.s3.receive()
receiver.write(file, () => {

Given libraries like s3rver and mock-aws-s3, connection to a locally instantiated bucket could be made possible by specifying the region parameter as local, like so:

const options =
      { // This is the usual stuff
        adapter: require('skipper-better-s3')
      , key: 'somekeyhere'
      , secret: 'dontsharethis'
      , bucket: 'my-s3-bucket'
      , region: 'local'  // Optional - default is 'us-standard'
        // Let's use the custom s3params to upload this file as publicly
        // readable by anyone
      , s3params:
        { ACL: 'public-read'
        }
        // And while we are at it, let's monitor the progress of this upload
      , onProgress: progress => sails.log.verbose('Upload progress:', progress)
      }

req.file('avatar').upload(options, (err, files) => {
  // ... Continue as usual
})

Your thoughts?