React Native Issue uploading smaller files about uploadcare-js-api-clients

[paulsizer]

We have changed to using the direct upload API for smaller file uploads (smaller than 100mb).

We now need a way to upload files greater than 100mb. I have tried to use the Javascript library above but that now also appears to be blowing up in my ReactNative app. I have tried changing the maxConcurrentRequests to 1 thinking it might be a resource issue but that still hasn't fixed it.

[paulsizer]

I am now getting file size mismatch when calling complete on the multipart upload. Not sure if this is something to do with the way I do the chunking in my ReactNative (Expo) app.

const resp = await fetch(image.uri);
const blob = await resp.blob();
const uriParts = image.uri.split('.');
const ext = uriParts[uriParts.length - 1];
const filename = `file.${ext}`;

if (blob.size > 99000000) {
  multipartUpload(blob, filename);
} else {
  directUpload(image, filename);
}

const getChunk = (file, index, filesize, chunkSize) => {
  const start = chunkSize * index
  const end = Math.min(start + chunkSize, filesize)

  return file.slice(start, end)
}

const multipartStart = async(blob, filename) => {

  const body = new FormData
  body.append("UPLOADCARE_PUB_KEY", "XXX")
  body.append("filename", filename)
  body.append("size", blob.data.size)
  body.append("content_type", blob.type)
  body.append("UPLOADCARE_STORE", "auto")

  const response = await fetch("https://upload.uploadcare.com/multipart/start/?jsonerrors=1", {
    method: "POST",
    body,
    headers: {
      "Content-Type": "multipart/form-data"
    }
  })

  return response.json();

};

const uploadParts = async(blob, parts, uuid) => {
  const CHUNK_SIZE = 5 * 1024 * 1024;
  const chunks = [];
  for (let i = 0; i < parts.length; i++) {
    chunks.push(getChunk(blob, i, blob.data.size, CHUNK_SIZE));
  }
  console.log(chunks);
  let completed = 0;
  for (let i = 0; i < parts.length; i++) {
    let endpoint = parts[i];
    axios(endpoint, {
      method: 'PUT',
      headers: {
        'Content-Type': blob.type
      },
      data: chunks[i]
    })
    .then(res => {
      if (res.status == 200) {
        completed++;
      }
      console.log('Parts uploaded: ', completed);
      if (completed == parts.length) {
        multipartComplete(uuid);
      }
    })
    .catch(function (error) {
      console.log(error);
    });
  }

};

const multipartComplete = async(uuid) => {

  const body = new FormData
  body.append("UPLOADCARE_PUB_KEY", "XXX")
  body.append("uuid", uuid)

  const response = await fetch("https://upload.uploadcare.com/multipart/complete/?jsonerrors=1", {
    method: "POST",
    body,
    headers: {
      "Content-Type": "multipart/form-data"
    }
  })

  const result = await response.json();
  console.log(result);
};

[nd0ut]

I'm not quite understand why you're doing your own implementation?

Until we release a fixed version, you can import needed methods directly and use react-native file-like object ({ name, type, uri, size }). It will work for both direct and multipart uploads.

Example:

import defaultSettings from '@uploadcare/upload-client/lib/defaultSettings';
import uploadFile from '@uploadcare/upload-client/lib/uploadFile';
import uploadMultipart from '@uploadcare/upload-client/lib/uploadFile/uploadMultipart';

const uri = image.uri;
const resp = await fetch(uri);
const blob = await resp.blob();
const name = 'filename.ext';
const size = blob.size
const type = blob.type

const populateOptionsWithSettings = (options) => ({
  ...defaultSettings,
  ...options,
});

const options = { publicKey: 'demopublickey' }
const asset = { size, name, type, uri }

uploadFile(asset, populateOptionsWithSettings(options));
uploadMultipart(asset, populateOptionsWithSettings(options))

[paulsizer]

@nd0ut using that approach the call to getChuck within uploadMultipart doesn't like the file.

This is the error I get using your example above.

Error - TypeError: file.slice is not a function

[paulsizer]

@nd0ut trying your approach, passing in the blob and extra options seems to do what I want but I get issues like a mentioned previously.

The whole reason for me trying my own approach was that it seems to be hit and miss and the app just completely crashes during an upload, doesn't drop in the the catch block just dumps out.

Like I said above I didn't know if it was something to do with the concurrent requests, this is why I get a failure without anyway to catch the error.

const options = {publicKey: 'XXX', fileName: name, contentType: type};

    uploadMultipart(blob, populateOptionsWithSettings(options))
    .then(value => 
      console.log(`Success - ${value}`)
    )
    .catch(error => 
      console.log(`Error - ${error}`)
    )

[nd0ut]

@paulsizer

using that approach the call to getChuck within uploadMultipart doesn't like the file.

Yep, I was wrong, uploadMultipart requires a Blob, sorry.

I was able to reproduce both problems with size mismatch and crashes on large files, I'll take a look into it. Thanks!

[paulsizer]

@nd0ut

I'm glad you were able to reproduce the errors and it wasn't me doing something wrong my end. I have been banging my head against this for a while now!

Any timescales when you think the fix can be rolled out? We are looking at getting this in the app asap as we were doing our own transcoding and uploading on our own server but it makes no sense for us to do it. We are better offloading this onto your service.

[nd0ut]

@paulsizer
This is the first time I've encountered with react-native and I can't figure out what's causing these crashes. There is nothing that can highlight the cause - no errors in the system log or in the console. Since your simplified approach works without crashes, the cause is clearly somewhere in our codebase. We need a lot more time to find out the problem.

Now to the good stuff. I ran your code and it seems that axios doesn't work well in react-native and doesn't upload the Blob to the server, it's just sends an empty request instead, which resulted in a size mismatch. I replaced it with fetch and that's worked for me.

fetch(endpoint, {
  method: 'PUT',
  headers: {
    'Content-Type': 'image/jpeg'
  },
  body: chunks[i]
})

[nd0ut]

Ha, I turned off the queue and retry mechanics and the crashes are gone. I'm close to a clue.

[nd0ut]

It will not crash if you set maxConcurrentRequests to value greater than parts number, it will upload all the parts at once without concurrency. Still can't figure out why concurrent queue causes crashes at react-native.

[paulsizer]

@nd0ut What would be the best way to do this? Is there a way to set that on the options for that you pass into populateOptionsWithSettings?

[nd0ut]

Catched it.

facebook/react-native#27543

When any of sliced blobs get collected, react-native will deallocate the whole blob, which is leading to the app crash.

@paulsizer for now you can temporarily set maxConcurrentRequests to some high enough value like 100 anywhere you want. I'll prepare a PR with the fix soon.

[paulsizer]

@nd0ut Excellent thanks Alex, I'll give it a go!

[paulsizer]

@nd0ut Hi Alex I tried the following but it seems to be even worse now! Crashes every single time, pretty much instantly now.

const options = {publicKey: 'XXX', fileName: name, contentType: type, maxConcurrentRequests: 100};

    uploadMultipart(blob, populateOptionsWithSettings(options))
    .then(value => 
      console.log(`Success - ${value}`)
    )
    .catch(error => 
      console.log(`Error - ${error}`)
    )

[nd0ut]

@paulsizer I'll release a new version soon, it should work. #314

[nd0ut]

@paulsizer Does version 1.1.3 work as expected?

[paulsizer]

@nd0ut Sorry it's taken a while to get back to you.

Myself and @stevesizer have tried to use the uploadFile method in the following ways:

const options = { publicKey: '882f02a0ca6451ab4795', fileName: name, contentType: type };
const asset = { size, name, type, uri };
var body = new FormData();
body.append('photo', asset);
uploadResponse = await uploadFile(body, populateOptionsWithSettings(options));

const options = { publicKey: '882f02a0ca6451ab4795', fileName: name, contentType: type };
const asset = { size, filename, type, uri };
uploadResponse = await uploadFile(asset, populateOptionsWithSettings(options));

But both result in the following:

TypeError: File uploading from "[object Object]" is not supported.

I have also tried to use the multipartUpload and I am still having issues...

const resp = await fetch(image.uri);
const blob = await resp.blob();
const options = { publicKey: 'XXX', fileName: name, contentType: type };
uploadResponse = await uploadMultipart(blob, populateOptionsWithSettings(options));

The above seemed to work the first time round. Tried it again with the same file and then it crashed as it has done in the past without any error.

I then remembered you mentioned about setting the maxConcurrentRequests. So I tried the following.

const resp = await fetch(image.uri);
const blob = await resp.blob();
 const options = { publicKey: 'XXX', fileName: name, contentType: type, maxConcurrentRequests: 100 };
uploadResponse = await uploadMultipart(blob, populateOptionsWithSettings(options));

As before this just dumps straight out no errors, just a complete crash...

Definitely using the new version 1.1.3

 "dependencies": {
    "@react-native-community/datetimepicker": "3.0.4",
    "@react-native-community/masked-view": "0.1.10",
    "@react-native-community/netinfo": "5.9.7",
    "@react-native-community/segmented-control": "2.2.1",
    "@react-native-community/slider": "3.0.3",
    "@react-navigation/bottom-tabs": "^5.9.1",
    "@react-navigation/native": "^5.7.5",
    "@react-navigation/stack": "^5.9.2",
    "@uploadcare/upload-client": "^1.1.3",
    "axios": "^0.21.1",
    "expo": "^40.0.1",

[nd0ut]

@paulsizer
Starting from 1.1.3 version, uploadFile doesn't support { size, filename, type, uri } input format, blobs only.

The above seemed to work the first time round. Tried it again with the same file and then it crashed as it has done in the past without any error.

Are you trying to upload the same blob or a new one a second time? React-native deallocates blob from the memory after upload complete, so you can't use the same blob after uploading and need to recreate it from uri again.

[paulsizer]

@nd0ut I am creating a new one again from uri

[nd0ut]

@paulsizer Is it the same uri, or a new one?

Strange thing. Probably, fresh blobs are created by reference, not by value and react-native clears memory that blob and uri pointing on.

I was tested double uploads using expo-asset's fromModule with no errors. But I don't know how it's works, seems that it creates new files in memory every time.

I'll dig into it.

[paulsizer]

Yeah it was the same uri

…

On Wed, 31 Mar 2021 at 15:01, Aleksandr Grenishin ***@***.***> wrote: @paulsizer <https://github.com/paulsizer> Is it the same uri, or a new one? Strange thing. Probably, fresh blobs are created by reference, not by value and react-native clears memory that blob and uri pointing on. I was tested double uploads using expo-asset's fromModule with no errors. But I don't know how it's works, seems that it creates new files in memory every time. I'll dig into it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#306 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARFCTMR2AMVHU3AZ623NALTGMTNHANCNFSM4YXXYMIA> .

[nd0ut]

@paulsizer According to this comment, new Blob([blob]) should create a copy of data in memory. I did some tests and it seems to work.

Example:

let blobCopy = new Blob([blob])
await client.uploadFile(blobCopy, options)
...

Please, check it on your end and I'll add this workaround to the upload-client codebase if it would work.

[nd0ut]

Hey @paulsizer,

did this workaround work?

[paulsizer]

Hey sorry @nd0ut I have been super busy trying to get other pieces of the app out.

I have just tested what you suggested and it dumped me straight out. No errors just a fatal crash.

[stevesizer]

Anymore news on this? Some clients are starting to get frustrated.

…

On 11 May 2021, at 12:26, Paul ***@***.***> wrote: Hey sorry @nd0ut <https://github.com/nd0ut> I have been super busy trying to get other pieces of the app out. I have just tested what you suggested and it dumped me straight out. No errors just a fatal crash. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#306 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARHUFU7RDJO4BDJNPDHPTTTNEH4ZANCNFSM4YXXYMIA>.

[nd0ut]

@paulsizer could you provide a minimal snippet? I can't reproduce it :(

[paulsizer]

@nd0ut I have been testing this morning and everything appears to be working for the larger files now if we set

maxConcurrentRequests = 1

We are doing the following to call the different APIs depending on the size, as there is a size limit for multipart uploads

if (size > 10000000) {
      uploadResponse = await uploadMultipart(blob, populateOptionsWithSettings(options));
      console.log(uploadResponse);
    } else {
      uploadResponse = await uploadFile(blob, populateOptionsWithSettings(options));
      console.log(uploadResponse)
    }

This works as expected on iOS but on Android for images that are under the 10Mb threshold we get an error that seems to come from using the uploadFile method.

Cannot create URL for blob!
- node_modules/react-native/Libraries/Blob/URL.js:120:12 in createObjectURL
* http://192.168.1.72:19000/node_modules/expo/AppEntry.bundle?platform=android&dev=true&hot=false&minify=false:147755:18 in Object.exports.transformFile
- node_modules/@uploadcare/upload-client/lib/tools/buildFormData.js:15:12 in _loop_1
* http://192.168.1.72:19000/node_modules/expo/AppEntry.bundle?platform=android&dev=true&hot=false&minify=false:147734:6 in Object.buildFormData [as default]
- node_modules/@uploadcare/upload-client/lib/api/base.js:19:38 in retryIfThrottled_1._default$argument_0
- node_modules/@uploadcare/upload-client/lib/tools/retryIfThrottled.js:14:34 in retry_1._default$argument_0
* http://192.168.1.72:19000/node_modules/expo/AppEntry.bundle?platform=android&dev=true&hot=false&minify=false:147925:13 in runAttempt
* http://192.168.1.72:19000/node_modules/expo/AppEntry.bundle?platform=android&dev=true&hot=false&minify=false:147931:11 in Object.retrier [as default]
- node_modules/@uploadcare/upload-client/lib/tools/retryIfThrottled.js:12:9 in retryIfThrottled
- node_modules/@uploadcare/upload-client/lib/api/base.js:16:438 in base

We do not get this error on iOS.

[nd0ut]

@paulsizer Try to use react-native-url-polyfill on Android to create URL for blobs.