[Errno 104] Connection reset by peer about b2_command_line_tool

[ppolewicz]

We are going to switch to requests and see what the situation is then

[bwbeach]

Are there any more details in the error?

If you catch the error you might be able to get more details:

try:
    ... your code ...
except urllib2.URLError as e:
    print e
    print e.reason

[pgonin]

Here is the full error stack.

It consistently happens after ~2 min on different connections (home and work) and different computers.

I usually manage to upload one file a day. And after I get those errors.
The files I am uploading are about 5 GB size.
Is there a daily upload cap ?

Getting upload URL...
Traceback (most recent call last):
  File "/home/someuser/bin/b2", line 949, in <module>
    main()
  File "/home/someuser/bin/b2", line 942, in main
    upload_file(args)
  File "/home/someuser/bin/b2", line 702, in upload_file
    response = post_file(url, headers, local_file)
  File "/home/someuser/bin/b2", line 313, in post_file
    with OpenUrl(url, data_file, headers) as response_file:
  File "/home/someuser/bin/b2", line 276, in __enter__
    self.file = urllib2.urlopen(request)
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1240, in https_open
    context=self._context)
  File "/usr/lib64/python2.7/urllib2.py", line 1197, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 104] Connection reset by peer>
Getting upload URL...
Traceback (most recent call last):
  File "/home/someuser/bin/b2", line 949, in <module>
    main()
  File "/home/someuser/bin/b2", line 942, in main
    upload_file(args)
  File "/home/someuser/bin/b2", line 702, in upload_file
    response = post_file(url, headers, local_file)
  File "/home/someuser/bin/b2", line 313, in post_file
    with OpenUrl(url, data_file, headers) as response_file:
  File "/home/someuser/bin/b2", line 276, in __enter__
    self.file = urllib2.urlopen(request)
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1240, in https_open
    context=self._context)
  File "/usr/lib64/python2.7/urllib2.py", line 1197, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 104] Connection reset by peer>

thx

[bwbeach]

That's puzzling. I just tested myself with a 5GB upload and it worked fine.

The spending cap feature on B2 accounts is now working, so if you're over your storage limit, it will deny uploads. You can check your status on the B2 accounts page.

Another thing to check is that your files are not bigger than 5,000,000,000 bytes. I should have done so before, but I just added an explicit check to the b2 tool to verify the file size before trying to upload.

[ppolewicz]

If the spending cap is hit, user should get the "storage_cap_exceeded" error instead of a closed socket.

[sentabi]

I got similiar error

$ /root/b2 sync /home/data/ b2:gulen
+ XXXX.tar.gz
/home/data/XXXX.tar.gz:  56%|██████████████████████████████████████████▊                                 | 6.23M/11.1M [00:03<00:02, 1.89MB/s]
Traceback (most recent call last):
  File "/root/b2", line 1742, in <module>
    main()
  File "/root/b2", line 1725, in main
    ct.sync(args)
  File "/root/b2", line 1667, in sync
    bucket.upload_file(filepath, b2_path, file_infos=file_infos)
  File "/root/b2", line 429, in upload_file
    progress_bar=not quiet,
  File "/root/b2", line 1042, in post_file
    with OpenUrl(url, data_file, headers) as response_file:
  File "/root/b2", line 916, in __enter__
    self.file = urllib2.urlopen(request)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 104] Connection reset by peer>

I replace file name with xxx.

I download the latest b2 from github and error become

$ ./b2 sync /home/data/ b2:gulen
Traceback (most recent call last):
  File "./b2", line 1702, in <module>
    main()
  File "./b2", line 1685, in main
    ct.sync(args)
  File "./b2", line 1581, in sync
    for file_info, __ in bucket.ls(prefix=bucket_prefix or None, max_entries=1000, recursive=True):
  File "./b2", line 426, in ls
    if not file_version_info.file_name.startswith(prefix):
TypeError: startswith first arg must be str, unicode, or tuple, not NoneType

both b2 file has b2 version 0.3.9

@bwbeach I only upload 100-200 MB file.

You can check your status on the B2 accounts page.

why not show these notification on b2 command line instead?

[ppolewicz]

I'll fix the second error tonight

[ppolewicz]

The second error is fixed in #39.

As for the main issue, I may have a fix but testing it is a bit tricky. It should be done in a few days.

[bwbeach]

Thanks, @ppolewicz, for fixing.

[bnyeggen]

Request this be reopened. I'm showing the same symptoms, with failure almost immediately after beginning the upload.

[ppolewicz]

@bnyeggen could you confirm you are using the current version (just retrieved from the repository) of b2 command line tool? Could you please paste the resulting stracktrace that you get now? (upload error reporting was greatly improved recently)

[bnyeggen]

Version 0.3.10, on Python 2.7.6, from https://docs.backblaze.com/public/b2_src_code_bundles/b2 yields:

Traceback (most recent call last):
  File "/vol1/bin/b2", line 1742, in <module>
    main()
  File "/vol1/bin/b2", line 1729, in main
    ct.upload_file(args)
  File "/vol1/bin/b2", line 1527, in upload_file
    quiet=quiet,
  File "/vol1/bin/b2", line 429, in upload_file
    progress_bar=not quiet,
  File "/vol1/bin/b2", line 1042, in post_file
    with OpenUrl(url, data_file, headers) as response_file:
  File "/vol1/bin/b2", line 916, in __enter__
    self.file = urllib2.urlopen(request)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 104] Connection reset by peer>

[bnyeggen]

Using version 0.3.9, on Python 2.7.6, from https://raw.githubusercontent.com/Backblaze/B2_Command_Line_Tool/master/b2 yields 5 retries, each consisting of:

[Errno 104] Connection reset by peer

Traceback (most recent call last):
  File "/vol1/bin/b2", line 1185, in __enter__
    self.file = urllib2.urlopen(request)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 104] Connection reset by peer>

[ppolewicz]

There should be a part which looks like this:

URL: https://pod-000-1004-00.backblaze.com/b2api/v1/b2_upload_file/424242424242424242424242/c001_v0001004_t0028
Params: None
Headers: {'X-Bz-Content-Sha1': '753ca1c2d0f3e8748320b38f5da057767029a036', 'X-Bz-File-Name': 'LICENSE', 'Content-Type': 'b2/x-auto', 'Content-Length': '1350'}

is it not shown?

[bnyeggen]

URL: https://pod-000-1012-03.backblaze.com/b2api/v1/b2_upload_file/f636459221c05d3d51200c10/c001_v0001012_t0024
Params: None
Headers: {'X-Bz-Content-Sha1': '4a51cb63f9bb32667f90d27f31533a550849d788', 'X-Bz-File-Name': 'calibre-2016-01-19.tar.aa.cpt', 'Content-Type': 'b2/x-auto', 'Content-Length': '4000000032'}

(stripped the authorization key)

[ppolewicz]

Does the same problem occur for very small files or only above certain treshold?

Could you please install tqdm (sudo pip install tqdm) and see if there is any progress in uploading or if it is failing right after start?

[bnyeggen]

Uploading small (10K) files failed with a quota-exceeded message. Setting my quota appropriately, small upload succeeded, and now large file upload appears to be proceeding. Hypothesizing that the small quota was the source of the earlier error, and the message was being swallowed.

[ppolewicz]

It was found in test that the upload progresses by 2.25MB and then freezes. A small upload fails instantly with an error (which is not wrapped in an exception).

[ppolewicz]

https://github.com/python-git/python/blob/master/Lib/urllib2.py#L1108

I have an idea for a fix

[ppolewicz]

In python 3 it is broken in the exact same way

[ppolewicz]

https://bitbucket.org/nikratio/python-dugong alternative library which might overcome this problem (requires python 3.3 or newer though)

[mdeboer]

I can confirm this still happens in v0.5.6. It retries 5 times and then just gives up. Also still giving exit code 0, see issue #174

I can hardly imagine the retry is handled as it is supposed to (new url and delay?). It seems to retry without delay.

[svonohr]

Do you have a way to reproduce ECONNRESET errors? I've tried to reproduce such or similar connectivity errors by dropping/corrupting/delaying packets and no method even lead to errors (just slower uploads). I've also tried some obscure Perl script which was supposed to close open sockets, but it was unable to do so.

[mdeboer]

Simply upload a bunch of files to b2 and wait for it to happen, shouldn't take long. It's just the infrastructure that can't handle it properly as I see a lot of 'server too busy' errors. Can't remember if that was with the b2 CLI tool or with rclone, but either way it happened. Both clients have to retry a lot of times and eventually give up.

Now I gave up until this is fixed (probably on the infrastructure side), only using S3 now.

[svonohr]

Exactly what is a "bunch of files"? I've uploaded some 100 of files and never had any problems with upload failures.

[mdeboer]

I upload large files randing from 1GB to 15GB to it every night and every night it's been the same. Probably like 40 files?

[ppolewicz]

@svonohr: is the ECONNRESET tracked in some other issue? If so, can we close #16?

[svonohr]

We have #172 which is similar, but is more about making sure that an ECONNRESET error is reported using the correct exception. The behavior reported by @mdeboer seems different.

[ppolewicz]

@svonohr since you have nailed this problem in python 2 and 3, will you take over this issue?

[mdeboer]

@svonohr trying again with v0.6.2 this week. Tried with a large singe file real quick and that went fine. Will know more in a few days 👍

[ionyx0]

Is anyone still working on this issue? I'm having the same problem as OP. My failure happens at upload of the duplicity signatures file (~2 GB). Archive upload in 100 MB file chunks works fine. I have no caps set at B2.

[svonohr]

No, currently there is nobody working on this issue. What's the exact error that you get? What operating system are you using?

[ionyx0]

Attempt 1 failed. B2ConnectionError: Connection error: ('Connection aborted.', error(104, 'Connection reset by peer'))

I'm using Ubuntu Server 16.04.

[dima-ser]

I'm also consistently having this error when uploading large files:

ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1099-17.backblaze.com/b2api/v1/b2_upload_part/4_z8d0f3a636baba47b55ee0a1b_f201d917d7aa63de7_d20171127_m112720_c001_v0001099_t0052/0009
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/b2/bucket.py", line 539, in _upload_part
    input_stream
  File "/usr/local/lib/python2.7/dist-packages/b2/raw_api.py", line 467, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "/usr/local/lib/python2.7/dist-packages/b2/b2http.py", line 259, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "/usr/local/lib/python2.7/dist-packages/b2/b2http.py", line 103, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "/usr/local/lib/python2.7/dist-packages/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: ('Connection aborted.', error(104, 'Connection reset by peer'))

b2 command line tool, version 0.7.2
Ubuntu Server 16.04.3 LTS

EDIT: just realized there's version 1.1.0 out, upgraded, will monitor if error is still there

[svonohr]

You could also try to reduce the number of threads. B2 CLI defaults to 10 threads which could be to much for some connections.

[KumKri]

I am getting following error while uploading files less than 50 MB using b2 command line tool:
ERROR: Connection error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

I am using CentOS Linux release 7.4.1708 OS

[ppolewicz]

This is an old issue with environmental-like errors and in the last release of the CLI, we've improved the retry policies, so hopefully similar situations should be less frequent. If something like that happens again, please open a new ticket so that we can investigate it properly.