upload_fileobj hangs indefinitely without returning error about boto3

[tim-finnigan]

Thanks for reaching out. In the issue you linked someone mentioned that changing the base Docker image fixed this. Have you tried reproducing the issue in environments other than RHEL? Also the latest Boto3 is 1.34.118 per the CHANGELOG, have you tried updating to that version? And are you using the CRT and if so have you tried uninstalling/reinstalling?

You could also try testing with the various S3 transfer configuration options documented here, for example:

import boto3
from boto3.s3.transfer import TransferConfig

# Create an S3 client
s3 = boto3.client('s3')

# Define the transfer config
transfer_config = TransferConfig(
    multipart_threshold=1024 * 25,  # 25 MB
    max_concurrency=10,
    multipart_chunksize=1024 * 1024,  # 1 MB
    num_download_attempts=5,
    max_io_queue=100,
    io_chunksize=1024 * 1024,  # 1 MB
    use_threads=True
)

# Open the local file
with open('file.txt', 'rb') as f:
    # Use the transfer config in an S3 upload operation
    s3.upload_fileobj(
        Fileobj=f,
        Bucket='test-bucket',
        Key='remote_file.txt',
        Config=transfer_config
    )

Otherwise I recommend following the S3 performance guideline optimizations documented here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html

If you're able to narrow down the conditions in which this issue occurs, please let us know.

[fitnycdigitalinitiatives]

Hello,

Not sure if this is related, but we recently got this error:

botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: AWSHTTPSConnectionPool(host='s3.us-east-1.amazonaws.com', port=443): Pool is closed.

[fitnycdigitalinitiatives]

More detailed log:

Traceback (most recent call last):
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.9/site-packages/urllib3/connectionpool.py", line 276, in _get_conn
    conn = self.pool.get(block=self.block, timeout=timeout)
  File "/usr/lib64/python3.9/queue.py", line 165, in get
    with self.not_empty:
  File "/usr/lib64/python3.9/threading.py", line 257, in __enter__
    return self._lock.__enter__()
  File "src/gevent/_semaphore.py", line 282, in gevent._gevent_c_semaphore.Semaphore.__enter__
  File "src/gevent/_semaphore.py", line 283, in gevent._gevent_c_semaphore.Semaphore.__enter__
  File "src/gevent/_semaphore.py", line 184, in gevent._gevent_c_semaphore.Semaphore.acquire
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib64/python3.9/site-packages/gevent/thread.py", line 112, in acquire
    acquired = BoundedSemaphore.acquire(self, blocking, timeout)
  File "src/gevent/_semaphore.py", line 184, in gevent._gevent_c_semaphore.Semaphore.acquire
  File "src/gevent/_semaphore.py", line 253, in gevent._gevent_c_semaphore.Semaphore.acquire
  File "src/gevent/_abstract_linkable.py", line 521, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait
  File "src/gevent/_abstract_linkable.py", line 487, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core
  File "src/gevent/_abstract_linkable.py", line 490, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core
  File "src/gevent/_abstract_linkable.py", line 442, in gevent._gevent_c_abstract_linkable.AbstractLinkable._AbstractLinkable__wait_to_be_notified
  File "src/gevent/_abstract_linkable.py", line 451, in gevent._gevent_c_abstract_linkable.AbstractLinkable._switch_to_hub
AttributeError: 'NoneType' object has no attribute 'switch'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.9/site-packages/botocore/httpsession.py", line 464, in send
    urllib_response = conn.urlopen(
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.9/site-packages/urllib3/connectionpool.py", line 704, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.9/site-packages/urllib3/connectionpool.py", line 279, in _get_conn
    raise ClosedPoolError(self, "Pool is closed.")
urllib3.exceptions.ClosedPoolError: AWSHTTPSConnectionPool(host='s3.us-east-1.amazonaws.com', port=443): Pool is closed.

[fitnycdigitalinitiatives]

@tim-finnigan Hello, I did some more digging into the system logs and saw this message popping up around these issues:
systemstats_linux: unexpected header length in /proc/net/snmp. 237 != 224
Which is related to this net-snmp bug

Do you think that could be causing the problem?

[tim-finnigan]

Thanks for following up, I'm not sure that could be causing the problem. Could you provide the full snippet you're using and complete debug logs (with sensitive info redacted) for us to investigate further? Adding boto3.set_stream_logger('') to your script will generate the full logs.

[fitnycdigitalinitiatives]

Hi Tim,

I don't want to speak too soon, but one thing that I recently tried was turning off threads in the TransferConfig and I haven't run into the error in the last several days. Perhaps there's something interrupting the threading process either with our application or in RHEL 9.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[fitnycdigitalinitiatives]

Verifying that turning off threading fixed the issue. I did see online that RHEL 9 changed something with it's threading approach, but it's way beyond my understanding to know if it could be related to what I'm experiencing.

[Susuvnkl]

I experienced the same issue,
no error log, the app just freezes.
s3_client.upload_fileobj(
file,
bucket_name,
filename,
ExtraArgs={"ContentType": file.content_type}
)

ATM using s3_client.put_object method instead.