files.list and files.delete are limited to 5000 files about aio-lib-files

[meryllblanchet]

Thanks for raising the issue, @amol-anand !
This looks like a default limit of the underlying storage, but ideally the SDK should handle the operation in the most appropriate manner.

cc @purplecabbage @sandeep-paliwal @moritzraho for feedback

[aiojbot]

JIRA issue created: https://jira.corp.adobe.com/browse/ACNA-946

[moritzraho]

After some investigation I can conclude that there are two issues here:

• 1. Azure returns max 5000 files when listing, list is internally used by delete and copy which explains the bug with delete. We need to support storage with more than 5000 files, we should expose some sort of pagination and need to fix the returned number of files in list based on the memory constraints in runtime actions. files.delete('/') should still delete all files but internally list with pagination once we support this. Some changes to handle > 5000 files will be required by files.copy too.
• 2. Runtime action crashes when handling many files (5000) because execution hits resource limits both for memory (default is 256 mb) and number of opened file descriptors for open requests to Azure. To solve this, delete and copy should handle files in fix size batches (like 200 or so).