truongleswe / export-dynamodb Goto Github PK

When I have a DynamoDB item like this:

{
  "id": "7e7e9de9-cb80-4697-bbed-b112caa34050",
  "minValue": 0,
  "maxValue": 10000,
  "readingType": "Power",
  "units": "Watts"
}

where the types of minValue and maxValue are numbers (not strings), the exported version records them as strings:

{
  "id": "7e7e9de9-cb80-4697-bbed-b112caa34050",
  "minValue": "0",
  "maxValue": "10000",
  "readingType": "Power",
  "units": "Watts"
}

I'm using python 3.10 and installed all the requirements.
The program fails immediatly on the import error, here is the stack trace:

C:\dci\data_migration\venv\Scripts\python.exe C:/dci/data_migration/exportdynamodb/entrypoints/main.py --profile dev -t Carriers -f csv -o carriers.csv
Traceback (most recent call last):
File "C:\dci\data_migration\exportdynamodb\entrypoints\main.py", line 7, in
from boto3 import Session
File "C:\dci\data_migration\venv\lib\site-packages\boto3_init_.py", line 16, in
from boto3.session import Session
File "C:\dci\data_migration\venv\lib\site-packages\boto3\session.py", line 17, in
import botocore.session
File "C:\dci\data_migration\venv\lib\site-packages\botocore\session.py", line 25, in
import botocore.configloader
File "C:\dci\data_migration\venv\lib\site-packages\botocore\configloader.py", line 19, in
from botocore.compat import six
File "C:\dci\data_migration\venv\lib\site-packages\botocore\compat.py", line 25, in
from botocore.exceptions import MD5UnavailableError
File "C:\dci\data_migration\venv\lib\site-packages\botocore\exceptions.py", line 15, in
from botocore.vendored.requests.exceptions import ConnectionError
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests_init_.py", line 58, in
from . import utils
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\utils.py", line 26, in
from .compat import parse_http_list as parse_list_header
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\compat.py", line 7, in
from .packages import chardet
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\packages_init.py", line 3, in
from . import urllib3
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\packages\urllib3_init_.py", line 10, in
from .connectionpool import (
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\packages\urllib3\connectionpool.py", line 38, in
from .response import HTTPResponse
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\packages\urllib3\response.py", line 9, in
from .collections import HTTPHeaderDict
File "C:\dci\data_migration\venv\lib\site-packages\botocore\vendored\requests\packages\urllib3_collections.py", line 1, in
from collections import Mapping, MutableMapping
ImportError: cannot import name 'Mapping' from 'collections' (C:\Users\diuppa\AppData\Local\Programs\Python\Python310\lib\collections_init.py)

Thoughts?

I don't know if this feature can be implemented.

I didn't find incremental export of dynamodb through google.

Hi from the looks of it all data is gathered in memory before writing to a file.
This works well for small tables. However for big tables the data should be written to file/disk instead of keeping it in memory.

I was reading through the source code of this package cause I needed to copy something, when I realized that:

cur_total = (len(items) + raw_data['Count'])

Why is this defined like this? It effectively makes the count of total items retrieved twice as much as it actually is, since items = raw_data['Items'].