skeenp / agol_backup Goto Github PK

2022-03-28 12:00:27,402 INFO: - 2b7098c4f6e84005bf160eaf7913bffd
2022-03-28 12:00:29,991 ERROR: Unexpected error occured backing up files, please review code and try again.
Traceback (most recent call last):
File "/volume1/WINSW/Backups/_agolbackup/backup_mgr.py", line 147, in run
res = bi.backup(ago.gis, itemid, backup_dir, options, fmt, skip_unmodified=skipunmod, logger=logger)
File "/volume1/WINSW/Backups/_agolbackup/backup_items.py", line 68, in backup
last_modified = datetime.fromtimestamp(last_modified/1000)
ValueError: year 292278994 is out of range
2022-03-28 12:00:30,080 INFO: Script finished at 03/28/2022 12:00:30 after 5 seconds

Check the once logic in app, looks like -1.0 may not be being converted to 0.0 on successful run

Popup window on console screen in cfg_manager when using 'run now' would be useful.

Get multithreading working for the scripts (mainly backup_mgr.py and backup_cfg_mgr.py)

Add a reset flag to bypass last_updated checks

Need to add in code to reload items on successful connection to AGOL

Move timestamp file creation to end of process

Bypass git and schedule

• Supply once only download flag or on demand > hide controls for schedule
• Run now from GUI > useful for testing

Need to verify config file exists before opening after backup_mgr.py ln 30

Sort items by name and folder in GUI

Split up large exports into multiple smaller ones by layer id or another mechanism

https://developers.arcgis.com/rest/users-groups-and-items/export-item.htm

Grab OIDs, sort and then order. Query then segments of 5000 records

{
"layers" :[ {"id" : {{itterate layers}} },
{"id" : 1, "where" : "OBJECTID> x"}
]
}

Provide functionality to move data to a remove git location as an option.

backup_mgr.py ln 214, message required in exception

Would be handy to be able to use tags for backups rather than Config files. Suggesting commandline args with multiple options in the format tag:frequency(hours). Allows custom tags to be used based on user preference.

Example {exec} -b {backup path} -t tier1:1 tier2:24 for tier one tagges items to be updated hourly, tier two items to be updated daily.

Backupmgr should reference the config file used in processing

File "C:\Users\skeenp01\Documents\GitHub\agol_backup\backup_cfg_mgr.py", line 501, in _loaditems
    if self._ago.gis:
AttributeError: 'NoneType' object has no attribute 'gis'

Need to check for self._ago first

GUI backgrounds for items needs updating to match new white background used in innerframe

Provide method to filter/search for items

2022-01-13 08:26:17,451 INFO: Connected to https://trade.maps.arcgis.com/
2022-01-13 08:26:17,451 INFO: Processing Items
2022-01-13 08:26:17,451 INFO:  - 02746f5e401c4734b36923cb20d0018b
2022-01-13 08:27:02,927 ERROR: Unexpected error occured backing up files, please review code and try again.
Traceback (most recent call last):
  File "C:\Users\skeenp01\Documents\GitHub\agol_backup\backup_mgr.py", line 163, in run
    util.set_ts(os.path.join(itmdir, 'lastupdate.ts'))
  File "C:\Users\skeenp01\Documents\GitHub\agol_backup\util.py", line 151, in set_ts
    export_obj(file, datetime.isoformat())
TypeError: unbound method datetime.isoformat() needs an argument

export_obj(file, datetime.now().isoformat())

Add a HTML summary to each out put folder to enable efficient access to backup content. Use item labels rather than ids.

Need to add a try/catch around export functions for items

# Setup export
            try:
                export = item.export(title=title, export_format=fmt, wait=True, overwrite=True)
                # Grab data
                file_data = export.download(item_dir)
                os.rename(file_data, file_data.replace(postfix, ''))
                # Delete export
                export.delete()
            except Exception as ex:
                logger.debug("  > Service Export Failed")
                return bla bla bla```

Remove reference to i['name'] when determining if exportable

Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Python39\lib\tkinter_init_.py", line 1885, in call
return self.func(*args)
File "C:\Users\skeenp01\Documents\GitHub\agol_backup\backup_cfg_mgr.py", line 739, in _runnow
backup_mgr.run([cfg_path], logger=self._logger)
File "C:\Users\skeenp01\Documents\GitHub\agol_backup\backup_mgr.py", line 47, in run
os.mkdirs(backup_dir)
AttributeError: module 'os' has no attribute 'mkdirs'

Remove -c flag, it is not consistent with the other apps