utils.strftime does not format the year properly about singer-python

[nick-mccoy]

Hi @bondgeek, I think we are going to change singer-python's strftime() to do something similar to what was done in this pull request.

I will link the change here once it happens to keep anyone following this thread in the loop.

[KAllan357]

Thanks for the report, @awm33.

After some research, it looks like this is dependent on the C libraries which back strftime.

On my Macbook:

$ python3
Python 3.6.3 (default, Oct  4 2017, 06:09:15)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import singer.utils as utils
>>> dobj = utils.strptime_with_tz("10/10/983")
>>> utils.strftime(dobj)
'4Y-10-10T00:00:00.000000Z'

On my Ubuntu VM:

Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import singer.utils as utils
>>> dobj = utils.strptime_with_tz("10/10/983")
>>> utils.strftime(dobj)
'0983-10-10T00:00:00.000000Z'

What I think we should do is revert to the normal %Y version of the format, and add an optional parameter to the function to provide your own format string. Does that seem like it would work for you?

[awm33]

@KAllan357 That works for me

[nick-mccoy]

I added the fix Kyle talked about, and did some testing to verify:

On my macbook:

$ python
Python 3.6.3 (v3.6.3:2c5fed86e0, Oct  3 2017, 00:32:08)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import singer.utils as utils
>>> dobj = utils.strptime_with_tz('01/01/990')
>>> utils.strftime(dobj)
'0990-01-01T00:00:00.000000Z'
>>> utils.strftime(dobj, '%04Y-%m-%dT%H:%M:%S.%fZ')
'4Y-01-01T00:00:00.000000Z' 

On my linux VM:

$ python
Python 3.4.3 (default, Nov 28 2017, 16:41:13)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import singer.utils as utils
>>> dobj = utils.strptime_with_tz(’01/01/990’)
>>> utils.strftime(dobj)
'990-01-01T00:00:00.000000Z'
>>> utils.strftime(dobj, '%04Y-%m-%dT%H:%M:%S.%fZ')
'0990-01-01T00:00:00.000000Z'

Basically, there are two cases:

1. The C Library's strftime() zero pads dates having a year <1000 and does not recognize the %04Y format. In this case, use singer.utils.strftime(dobj)

2. The C Library's strftime() does not zero pad dates having a year <1000, but does recognize the %04Y format. In this case, use singer.utils.strftime(dobj, '%04Y-%m-%dT%H:%M:%S.%fZ')

[nick-mccoy]

I'm going to merge these changes shortly, and they should be pulled into singer-python version 5.0.3

[bondgeek]

So, this fix was reverted here:

9202323

This relies on behavior that is not part of the documented behavior of the datetime library. https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior

This also renders the time_extracted feature of the write_record function unusable on platforms that don't support the desired behavior for years < 1000 (which would seem to meet the definition of an edge case).

Is there still a chance that the reliance on the documented behavior for strptime / strftime will be restored?

[sminnee]

FYI I worked around this by removing "04" in this line: https://github.com/singer-io/singer-python/blob/master/singer/utils.py#L14

[nick-mccoy]

This has been resolved by #69, and utils.strftime should now work on any platform.

[dipamacharyya]

I'm running tap-onfleet --config config.json --catalog catalog.json but getting an unknown string format for my system time

INFO administrators: Starting sync
INFO GET request to https://onfleet.com/api/v2/admins
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"endpoint": "administrators"}}
CRITICAL Unknown string format: 2022-10-10 11:13:18 Pacific Daylight Time
Traceback (most recent call last):
File "C:\python39\lib\runpy.py", line 197, in run_module_as_main
return run_code(code, main_globals, None,
File "C:\python39\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Python39\Scripts\tap-onfleet.exe_main.py", line 7, in
File "C:\python39\lib\site-packages\singer\utils.py", line 192, in wrapped
return fnc(*args, **kwargs)
File "C:\python39\lib\site-packages\tap_onfleet_init.py", line 91, in main
sync(client, parsed_args.catalog, state)
File "C:\python39\lib\site-packages\tap_onfleet_init.py", line 66, in sync
counter_value = sync_stream(state, instance)
File "C:\python39\lib\site-packages\tap_onfleet\sync.py", line 19, in sync_stream
for (stream, record) in instance.sync(state):
File "C:\python39\lib\site-packages\tap_onfleet\streams.py", line 103, in sync
self.update_bookmark(state, item[self.replication_key])
File "C:\python39\lib\site-packages\tap_onfleet\streams.py", line 55, in update_bookmark
if self.is_bookmark_old(state, value):
File "C:\python39\lib\site-packages\tap_onfleet\streams.py", line 61, in is_bookmark_old
return utils.strptime_with_tz(value) > utils.strptime_with_tz(current_bookmark)
File "C:\python39\lib\site-packages\singer\utils.py", line 21, in strptime_with_tz
d_object = dateutil.parser.parse(dtime)
File "C:\python39\lib\site-packages\dateutil\parser_parser.py", line 1368, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "C:\python39\lib\site-packages\dateutil\parser_parser.py", line 643, in parse
raise ParserError("Unknown string format: %s", timestr)
dateutil.parser._parser.ParserError: Unknown string format: 2022-10-10 11:13:18 Pacific Daylight Time

Can someone please help?