avoid overhead of strftime? about libosmium

[springmeyer]

only place I see it being called is https://github.com/osmcode/libosmium/blob/master/include/osmium/utils/timestamp.hpp#L65-L77

[joto]

Is this actually a problem, ie does it actually take that much time? It could be it just comes up because almost everything else in that part can be inlined, so you don't see that.

[springmeyer]

Yes, it is a problem. With this code commented:

diff --git a/include/osmium/utils/timestamp.hpp b/include/osmium/utils/timestamp.hpp
index aa03893..71d3604 100644
--- a/include/osmium/utils/timestamp.hpp
+++ b/include/osmium/utils/timestamp.hpp
@@ -73,7 +73,7 @@ namespace osmium {
                the year will have 5 digits). And by setting the size
                afterwards from the result of strftime we make sure thats set
                right, too. */
-            s.resize(strftime(const_cast<char*>(s.c_str()), timestamp_length, timestamp_format(), tm));
+            //s.resize(strftime(const_cast<char*>(s.c_str()), timestamp_length, timestamp_format(), tm));
             return s;
         }

The test runs nearly twice as fast, finishing in 11 seconds rather than 18 seconds.

[joto]

I tried it on Linux and commenting out that line makes the osmium_debug about 4% faster. I tried with

s.resize(snprintf(const_cast<char*>(s.c_str()), timestamp_length, "%04d-%02d-%02dT%02d:%02d:%02dZ",
tm->tm_year+1900, tm->tm_mon+1, tm->tm_mday,
tm->tm_hour, tm->tm_min, tm->tm_sec));

which makes the program slower.

So I don't see nearly as much difference as you see. This might be related to the TZ environment variable (see http://stackoverflow.com/questions/8174147/strftime-performance-vs-snprintf and https://bugzilla.redhat.com/show_bug.cgi?id=171351). On my system TZ wasn't set and strace indicates that /etc/localtime was only opened and read once.

This all brings the question: Do we actually want to convert this time depending on some environment variables? I think the answer is "no", it should always be UTC. So maybe strftime is not the right function for this.

[joto]

@springmeyer: can you check whether this is still a problem on OSX? You can use this little test program:

#include <assert.h>
#include <time.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
    char outstr[200];
    time_t t;
    struct tm *tmp;

    t = time(NULL);
    tmp = gmtime(&t);
    assert(tmp != NULL);

    for (int i=0; i < 100000; ++i) {
        assert(strftime(outstr, sizeof(outstr), "%Y-%m-%dT%H:%M:%SZ", tmp) != 0);
    }

    printf("%s\n", outstr);
}

Can you strace that and see whether it opens files for each run?

[springmeyer]

There is no strace on OS X but I tried dtruss and got:

$ sudo dtruss -f ./s
2014-09-26T15:45:46Z
    PID/THRD  SYSCALL(args)          = return
15765/0x2565a:  open("/dev/dtracehelper\0", 0x2, 0x7FFF564D71C0)         = 3 0
15765/0x2565a:  ioctl(0x3, 0x80086804, 0x7FFF564D7148)       = 0 0
15765/0x2565a:  close(0x3)       = 0 0
15765/0x2565a:  issetugid(0x7FFF95E5CAE9, 0x7F9EE9800000, 0x7FFF564D7670)        = 0 0
15765/0x2565a:  open_nocancel("/usr/share/zoneinfo/UTC\0", 0x0, 0x8)         = 3 0
15765/0x2565a:  fstat64(0x3, 0x7FFF564D7580, 0x0)        = 0 0
15765/0x2565a:  read_nocancel(0x3, "TZif\0", 0x2A64)         = 56 0
15765/0x2565a:  close_nocancel(0x3)      = 0 0
15765/0x2565a:  getrlimit(0x1008, 0x7FFF564D71B0, 0x7FFF95E06E7C)        = 0 0
15765/0x2565a:  fstat64(0x1, 0x7FFF564D7178, 0x7FFF564D723C)         = 0 0
15765/0x2565a:  ioctl(0x1, 0x4004667A, 0x7FFF564D7204)       = 0 0
15765/0x2565a:  write_nocancel(0x1, "2014-09-26T15:45:46Z\n\0", 0x15)        = 21 0
15765/0x2565a:  thread_selfid(0x0, 0x1DC0, 0x7FFF6D566550)       = 153178 0
15765/0x2565a:  open(".\0", 0x0, 0x1)        = 3 0
15765/0x2565a:  fstat64(0x3, 0x7FFF564D7050, 0x0)        = 0 0
15765/0x2565a:  fcntl(0x3, 0x32, 0x7FFF564D7310)         = 0 0
15765/0x2565a:  close(0x3)       = 0 0
15765/0x2565a:  stat64("/Users/dane/projects/mapnik/master\0", 0x7FFF564D6FC0, 0x0)      = 0 0
15765/0x2565a:  csops(0x0, 0x0, 0x7FFF564D72D4)      = 0 0
15765/0x2565a:  issetugid(0x0, 0x0, 0x0)         = 0 0
15765/0x2565a:  shared_region_check_np(0x7FFF564D5208, 0x109728000, 0x4)         = 0 0
15765/0x2565a:  stat64("/usr/lib/dtrace/libdtrace_dyld.dylib\0", 0x7FFF564D63B8, 0x7FFF564D72F0)         = 0 0
15765/0x2565a:  open("/usr/lib/dtrace/libdtrace_dyld.dylib\0", 0x0, 0x0)         = 3 0
15765/0x2565a:  pread(0x3, "\312\376\272\276\0", 0x1000, 0x0)        = 4096 0
15765/0x2565a:  pread(0x3, "\317\372\355\376\a\0", 0x1000, 0x1000)       = 4096 0
15765/0x2565a:  fcntl(0x3, 0x3D, 0x7FFF564D4720)         = 0 0
15765/0x2565a:  mmap(0x10972E000, 0x2000, 0x5, 0x12, 0x3, 0x1000)        = 0x10972E000 0
15765/0x2565a:  mmap(0x109730000, 0x1000, 0x3, 0x12, 0x3, 0x3000)        = 0x109730000 0
15765/0x2565a:  mmap(0x109731000, 0x2050, 0x1, 0x12, 0x3, 0x4000)        = 0x109731000 0
15765/0x2565a:  close(0x3)       = 0 0
15765/0x2565a:  stat64("/usr/lib/libSystem.B.dylib\0", 0x7FFF564D61D8, 0x7FFF564D7070)       = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libcache.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)       = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libcommonCrypto.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)        = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libcompiler_rt.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)         = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libcopyfile.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)        = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libcorecrypto.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)      = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libdispatch.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)        = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libdyld.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)        = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libkeymgr.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)      = 0 0
15765/0x2565a:  stat64("/usr/lib/system/liblaunch.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)      = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libmacho.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)       = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libquarantine.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)      = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libremovefile.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)      = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_asl.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)      = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_blocks.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)       = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_c.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)        = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_configuration.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)        = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_dnssd.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)        = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_info.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)         = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_kernel.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)       = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_m.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)        = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_malloc.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)       = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_network.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)      = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_notify.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)       = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_platform.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)         = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_pthread.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)      = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_sandbox.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)      = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libsystem_stats.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)        = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libunc.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)         = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libunwind.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)      = 0 0
15765/0x2565a:  stat64("/usr/lib/system/libxpc.dylib\0", 0x7FFF564D5E38, 0x7FFF564D6CD0)         = 0 0
15765/0x2565a:  stat64("/usr/lib/libobjc.A.dylib\0", 0x7FFF564D5078, 0x7FFF564D5F10)         = 0 0
15765/0x2565a:  stat64("/usr/lib/libauto.dylib\0", 0x7FFF564D5078, 0x7FFF564D5F10)       = 0 0
15765/0x2565a:  stat64("/usr/lib/libc++abi.dylib\0", 0x7FFF564D4B28, 0x7FFF564D59C0)         = 0 0
15765/0x2565a:  stat64("/usr/lib/libc++.1.dylib\0", 0x7FFF564D4B28, 0x7FFF564D59C0)      = 0 0
15765/0x2565a:  stat64("/usr/lib/libDiagnosticMessagesClient.dylib\0", 0x7FFF564D4A08, 0x7FFF564D58A0)       = 0 0
15765/0x2565a:  getpid(0x7FFF564D70C8, 0x10972B004, 0xEA60)      = 15765 0
15765/0x2565a:  __sysctl(0x7FFF564D6B68, 0x2, 0x7FFF564D6B78)        = 0 0
15765/0x2565a:  thread_selfid(0x7FFF7CD8E310, 0x7FFF7CD8E258, 0x10101)       = 153178 0
15765/0x2565a:  bsdthread_register(0x7FFF8CF8CFBC, 0x7FFF8CF8CFAC, 0x2000)       = 0 0
15765/0x2565a:  mprotect(0x10972C000, 0x88, 0x1)         = 0 0
15765/0x2565a:  mprotect(0x109734000, 0x1000, 0x0)       = 0 0
15765/0x2565a:  mprotect(0x10974A000, 0x1000, 0x0)       = 0 0
15765/0x2565a:  mprotect(0x10974B000, 0x1000, 0x0)       = 0 0
15765/0x2565a:  mprotect(0x109761000, 0x1000, 0x0)       = 0 0
15765/0x2565a:  mprotect(0x109762000, 0x1000, 0x1)       = 0 0
15765/0x2565a:  mprotect(0x10972C000, 0x88, 0x3)         = 0 0
15765/0x2565a:  mprotect(0x10972C000, 0x88, 0x1)         = 0 0
15765/0x2565a:  issetugid(0x7FFF7F281480, 0x7FFFFFE00034, 0x7FFFFFE00036)        = 0 0
15765/0x2565a:  getpid(0x1, 0x109763000, 0x49656E69)         = 15765 0
15765/0x2565a:  __mac_syscall(0x7FFF93007E47, 0x2, 0x7FFF564D69E8)       = 0 0
15765/0x2565a:  stat64("/AppleInternal\0", 0x7FFF564D6A68, 0x0)      = -1 Err#2
15765/0x2565a:  audit_session_self(0x7FFF564D6920, 0x7FFF564D6758, 0x4)      = 4099 0
15765/0x2565a:  geteuid(0x7FFF564D6920, 0x7FFF564D6758, 0x0)         = 0 0
15765/0x2565a:  getegid(0x7FFF564D6920, 0x7FFF564D6758, 0x0)         = 0 0
15765/0x2565a:  getaudit_addr(0x7FFF564D69F8, 0x30, 0x0)         = 0 0
15765/0x2565a:  csops(0x3D95, 0x7, 0x7FFF564D65E0)       = -1 Err#22
15765/0x2565a:  shm_open(0x7FFF9077CCE4, 0x0, 0x0)       = 3 0
15765/0x2565a:  mmap(0x0, 0x1000, 0x1, 0x1, 0x3, 0x0)        = 0x109764000 0
15765/0x2565a:  close_nocancel(0x3)      = 0 0
15765/0x2565a:  access("/etc/localtime\0", 0x4, 0x7)         = 0 0
15765/0x2565a:  open_nocancel("/etc/localtime\0", 0x0, 0x0)      = 3 0
15765/0x2565a:  fstat64(0x3, 0x7FFF564D7110, 0x0)        = 0 0
15765/0x2565a:  read_nocancel(0x3, "TZif\0", 0x2A64)         = 1017 0
15765/0x2565a:  close_nocancel(0x3)      = 0 0

[joto]

That looks fine to me. Only opened /etc/localtime once. Can you benchmark against this change?

assert(snprintf(outstr, sizeof(outstr), "%04d-%02d-%02dT%02d:%02d:%02dZ", tmp->tm_year+1900, tmp->tm_mon+1, tmp->tm_mday, tmp->tm_hour, tmp->tm_min, tmp->tm_sec) > 0);

On my (Linux) machine strftime is about twice as fast as snprintf in this case. You might have to do more iterations to get results. Nothing beats hand-crafted code doing the integer-to-string conversion myself: Basically doesn't take any time at all compared to both alternatives.

[springmeyer]

With 10000000 iterations and -O3 on OSX 10.9 and clang++:

• snprintf: 3.627s
• strftime: 8.552s

$ clang++ -v
Apple LLVM version 6.0 (clang-600.0.51) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin13.4.0
Thread model: posix

[springmeyer]

perhaps resolved by cf854e9

[joto]

@springmeyer no, this is the other direction, parsing the timestamp instead of writing it.