strange parsing behavior about nanotime

[lsilvest]

I've got an old MacBook (but with a recent OS) lying around here, I'll look at why the fix I suggested didn't work.

[lsilvest]

I was able to reproduce this issue on the MacBook. I recompiled and installed RcppCCTZ with the changes I suggested, and this solves the issue. @ijlyttle, you might not have fully recompiled the package or it didn't reload?

[ijlyttle]

Yes!

Just to confirm what you both have found, I have installed GH version of RcppCCTZ and CRAN version of nanotime on my Mac:

> library("nanotime")
> nanotime("2015-11-18T05:33:03.765333000+00:00")
[1] "2015-11-18T05:33:03.765333000+00:00"

OK for me to close the issue.

Would have responded sooner, but I was trying to confirm on Windows, but (I think) I ran into the Rtools problem described here: eddelbuettel/rcppcctz#9 - which is a different problem.

Thanks @eddelbuettel and @lsilvest!

[eddelbuettel]

I just needed add a little bit of conditional logic to have the local-for-g++-4.9-on-windows backport activated. All good, new package should be on CRAN "shortly" so this can now be closed.

[eddelbuettel]

On Ubuntu 16.10:

R> library(nanotime)
R> x <- nanotime("1970-01-01T00:00:00.000000001+00:00")   # from ?nanotime
R> x
[1] "1970-01-01T00:00:00.000000001+00:00"
R> y <- nanotime("2015-11-18T05:33:03.765333000+00:00")
R> y
[1] "2015-11-18T05:33:03.765333000+00:00"
R> 

We rely on the CCTZ library and the RcppCCTZ package I wrapped around that. Maybe poke there, put some print statements in etc?

[ijlyttle]

Just to confirm that I find the same as you on Ubuntu 16.04.

I'll do as you suggest, and poke around the supporting libraries on the Mac.

[eddelbuettel]

Sounds good. Just for kicks, the nanotime ctor does this:

##' @rdname nanotime
##' @export
setMethod("nanotime",
          "character",
          function(x) {
            fmt <- getOption("nanotimeFormat", default="%Y-%m-%dT%H:%M:%E9S%Ez")
            tz <- getOption("nanotimeTz", default="UTC")
            d <- RcppCCTZ::parseDouble(x, fmt=fmt, tz=tz)
            new("nanotime", as.integer64(d[,1]) * 1e9 + as.integer64(d[, 2]))
          })

so we check all the ingredients. And it is good that you have another box to validate against.

(Mind you that is from GH with the PR by @lsilvest which adds S4 methods. The initial parsing issue should be ortogonal to all this. If anything it may be a CCTZ and clang interaction....)

[ijlyttle]

I see a strange pattern here, it seems that on the mac the digits are being transposed from one element of the matrix to another.

Ubuntu:

> fmt <- getOption("nanotimeFormat", default="%Y-%m-%dT%H:%M:%E9S%Ez")
> tz <- getOption("nanotimeTz", default="UTC")
> d <- RcppCCTZ::parseDouble("2015-11-18T05:33:03.765333000+00:00", fmt=fmt, tz=tz)
> d
           [,1]      [,2]
[1,] 1447824783 765333000

Mac:

> fmt <- getOption("nanotimeFormat", default="%Y-%m-%dT%H:%M:%E9S%Ez")
> tz <- getOption("nanotimeTz", default="UTC")
> d <- RcppCCTZ::parseDouble("2015-11-18T05:33:03.765333000+00:00", fmt=fmt, tz=tz)
> d
        [,1]      [,2]
[1,] 1447824 783765333

[eddelbuettel]

Let me dig through my code snippets when I get home and see if we can set up a C++ four-liner we can show to the CCTZ guys -- who are pretty friendly and quick to respond. In the meantime hang on to your Linux box :)

[eddelbuettel]

I am about to head out and already late ... but here is that function. Before we get to CCTZ itself, can you see what you get here for intermediate results? Its from RcppCCTZ.

//' @rdname parseDatetime
// [[Rcpp::export]]
Rcpp::NumericMatrix parseDouble(Rcpp::CharacterVector svec,
                                std::string fmt = "%Y-%m-%dT%H:%M:%E*S%Ez",
                                std::string tzstr = "UTC") {
    cctz::time_zone tz;
    load_time_zone(tzstr, &tz);
    sc::system_clock::time_point tp;

    // Rcpp::Rcout << cctz::format(fmt, tp, tz) << std::endl;

    cctz::time_point<cctz::sys_seconds> unix_epoch =
        sc::time_point_cast<cctz::sys_seconds>(sc::system_clock::from_time_t(0));

    auto n = svec.size();
    Rcpp::NumericMatrix dm(n, 2);
    for (auto i=0; i<n; i++) {
        std::string txt(svec(i));
    
        if (!cctz::parse(fmt, txt, tz, &tp)) Rcpp::stop("Parse error on %s", txt);

        auto nanoseconds = (tp - unix_epoch).count();
        auto secs = nanoseconds / 1000000000;
        auto nanos = nanoseconds - secs * 1000000000;
        //Rcpp::Rcout << nanoseconds << " " << secs << " " << nanos << std::endl;
        dm(i, 0) = secs;
        dm(i, 1) = nanos;
    }
    return dm;
}

[ijlyttle]

I did some digging around on CCTZ issues and found this:

google/cctz#29 (comment)

From this I learned that the Linux clock implementation uses nanoseconds, while OSX uses microseconds.

My guess is that (tp - unix_epoch).count() is returning microseconds on the Mac, which would explain the difference.

FWIW, parsing works as expected on Windows.

[lsilvest]

From the discussion of google/cctz#29, it looks like changing tp to be of type std::chrono::time_point<std::chrono::system_clock, std::chrono::nanoseconds> should work. Btw, unix_epoch is superfluous, one can use directly tp.time_since_epoch().count().

[eddelbuettel]

Good sleuthing, both of you. I guess we have work to do to ensure we actually have nanoseconds on that dreaded OS.

[ijlyttle]

No problem (although I quite like my OS :)). I stand ready to help, and certainly to test whenever needed.

[eddelbuettel]

@lsilvest Hm, on first try that did not compile:

ccache g++ -std=c++11 -I/usr/share/R/include -DNDEBUG   -I"/usr/local/lib/R/site-library/Rcpp/include"  -I../inst/include -fpic  -O3 -Wall -pipe -Wno-unused -pedantic  -c utilities.cpp -o utilities.o
utilities.cpp: In function ‘Rcpp::NumericMatrix parseDouble(Rcpp::CharacterVector, std::__cxx11::string, std::__cxx11::string)’:
utilities.cpp:260:23: error: ‘std::chrono::_V2::system_clock::time_point {aka std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> > >}’ is not a template
     sc::system_clock::time_point<std::chrono::system_clock, std::chrono::nanoseconds> tp;
                       ^~~~~~~~~~
/usr/lib/R/etc/Makeconf:141: recipe for target 'utilities.o' failed
make: *** [utilities.o] Error 1
ERROR: compilation failed for package ‘RcppCCTZ’

Edit: Never mind. Had an extra system_clock in there.

[eddelbuettel]

@ijlyttle so with that, can you test this version please -- it is a drop in for the same function in src/utilities.cpp but now with the two suggestions by @lsilvest :

//' @rdname parseDatetime
// [[Rcpp::export]]
Rcpp::NumericMatrix parseDouble(Rcpp::CharacterVector svec,
                                std::string fmt = "%Y-%m-%dT%H:%M:%E*S%Ez",
                                std::string tzstr = "UTC") {
    cctz::time_zone tz;
    load_time_zone(tzstr, &tz);
    sc::time_point<sc::system_clock, sc::nanoseconds> tp;

    // Rcpp::Rcout << cctz::format(fmt, tp, tz) << std::endl;

    auto n = svec.size();
    Rcpp::NumericMatrix dm(n, 2);
    for (auto i=0; i<n; i++) {
        std::string txt(svec(i));
    
        if (!cctz::parse(fmt, txt, tz, &tp)) Rcpp::stop("Parse error on %s", txt);

        auto nanoseconds = tp.time_since_epoch().count();
        auto secs = nanoseconds / 1000000000;
        auto nanos = nanoseconds - secs * 1000000000;
        //Rcpp::Rcout << nanoseconds << " " << secs << " " << nanos << std::endl;
        dm(i, 0) = secs;
        dm(i, 1) = nanos;
    }
    return dm;
}

[ijlyttle]

I forked RcppCCTZ, patched the function into utilities.cpp, then rebuilt and restarted R.

Unfortunately, I get the same thing:

> nanotime("2015-11-18T05:33:03.765333000+00:00")
[1] "1970-01-17T18:10:24.783765333+00:00"

[eddelbuettel]

I don't have access to OS X so I won't be able to help you. Maybe with some careful reading of the aforementioned thread #29 and some experiments you can get somewhere. Seems like a difference in the C++ library implementation as Apple in their infinite wisdom needs to depart from the g++ family. Sorry. In theory C++11 should have that covered.

[ijlyttle]

I understand - my workaround for now is to use a regular expression to remove "fractional-second" part of the timestamp, then parse to POSIXct, then to nanosecond, then to add the fractional part of of the second. (I hope you are not eating your breakfast right now.)

Kidding aside, I suspect that is not the optimal way to do things. I'll poke around some, maybe @lsilvest has some more ideas.

[eddelbuettel]

Correct. We need to get this squared. My first instinct would be to establish what we would need to do on

• C++(11) only
• then CCTZ
• then RcppCCTZ

but I have no ready-made examples handy. I guess whatever examples I extracted from CCTZ are in RcppCCTZ.

[eddelbuettel]

@ijlyttle Are you using GitHub RcppCCTZ or CRAN? 0.2.0 or 0.2.0.{1,2} ? There was actually another bug fix. Edit: That should be 0.2.1 or 0.2.1.*

Also, I am about to drop an update to upstream CCTZ, first into a branch.

[eddelbuettel]

I had made a minor hash of things leaving a minor commit on the server at home while updating on the laptop while on the train.

RcppCCTZ was 0.2.1 at CRAN.

I have now updated it, reflecting that version and calling it 0.2.1.1 as a release candidate to 0.2.2. Can you test with that, ie RcppCCTZ master as of GH pushed just minutes ago.

[eddelbuettel]

Oh, and one more so master now has the change suggested by Leonardo.

Now we should be all current and good.

[lsilvest]

Ran an R CMD check on these latest versions for both packages on the MacBook and they looked fine.

[eddelbuettel]

Thanks for all your help on this. I'd love to hear from @ijlyttle that it helped him too...

[eddelbuettel]

Nice work, gentlemen. Will push this as 0.2.2 tomorrow or so.

[eddelbuettel]

BTW @lsilvest if one uses R-devel CMD check --as-cran it is bemoaning the lack of registration, but I'll of course add the before I ship.

[eddelbuettel]

I just noticed the Windows regression mentioned by @ijlyttle above. That is new -- 0.2.1 actually built on Windows thanks to pretty heroic backporting by @dcdillon. The CCTZ upstream change may have move a few deck chairs around. I'll need to fix that before I can release :-/