ODBC getting datetime from SQL arriving in wrong time zone about odbc

[dancrocker]

The devtools version resolved the timezone issue when connecting to an MS Access db as well. My system time and data tz are both America/New_York, and the data has the proper tz now.

con <- dbConnect(odbc::odbc(),
                     .connection_string = "driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:/WQDatabase/WaterQualityDB_fe.mdb;Uid=Admin;Pwd=;",
timezone = "America/New_York")
                   )
storms <- dbReadTable(con,"qryStormSamples") # data comes in as UTC
with_tz(c(storms$dt_start, storms$dt_end), tzone = "America/New_York") # update to local time

It would be great if these changes are carried forward to the CRAN version of the package.

Thanks

[tgb417]

This looks to be the same as issue #37

[tgb417]

Unfortunately, I'm getting the following error when trying to install the version from

devtools::install_github("rstats-db/odbc")

• installing source package 'odbc' ...
  ** libs
  C:/RBuildTools/3.4/mingw_64/bin/g++ -std=c++0x -I"C:/PROGRA1/R/R-331.3/include" -DNDEBUG -I"C:/Program Files/R/R-3.3.3/library/Rcpp/include" -I"C:/Program Files/R/R-3.3.3/library/BH/include" -I"d:/Compiler/gcc-4.9.3/local330/include" -I. -DNANODBC_USE_BOOST_CONVERT -Icctz/include -O2 -Wall -mtune=core2 -c connection.cpp -o connection.o
  connection.cpp:3:21: fatal error: nanodbc.h: No such file or directory
  #include "nanodbc.h"
  ^
  compilation terminated.
  make: *** [connection.o] Error 1
  Warning: running command 'make -f "Makevars.win" -f "C:/PROGRA1/R/R-331.3/etc/x64/Makeconf" -f "C:/PROGRA1/R/R-331.3/share/make/winshlib.mk" CXX='$(CXX1X) $(CXX1XSTD)' CXXFLAGS='$(CXX1XFLAGS)' CXXPICFLAGS='$(CXX1XPICFLAGS)' SHLIB_LDFLAGS='$(SHLIB_CXX1XLDFLAGS)' SHLIB_LD='$(SHLIB_CXX1XLD)' SHLIB="odbc.dll" WIN=64 TCLBIN=64 ' had status 2
  ERROR: compilation failed for package 'odbc'
• removing 'C:/Program Files/R/R-3.3.3/library/odbc'
• restoring previous 'C:/Program Files/R/R-3.3.3/library/odbc'
  Error: Command failed (1)

[jimhester]

@tgb417 Just pushed a commit that should fix the windows compilation errors (a9fd87d) sorry it was broken temporarily.

[tgb417]

Cool. Will look at it in an hour or two. Will comment back.

…

--Tom On Mon, Apr 24, 2017 at 10:36 AM, Jim Hester <[email protected]> wrote: @tgb417 [https://github.com/tgb417] Just pushed a commit that should fix the windows compilation errors ( a9fd87d [a9fd87d] ) sorry it was broken temporarily. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [#54 (comment)] , or mute the thread [https://github.com/notifications/unsubscribe-auth/AIL89iYv9imK24q5HB2ds68zqsamJrkEks5rzLNpgaJpZM4NE2Xu] .

[tgb417]

Between my comments two hours ago and now I've installed R 3.4.

R version 3.4.0 (2017-04-21) -- "You Stupid Darkness" Copyright (C) 2017 The R Foundation for Statistical Computing Platform: x86_64-w64-mingw32/x64 (64-bit)

Thinking that R.Version might be the issue.

Then I got your note about and update. And I just tried to run:

devtools::install_github("rstats-db/odbc")

I ended up with a different set of errors that lead to an inability to install.

`installing to C:/Program Files/R/R-3.4.0/library/odbc/libs/x64
** R
** inst
** tests
** preparing package for lazy loading
Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]) :
there is no package called 'bit'
ERROR: lazy loading failed for package 'odbc'

• removing 'C:/Program Files/R/R-3.4.0/library/odbc'
  Error: Command failed (1)`

Looks like the R package 'bit' was not installed in my new 3.4 environment.

Once I

install.packages('bit')

and re-ran,

devtools::install_github("rstats-db/odbc")

the code seemed to install. There were a number of warnings "ERROR" in the process so, I'm including some more of the logs here.

Will test to see if this resolves my issue with the time zone as described in #37 .

/include"     -O2 -Wall  -mtune=core2 -c barrier.cpp -o barrier.o
C:/RBuildTools/3.4/mingw_64/bin/g++ -shared -s -static-libgcc -o Rcpp.dll tmp.def Date.o Module.o Rcpp_init.o api.o attributes.o barrier.o -Ld:/Compiler/gcc-4.9.3/local330/lib/x64 -Ld:/Compiler/gcc-4.9.3/local330/lib -LC:/PROGRA~1/R/R-34~1.0/bin/x64 -lR
installing to C:/Program Files/R/R-3.4.0/library/Rcpp/libs/x64
** R
** inst
** tests
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
*** arch - i386
*** arch - x64
* DONE (Rcpp)
"C:/PROGRA~1/R/R-34~1.0/bin/x64/R" --no-site-file --no-environ --no-save --no-restore --quiet CMD INSTALL  \
  "C:/Users/tbrown/AppData/Local/Temp/RtmpQFU9DT/devtools1e2c572744d4/rstats-db-odbc-a9fd87d" --library="C:/Program  \
  Files/R/R-3.4.0/library" --install-tests 

* installing *source* package 'odbc' ...
** libs

*** arch - i386
(cd cctz && \
make libcctz.a PREFIX="../" CC="C:/RBuildTools/3.4/mingw_32/bin/gcc " CXX="C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11" AR="C:/RBuildTools/3.4/mingw_32/bin/ar" ARFLAGS=rv)
make[1]: Entering directory `/cygdrive/c/Users/tbrown/AppData/Local/Temp/RtmpQFU9DT/devtools1e2c572744d4/rstats-db-odbc-a9fd87d/src-i386/cctz'
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_format.o src/time_zone_format.cc
src/time_zone_format.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_if.o src/time_zone_if.cc
src/time_zone_if.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_impl.o src/time_zone_impl.cc
src/time_zone_impl.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_info.o src/time_zone_info.cc
src/time_zone_info.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_libc.o src/time_zone_libc.cc
src/time_zone_libc.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_lookup.o src/time_zone_lookup.cc
src/time_zone_lookup.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_posix.o src/time_zone_posix.cc
src/time_zone_posix.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_32/bin/ar rv libcctz.a time_zone_format.o time_zone_if.o time_zone_impl.o time_zone_info.o time_zone_libc.o time_zone_lookup.o time_zone_posix.o
a - time_zone_format.o
a - time_zone_if.o
a - time_zone_impl.o
a - time_zone_info.o
a - time_zone_libc.o
a - time_zone_lookup.o
a - time_zone_posix.o
C:\RBuildTools\3.4\mingw_32\bin\ar.exe: creating libcctz.a
make[1]: Leaving directory `/cygdrive/c/Users/tbrown/AppData/Local/Temp/RtmpQFU9DT/devtools1e2c572744d4/rstats-db-odbc-a9fd87d/src-i386/cctz'
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11 -I"C:/PROGRA~1/R/R-34~1.0/include" -DNDEBUG  -I"C:/Program Files/R/R-3.4.0/library/Rcpp/include" -I"C:/Program Files/R/R-3.4.0/library/BH/include"   -I"d:/Compiler/gcc-4.9.3/local330/include"  -I. -DNANODBC_USE_BOOST_CONVERT -Icctz/include -Inanodbc   -O2 -Wall  -mtune=core2 -c connection.cpp -o connection.o
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11 -I"C:/PROGRA~1/R/R-34~1.0/include" -DNDEBUG  -I"C:/Program Files/R/R-3.4.0/library/Rcpp/include" -I"C:/Program Files/R/R-3.4.0/library/BH/include"   -I"d:/Compiler/gcc-4.9.3/local330/include"  -I. -DNANODBC_USE_BOOST_CONVERT -Icctz/include -Inanodbc   -O2 -Wall  -mtune=core2 -c nanodbc/nanodbc.cpp -o nanodbc.o
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11 -I"C:/PROGRA~1/R/R-34~1.0/include" -DNDEBUG  -I"C:/Program Files/R/R-3.4.0/library/Rcpp/include" -I"C:/Program Files/R/R-3.4.0/library/BH/include"   -I"d:/Compiler/gcc-4.9.3/local330/include"  -I. -DNANODBC_USE_BOOST_CONVERT -Icctz/include -Inanodbc   -O2 -Wall  -mtune=core2 -c result.cpp -o result.o
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11 -I"C:/PROGRA~1/R/R-34~1.0/include" -DNDEBUG  -I"C:/Program Files/R/R-3.4.0/library/Rcpp/include" -I"C:/Program Files/R/R-3.4.0/library/BH/include"   -I"d:/Compiler/gcc-4.9.3/local330/include"  -I. -DNANODBC_USE_BOOST_CONVERT -Icctz/include -Inanodbc   -O2 -Wall  -mtune=core2 -c odbc_connection.cpp -o odbc_connection.o
In file included from C:/PROGRA~1/R/R-34~1.0/include/R.h:91:0,
                 from C:/Program Files/R/R-3.4.0/library/Rcpp/include/Rcpp/r/headers.h:52,
                 from C:/Program Files/R/R-3.4.0/library/Rcpp/include/RcppCommon.h:29,
                 from C:/Program Files/R/R-3.4.0/library/Rcpp/include/Rcpp.h:27,
                 from odbc_connection.h:6,
                 from odbc_connection.cpp:1:
C:/PROGRA~1/R/R-34~1.0/include/R_ext/RS.h:55:0: warning: "ERROR" redefined
 #define ERROR   ),error(R_problem_buf);}
 ^
In file included from C:/RBuildTools/3.4/mingw_32/i686-w64-mingw32/include/windows.h:71:0,
                 from sql_types.h:6,
                 from odbc_connection.h:4,
                 from odbc_connection.cpp:1:
C:/RBuildTools/3.4/mingw_32/i686-w64-mingw32/include/wingdi.h:75:0: note: this is the location of the previous definition
 #define ERROR 0
 ^
C:/RBuildTools/3.4/mingw_32/bin/g++  -std=gnu++11 -I"C:/PROGRA~1/R/R-34~1.0/include" -DNDEBUG  -I"C:/Program Files/R/R-3.4.0/library/Rcpp/include" -I"C:/Program Files/R/R-3.4.0/library/BH/include"   -I"d:/Compiler/gcc-4.9.3/local330/include"  -I. -DNANODBC_USE_BOOST_CONVERT -Icctz/include -Inanodbc   -O2 -Wall  -mtune=core2 -c RcppExports.cpp -o RcppExports.o
C:/RBuildTools/3.4/mingw_32/bin/g++ -shared -s -static-libgcc -o odbc.dll tmp.def connection.o nanodbc.o result.o odbc_connection.o RcppExports.o -lodbc32 -Lcctz -lcctz -Ld:/Compiler/gcc-4.9.3/local330/lib/i386 -Ld:/Compiler/gcc-4.9.3/local330/lib -LC:/PROGRA~1/R/R-34~1.0/bin/i386 -lR
installing to C:/Program Files/R/R-3.4.0/library/odbc/libs/i386

*** arch - x64
(cd cctz && \
make libcctz.a PREFIX="../" CC="C:/RBuildTools/3.4/mingw_64/bin/gcc " CXX="C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11" AR="C:/RBuildTools/3.4/mingw_64/bin/ar" ARFLAGS=rv)
make[1]: Entering directory `/cygdrive/c/Users/tbrown/AppData/Local/Temp/RtmpQFU9DT/devtools1e2c572744d4/rstats-db-odbc-a9fd87d/src-x64/cctz'
C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_format.o src/time_zone_format.cc
src/time_zone_format.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_if.o src/time_zone_if.cc
src/time_zone_if.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_impl.o src/time_zone_impl.cc
src/time_zone_impl.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_info.o src/time_zone_info.cc
src/time_zone_info.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_libc.o src/time_zone_libc.cc
src/time_zone_libc.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_lookup.o src/time_zone_lookup.cc
src/time_zone_lookup.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11  -Wall -Iinclude -std=c++11 -pthread  -O -fPIC -MMD  -c -o time_zone_posix.o src/time_zone_posix.cc
src/time_zone_posix.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc. All Rights Reserved.
 ^
C:/RBuildTools/3.4/mingw_64/bin/ar rv libcctz.a time_zone_format.o time_zone_if.o time_zone_impl.o time_zone_info.o time_zone_libc.o time_zone_lookup.o time_zone_posix.o
a - time_zone_format.o
a - time_zone_if.o
a - time_zone_impl.o
a - time_zone_info.o
a - time_zone_libc.o
a - time_zone_lookup.o
a - time_zone_posix.o
C:\RBuildTools\3.4\mingw_64\bin\ar.exe: creating libcctz.a
make[1]: Leaving directory `/cygdrive/c/Users/tbrown/AppData/Local/Temp/RtmpQFU9DT/devtools1e2c572744d4/rstats-db-odbc-a9fd87d/src-x64/cctz'
C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11 -I"C:/PROGRA~1/R/R-34~1.0/include" -DNDEBUG  -I"C:/Program Files/R/R-3.4.0/library/Rcpp/include" -I"C:/Program Files/R/R-3.4.0/library/BH/include"   -I"d:/Compiler/gcc-4.9.3/local330/include"  -I. -DNANODBC_USE_BOOST_CONVERT -Icctz/include -Inanodbc   -O2 -Wall  -mtune=core2 -c connection.cpp -o connection.o
C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11 -I"C:/PROGRA~1/R/R-34~1.0/include" -DNDEBUG  -I"C:/Program Files/R/R-3.4.0/library/Rcpp/include" -I"C:/Program Files/R/R-3.4.0/library/BH/include"   -I"d:/Compiler/gcc-4.9.3/local330/include"  -I. -DNANODBC_USE_BOOST_CONVERT -Icctz/include -Inanodbc   -O2 -Wall  -mtune=core2 -c nanodbc/nanodbc.cpp -o nanodbc.o
C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11 -I"C:/PROGRA~1/R/R-34~1.0/include" -DNDEBUG  -I"C:/Program Files/R/R-3.4.0/library/Rcpp/include" -I"C:/Program Files/R/R-3.4.0/library/BH/include"   -I"d:/Compiler/gcc-4.9.3/local330/include"  -I. -DNANODBC_USE_BOOST_CONVERT -Icctz/include -Inanodbc   -O2 -Wall  -mtune=core2 -c result.cpp -o result.o
C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11 -I"C:/PROGRA~1/R/R-34~1.0/include" -DNDEBUG  -I"C:/Program Files/R/R-3.4.0/library/Rcpp/include" -I"C:/Program Files/R/R-3.4.0/library/BH/include"   -I"d:/Compiler/gcc-4.9.3/local330/include"  -I. -DNANODBC_USE_BOOST_CONVERT -Icctz/include -Inanodbc   -O2 -Wall  -mtune=core2 -c odbc_connection.cpp -o odbc_connection.o
In file included from C:/PROGRA~1/R/R-34~1.0/include/R.h:91:0,
                 from C:/Program Files/R/R-3.4.0/library/Rcpp/include/Rcpp/r/headers.h:52,
                 from C:/Program Files/R/R-3.4.0/library/Rcpp/include/RcppCommon.h:29,
                 from C:/Program Files/R/R-3.4.0/library/Rcpp/include/Rcpp.h:27,
                 from odbc_connection.h:6,
                 from odbc_connection.cpp:1:
C:/PROGRA~1/R/R-34~1.0/include/R_ext/RS.h:55:0: warning: "ERROR" redefined
 #define ERROR   ),error(R_problem_buf);}
 ^
In file included from C:/RBuildTools/3.4/mingw_64/x86_64-w64-mingw32/include/windows.h:71:0,
                 from sql_types.h:6,
                 from odbc_connection.h:4,
                 from odbc_connection.cpp:1:
C:/RBuildTools/3.4/mingw_64/x86_64-w64-mingw32/include/wingdi.h:75:0: note: this is the location of the previous definition
 #define ERROR 0
 ^
C:/RBuildTools/3.4/mingw_64/bin/g++  -std=gnu++11 -I"C:/PROGRA~1/R/R-34~1.0/include" -DNDEBUG  -I"C:/Program Files/R/R-3.4.0/library/Rcpp/include" -I"C:/Program Files/R/R-3.4.0/library/BH/include"   -I"d:/Compiler/gcc-4.9.3/local330/include"  -I. -DNANODBC_USE_BOOST_CONVERT -Icctz/include -Inanodbc   -O2 -Wall  -mtune=core2 -c RcppExports.cpp -o RcppExports.o
C:/RBuildTools/3.4/mingw_64/bin/g++ -shared -s -static-libgcc -o odbc.dll tmp.def connection.o nanodbc.o result.o odbc_connection.o RcppExports.o -lodbc32 -Lcctz -lcctz -Ld:/Compiler/gcc-4.9.3/local330/lib/x64 -Ld:/Compiler/gcc-4.9.3/local330/lib -LC:/PROGRA~1/R/R-34~1.0/bin/x64 -lR
installing to C:/Program Files/R/R-3.4.0/library/odbc/libs/x64
** R
** inst
** tests
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
*** arch - i386
*** arch - x64
* DONE (odbc)

[tgb417]

So, once I got the dev version installed. I was able to

con <- DBI::dbConnect(odbc::odbc(),
                     driver = "{SQL Server Native Client 11.0}",
                     server = "myserver",
                     database = "mydatabase",
                     Trusted_Connection = "yes",
                     timezone = "America/New_York")

then

orders <-
con %>% 
  dbReadTable("Order_Sum_table" ) %>%
  head

then as advertised, the times in the returned date were really UTC.

But simply doing a:

with_tz(orders$order_dt_only,tzone = "America/New_York")

showed the right results for NYC. (This makes a lot more sense than what was happening.)

I don't know how to do a feature request. But...

It would be great if dbConnect would read:

> Sys.timezone()
[1] "America/New_York"

as default for the timezone parameter of dbConnect(), Then only make folks have to set another value if the SQL servers happen to be in a different time zone from the workstation in question.

It would also be great if dbReadTable() did the same kind of thing changing the date time results stored to Local time by default. Again only causing someone to use a parameter if they had to change the timezone to something other than the local timezone. Just a thought. This would lead to a way more intuitive experience for new users like myself. And allow the power to override when it becomes necessary in a very flexible ways.

Thanks for the help. I'm now going to move along to the analysis I'm trying to do.

[jimhester]

Using the default of Sys.timezone() was done deliberately, as it makes the results non-reproducible across different default timezones. The help for ?dbConnect mentions using Sys.timezone()

The Server Time Zone. Useful if the database has an internal timezone that is not 'UTC'. If the database is in your local timezone set to Sys.timezone(). See OlsonNames() for a complete list of available timezones on your system.