westat-transportation / summarizenhts

"An open-source, survey-specific toolkit capable of processing weighted data, computing common statistics and errors, and producing interactive web visualizations will increase transparency in analyses, eliminate common technical stumbling blocks, and ultimately attract more interest to these powerful datasets." - summarizeNHTS Team

To read more about the National Household Travel Survey, visit this documentation page.

Data

This package handles the downloading, organizing, and loading of NHTS datasets for you. It reads directly from the Oak Ridge National Laboratory NHTS data page, and currently supports the 2001, 2009, and 2017 surveys.

Install

Reference the install readme for specific instructions. We recommend using RStudio Desktop to develop analyses with this package. Your computer should have at least 8GB of memory and a recent 64 bit version of R.

install.packages('devtools')
devtools::install_github('Westat-Transportation/summarizeNHTS')

Demo

library(summarizeNHTS)
download_nhts_data("2017", exdir="C:/NHTS")
dataset <- read_data("2017", csv_path="C:/NHTS")
statistic <- summarize_data(
    data = dataset,
    agg = "household_count",
    by = c("HHSIZE","HHVEHCNT")
)
make_chart(statistic)

Extended Demo

Review our tutorial workshop, Exploring the NHTS in R, for more guidance on using this package.

Trend Analysis

Analyze trends from year-to-year with statistical confidence using a multi-year report generator that utilizes summarizeNHTS. Review the demo report and generate your own using the program available here.

summarizenhts's Issues

read_data error: MSACAT

When reading in the 2009 data, I get an error (attached) about the MSACAT object not found. I searched the .csv files and found that field on the non-weights files. It looks like this error is coming from the "derived_variable_config.csv" file Cell D2. Do you have suggestions?

MSACAT_error.txt

Issue in installation

I try to install it on my Windows 10 laptop, but I found the error when I ran the following code in Rstudio.

devtools::install_github('Westat-Transportation/summarizeNHTS')

Downloading GitHub repo Westat-Transportation/summarizeNHTS@master
✓  checking for file 'C:\Users\rqbb\AppData\Local\Temp\RtmpqSBa8H\remotes2a8420e70f8\Westat-Transportation-summarizeNHTS-02ceacc/DESCRIPTION' (359ms)
─  preparing 'summarizeNHTS': (1.7s)
✓  checking DESCRIPTION meta-information ... 
─  cleaning src
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  looking to see if a 'data/datalist' file should be added
─  building 'summarizeNHTS_0.0.0.9000.tar.gz' (1.3s)
   Warning in utils::tar(filepath, pkgname, compression = compression, compression_level = 9L,  :
     storing paths of more than 100 bytes is not portable:
     'summarizeNHTS/inst/multi_year_report_generator/derived variable files/2001/derived_variable_config.csv'
   Warning in utils::tar(filepath, pkgname, compression = compression, compression_level = 9L,  :
     storing paths of more than 100 bytes is not portable:
     'summarizeNHTS/inst/multi_year_report_generator/derived variable files/2009/derived_variable_config.csv'
   Warning in utils::tar(filepath, pkgname, compression = compression, compression_level = 9L,  :
     storing paths of more than 100 bytes is not portable:
     'summarizeNHTS/inst/multi_year_report_generator/derived variable files/2017/derived_variable_config.csv'
   Warning in utils::tar(filepath, pkgname, compression = compression, compression_level = 9L,  :
     storing paths of more than 100 bytes is not portable:
     'summarizeNHTS/inst/tutorials/workshop/resources/derived_variables/derived_variable_config(example).csv'
   
Installing package into ‘C:/Users/rqbb/Documents/R/win-library/3.6’
(as ‘lib’ is unspecified)
* installing *source* package 'summarizeNHTS' ...
** using staged installation
** libs

*** arch - i386
c:/Rtools/mingw_32/bin/g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-36~1.2/include" -DNDEBUG  -I"C:/Program Files/R/R-3.6.2/library/Rcpp/include"        -O2 -Wall  -mtune=generic -c RcppExports.cpp -o RcppExports.o
c:/Rtools/mingw_32/bin/g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-36~1.2/include" -DNDEBUG  -I"C:/Program Files/R/R-3.6.2/library/Rcpp/include"        -O2 -Wall  -mtune=generic -c cpp_helper_functions.cpp -o cpp_helper_functions.o
cpp_helper_functions.cpp: In function 'double Rcpp_wgtmed(Rcpp::NumericVector, Rcpp::NumericVector)':
cpp_helper_functions.cpp:21:59: error: no matching function for call to 'make_pair(Rcpp::traits::storage_type<14>::type&, Rcpp::traits::storage_type<14>::type&)'
     vals[i] = std::make_pair<double, double>(x[i], wgts[i]);
                                                           ^
cpp_helper_functions.cpp:21:59: note: candidate is:
In file included from C:/Rtools/mingw_32/i686-w64-mingw32/include/c++/utility:70:0,
                 from C:/Rtools/mingw_32/i686-w64-mingw32/include/c++/unordered_map:38,
                 from C:/Program Files/R/R-3.6.2/library/Rcpp/include/Rcpp/platform/compiler.h:153,
                 from C:/Program Files/R/R-3.6.2/library/Rcpp/include/Rcpp/r/headers.h:67,
                 from C:/Program Files/R/R-3.6.2/library/Rcpp/include/RcppCommon.h:29,
                 from C:/Program Files/R/R-3.6.2/library/Rcpp/include/Rcpp.h:27,
                 from cpp_helper_functions.cpp:1:
C:/Rtools/mingw_32/i686-w64-mingw32/include/c++/bits/stl_pair.h:276:5: note: template<class _T1, class _T2> constexpr std::pair<typename std::__decay_and_strip<_Tp>::__type, typename std::__decay_and_strip<_T2>::__type> std::make_pair(_T1&&, _T2&&)
     make_pair(_T1&& __x, _T2&& __y)
     ^
C:/Rtools/mingw_32/i686-w64-mingw32/include/c++/bits/stl_pair.h:276:5: note:   template argument deduction/substitution failed:
cpp_helper_functions.cpp:21:59: note:   cannot convert 'x.Rcpp::Vector<RTYPE, StoragePolicy>::operator[]<14, Rcpp::PreserveStorage>(i)' (type 'Rcpp::traits::storage_type<14>::type {aka double}') to type 'double&&'
     vals[i] = std::make_pair<double, double>(x[i], wgts[i]);
                                                           ^
make: *** [C:/PROGRA~1/R/R-36~1.2/etc/i386/Makeconf:215: cpp_helper_functions.o] Error 1
ERROR: compilation failed for package 'summarizeNHTS'
* removing 'C:/Users/rqbb/Documents/R/win-library/3.6/summarizeNHTS'
error: Failed to install 'summarizeNHTS' from GitHub:
installation of package ‘C:/Users/rqbb/AppData/Local/Temp/RtmpqSBa8H/file2a8472186f1/summarizeNHTS_0.0.0.9000.tar.gz’ had non-zero exit status

I am sure that I followed the Install Instructions, installed the latest version of R, Rtools and Rstudio.
I don't know why it doesn't work.
My OS is Windows 10 professional, version 1903.
I will be very appreciate if you could provide any help.

Install Error

Install Error with 3.4.3 and stringi -

library(summarizeNHTS)
Loading required package: ggplot2
Error: package or namespace load failed for ‘summarizeNHTS’ in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]):
there is no package called ‘stringi’
In addition: Warning message:
package ‘ggplot2’ was built under R version 3.4.4

install.packages("summarizeNHTS")
Installing package into ‘C:/Users/DOTJEM/Documents/R/win-library/3.4’
(as ‘lib’ is unspecified)
Warning in install.packages :
package ‘summarizeNHTS’ is not available (for R version 3.4.3)

Addon agency issue running with 5day weights

Hi - when running the tool using the INCOG 5-day weight in the trip file, the results are consistently 1.44 times the weighted results using the ORNL Data Explorer or using SPSS. The % distribution is the same but the actual numbers (sums) are higher. Attached is a summary excel file.
NHTS2017_R_Output sbcomparison.xlsx. Is there something in the code that is causing this?

prepare_add_on_files(
directory = "C:/Data/R/R_New/csv/2017",
weight_type = "5day"
)

nhts_data <- read_data(
dataset = "2017",
csv_path = "C:/Data/R/R_New"
)

Missing variables in 2017 data

Hello,
I've been using the library to do some analysis on the 2017 data and it is working great aside from a few missing variables. I am particularly interested in the questions related to why respondents don't bike or walk more.

Here is a list of variables that appear in the data dictionary and csvs, but not the data imported using read_data (not counting ID and weight columns):

ALT_16 | Alternative Mode of Transportation: Public Transportation or Taxi
ALT_23 | Alternative Mode of Transportation: Passenger to Friend/Family Member or Rental Car
ALT_45 | Alternative Mode of Transportation: Bicycle or Walk
BIKE_DFR | Reason for Not Biking More: Infrastructure
BIKE_GKP | Reason for Not Biking More: Safety
SMPLSRCE | Sample where the case originated
TDCASEID | Unique identifier for every trip record in the file
TDTRPNUM | Incrementing travel day trip number, starting at 1 for each person in the file
TRACC_BUS | Bus as mode used to get to public transit?
TRACC_CRL | Rail as mode used to get to public transit?
TRACC_OTH | Other mode used to get to public transit?
TRACC_POV | POV as mode used to get to public transit?
TRACC_SUB | Subway as mode used to get to public transit?
TRACC_WLK | Walk as mode used to get to public transit?
TREGR_BUS | Bus as mode used to get from public transit?
TREGR_CRL | Rail as mode used to get from public transit?
TREGR_OTH | Other mode used to get from public transit?
TREGR_POV | POV as mode used to get from public transit?
TREGR_SUB | Subway as mode used to get from public transit?
TREGR_WLK | Walk as mode used to get from public transit?
WALK_DEF | Reason for Not Walking More: Infrastructure
WALK_GKQ | Reason for Not Walking More: Safety

If it's an error on my end, I'm happy to provide more code or information.

Thanks!

Maggie

3 Weird Characters and

This happens before I run the following step in prepping the data

nhts_data <- read_data (‘2017’)
For example in the household.csv, column A should read HOUSEID, not Ï»¿HOUSEID. The household, person, and trip csv files all had the 3 weird characters in the first column name HOUSEID. The other files (household_weights.csv, location.csv, person_weights.csv, and vehicle.csv) did not have these weird characters.

Later when looking at the person and trip files, I have noticed that the number of observations and weighted statistic are not OK.

Incorrect variable labels in 2017 codebook

In the package's internal codebook for the 2017 NHTS, the variable labels for the "medical condition results in..." variables (CONDNIGH, CONDPUB, CONDRIVE, CONDRIDE, CONDSPEC, CONDTAX, CONDTRAV) appear to be incorrect. When I use summarize_data(), they all return labels with "Reduced [$YOUR2] day-to-day travel" (for code "01") or "Asked others for rides" (for code "02"). As a result, in the case of CONDRIDE, "asked others for rides" appears to have the opposite meaning.

EDIT: I'm using the latest version of summarizeNHTS (0.0.0.9000).

Code to reproduce the issue:

library(summarizeNHTS)

download_nhts_data("2017")
nhts2017 <- read_data("2017")

summarize_data(
  data = nhts2017,
  agg = "person_count",
  by = c("CONDNIGH")
)

summarize_data(
  data = nhts2017,
  agg = "person_count",
  by = c("CONDPUB")
)

summarize_data(
  data = nhts2017,
  agg = "person_count",
  by = c("CONDRIDE")
)

summarize_data(
  data = nhts2017,
  agg = "person_count",
  by = c("CONDRIVE")
)

summarize_data(
  data = nhts2017,
  agg = "person_count",
  by = c("CONDSPEC")
)

summarize_data(
  data = nhts2017,
  agg = "person_count",
  by = c("CONDTAX")
)

summarize_data(
  data = nhts2017,
  agg = "person_count",
  by = c("CONDTRAV")
)

Issue in installation with R 3.6 in Conda environment

Hi. I am trying to install the package in my conda environment on an Archlinux machine.
The build tools is failing and following is the relevant stacktrace.

Could it be an Rtools issue? Conda has updated to R 3.6 and I saw some workarounds to get Rtools for 3.6.

Your help will be deeply appreciated.
I intend to call the methods via Rpy2 to export the data into Pandas dataframes.

cpp_helper_functions.cpp: In function 'double Rcpp_wgtmed(Rcpp::NumericVector, Rcpp::NumericVector)':
cpp_helper_functions.cpp:21:59: error: no matching function for call to 'make_pair<double, double>(Rcpp::traits::storage_type<14>::type&, Rcpp::traits::storage_type<14>::type&)'
     vals[i] = std::make_pair<double, double>(x[i], wgts[i]);
                                                           ^
In file included from /home/darshan/miniconda3/envs/nhts-env/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/utility:70:0,
                 from /home/darshan/miniconda3/envs/nhts-env/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/unordered_map:38,
                 from /home/darshan/miniconda3/envs/nhts-env/lib/R/library/Rcpp/include/Rcpp/platform/compiler.h:162,
                 from /home/darshan/miniconda3/envs/nhts-env/lib/R/library/Rcpp/include/Rcpp/r/headers.h:59,
                 from /home/darshan/miniconda3/envs/nhts-env/lib/R/library/Rcpp/include/RcppCommon.h:29,
                 from /home/darshan/miniconda3/envs/nhts-env/lib/R/library/Rcpp/include/Rcpp.h:27,
                 from cpp_helper_functions.cpp:1:
/home/darshan/miniconda3/envs/nhts-env/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/bits/stl_pair.h:519:5: note: candidate: template<class _T1, class _T2> constexpr std::pair<typename std::__decay_and_strip<_Tp>::__type, typename std::__decay_and_strip<_T2>::__type> std::make_pair(_T1&&, _T2&&)
     make_pair(_T1&& __x, _T2&& __y)
     ^~~~~~~~~
/home/darshan/miniconda3/envs/nhts-env/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/bits/stl_pair.h:519:5: note:   template argument deduction/substitution failed:
cpp_helper_functions.cpp:21:59: note:   cannot convert 'x.Rcpp::Vector<14, Rcpp::PreserveStorage>::operator[](((R_xlen_t)i))' (type 'Rcpp::traits::storage_type<14>::type {aka double}') to type 'double&&'
     vals[i] = std::make_pair<double, double>(x[i], wgts[i]);
                                                           ^
make: *** [/home/darshan/miniconda3/envs/nhts-env/lib/R/etc/Makeconf:175: cpp_helper_functions.o] Error 1
ERROR: compilation failed for package ‘summarizeNHTS’

Error in fread

Hi Anthony, Alex,

I am trying to use read_data function of summerizeNHTS. But it is giving me following error -

Error in fread(input = file.path(self$path, table_csv), select = c(table_key, :
File 'C:/Data/R/Project_Test_1/R_test_1/csv/2017/person_weights.csv' does not exist. Include one or more spaces to consider the input a system command.

When I change the name of the file in csv folder to person_weights.csv, it gives me a different error -

Error in fread(input = file.path(self$path, table_csv), select = c(table_key, :
Column name 'HOUSEID' in colClasses[[1]] not found

Is your read_data function has something hard coded in it or why I am getting these errors?

Thank you,
Nimish

summarizeNHTS

I can't load the summarizeNHTS.

I was trying " devtools::install_github('Westat-Transportation/summarizeNHTS') " but got following error: Is it related to data download or something else ?

Downloading GitHub repo Westat-Transportation/summarizeNHTS@master
from URL https://api.github.com/repos/Westat-Transportation/summarizeNHTS/zipball/master
Installing summarizeNHTS
"C:/PROGRA1/R/R-341.3/bin/x64/R" --no-site-file --no-environ --no-save --no-restore --quiet CMD INSTALL
"C:/Users/ndharmadhikari/AppData/Local/Temp/RtmpMFK3Q3/devtools2d80322f2463/Westat-Transportation-summarizeNHTS-7bccbaf"
--library="\server-file/users/ndharmadhikari/R/win-library/3.4" --install-tests

• installing source package 'summarizeNHTS' ...
  ** libs

*** arch - i386
C:/RBuildTools/3.4/mingw_32/bin/g++ -I"C:/PROGRA1/R/R-341.3/include" -DNDEBUG -I"\server-file/users/ndharmadhikari/R/win-library/3.4/Rcpp/include" -O2 -Wall -mtune=generic -c RcppExports.cpp -o RcppExports.o
RcppExports.cpp:4:18: fatal error: Rcpp.h: No such file or directory
#include <Rcpp.h>
^
compilation terminated.
make: *** [RcppExports.o] Error 1
Warning: running command 'make -f "C:/PROGRA1/R/R-341.3/etc/i386/Makeconf" -f "C:/PROGRA1/R/R-341.3/share/make/winshlib.mk" SHLIB_LDFLAGS='$(SHLIB_CXXLDFLAGS)' SHLIB_LD='$(SHLIB_CXXLD)' SHLIB="summarizeNHTS.dll" OBJECTS="RcppExports.o cpp_helper_functions.o"' had status 2
ERROR: compilation failed for package 'summarizeNHTS'

• removing '\server-file/users/ndharmadhikari/R/win-library/3.4/summarizeNHTS'
  In R CMD INSTALL
  Installation failed: Command failed (1)