janmarvin / readspss

Read the SPSS file formats

readspss

R package using Rcpp to parse an SPSS file into a data.frame(). Currently read.sav and read.por are the main functions and feature of this package. Writing of SPSS files is provided by write.por and write.sav. Writing is limited to uncompressed por and sav files and to compressed sav file.

It works. Its read function is extensively tested on approximately 600+ sav-files and ~100 por-files. The code is maturing and is frequently tested. The read functions imports everything into a data frame. Including long strings and labels. Various features such as importing of value label or missings are tested and are working as intended.

The package features reading of

• sav files,
• zsav files,
• encrypted sav files and
• por files

and (experimental) writing support of (un)compressed

• sav files and
• zsav files and
• por files.

Because of the R code wrapped Rcpp functions the package is pretty fast. The R code for factor conversion slows things down a bit, changing the encoding a bit more.

In comparison to haven and foreign this package preforms pretty well. It reads more files than each of its predecessors, some are only readable using readspss and it covers a few more cases of missing values.

Focus was not so much on winning every benchmark, but reading all features of an SPSS file and to be as exactly as possible. So some benchmarks are won and others are lost. It is entirely up to the task. Besides the data itself readspss ships additional information provided by the SPSS files like the data label, documentation, date and timestamp.

Reading of sav and por files is considered feature complete.

Writing of (un)compressed sav and por files is implemented and considered working. Unsupported features are reading and writing of dates (might no longer be true) and writing of long strings.

Installation

With remotes:

remotes::install_github("JanMarvin/readspss")

Or from r-universe with:

options(repos = c(
  janmarvin = 'https://janmarvin.r-universe.dev',
  CRAN = 'https://cloud.r-project.org'))
install.packages('readspss')

Usage

fls <- system.file("extdata", "electric.sav", package = "readspss")
flp <- system.file("extdata", "electric.por", package = "readspss")

df_s <- read.sav(fls)
df_p <- read.por(flp)

all.equal(df_s, df_p, check.attributes = FALSE)
#> [1] TRUE

readspss's Issues

Error install in windows

install using
devtools::install_git("https://github.com/JanMarvin/readspss.git")

error obtain for Rtools, using R version 3.2.2 and R-devel 4.0

WARNING: Rtools is required to build R packages, but is not currently installed.

Please download and install Rtools custom from http://cran.r-project.org/bin/windows/Rtools/.
Downloading git repo https://github.com/JanMarvin/readspss.git
WARNING: Rtools is required to build R packages, but is not currently installed.

Please download and install Rtools custom from http://cran.r-project.org/bin/windows/Rtools/.
Error: Failed to install 'unknown package' from Git:
  Could not find tools necessary to compile a package
Call `pkgbuild::check_build_tools(debug = TRUE)` to diagnose the problem.

Mimic SPSS when writing missings for uncompressed sav-files

As the title says. This would help PSPP to handle such files. SPSS itself does not bother with our sav-files as one can see here. Otherwise the issue it not urgent.

dd <- data.frame(x = 1, y = NA)

write.sav(dd, "/tmp/test.sav", compress = F)
read.sav("/tmp/test.sav")

Improve por detection

Read header and check for the ASCII SPSS Port File string. Just read the full header and do the regex detection. No iconv required.

FR: allow `convert.times = TRUE` for `read.sav`

Currently the option convert.dates = TRUE forgets the times in what would be date-time data (with haven:read_sav I get POSIXct). I see that varmat=22 in my dataset.

read.sav never finishes

Hi,
Thanks for developing readspss!

I use it once in a while to import SPSS files from collaborators into R.
Last time (couple of months ago) I used it, everything worked fine. However, today I would like to import a file, but now R hangs for hours... Could you please have a look?

What did i do?

I installed (the latest version of) readspss through:
remotes::install_github("JanMarvin/readspss")

R hangs also when reading the included sample file. In other words; the code below ran for > 30 minutes without completing, and then I killed R...

> library(readspss)
> 
> fls <- system.file("extdata", "electric.sav", package="readspss")
> fls
[1] "C:/Program Files/R/R-4.2.1/library/readspss/extdata/electric.sav"
> 
> df_s <- read.sav(fls) # had to kill R session after 30 mins

> sessionInfo()
R version 4.2.1 (2022-06-23 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19042)

Matrix products: default

Random number generation:
 RNG:     Mersenne-Twister 
 Normal:  Inversion 
 Sample:  Rounding 
 
locale:
[1] LC_COLLATE=English_United States.utf8 
[2] LC_CTYPE=English_United States.utf8   
[3] LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] readspss_0.16.1

loaded via a namespace (and not attached):
[1] compiler_4.2.1 tools_4.2.1    Rcpp_1.0.9    
> 

ii should be a character vector

readspss/R/writesav.R

Lines 181 to 186 in 135716c

	ii <- sapply(dat, function(x) { !(is.integer(x) & all(is.na(x))) })
	gg <- sapply(dat[ii], function(x) {
	is.integer(x) & (min(x, na.rm = TRUE) >= -100 &
	max(x, na.rm = TRUE) < 151)
	})
	gg <- gg[names(ii)]

at least 186 looks sketchy

check header of encrypted files

the first 36 chars are some unique string, test this

Implement reading of files without a valid n

Pspp documentation states that n must not be valid. It can be -1 if not set correctly when writing the file. Such files are currently not read. All files at hand have a valid n which makes me believe, that it's a minor problem. Still it requires to either implement some kind of increasing vectors (convert to CharacterVector or NumericVector at the last possible time) or to calculate the number of cases using the file length left to read divided by the width of a row. Latter approach seems easier and less troublesome, though requires every row to be of identical width.

interpretation of label and haslabel lists

Dear Jan,

Thank you very much for the package! I founded to it since I had to work with a *.sav with password, and no other R package I know (haven, `foreign, etc.) seem to have this option.

The purpose of this issue is asking for a bit more of documentation on the values of the object returned by read.sav, since, for example, I cannot understand the values under the varmatrix attribute.

But I am intrigued since the data.frame I opened has 219 variables, while the attribute label is a list of length 183, as the attribute haslabel. I would like to know to which variable correspond each of the labels, so I tried to look at haslabel, but the components of the list arrive to 309, so they are not the indices of the variables in the data.frame, as I expected. How can I decipher this problem?

Thank you again!

decrypt encrypted sav-files

currently possible using pspp-convert

spss uses AES which could be provided by the openssl package?

file consists of 36 byte header followed by a aes 256 byte encoded sav-file

evaluate if it is possible. should be an interesting learning project on the other hand dependencies will increase (again) for an somewhat unimportant feature

Fix zsav compression on Windows

Apparently it works in Mac and Linux, but is broken on Windows. Not sure what is broken, but my guess is writing. Files appear cut of.

Read integers as integers

Right now if the sav file contains a cflag, integers are read, but stored as numerics. Still if the type is known prior, it should be possible to store the int correctly.

Missing variable labels in read.sav?

Greetings. In the description of the returned data.frame objects in read.sav, the following are defined:

val.label = value labels

var.label = variable labels

Within the actual R code, the following exists:

val.labels <- attribs$vallabels # line 155

attr(data, "var.label") <- val.labels # line 514

Shouldn't var.label (variable labels) be a different attribute than val.label (value labels)?

Thank you in advance.

Write sav files

• vtype needs to be expanded to add some -1 for strings longer than 8 chars
• names need to be expanded for >255 chars will require longnames handling
• var.labels implementation
• missings need handling kinda work already. could provide a missing table, though not sure if really required or beneficial
• value labels need handling
• long value labels need handling they are not long labels, just labels for long strings. nothing useful for the R world

Add checks

Add checks to different segments of the loop (possibly move code out to functions), to avoid looping into oblivion.

write por

Should be fairly easy to implement now that reading works.

• extract pnum function from TDA to print numerics
• put everything together for writing integers, numerics and strings < 255 chars
• add data and var label
• add value labels
• handle missings
• test everything!

Read por files

Por files are structured differently and less familiar e.g. one does find fewer por files in the wild.

Iconv on the full vector

Currently after a string is read, iconv is called. This already reduced the iconv overhead, still iconv is called for every n. Since R's implementation of iconv is used, it could be run on the full vector after it was read.

Check if EOF is reached

Otherwise the loops have the potential to read into oblivion.