cloudyr / limer

A LimeSurvey R Client

limer: A LimeSurvey R client

limer provides access to LimeSurvey's RemoteControl 2 API, allowing you to collect and analyze survey data in a simple, reproducible workflow.

Setup

Make sure you have enabled LimeSurvey's RPC interface, found in the administration section: Global settings > Interfaces > RPC interface enabled = JSON-RPC (not XML-RPC). You don't need to publish the API on admin/remotecontrol—all those details are available elsewhere. The API URL should look something like http://example.com/limesurvey/admin/remotecontrol.

Load your API details and user credentials into R using options():

options(lime_api = 'http://example.com/limesurvey/admin/remotecontrol')
options(lime_username = 'put_username_here')
options(lime_password = 'put_password_here')

Before calling the API, you need to generate an access token with get_session_key() (examples of how to do this are shown below). Many services provide tokens that last indefinitely, but by default LimeSurvey's will only last for two hours. (this can be modified by editing limesurvey/application/config/config-default.php and changing $config['iSessionExpirationTime'] = 7200; to something else).

Code examples

Here's how easy it is to export the results from a survey:

# Load library
library(limer)

# Setup API details
options(lime_api = 'http://example.com/limesurvey/admin/remotecontrol')
options(lime_username = 'put_username_here')
options(lime_password = 'put_password_here')

# Do stuff with LimeSurvey API
get_session_key()  # Log in
responses <- get_responses(12345)  # Get results from survey

You can also run any arbitrary API call using call_limer() (get_responses() and other functions are just convenient wrappers around call_limer()):

# Get a list of all the surveys
call_limer(method = "list_surveys")

# Get the number of completed responses for a survey
call_limer(method = "get_summary", 
           params = list(iSurveyID = 12345,
                         sStatname = "completed_responses"))

If the API call returns base64 encoded text, you can convert it to a data frame with base64_to_df():

raw_data <- call_limer(method = "export_responses", 
                       params = list(iSurveyID = 12345, 
                                     sDocumentType = "csv", 
                                     sLanguageCode = "en", 
                                     sCompletionStatus = "complete", 
                                     sHeadingType = "code", 
                                     sResponseType = "long"))
base64_to_df(raw_data)

At the end of your script or session, it's nice to release the session key. If you don't release the session key, LimeSurvey will eventually clean it up.

release_session_key()

Roadmap

For now, the only special thing this package does is provide get_responses(). In the future I hope to add convenience functions for creating new surveys, managing responses, and running other common API operations. Regardless, all API functions are accessible with call_limer(method = "whatever").

get_responses() currently defaults to CSV, but the API can also return JSON, which might actually be better for transferring data over the internet. I'll eventually add JSON support.

Installation

limer isn't on CRAN (yet), but it's easy to install directly from GitHub using devtools:

if (!require("devtools")) {
  install.packages("devtools")
  library("devtools")
}
install_github("cloudyr/limer")

limer's Issues

Does limer work with limesurvey 5.3?

Hi guys.
I was using limer to access limesurvey ver. 3.....
Now we passed to limesurvey 5.3, but get_session_key() returns an error:
Error: Argument txt must be a JSON string, URL or file.

Did someone use limer with limesurvey 5.3?

TX a lot

update_response

I am having trouble using call_limer to update a response. I have both the response id and token as headers, along with the response to update. I get the following error message:

> update_response <-call_limer(method = "update_response", 
                            params = list(iSurveyID = 123456,
                                          ResponseData = df))

> update_response
[1] "Error: Missing response identifier (id|token)."

Any ideas would be helpful!

Error get_session_key()

Thank you for the limer package.
When using get_session_key() I get the following error message:

Error: Argument txt must be a JSON string, URL or file.

I have installed the latest version of the limer package and also already looked at possible solutions to the problem here on github. The script is also run regularly and in mid-May, it was still working without error message.

Are there any other ideas how I can fix the problem?

Thanks
Sebastian

get_session returns "Argument 'txt' must be a JSON string, URL or file."

Any issues known with using https under R? Or is the UTF8 format broken in certain R versions?

Limer is not able to get a connection via https to a RPC-JSON Limesurvey (LS 2.06LTS installation).
When trying to access the Limesurvey API via e.g. C# the connection is working.

The same issue was described here:
https://gist.github.com/andrewheiss/fe64718c41991b1620f5

R textConnection

I am trying to use the folowing code to download some forms from (https://www.sec.gov/Archives/edgar/full-index/ ).
I am getting this error :
Error in fread(textConnection(master), sep = "|") :
input= must be a single character string containing a file name, a system command containing at least one space, a URL starting 'http[s]://', 'ftp[s]://' or 'file://', or, the input data itself containing at least one \n or \r

this the code
start_year <- 2016
start_QTR <- 1

end_year <- 2016
end_QTR <- 4
setwd("D:/Requests/SC13D")
require(data.table)
require(Hmisc)
require(data.table)

###########################################
####### construct SEC master file ########
###########################################
qtr.master.file <- function(year, QTR)
{
require(data.table)
name <- paste0("https://www.sec.gov/Archives/edgar/full-index/", year,"/QTR",QTR,"/master.idx")
print(sprintf("Downloading master file for quarter %d of year %s...", QTR, year))
master <- readLines(url(name))
master <- master[grep("SC 13(D|G)", master)]
master <- gsub("#", "", master) # R does not treat a comment sign well
master_table <- fread(textConnection(master), sep = "|")
rm(master)
colnames(master_table) <- c("cik", "name", "type", "date", "link")
master_table <- as.data.table(master_table)
master_table[, link := paste0("https://www.sec.gov/Archives/", link)]
master_table[, file := gsub(".*/", "", link)]
closeAllConnections()
return(master_table)
}
###########################################

download all files into temp dir

###########################################
dwnld.files <- function(master)
{
require(RCurl)
dir.create("temp_dir")
master <- as.data.frame(master)
master <- master[!duplicated(master$file),]
for(j in 1:length(master$file))
{
file <- NA
file_url <- as.character(master$link[j])
file_name <- paste0("./temp_dir/",master$file[j])
try(file <- getURL(file_url))
write(file, file_name)
}
}
###########################################
####### put all forms in SQdatabase #######
###########################################
put.files.in.sql <- function(dbname)
{
library(DBI)
library(RSQLite)
together <- function(x)
{
return(paste(x, collapse = "\n"))
}

con = dbConnect(SQLite(), dbname=dbname)
dbSendQuery(conn=con,
"CREATE TABLE compsubm
(FILENAME TEXT, COMLSUBFILE TEXT)")
path <- paste0("./temp_dir/")
files <- list.files(path)
n <- length(files)
step <- 500
for(i in 1:(n %/% step + 1))
{
start <- 1 + (i-1)step
end <- i(step)
ind <- start:min(end,n)
objects <- lapply(paste0(path,files[ind]), readLines)
clean <- lapply(objects, together)
data <- NULL
data$FILENAME <- files[ind]
data <- as.data.frame(data)
data$COMLSUBFILE <- unlist(clean)
dbWriteTable(conn=con, name = "compsubm", data, append = T)
}
dbDisconnect(con)
unlink("temp_dir", recursive = T)
}
get_dates <- function(start_year, start_QTR, end_year, end_QTR)
{
require(data.table)
all_dates <- data.table(year = rep(1993:2050, 4))
setkey(all_dates,year)
all_dates[, QTR := 1:.N, by = year]
all_dates <- as.data.frame(all_dates)

x <- paste0(all_dates$year, all_dates$QTR) >= paste0(start_year, start_QTR) & paste0(all_dates$year, all_dates$QTR) <= paste0(end_year, end_QTR)
return(all_dates[x,])
}

dates <- get_dates(start_year, start_QTR, end_year, end_QTR)
for(i in 1:length(dates$QTR))
{
print(Sys.time())
master <- qtr.master.file(dates$year[i], dates$QTR[i])
write.csv(master, paste0("./Master/master_", dates$year[i], dates$QTR[i], ".csv"), row.names = F)
print("Dowloading files, it takes up to 4 hours")
dwnld.files(master)
print("Putting all files into SQL & cleaning")
put.files.in.sql(paste0("./Forms/",dates$year[i],dates$QTR[i], ".sqlite"))
}

get_responses() returns multiple observations but only one variable

Hi guys! Thanks for developing the package

I have encountered the following problem:
Whenever I download the responses I get all the entries but in a strange format where all the columns are combined into one.

I tried to circumvent this using direct call using call_limer, but it basically gives me raw data which if I push to base64_to_df - gives just the same results as using get_response.

This guy has the same issue - so at least I'm not alone:
https://stackoverflow.com/questions/51989013/trying-to-pull-down-data-in-r-from-limesurvey-its-failing-why

All the variables are squashed into one line with points as separators:
as :
q620.cortico_priorv5..q620.cortico_acute..q620.cortico_primcarev3..q620.cortico_centre..q620.cortico_no..q620.other_priorv5..q620.other_acute..q620.other_primcarev3..q620.other_centre..q620.other_no..q631autoinjbrandv7.q632autoinjnumberv7.q633autoinjdosisv7.qcommentv5
I have underscores in my variable names - could this be a problem?

This is my session info:

R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.10

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=de_DE.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=de_DE.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=de_DE.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] limer_0.1

loaded via a namespace (and not attached):
[1] httr_1.4.0      compiler_3.5.1  R6_2.4.0        tools_3.5.1     base64enc_0.1-3 curl_3.3        yaml_2.2.0      jsonlite_1.6  

SQL SELECT query limesurvey using limer?

We are in the process of setting up a demo version of limesurvey, with a view to testing it out for use in our organisation to hold routine surveys. These surveys will always be open (collecting new responses daily). As such, analysts will not normally want to import all the data in one go as e.g. after one year there will be several thousand responses. More typically we would want to import a subset of the data that falls within a specific range for a given analysis pipeline.

It would therefore be useful to be able to perform a select SQL query for e.g., responses for a particular survey ID within a specific date range, or age range or whatever. I (and an increasing number of my colleagues) use R, and ultimately we want to be able to create automated reports from the survey data with R markdown which we might run on a regular basis (weekly / quarterly / annual etc).

Is it possible to perform an SQL select query with limer and if so would it be possible to demonstrate with a vignette how to do this?

Consider putting this package on LimeStore!

This looks like a nice package. You should consider putting it on LimeStore for greater visibility!

https://www.limesurvey.org/limestore

Using insecure server connections

I'm trying to import from an insecure server connection, how can I do it?

Error in curl::curl_fetch_memory(url, handle = handle) : SSL certificate problem: unable to get local issuer certificate

get_participants throws an error when debug=1

All good when debug=0 (in application/config/config.php), but when debug=1 I receive an error in RStudio:

Error: lexical error: invalid char in json text.
<!DOCTYPE html PUBLIC "-//W3C/
(right here) ------^

Workaround for now is debug=0, but debug=1 is necessary at certain times.

Not getting First name and Last name data responce on export responces api call

Hello..

in below api i want also First name and Last name data . so what param i need to add ?
i have try to adding sField = "First name" but not getting data.

raw_data <- call_limer(method = "export_responses",
params = list(iSurveyID = 12345,
sDocumentType = "csv",
sLanguageCode = "en",
sCompletionStatus = "complete",
sHeadingType = "code",
sResponseType = "long"))
base64_to_df(raw_data)

Thanks
Dipen Dalsaniya

Documentation: expand on not using session_key

Today I spend several hours getting something to work.

As it turns out, when using call_limer, you do not need to send the session_key, although this is indicated in the LimeSurvey RPC page at: https://api.limesurvey.org/classes/remotecontrol_handle.html

For example (to get response per token)

NOT GOOD
sessionKey<-get_session_key()
response_raw<-call_limer(method= 'export_responses_by_token',
params = list(sSessionKey=sessionKey,
iSurveyID = 111111,
sDocumentType = 'csv',
sToken = 'chg2h4460',
sLanguageCode= ''
))
response<-base64_to_df(response_raw)

GOOD
get_session_key()
response_raw<-call_limer(method= 'export_responses_by_token',
params = list(iSurveyID = 111111,
sDocumentType = 'csv',
sToken = 'chg2h4460',
sLanguageCode= ''
))
response<-base64_to_df(response_raw)

Fetching all available data (including incomplete)

Hi,

thanks so much for the Limer package which makes data fetching so easy with Limesurvey. I would like to fetch not only complete responses but all responses. How is this possible?

Thanks!

Note about alternative package

Just a note that there appears to be an alternative LimeSurvey package being developed here: https://github.com/kalimu/LimeRick

additional closing bracket in base64_to_df.R

There seems to be an error in "limer/R/base64_to_df.R" on line 14. You added an additional closing bracket in your latest commit. Now the installation fails.

• return(read.csv(textConnection(raw_csv), stringsAsFactors = FALSE))

• return(read.csv(textConnection(raw_csv), stringsAsFactors = FALSE, sep = ";")))

add user agent to avoid problems with modsecurity

I may be wrong, but it appears that modsecurity is blocking access (rule 990011 in my case) to limesurvey remotecontrol on my survey because the httr call is not providing a user-agent parameter.

Unless there is a reason not to, if my guess is correct, might it be possible to add a user-agent of "httr" to the code that interacts with remotecontrol?

Thanks.

Submit to the LimeSurvey extension directory

Hi,

you should submit this script to our extensions directory, so it receives even more attention...
https://www.limesurvey.org/community-services/extensions

not getting full heading in response

Hello,

i used below code for get survey response as dataframe,

raw_data <- call_limer(method = "export_responses",
params = list(iSurveyID = 12345,
sDocumentType = "csv",
sLanguageCode = "en",
sCompletionStatus = "complete",
sHeadingType = "code",
sResponseType = "long"))
base64_to_df(raw_data)

but I get only question code as column name of dataframe , I'm not getting full question.
how can I get full question as column name of dataframe...
Please help

Thanks
dipen

stop double parsing json

recent limesurvey update changed response content type to appliction/json, httr automagically parses this. in get_session_key function, limer parses it again, resulting in error

fix similar to call_limer.R:L39

Changes after updating to 2.5?

I updated to limesurvey v2.5 today and it appears that the url for remotecontrol has changed. It appears to be at /admin/remotecontrol now.

That is fine, but when I try to use that in options() it returns NULL and doesn't allow a connection. Is there something else I need to do on my end?

Thanks for a really useful interface, in any case.

example using set_survey_properties

Hi guys. Using LS 5.3, I'm trying to set via API the expires date for a survey, and to do this I'm using the method set_survey_properties in R.
Unfortunately it seems not working, the value of the filed doesn't change.
Please, has someone a working example of successfully invoking set_survey_properties in R?
Thank you so much
Manlio

Log in not working

When I use the function get_session_key after defining the username, password, and API, I get the following error:

Error: Argument 'txt' must be a JSON string, URL or file.

Any thoughts?

expose separator value in base64_to_df()

This is linked to my question on limesurvey.org (which is apperantly the wrong forum to place questions on the LSRC2 API or more specifically the limer package.

My issue was that get_responses() did not seem to expect the comma (",") as a delimiter value (see the screenshot).

I found the culprit in this case: The function base64_to_df() (base64_to_df.R, see d7c0c3e) has sep = ; hardcoded, whereas in my case sep = "," would be appropriate. Would it be possible to expose the sep-value as an option?

Mail a specific participant with mail_registered_participants

I try to mail a specific participant of a survey.

According to the API documentation, it is possible using the overrideAllConditions parameter :

/**
* Send register mails to participants in a survey
*
* Returns array of results of sending
*
* Default behaviour is to send register emails to not invited, not reminded, not completed and in valid frame date participant.
*
* $overrideAllConditions replaces this default conditions for selecting the participants. A typical use case is to select only one participant
* * $overrideAllConditions = Array(); * $overrideAllConditions[] = 'tid = 2'; * $response = $myJSONRPCClient->mail_registered_participants( $sessionKey, $survey_id, $overrideAllConditions ); *
*
* @access public
* @param string $sSessionKey Auth credentials
* @param int $iSurveyID ID of the Survey that participants belong
* @param array $overrideAllConditions replace the default conditions
* @return array Result of the action
*/

I tried these three syntax with limer to mail the participant with the token id 1 :

call_limer(method = "mail_registered_participants", params = list("iSurveyID" = 123456, "overrideAllConditions" = list("tid" = "1")))
call_limer(method = "mail_registered_participants", params = list("iSurveyID" = 123456, "overrideAllConditions" = list("tid" = 1)))
call_limer(method = "mail_registered_participants", params = list("iSurveyID" = 123456, "overrideAllConditions" = "tid = 1"))

But none works :(
Does anyone have an idea ?

call_limer - prefix POST with httr package

Would it be possible to prefix the POST in the call_limer function with the httr package ?
POST -> httr::POST

Otherwise, it is necessary to load the httr library.

Not Getting Answer Code in Responce

Hello..
I have did answer option of question .
so now issue is when i am taking get response of answer then getting answer text data not getting answer code.

Because i want to compare Answer Code with My Answer Key.
So how i can do that ?

Here i have attached screenshot of Answer and Answer code Seen By Yellow Highlight.
That Yellow Highlighted code i not getting in API response.
so thats why i am not able to compare with my answer key.

So please suggest me right solution

I am using R limer Plugin and Rest RemoteAPI

Thanks

fix for encoding in base64_to_df

With a simple "encoding" added, we can tell R to encode in UTF-8.

base64_to_df <- function(x) {
  raw_csv <- rawToChar(base64enc::base64decode(x))
  return(read.table(textConnection(raw_csv), stringsAsFactors = FALSE, header=TRUE,
                  sep = ";", encoding = "UTF-8" ))
}

get_session_key() returns Error: Argument 'txt' must be a JSON string, URL or file.

list_participants returns a data.frame with a column that is a data.frame

Received via e-mail:

---

Is this intended behaviour?

> req2 <- call_limer("list_participants", list(971744))
> names(req2)
[1] "tid"              "token"            "participant_info"
> get_session_key(password=LSPASSWORD,username = LSUSER)
[1] "xwi3uhr5kf4wapmv255i8k8s5gkcvhgr"
> req2 <- call_limer("list_participants", list(971744))
> names(req2)
[1] "tid"              "token"            "participant_info"
> class(req2$participant_info)
[1] "data.frame"
> names(req2$participant_info)
[1] "firstname" "lastname"  "email"

So list_participants returns a dataframe of 3 columns, one of which is a dataframe on itself.

error when get _responses

get_session_key()
[1] "ddrvaxfczqv9i5bhf4h34ukq5h86f4jt"
responses <- get_responses(913762)
Error in make.names(col.names, unique = TRUE) :
invalid multibyte string 1

could you pls tell me what's the problem may be, thanks!

Limer dont work in R?

Hello, when trying to install the limer package on a new instance of R, it gives me an error... is it possible that the repository has been deactivated? I don't know if I'm the only one with this problem... it worked perfectly before... thank you!

Legacy issues to fix

Received from e-mail:

---

For instance: before call_limer(method='list_surveys') returned (on my system) a vector, now a dataframe. Am I correct, or is something changed in rpc2 itself (although I do’t believe I upgraded the system. But I did reinstall limer). On itself it is much easier that it returns a dataframe, but I had to change my functions in order to get them working again. And also the manual information on the limesurvey page has to be updated. May I suggest you to change the version number (from 0.1 to 0.1.1, or something lime that, if something changes in the package?

In addition I noticed you added a function get_participants. I would like to let you know that this function gives an error on my system (R version 3.1.2):

get_participants(975155)
Error: No method asJSON S3 class: name

Are there any recent changes for get_responses and export_responses to work now?

When I use get_responses from limer now, the first line of variable names seems to be delimited by "." and the rest of the data seems to be delimited by ";".

If I use call_limer with "export responses" and base64_to_df to create a dataframe, I'm getting 1 variable rather than multiple variables for each responses (I'm guessing, for the same reason as above.

Is there something I need to be doing differently now? I don't think that I had tried this until updating from an older version of limesurvey and remotecontrol.

Thanks for any help.

not an issue - close / delte: WAS get_responses > Error in make.names

SORRY -DELETE THIS - base64_to_df handles this ... sorry reported by mistake.

Maybe encoding issue:
get_response returns the following error:

Error in make.names(col.names, unique = TRUE) : 
  invalid multibyte string 1

default languae for the survey is lang=da

Other methods, e.g. call_limer(method='list_surveys') produces the expected results.

Encoding error

When importing Portuguese characters using Rstudio (windows 10), I end up with a lot of badly encoded letters. Although in Rstudio Linux version, that doesn't happen. Any solution? I both cases I use exactly the same code and the same version of limer.

Error when getting data

when running get_responses(surveyID) I get the following error on limesurvey Version 2.06lts Build 160801:

Error in make.names(col.names, unique = TRUE) :
invalid multibyte string 1

Any ideas, how to solve the problem?
(there is no error on my home computer with limesurvey Version 2.62.0+170124 )

Base language for get_responses should be default=all

Currently when you want to get_responses from a limesurvey survey, you get by default only the english responses.

In order to accommodate all users in an equal way, I would suggest to specify that you get all languages, unless you specify a language.

I guess that specifying sLanguangeCode = "" in get_responses.R will do the trick.

Use Travis to create LimeSurvey instance for testing

Create an on-the-fly instance of LimeSurvey with the API enabled for creating unit tests

get_participants No method asJSON S3 class: name

I get the error No method asJSON S3 class: name when i try to use the get_participants function.
Do i need to install/load a library besides jsonlite and RJSONIO ?

It works if I use the call_limer function.

getResponses() for one question

Hello,
I've tested the limer package and found it very useful and promising, thak you very much for your work.
Is there possibility to export results only for one question? I'm working on a project to visualize stats for active surveys grouped by one variable. Exporting all results is too slow, taking into account that I need only one question to examine.

Best regards,
Daria Omelchenko