asitav-sen / koboconnectr

Download data from Kobotoolbox to R

KoboconnectR

The goal of KoboconnectR is to facilitate data extraction from Kobotoolbox projects/assets into R, instead of downloading the individual spreadsheet files.

Installation

CRAN version can be downloaded with:

install.packages("KoboconnectR") 

The effort is still in its early stage. The development version can be installed from GitHub with:

# install.packages("devtools")
devtools::install_github("asitav-sen/KoboconnectR")

Getting started

Before being able to use the kobotoolbox API, one needs to set up API token. The kobotoolbox documentation lists multiple ways for one to extract the token value. If there is a token, the easiest way to retrieve it is by using get_kobo_token() function.

kobotools_kpi_data(assetid = "assetid", url = "eu.kobotoolbox.org", uname = "username", pwd = "password")

To find the asset id, one can use the kobotools_api() function that returns a list of assets the user has access to.

kobotools_api(url = "eu.kobotoolbox.org", simplified = T, uname = "userid", pwd = "password")

Downloading data

KoboconnectR::kobo_df_download(
  uname = "username",
  pwd = "password", assetid = "assetid",
  all = "false", lang = "_default",
  hierarchy = "false", include_grp = "true", grp_sep = "/", fsep = ";",
  multi_sel = "both", media_url = "true", fields = NULL, sub_ids = NULL, sleep = 2
)

IF THE DOWNLOAD FAILS, TRY INCREASING THE SLEEP PARAMETER (use the latest version please, dev version, if required).

For further details, check the vignette.

Contribution

The package is stable but still in early stages of development. Any help to enhance it further will be highly appreciated. Please feel free to fork, add/modify and create pull requests.

Issues and Suggestions

Please feel free to add issue of suggestions in github

Dependencies

The following packages are required to install and run this package.

1. Package httr2
2. Package jsonlite
3. Package curl
4. Package mime
5. Package openssl
6. Package R6
7. Package methods
8. Package readxl
9. Package rlang
10. Package purrr

koboconnectr's Issues

Posting csv files

Many thanks for this very useful package.
Are you planning a function to post via API .csv file(s) to the 'media' folder where Kobotoolbox stores data that is used in the survey form "pulldata" function?
Thanks for considering. Xavier

include_grp="false" does not reflect

Continuation from last issue.
Using include_grp="false"does not change the export in Kobotoolbox.
@zxrey

kobotools_api error retrieving ID

Hello,

I was wondering if anyone encountered the following issue. When running the kobotools_api function to retrieve the list of assets IDs, I am receiving this error:
Downloading: 260 kB Error in parsed$results[[i]] : subscript out of bounds

I tried the kobotools_api function with another account that has less data (surveys and submissions) and it worked. I am thinking that perhaps my kobo account has too much data on it for the function to handle. Before I delete some archived data, I was wondering if others have encountered this issue and found some solutions. Or if deleting is the only solution, how much would I need to delete??

I am using:
kobotools_api(url="kobo.humanitarianresponse.info", simplified=T, uname="username", pwd=pw)
#username is actually replaced with my username*

If I set simplified to F then I can actually download the asset IDs but it comes in the form of a large list (4 elements)

Thanks

image link

Hi @asitav-sen , thanks for this great package.

I´m transferring the data with the commands:

idesam_export <- KoboconnectR::kobo_export_create(uname = "myUser", pwd = "myPass",
                                                  assetid=idesam_assets$asset[1], type= "xls",
                                                  all="true", lang="Portuguese",
                                                  hierarchy="true", include_grp="true", grp_sep="/")

httr::GET(idesam_export, httr::authenticate(user="myUser", password ="myUser"),
          httr::write_disk(tmp_file, overwrite = TRUE))

But the downloaded spreadsheet doesn't have a column with the images link.

Is there any way to fix it?

Unable to use function kobo_df_download

Hi, Thank you for creating this package. I am trying to use it to download data from kobotools into R to save it as xls later on. I want to be able to download the data in long format in R (maybe through creating separate dta files). But, when I try to run the function kobo_df_download, it gives the following error "Downloading: 7.9 kB Export instruction sent successfully. Waiting for result.
Downloading: 7.9 kB Execution in Progress...Timeout[1] "export creation was not successful"
NULL"

I have tried to increase the sleep time as well, but it does not solve the problem.

Error in Running KoboConnect.R Script

When I run the code chunks separately, the data finally downloads. However, when I put the script on Shiny apps or in a similar environment, and run the R-script, the download fails.

Any reason for this? (See screenshot)

Downloading audit data

Hi, great package. Worked nicely since the start.
It seems that the package does NOT target the download of "Media attachments", which include the 'audit' information of the filed type 'audit'. These data can be found in KoBoToolbox (your project) > Data > Downloads > Select export type > Media attachments (ZIP).
Thanks for considering
xboschc

Unable to download more than 30k rows with kobotools_kpi_data

Hi @asitav-sen
Please let me know if there is a workaround to download more than 30k rows, I have noticed the api allows for only 30k, so the function only retrieves 30k observations. I see the solution here for other contexts
https://kc.humanitarianresponse.info/api/v1/data/XXXX?format=csv&sort=%7B%22_id%22:1%7D&start=30000

Test failure with latest 'curl' due to redirect to eu.kobotoolbox.org

The latest version of the curl package changes the default for CURLOPT_UNRESTRICTED_AUTH to 0, which causes the CRAN version of KoboconnectR to fail its tests:

* checking tests ...
  Running ‘spelling.R’
  Running ‘testthat.R’
 ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
    'test-kobo_export_create.R:21:3', 'test-kobo_export_create.R:30:3',
    'test-kobo_exports.R:2:3', 'test-kobo_xls_dl.R:2:3',
    'test-kobotools_api.R:31:3', 'test-kobotools_kpi_data.R:38:3'

  ══ Failed tests ════════════════════════════════════════════════════════════════
  ── Failure ('test-kobo_df_download.R:10:3'): invalid input fails ───────────────
  `is.null(...)` did not throw the expected error.
  ── Failure ('test-kobo_media_downloader.R:10:3'): invalid input fails ──────────
  `is.null(kobo_media_downloader(uname = "scary_", pwd = "sy", assetid = "ahwQoXtnbQJB"))` did not throw the expected error.
  ── Failure ('test-kobo_xls_dl.R:12:3'): invalid input fails ────────────────────
  `is.null(kobo_xls_dl(uname = "scary_", pwd = "sy", assetid = "ahwQoXtnbQJB"))` did not throw the expected error.

  [ FAIL 3 | WARN 3 | SKIP 10 | PASS 25 ]
  Error: Test failures
  Execution halted

The problem is that your package makes an authenticated HTTP request to https://kobo.humanitarianresponse.info/ but this API seems to have moved and gets redirected to to https://eu.kobotoolbox.org/.

For security reasons, the new version of curl no longer automatically sends the credentials to the redirected host.

The easiest solution is to search-and-replace kobo.humanitarianresponse.info with eu.kobotoolbox.org in your code.

I cant set set_kobo_token("this is a token value")

I install all dependences and the package, but when I try set the code:
set_kobo_token("this is a token value")
I get this message :
Error in set_kobo_token("b80c73109cb88e1f9b5ec94491f64badf3c3d907") :
function could not be found "set_kobo_token"

column name with kobo_df_download

Hi @asitav-sen,
Thank you very much for your wonderful work. I'm try to use kobo_df_download function but the dataset come out with column names as the labels of the questions instead of the name I gave to them in Kobo.
Manh

Issue on downloading data from Kobo toolbox

After running this code:

kobo_df_download(url=url,
uname = uname,
pwd=pwd,
assetid = asset$asset[1],
all = "false",
lang = "English (en)",
grp_sep = "/",
multi_sel = "both",
media_url = "true",
fields = NULL,
sub_ids = NULL,
sleep = 2
)

I have these outcomes

Execution in Progress...Timeout[1] "export creation was not successful"
NULL

kobo_export_create doesn't work

Hi @asitav-sen l see that you worked to doing the new version of Kobocollect, that is wonderful. l tried to get my xls, l update the version but l can't get when l put the commands

kobotools_api(
url = "kobo.humanitarianresponse.info",
simplified = TRUE,
uname = "",
pwd = "",
encoding = "UTF-8"
)

then

kobotools_kpi_data(
assetid,
url = "kobo.humanitarianresponse.info",
uname = "",
pwd = "",
encoding = "UTF-8"
)

and

kobo_export_create(
url = "kobo.humanitarianresponse.info",
uname = "",
pwd = "",
assetid = "",
type = "csv",
all = "false",
lang = "_default",
hierarchy = "false",
include_grp = "true",
grp_sep = "/",
multi_sel = "both",
fields = NULL,
media_url = "true",
sub_ids = NULL,
qry = NULL,
flatten = "true",
sleep = 2
)

with the last one command, creates a empty value

Maybe did l something wrong

kobo_df_download for XLS type

Hi @asitav-sen, It was great and save me a lot of time with this download function. Thank you so much for your work. But now My boss asks me to have some repeat questions type and cannot be downloaded with csv type. Could you please consider making similar function that we can download and import to R with xls type.
Thank you so much.
Mạnh

Failure with kobo_df_download

Hello,

I am having trouble downloading the survey results as a data frame. The process has worked well to date but I noticed it is not working-perhaps after the update? Previously, on occasion there were some time outs and failures when accessing exports but this seems a more definite failure. I tried increasing the sleep= amount but it still fails. I used dev version, cran version and then made a fresh download but no success.

Error is:
'''
Downloading: 7.2 kB     Export instruction sent successfully. Waiting for result.
Downloading: 7.2 kB     Error in if (post_export$results$status[post_count] == "error") { :
  missing value where TRUE/FALSE needed"
'''
Using function (with name, Id, and pw omitted)

'''
cm_dat <- KoboconnectR::kobo_df_download(uname = "name",
                                         pwd=pw, assetid = "id#",
                                         all="false", lang="_default",
                                         hierarchy="false", include_grp="true",grp_sep="/",fsep=";",
                                         multi_sel="both", media_url="true", fields=NULL, sub_ids=NULL, sleep=4)

'''

Error when exporting DF: "Export creation was not successful"

@asitav-sen

I could fetch the token using get_kobo_token() and asset_id using kobotools_api().
When using kobo_df_download() to download the DF to my workspace, I get the following error on the console (See screenshot). This is despite using the dev version of the package and increasing the sleep value=4.

Any reason to the error

kobo_kf_download problem: Could not resolve host

Hi, I am also having troubles with kobo_kf_donwnload. But a little different from the previous issues. I am receiving this message:

df <- kobo_df_download(uname, pwd, assetid, all="false", lang="English (em)", include_grp="false", grp_sep=NULL)
Error in curl::curl_fetch_memory(url, handle = handle) :
Could not resolve host: andresimi

What this could be?

kobo_df_download field argument suggestion

Hi people,
I have an suggestion for the fields argument in the command kobo_df_download:
Instead of accept an array of column names to be included in the export, i think it would be nice to have an option to set only the excluded variables from the export. It is a nice feature to not download identifiable data.
Best!

Read existing exports to R using the URL from kobo_exports() function

Hi, I wanted to use your package for my project. I find it would be very convenient for me if I can read or import the export dataset directly to R. I use the URL from kobo_exports() function but I could not do it:

For the excel file, it keep saying the path does not exist, but if I copy and paste the link to web it can be downloaded.
library(openxlsx)
df <- read_excel("https://kf.kobotoolbox.org/private-media/manh_nguyen/exports/Contact_tracing_questionnaires_-_all_versions_-_False_-_2022-04-01-03-49-32.xlsx")
Error: path does not exist: ‘https://kf.kobotoolbox.org/private-media/manh_nguyen/exports/Contact_tracing_questionnaires_-_all_versions_-_False_-_2022-04-01-03-49-32.xlsx’

For the csv file it gives the error 403
library(readr)
df <- read_csv("https://kf.kobotoolbox.org/private-media/manh_nguyen/exports/Contact_tracing_questionnaires_-_all_versions_-_False_-_2022-04-15-02-36-24.csv")
Error in open.connection(3L, "rb") : HTTP error 403.

Thank you very much

Hi @mbk7162 ,

Unable to use function kobo_df_download

Hi, Asitav-sen
Many thanks for providing this awesome package,
I have come across the same issue however I was able to figure it out that is why I feel it is good to share so that others can benefit
kindly the see method I apply below
data <- kobo_df_download(url="xxxxxxx",
uname="xxxxxxxx",
pwd="xxxxxxx",
assetid = "xxxxxx",
lang="_default")

I replace ( lang="English") to lang="_default") and it works for me

Best regards