aldefouw / redcap-api-transfer Goto Github PK

We are migrating the Redcap projects using this utility.

Issue: While importing the files other than Text files, file are uploaded successfully, however they are getting corrupted.

Cause: While importing, the file is being read using File.read(file) which corrupts the file if it's not a text file.

Importing a DOC/PDF/EXCEL file from API playground or Postman works fine. The code which is used is the same code which is being provided in the Redcap samples https://redcaptest.cc.nih.gov/api/help/?content=examples

Below is the code which is causing the issue, any help would be appreciated.

def body(boundary, id, field, event_name, file)
<<-EOF
--#{boundary}
Content-Disposition: form-data; name="file"; filename="#{File.basename(file)}"
Content-Type: application/octet-stream
#{File.read(file)}
--#{boundary}
#{import_file_fields(id, field, event_name).collect{|k,v|"Content-Disposition: form-data; name=\"#{k.to_s}\"\n\n#{v}\n--#{boundary}\n"}.join}
EOF
end

Not sure if you're still actively maintaining this project, but I'm getting the below error when trying to transfer a project. Note that I previously posted a bug report as "senorrojo" a few years ago (which you solved - thank you) and I've just started using this package again because we're migrating from one REDCap server to another. Thank you so much for creating this project - it truly saved me the last time I needed it and I had many, many successful transfers of large projects.

I get the same error for transferring a single record or all records:

[matt@localhost redcap-api-transfer]# sh transfer_single_record.sh Test 25
/app/library/export_data.rb:50:in throw': uncaught throw "Unable to find a destination data template file for the project /app/export_data/dest_template-1708116087.tmp." (UncaughtThrowError) from /app/library/export_data.rb:50:in fetch_dest_data_template'
from /app/library/export_data.rb:43:in dest_data_cols' from /app/library/export_data.rb:15:in initialize'
from /app/library/transfer_records.rb:50:in new' from /app/library/transfer_records.rb:50:in initialize'
from transfer_single_record.rb:7:in new' from transfer_single_record.rb:7:in <main>'
Loading Configuration ...
Downloading SOURCE Project Template ...
Downloading DESTINATION Project Template ...
Downloading Data Dictionary ...
Loading Export Data ...
[matt@localhost redcap-api-transfer]#

[matt@localhost redcap-api-transfer]# ls app/export_data/
data.csv

data.csv has a CSV export of the project data from the SOURCE. I also tried putting the data dictionary in that export_data directory to no avail. Below is my config file. I'm happy to provide verbose logging or any other information that might be helpful.

settings:
  processes: 8 # Default number of processes the script will fallback to if none is specified in specific projects below
  verbose: false # When true, outputs additional information to logs about the HTTP requests being made

projects:
  Test:
    source:
      url: https://redcap.healthsciencessc.org/api/
      token: XXXX

    destination:
      url: https://redcap.research.sc.edu/api/
      token: YYYY

    transfer_new_records_only: true # Must have API Export Rights on Destination REDCap Server!
    processes: 8 # Takes precedence over the processes listed in settings
    verbose: false  # Takes precedence over the verbose flag listed in settings

Again, thank you!

When trying to transfer records (single or all records) with an uploaded file attached, I get the below error:

$ ruby -v
ruby 2.3.5p376 (2017-09-14 revision 59905) [x86_64-linux]

$ ruby single_record.rb
Loading Configuration ...
Loading Export Data ...
Loading Reporting Options ...
Completed request for 2a206 on source.
Successfully fetched 2a206 from source.
Completed request for 2a206 on destination.
Successfully created 2a206 on destination.
Completed request for 2a206 on source file.
Successfully fetched source file called 2a206 form.pdf from 2a206 / source.
/home/matt/.rvm/gems/ruby-2.3.5/gems/curb-0.9.4/lib/curl/easy.rb:68: warning: uncaught exception from callback

...and the file is not uploaded on the destination. I get the same thing if I try ruby-2.6.3. The only corresponding log entry is in info.log (nothing in error.log):

Successfully fetched source file called 2a206 form.pdf from 2a206 / source.