instructure / canvas-data-loader Goto Github PK

--- Delete everything above this line ---

Unable to import data to DB

Installed all dependencies and database(s), ran import, waited almost 2 days for it to complete.

Additional Information

INFO 2019-10-04T22:38:17Z: cdl_runner: Connecting to the DB
ERROR 2019-10-06T13:15:22Z: cdl_runner::db_client: insert error
ERROR 2019-10-06T13:15:22Z: cdl_runner::db_client: MySqlError { ERROR 1264 (22003): Out of range value for column '(null)' at row 1 }
ERROR 2019-10-06T13:15:22Z: cdl_runner::importer: process -> for line in finalized_string -> !is_volatile -> ins_res -> is_err
ERROR 2019-10-06T13:15:22Z: cdl_runner::importer: Error(MysqlErr, State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })

• Rust Version: rustc 1.38.0 (625451e37 2019-09-23)
• Platform: Windows Server 2012 R2
• Logs: (If you can please run the CLI with: RUST_LOG=trace at the beginning and provide us the debug logs.)

Encountering some odd behaviour with Canvas Data Loader. CDL runs but files aren't downloaded and no data is loaded into the database. DumpIDs are recorded in rocksdb and it opens a connection to postgres, but with no files downloaded there's nothing to load.

I've never used Rust, but I've managed to trace it to api_client.rs:

          let uri = try!(file_to_download.url.parse());
          work_to_do.push(client.get(uri).and_then(move |res| {
            let download_path = cloned_download_path;
            let mut file = File::create(Path::new(&download_path)).expect("Failed to create download file");
            res.body().for_each(move |chunk| {
              let _ = file.write(&*chunk);
              Ok(())
            })
          })
          );
        }
      }
    }
    try!(core.run(join_all(work_to_do)));

If I modify core.run to use expect instead of try: core.run(join_all(work_to_do)).expect("Failed"); an Io Error is thrown.

thread 'main' panicked at 'Failed: Io(Error { repr: Custom(Custom { kind: Other, error: StringError("failed to lookup address information: nodename nor servname provided, or not known") }) })', src/libcore/result.rs:906:4
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic
   6: std::panicking::begin_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::result::unwrap_failed
  10: cdl_runner::api_client::CanvasDataApiClient::download_files_for_dump
  11: cdl_runner::main::{{closure}}
  12: cdl_runner::main
  13: __rust_maybe_catch_panic
  14: std::rt::lang_start

It seems that I can get CDL working if I download each file individually:

          let mut work = client.get(uri).and_then(move |res| {
            let download_path = cloned_download_path;
            let mut file = File::create(Path::new(&download_path)).expect("Failed to create download file");
            res.body().for_each(move |chunk| {
              let _ = file.write(&*chunk);
              Ok(())
            })
          });
          try!(core.run(work));

I'm happy to chase this down further, but being new to Rust, I could use some guidance. The lookup failure just seems particularly weird to me.

Additional Information

• Rust Version 1.22.1
• Platform: OSX 10.11.6
• Logs: Trace output is 211,845 lines. Can provide if desired.

Description

When I run 'cargo build --release' I get the following error:

error[E0599]: no method named expectfound for type()` in the current scope
--> src/main.rs:57:22
|
57 | env_logger::init().expect("Failed to setup logger!");
| ^^^^^^

error: aborting due to previous error

error: Could not compile cdl-runner.`

I am not proficient in rust and am not sure where to start debugging.

Thanks in advance for any help.

Additional Information

• Rust Version: rustc 1.23.0 (766bd11c8 2018-01-01)
• Platform: Mac running OS version 10.13.2
• Logs: I can't run the app with debug info since the issue is in the compile phase. Attached
  is the output of cargo build --release --verbose
  Verbose Output.txt

Not really an issue, just more of a question. What is the expected performance of the data loader and what size of dumps has it been tested against?

I'm testing using skip_historical_imports = true and only_load_final = true which amounts to a 1.1 GB dump for our institution. I've yet to successfully complete a full load, locally or on AWS. My local host is a very dated Macbook Pro, so I decided to try on AWS. Even with the loader running on d2.xlarge and RDS instances on db.r4.2xlarge, the loads are too slow to be useful.

Before I investigate too much further, I figured I'd touch base to find out more about how CDL has been tested.

Description

When I run cargo build --release, I am receiving the following error:

= note: liblibrocksdb_sys-4fe83a8a2d9f911a.rlib(env_win.o) : error LNK2019: unresolved external symbol __imp_PathIsRelativeA referenced in function "public: virtual class rocksdb::Status __cdecl rocksdb::port::WinEnvIO::GetAbsolutePath(class std::basic_string<char,struct std::char_traits,class std::allocator > const &,class std::basic_string<char,struct std::char_traits,class std::allocator > *)" (?GetAbsolutePath@WinEnvIO@port@rocksdb@@UEAA?AVStatus@3@AEBV?$basic_string@DU?$char_traits@D@std@@v?$allocator@D@2@@std@@PEAV56@@z)
C:\canvas-data-loader\target\release\deps\cdl_runner-7e809c944750d409.exe : fatal error LNK1120: 1 unresolved externals

error: aborting due to previous error

error: Could not compile cdl-runner.

Additional Information

• Rust Version: rustc 1.30.1 (1433507eb 2018-11-07)
• Platform: Windows 7, SP1. I'm running VS 2017 with the C++ build tools.
• Logs: Attached is the output from running cargo build --release --verbose
  Build Output.txt

I've managed to work through a bunch of compile errors by installing several other requirements not mentioned in your documentation, but I have come to a point where I can't find a solution. We are running Ubuntu 14.04. Any help would be very much appreciated.

This is the very long error message:

error: failed to run custom build command for librocksdb-sys v5.14.2
process didn't exit successfully: /home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-7a8e65a237a57e67/build-script-build (exit code: 101)
--- stdout
cargo:rerun-if-changed=build.rs
cargo:rerun-if-changed=rocksdb/
cargo:rerun-if-changed=snappy/
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/cache/clock_cache.o" "-c" "rocksdb/cache/clock_cache.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/cache/lru_cache.o" "-c" "rocksdb/cache/lru_cache.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/cache/sharded_cache.o" "-c" "rocksdb/cache/sharded_cache.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/builder.o" "-c" "rocksdb/db/builder.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/c.o" "-c" "rocksdb/db/c.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/column_family.o" "-c" "rocksdb/db/column_family.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/compacted_db_impl.o" "-c" "rocksdb/db/compacted_db_impl.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/compaction.o" "-c" "rocksdb/db/compaction.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/compaction_iterator.o" "-c" "rocksdb/db/compaction_iterator.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/compaction_job.o" "-c" "rocksdb/db/compaction_job.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/compaction_picker.o" "-c" "rocksdb/db/compaction_picker.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/compaction_picker_universal.o" "-c" "rocksdb/db/compaction_picker_universal.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/convenience.o" "-c" "rocksdb/db/convenience.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/db_filesnapshot.o" "-c" "rocksdb/db/db_filesnapshot.cc"
exit code: 0
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
DEBUG = Some("false")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/db_impl.o" "-c" "rocksdb/db/db_impl.cc"
cargo:warning=c++: internal compiler error: Killed (program cc1plus)
cargo:warning=Please submit a full bug report,
cargo:warning=with preprocessed source if appropriate.
cargo:warning=See file:///usr/share/doc/gcc-4.8/README.Bugs for instructions.
exit code: 4

--- stderr
rocksdb/include/rocksdb/c.h:48:9: warning: #pragma once in main file [-Wpragma-once-outside-header], err: false
thread 'main' panicked at '

Internal error occurred: Command "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.7.0/fused-src/" "-I" "snappy/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DNDEBUG=1" "-DSNAPPY=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/ubuntu/canvas-data-loader/target/release/build/librocksdb-sys-1366b337e61dd3f0/out/rocksdb/db/db_impl.o" "-c" "rocksdb/db/db_impl.cc" with args "c++" did not execute successfully (status code exit code: 4).

', /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.23/src/lib.rs:2253:5
note: Run with RUST_BACKTRACE=1 for a backtrace.

Description

The script has been running okay, but recently, we started getting this error, here is the full log:

INFO 2018-08-23T11:55:28Z: cdl_runner: Setting up API Client...
INFO 2018-08-23T11:55:28Z: cdl_runner: Connecting to RocksDB Store....
thread 'main' panicked at 'Failed to open RocksDB: Error { message: "IO error: No space left on deviceWhile appending to file: /tmp/cdl-rocksdb/000302.sst: No space left on device" }', /checkout/src/libcore/result.rs:916:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

We increased the disk space but still getting the same error.

The biggest bottlenecks of the loader are really:

• The fact we have to iterate through every record in order to determine if we need to delete a specific value.
• The fact we don't batch together DELETEs/INSERTs at all.
• The fact we're processing each table one at a time, without threading out.

All of these facts do slow it down considerably, and we've in the past avoided fixing these bottlenecks because we don't want people to rely on this full time working with Canvas Data. It's meant to be an example application showing users how to interact with the API, and how they could do things like "avoid historical refreshes".

We could one day down the road, fix these features, and turn the CDL into a fully supported solution, but that would have to be a product decision.

Originally posted by @securityinsanity in #6 (comment)

@securityinsanity, @UniversalsuperB made a remark on finding out that some id columns gets padded with 555555. I believe this is why I get the following error when I tried to run this command:
[root@localhost home]# RUST_LOG=info ./target/release/cdl-runner

Error: 
 INFO 2018-11-13T16:42:52Z: cdl_runner: Setting up API Client...
 INFO 2018-11-13T16:42:52Z: cdl_runner: Connecting to RocksDB Store....
 INFO 2018-11-13T16:42:52Z: cdl_runner: Skipping dump: f2861f59-f11c-4c97-8d09-c349e7f4f3f6 
 INFO 2018-11-13T16:42:52Z: cdl_runner: Skipping dump: 259dfc10-69b7-4f9f-bd89-013082e6c1ef due to only final selected
 INFO 2018-11-13T16:42:52Z: cdl_runner: Skipping dump: a8d4271f-2013-4ff0-8813-b8aa79e85e0b due to only final selected
 INFO 2018-11-13T16:42:53Z: cdl_runner: Connecting to the DB
ERROR 2018-11-13T16:50:35Z: cdl_runner::db_client: insert error
ERROR 2018-11-13T16:50:35Z: cdl_runner::db_client: MySqlError { ERROR 1264 (22003): Out of range value for column 'position' at row 1 }
ERROR 2018-11-13T16:50:35Z: cdl_runner::importer: process -> for line in finalized_string -> !is_volatile -> ins_res -> is_err
ERROR 2018-11-13T16:50:35Z: cdl_runner::importer: Error(MysqlErr, State { next_error: None, backtrace: None })
 INFO 2018-11-13T16:50:35Z: cdl_runner: Done!

My question is: Is there a fix for this?

As of yesterday I get cdl_runner::db_client: Error(Db(DbError { severity: "ERROR", parsed_severity: Some(Error), code: SqlState("42703"), message: "column \"external_tool_id\" of relation \"assignment_dim\" does not exist", detail: None, hint: None, position: Some(Normal(182)), where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("parse_target.c"), line: Some(954), routine: Some("checkInsertTargets") })) when running the loader.

Description

I am following the steps in canvas DOC-11943
at Step 9, I can see that %USERPROFILE%.cargo\bin is added to my Windows USER path instead of SYSTEM path.

next, I don't know what "to configure current shell run" mean. I blindly followed and executed source "%USERPROFILE%.cargo\env" and it gave this error

'source' is not recognized as an internal or external command,
operable program or batch file.

probably there should be some window's way of doing it that I don't know.

at Step 10, I initially faced Virtual studio error and then I installed Build tools with C++ and now building release version it gave me following error

[2019-06-12T19:35:34Z INFO  cargo::core::compiler::job_queue] start: librocksdb-sys v5.18.3 => Target(script)/Profile() => Host
   Compiling librocksdb-sys v5.18.3
[2019-06-12T19:35:34Z INFO  cargo::core::compiler::job_queue] start: cookie_store v0.7.0 => Target(lib)/Profile(release) => Host
[2019-06-12T19:35:34Z INFO  cargo::core::compiler::job_queue] end: cookie_store v0.7.0 => Target(lib)/Profile(release) => Host
[2019-06-12T19:35:34Z INFO  cargo::core::compiler::job_queue] start: reqwest v0.9.18 => Target(lib)/Profile(release) => Host
[2019-06-12T19:35:34Z INFO  cargo::core::compiler::job_queue] end: reqwest v0.9.18 => Target(lib)/Profile(release) => Host
[2019-06-12T19:35:34Z INFO  cargo::core::compiler::job_queue] end: librocksdb-sys v5.18.3 => Target(script)/Profile() => Host
error: failed to run custom build command for `librocksdb-sys v5.18.3`
process didn't exit successfully: `C:\Softwares\CanvasDataLoader\canvas-data-loader\target\release\build\librocksdb-sys-10a20a9d59bb2f10\build-script-build` (exit code: 101)
--- stdout
cargo:rerun-if-changed=build.rs
cargo:rerun-if-changed=rocksdb/
cargo:rerun-if-changed=snappy/
cargo:rerun-if-changed=lz4/
cargo:rerun-if-changed=zstd/
cargo:rerun-if-changed=zlib/
cargo:rerun-if-changed=bzip2/

--- stderr
thread 'main' panicked at 'Unable to find libclang: "couldn\'t find any valid shared libraries matching: [\'clang.dll\', \'libclang.dll\'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"', src\libcore\result.rs:997:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.


C:\Softwares\CanvasDataLoader\canvas-data-loader>

I am not sure if I didn't do something right in step 9 which gave me this error in 10 or something else. but I am stuck and cannot move forward.

do I need to wait for successful completion of this step to see cdl-runner.bat/cdl-runner.exe file under my C:\Softwares\CanvasDataLoader\canvas-data-loader\target\release directory?
because I am not seeing anything currently

Additional Information

• Rust Version

rustc 1.35.0 (3c235d560 2019-05-20)

• Platform:

Windows 10 Desktop

*Logs:

Attached the cargo --build --verbose
BuildOutput.txt

Hello @securityinsanity, @UniversalSuperB made a remark on finding out that some id columns gets padded with 555555. I believe this is why I get the following error when I tried to run this command:

[root@localhost home]# RUST_LOG=info ./target/release/cdl-runner

Error: 
 INFO 2018-11-13T16:42:52Z: cdl_runner: Setting up API Client...
 INFO 2018-11-13T16:42:52Z: cdl_runner: Connecting to RocksDB Store....
 INFO 2018-11-13T16:42:52Z: cdl_runner: Skipping dump: f2861f59-f11c-4c97-8d09-c349e7f4f3f6 
 INFO 2018-11-13T16:42:52Z: cdl_runner: Skipping dump: 259dfc10-69b7-4f9f-bd89-013082e6c1ef due to only final selected
 INFO 2018-11-13T16:42:52Z: cdl_runner: Skipping dump: a8d4271f-2013-4ff0-8813-b8aa79e85e0b due to only final selected
 INFO 2018-11-13T16:42:53Z: cdl_runner: Connecting to the DB
ERROR 2018-11-13T16:50:35Z: cdl_runner::db_client: insert error
ERROR 2018-11-13T16:50:35Z: cdl_runner::db_client: MySqlError { ERROR 1264 (22003): Out of range value for column 'position' at row 1 }
ERROR 2018-11-13T16:50:35Z: cdl_runner::importer: process -> for line in finalized_string -> !is_volatile -> ins_res -> is_err
ERROR 2018-11-13T16:50:35Z: cdl_runner::importer: Error(MysqlErr, State { next_error: None, backtrace: None })
 INFO 2018-11-13T16:50:35Z: cdl_runner: Done!

My question is: Is there a fix for this?

Originally posted by @latifatbraimah in #6 (comment)

Description

I have installed the Canvas-data-loader on a Windows 10 platform.

Version of rust
rustc 1.32.0 (9fda7c223 2019-01-16)
When I get to step 11 from the command prompt.
I have changed directories to canvas-data-loader-master
Everything compiled properly
Compiling librocksdb-sys v5.14.3
Compiling rocksdb v0.10.1
Compiling cdl-runner v0.4.0 (I:\CanvasData\canvas-data-loader-master)
Finished release [optimized] target(s) in 7m 00s

Is there something else that needs to be installed on a Windows machine?

Additional Information

• Rust Version: rustc 1.32.0 (9fda7c223 2019-01-16)
• Platform: Windows 10
• Logs: None

When running cargo build --release several errors are thrown for db_client.rs:
the trait r2d2::ManageConnection is not implemented for r2d2_postgres::PostgresConnectionManager

• Rust Version 1.22.1
• Platform: OSX 10.11.6

The errors seem to be related to changes in r2d2_postgres 0.14 to support r2d2 0.8.