This project forked from facebook/rocksdb
A library that provides an embeddable, persistent key-value store for fast storage optimized for AWS
Home Page: http://rocksdb.org
License: GNU General Public License v2.0
in case you would be interrested I created a binding of rocksdb-cloud for erlang based on the normal binding:
https://gitlab.com/barrel-db/erlang-rocksdb/wikis/Rocksdb-Cloud
Hope it would be useful for some.
How do I use zero copy? Does the so-called zero copy share s3 storage or not share it? I see ppts introduction of both situations, very confused. Hope to answer.
Thanks for open-sourcing this library!
Using g++ on Ubuntu 18.04, I found it necessary to compile with
CXXFLAGS=-Wno-unused-parameter make -j8
The version of g++ is
g++ (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0
The error with make -j8 was:
cloud/cloud_env.cc: In static member function ‘static rocksdb::Status rocksdb::CloudEnv::NewAwsEnv(rocksdb::Env*, const string&, const string&, const string&, const string&, const string&, const string&, const rocksdb::CloudEnvOptions&, std::shared_ptr<rocksdb::Logger>, rocksdb::CloudEnv**)’:
cloud/cloud_env.cc:146:33: error: unused parameter ‘base_env’ [-Werror=unused-parameter]
Status CloudEnv::NewAwsEnv(Env* base_env, const std::string& src_cloud_storage,
^~~~~~~~
cloud/cloud_env.cc:146:62: error: unused parameter ‘src_cloud_storage’ [-Werror=unused-parameter]
Status CloudEnv::NewAwsEnv(Env* base_env, const std::string& src_cloud_storage,
^~~~~~~~~~~~~~~~~
cloud/cloud_env.cc:147:47: error: unused parameter ‘src_cloud_object_prefix’ [-Werror=unused-parameter]
const std::string& src_cloud_object_prefix,
^~~~~~~~~~~~~~~~~~~~~~~
cloud/cloud_env.cc:148:47: error: unused parameter ‘src_cloud_region’ [-Werror=unused-parameter]
const std::string& src_cloud_region,
^~~~~~~~~~~~~~~~
cloud/cloud_env.cc:149:47: error: unused parameter ‘dest_cloud_storage’ [-Werror=unused-parameter]
const std::string& dest_cloud_storage,
^~~~~~~~~~~~~~~~~~
cloud/cloud_env.cc:150:47: error: unused parameter ‘dest_cloud_object_prefix’ [-Werror=unused-parameter]
const std::string& dest_cloud_object_prefix,
^~~~~~~~~~~~~~~~~~~~~~~~
cloud/cloud_env.cc:151:47: error: unused parameter ‘dest_cloud_region’ [-Werror=unused-parameter]
const std::string& dest_cloud_region,
^~~~~~~~~~~~~~~~~
cloud/cloud_env.cc:152:51: error: unused parameter ‘options’ [-Werror=unused-parameter]
const CloudEnvOptions& options,
^~~~~~~
cloud/cloud_env.cc:153:52: error: unused parameter ‘logger’ [-Werror=unused-parameter]
std::shared_ptr<Logger> logger, CloudEnv** cenv) {
^~~~~~
cloud/cloud_env.cc:153:71: error: unused parameter ‘cenv’ [-Werror=unused-parameter]
std::shared_ptr<Logger> logger, CloudEnv** cenv) {
^~~~
Maybe you can fix these unused parameters?
I'm maintaining a binding of rocksdb-cloud in erlang. For now it's maintainned as a branch of the rocksdb binding but I'm considering to split it in its own repository. Do you think rocksdb-cloud will at some point completely difffers from rocksdb? does it makes sense to consider they are different project? I'm wondering if t wouldn't possible to have only the env differing. Thoughts?
Fresh debug build, make check starts failing in db_basic_test.
$ uname -a
Linux ip-10-0-1-81 4.4.0-70-generic #91-Ubuntu SMP Wed Mar 22 12:47:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ g++ --version
g++ (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Running in an ec2 instance, output from gdb:
db/db_test_util.cc:564: Failure
TryReopen(options)
IO error: /tmp/rocksdbtest-1000/db_basic_test: Access Denied
Thread 1 "db_basic_test" received signal SIGSEGV, Segmentation fault.
0x00000000004e20d4 in rocksdb::DBTestBase::CreateColumnFamilies (this=this@entry=0xceef10,
cfs=std::vector of length 1, capacity 1 = {...}, options=...) at db/db_test_util.cc:507
507 ASSERT_OK(db_->CreateColumnFamily(cf_opts, cf, &handles_[cfi++]));
(gdb) p cfi
$1 =
(gdb) p cfs
$2 = std::vector of length 1, capacity 1 = {"pikachu"}
(gdb) p options
$3 = (const rocksdb::Options &) @0x7fffffffddb0: {rocksdb::DBOptions = {create_if_missing = true,
create_missing_column_families = false, error_if_exists = false, paranoid_checks = true, env = 0xcd0fd0,
rate_limiter = std::shared_ptr (empty) 0x0, sst_file_manager = std::shared_ptr (empty) 0x0,
info_log = std::shared_ptr (empty) 0x0, info_log_level = rocksdb::DEBUG_LEVEL, max_open_files = 5000,
max_file_opening_threads = 16, max_total_wal_size = 0, statistics = std::shared_ptr (empty) 0x0,
use_fsync = false, db_paths = std::vector of length 0, capacity 0, db_log_dir = "", wal_dir = "",
delete_obsolete_files_period_micros = 21600000000, base_background_compactions = -1,
max_background_compactions = 1, max_subcompactions = 1, max_background_flushes = 1, max_log_file_size = 0,
log_file_time_to_roll = 0, keep_log_file_num = 1000, recycle_log_file_num = 0,
max_manifest_file_size = 18446744073709551615, table_cache_numshardbits = 6, WAL_ttl_seconds = 0,
WAL_size_limit_MB = 0, manifest_preallocation_size = 4194304, allow_mmap_reads = false,
allow_mmap_writes = false, use_direct_reads = false, use_direct_writes = false, allow_fallocate = true,
is_fd_close_on_exec = true, skip_log_error_on_recovery = false, stats_dump_period_sec = 600,
advise_random_on_open = true, db_write_buffer_size = 0, write_buffer_manager = std::shared_ptr (empty) 0x0,
access_hint_on_compaction_start = rocksdb::DBOptions::NORMAL, new_table_reader_for_compaction_inputs = false,
compaction_readahead_size = 0, random_access_max_buffer_size = 1048576, writable_file_max_buffer_size = 1048576,
use_adaptive_mutex = false, bytes_per_sync = 0, wal_bytes_per_sync = 0,
listeners = std::vector of length 0, capacity 0, enable_thread_tracking = false, delayed_write_rate = 16777216,
allow_concurrent_memtable_write = true, enable_write_thread_adaptive_yield = true,
write_thread_max_yield_usec = 100, write_thread_slow_yield_usec = 3, skip_stats_update_on_db_open = false,
wal_recovery_mode = rocksdb::WALRecoveryMode::kTolerateCorruptedTailRecords, allow_2pc = false,
row_cache = std::shared_ptr (empty) 0x0, wal_filter = 0x0, fail_if_options_file_error = true,
dump_malloc_stats = false, avoid_flush_during_recovery = false,
avoid_flush_during_shutdown = false}, rocksdb::ColumnFamilyOptions = {rocksdb::AdvancedColumnFamilyOptions = {
max_write_buffer_number = 2, min_write_buffer_number_to_merge = 1, max_write_buffer_number_to_maintain = 0,
inplace_update_support = false, inplace_update_num_locks = 10000, inplace_callback = 0x0,
memtable_prefix_bloom_size_ratio = 0, memtable_huge_page_size = 0,
arena_block_size = 0, compression_per_level = std::vector of length 0, capacity 0, num_levels = 7,
---Type to continue, or q to quit---
level0_slowdown_writes_trigger = 20, level0_stop_writes_trigger = 36, target_file_size_base = 2097152,
target_file_size_multiplier = 1, level_compaction_dynamic_level_bytes = false,
max_bytes_for_level_multiplier = 10,
max_bytes_for_level_multiplier_additional = std::vector of length 7, capacity 7 = {1, 1, 1, 1, 1, 1, 1},
max_compaction_bytes = 0, soft_pending_compaction_bytes_limit = 68719476736,
hard_pending_compaction_bytes_limit = 274877906944, compaction_style = rocksdb::kCompactionStyleLevel,
compaction_pri = rocksdb::kByCompensatedSize, compaction_options_universal = {size_ratio = 1,
min_merge_width = 2, max_merge_width = 4294967295, max_size_amplification_percent = 200,
compression_size_percent = -1, stop_style = rocksdb::kCompactionStopStyleTotalSize,
allow_trivial_move = false}, compaction_options_fifo = {max_table_files_size = 1073741824},
max_sequential_skip_in_iterations = 8, memtable_factory = std::shared_ptr (count 2, weak 0) 0xd79020,
table_properties_collector_factories = std::vector of length 0, capacity 0, max_successive_merges = 0,
optimize_filters_for_hits = false, paranoid_file_checks = false, force_consistency_checks = false,
report_bg_io_stats = false, max_mem_compaction_level = 0, soft_rate_limit = 0, hard_rate_limit = 0,
rate_limit_delay_max_milliseconds = 100, purge_redundant_kvs_while_flush = true}, comparator =
0xae8680 rocksdb::BytewiseComparator()::bytewise, merge_operator = std::shared_ptr (empty) 0x0,
compaction_filter = 0x0, compaction_filter_factory = std::shared_ptr (empty) 0x0, write_buffer_size = 16752640,
compression = , bottommost_compression = , compression_opts = {
window_bits = -14, level = -1, strategy = 0, max_dict_bytes = 0}, level0_file_num_compaction_trigger = 4,
prefix_extractor = std::shared_ptr (count 2, weak 0) 0xcd1210, max_bytes_for_level_base = 10485760,
disable_auto_compactions = false, table_factory = std::shared_ptr (count 2, weak 0) 0xe024b0}, }
(gdb) l
502 const Options& options) {
503 ColumnFamilyOptions cf_opts(options);
504 size_t cfi = handles_.size();
505 handles_.resize(cfi + cfs.size());
506 for (auto cf : cfs) {
507 ASSERT_OK(db_->CreateColumnFamily(cf_opts, cf, &handles_[cfi++]));
508 }
509 }
510
511 void DBTestBase::CreateAndReopenWithCF(const std::vectorstd::string& cfs,
(gdb) p cf
$4 = "pikachu"
(gdb) p handles_.size()
$5 = 1
(gdb) p cfs.size()
$6 = 1
(gdb) p handles_
$7 = std::vector of length 1, capacity 2 = {0x0}
(gdb) p cf_opts
$8 = {rocksdb::AdvancedColumnFamilyOptions = {max_write_buffer_number = 2, min_write_buffer_number_to_merge = 1,
max_write_buffer_number_to_maintain = 0, inplace_update_support = false, inplace_update_num_locks = 10000,
inplace_callback = 0x0, memtable_prefix_bloom_size_ratio = 0, memtable_huge_page_size = 0,
memtable_insert_with_hint_prefix_extractor = std::shared_ptr (empty) 0x0, bloom_locality = 0,
arena_block_size = 0, compression_per_level = std::vector of length 0, capacity 0, num_levels = 7,
level0_slowdown_writes_trigger = 20, level0_stop_writes_trigger = 36, target_file_size_base = 2097152,
target_file_size_multiplier = 1, level_compaction_dynamic_level_bytes = false,
max_bytes_for_level_multiplier = 10,
max_bytes_for_level_multiplier_additional = std::vector of length 7, capacity 7 = {1, 1, 1, 1, 1, 1, 1},
max_compaction_bytes = 0, soft_pending_compaction_bytes_limit = 68719476736,
hard_pending_compaction_bytes_limit = 274877906944, compaction_style = rocksdb::kCompactionStyleLevel,
compaction_pri = rocksdb::kByCompensatedSize, compaction_options_universal = {size_ratio = 1,
min_merge_width = 2, max_merge_width = 4294967295, max_size_amplification_percent = 200,
compression_size_percent = -1, stop_style = rocksdb::kCompactionStopStyleTotalSize,
allow_trivial_move = false}, compaction_options_fifo = {max_table_files_size = 1073741824},
max_sequential_skip_in_iterations = 8, memtable_factory = std::shared_ptr (count 2, weak 0) 0xd79020,
table_properties_collector_factories = std::vector of length 0, capacity 0, max_successive_merges = 0,
optimize_filters_for_hits = false, paranoid_file_checks = false, force_consistency_checks = false,
report_bg_io_stats = false, max_mem_compaction_level = 0, soft_rate_limit = 0, hard_rate_limit = 0,
rate_limit_delay_max_milliseconds = 100, purge_redundant_kvs_while_flush = true},
comparator = 0xae8680 rocksdb::BytewiseComparator()::bytewise, merge_operator = std::shared_ptr (empty) 0x0,
compaction_filter = 0x0, compaction_filter_factory = std::shared_ptr (empty) 0x0, write_buffer_size = 16752640,
compression = , bottommost_compression = , compression_opts = {window_bits = -14,
level = -1, strategy = 0, max_dict_bytes = 0}, level0_file_num_compaction_trigger = 4,
prefix_extractor = std::shared_ptr (count 2, weak 0) 0xcd1210, max_bytes_for_level_base = 10485760,
disable_auto_compactions = false, table_factory = std::shared_ptr (count 2, weak 0) 0xe024b0}
Any suggestions?
Chatz
From https://travis-ci.org/rockset/rocksdb-cloud/jobs/203666423
[ RUN ] DBTest.GetPicksCorrectFile
db/db_test_util.cc:896: Failure
db_->CompactRange(CompactRangeOptions(), handles_[cf], &start, &limit)
Corruption: Bad table magic number
terminate called after throwing an instance of 'testing::internal::GoogleTestFailureException'
what(): db/db_test_util.cc:896: Failure
db_->CompactRange(CompactRangeOptions(), handles_[cf], &start, &limit)
Corruption: Bad table magic number
Received signal 6 (Aborted)
#0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x2ac33757ec37] ?? ??:0
#1 /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x2ac337582028] ?? ??:0
#2 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x15d) [0x2ac336d84bbd] ?? ??:0
#3 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x69b96) [0x2ac336d82b96] ?? ??:0
#4 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x69be1) [0x2ac336d82be1] ?? ??:0
#5 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__cxa_rethrow+0x46) [0x2ac336d82e46] ?? ??:0
#6 ./db_test() [0x829ea5] HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> /home/travis/build/rockset/rocksdb-cloud/third-party/gtest-1.7.0/fused-src/gtest/gtest-all.cc:3865
#7 ./db_test() [0x41004d] main /home/travis/build/rockset/rocksdb-cloud/db/db_test.cc:5809
#8 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x2ac337569f45] ?? ??:0
#9 ./db_test() [0x4ca7fa] _start ??:?
/bin/bash: line 1: 7483 Aborted (core dumped) ./$t
make: *** [check_some] Error 1
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:28.920444 2ac33e6d4700 (Original Log Time 2017/02/21-03:18:28.802884) [pikachu] Manual compaction from level-0 to level-1 from 'a' @ 72057594037927935: 7 .. 'b' @ 0: 0; will stop at (end)
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:28.920448 2ac33e6d4700 (Original Log Time 2017/02/21-03:18:28.802889) [pikachu] Moving #13 to level-1 894 bytes
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:28.920450 2ac33e6d4700 (Original Log Time 2017/02/21-03:18:28.920408) EVENT_LOG_v1 {"time_micros": 1487647108920397, "job": 4, "event": "trivial_move", "destination_level": 1, "files": 1, "total_files_size": 894}
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:28.920451 2ac33e6d4700 (Original Log Time 2017/02/21-03:18:28.920423) [pikachu] Moved #1 files to level-1 894 bytes OK: base level 1 max bytes base 10485760 files[0 1 0 0 0 0 0] max score 0.00
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:28.920683 2ac335763cc0 [pikachu] New memtable created with log file: #14. Immutable memtables: 0.
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:28.920734 2ac33e8d5700 [JOB 5] Syncing log #12
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:28.923178 2ac33e8d5700 (Original Log Time 2017/02/21-03:18:28.920726) Calling FlushMemTableToOutputFile with column family [pikachu], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:28.923183 2ac33e8d5700 [pikachu] [JOB 5] Flushing memtable with next log file: 14
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:28.923200 2ac33e8d5700 EVENT_LOG_v1 {"time_micros": 1487647108923193, "job": 5, "event": "flush_started", "num_memtables": 1, "num_entries": 1, "num_deletes": 0, "memory_usage": 816}
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:28.923203 2ac33e8d5700 [pikachu] [JOB 5] Level-0 flush table #15: started
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.332420 2ac33e8d5700 EVENT_LOG_v1 {"time_micros": 1487647109332402, "cf_name": "pikachu", "job": 5, "event": "table_file_creation", "file_number": 15, "file_size": 894, "table_properties": {"data_size": 27, "index_size": 27, "filter_size": 0, "raw_key_size": 9, "raw_average_key_size": 9, "raw_value_size": 2, "raw_average_value_size": 2, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}}
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.332467 2ac33e8d5700 [pikachu] [JOB 5] Level-0 flush table #15: 894 bytes OK
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.453320 2ac33e8d5700 (Original Log Time 2017/02/21-03:18:29.334320) [pikachu] Level-0 commit table #15 started
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.453323 2ac33e8d5700 (Original Log Time 2017/02/21-03:18:29.453261) [pikachu] Level-0 commit table #15: memtable #1 done
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.453324 2ac33e8d5700 (Original Log Time 2017/02/21-03:18:29.453281) EVENT_LOG_v1 {"time_micros": 1487647109453271, "job": 5, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.453326 2ac33e8d5700 (Original Log Time 2017/02/21-03:18:29.453306) [pikachu] Level summary: base level 1 max bytes base 10485760 files[1 1 0 0 0 0 0] max score 0.25
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.453333 2ac33e8d5700 [JOB 5] Try to delete WAL files size 19, prev total WAL file size 19, number of live WAL files 2.
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.453367 2ac33e8d5700 [DEBUG] [JOB 5] Delete /tmp/rocksdbtest-1000/db_test//000012.log type=0 #12 -- OK
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.453540 2ac335763cc0 [pikachu] Manual compaction starting
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.574306 2ac33e6d4700 (Original Log Time 2017/02/21-03:18:29.453614) [pikachu] Manual compaction from level-0 to level-1 from 'x' @ 72057594037927935: 7 .. 'y' @ 0: 0; will stop at (end)
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.574309 2ac33e6d4700 (Original Log Time 2017/02/21-03:18:29.453620) [pikachu] Moving #15 to level-1 894 bytes
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.574311 2ac33e6d4700 (Original Log Time 2017/02/21-03:18:29.574277) EVENT_LOG_v1 {"time_micros": 1487647109574265, "job": 6, "event": "trivial_move", "destination_level": 1, "files": 1, "total_files_size": 894}
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.574312 2ac33e6d4700 (Original Log Time 2017/02/21-03:18:29.574287) [pikachu] Moved #1 files to level-1 894 bytes OK: base level 1 max bytes base 10485760 files[0 2 0 0 0 0 0] max score 0.00
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.574546 2ac335763cc0 [pikachu] New memtable created with log file: #16. Immutable memtables: 0.
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.574603 2ac33e8d5700 [JOB 7] Syncing log #14
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.576963 2ac33e8d5700 (Original Log Time 2017/02/21-03:18:29.574595) Calling FlushMemTableToOutputFile with column family [pikachu], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.576968 2ac33e8d5700 [pikachu] [JOB 7] Flushing memtable with next log file: 16
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.576982 2ac33e8d5700 EVENT_LOG_v1 {"time_micros": 1487647109576975, "job": 7, "event": "flush_started", "num_memtables": 1, "num_entries": 1, "num_deletes": 0, "memory_usage": 816}
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.576986 2ac33e8d5700 [pikachu] [JOB 7] Level-0 flush table #17: started
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.950814 2ac33e8d5700 [pikachu] [JOB 7] Level-0 flush table #17: 894 bytes Corruption: Bad table magic number
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.952749 2ac33e8d5700 [ERROR] Waiting after background flush error: Corruption: Bad table magic numberAccumulated background error counts: 1
/tmp/rocksdbtest-1000/db_test/LOG:2017/02/21-03:18:29.952754 2ac33e8d5700 (Original Log Time 2017/02/21-03:18:29.952727) EVENT_LOG_v1 {"time_micros": 1487647109952717, "job": 7, "event": "flush_finished", "lsm_state": [0, 2, 0, 0, 0, 0, 0], "immutable_memtables": 1}
Per discussion with Dhruba Borthakur, please add teddybearz to the committer list to make changes related to the Applatix cluster. Attached is the signed CLA form. Thanks!
Exciting project! I see that you use S3 quite heavily. For those of us on humble budgets, how would you describe the costs that would be incurred by using S3 + RocksDB-Cloud? Thanks!
This is a design question, not necessarily a bug issue.
Let's say I'm designing a multi-tenant system, with separate accounts.
Is it better for scaling and performance to use one RocksDB and always add the tenant id as prefix in the key?
Or is it better for scaling and performance to use a separate RocksDB for each tenant and just have all the DBs opened as needed?
In the tenant id as key prefix scenario, I think there would also be the overhead of separate Kafka WAL for each tenant. What are the key architecture overhead that are shared if I used key prefix instead of keeping separate tenant RocksDBs?
Obviously, in the one db per tenant world, there's much stronger isolation, and purging tenant data becomes a lot easier, but I'm worried about any hidden gotchas with regard to overhead of keeping multiple DBs open simultaneously.
Could you pls. add extra parameters for AWS client and SDK configuration?
I have to use an alternative S3 server address:port instead of external AWS - own Minio server.
Proposing more generic way to do this + specify extra useful stuff like proxy support, timeout and AWS logging.
To extend CloudEnvOptions class adding aws_options object of new AwsOptions class:
--- ./server-10.4/storage/rocksdb/rocksdb-cloud_public_master_2018-11-13_01-57-33/include/rocksdb/cloud/cloud_env_options.h.orig 2018-11-13 02:59:46.997484586 +0300
+++ ./server-10.4/storage/rocksdb/rocksdb-cloud_public_master_2018-11-13_01-57-33/include/rocksdb/cloud/cloud_env_options.h 2018-11-13 05:53:37.681321957 +0300
@@ -46,6 +46,47 @@
std::unordered_map<std::string, std::string> client_config_params;
};
+// Extra data to fill-in Aws::SDKOptions and Aws::Client::ClientConfiguration
+// inside AwsEnv::AwsEnv()
+// Not usable values, using defaults instead:
+// -1 for int/long
+// "" for std::string
+class AwsOptions {
+ public:
+ // General SDK options: logging
+ int logLevel; // see https://sdk.amazonaws.com/cpp/api/0.14.3/aws-cpp-sdk-core_2include_2aws_2core_2utils_2logging_2_log_level_8h_source.html
+
+ // Timeout control
+ long requestTimeoutMs;
+ long connectTimeoutMs;
+
+ // Endpoint override
+ std::string endpointOverride; // local S3/Minio server "127.0.0.1:9000"
+ std::string scheme; // "http" or "https"
+ int verifySSL; // 0 - don't verify, 1 - do verify
+
+ // Proxy settings, not using by default and if proxyHost == ""
+ std::string proxyScheme;
+ std::string proxyHost;
+ unsigned proxyPort;
+ std::string proxyUserName;
+ std::string proxyPassword;
+
+ // Setting not usable values as defaults (librocksdb/AWS defaults will be used)
+ AwsOptions() :
+ logLevel(-1),
+ requestTimeoutMs(-1),
+ connectTimeoutMs(-1),
+ endpointOverride(""),
+ scheme(""),
+ verifySSL(-1),
+ proxyScheme(""),
+ proxyHost(""),
+ proxyPort(0),
+ proxyUserName(""),
+ proxyPassword("") {}
+};
+
enum class CloudRequestOpType {
kReadOp,
kWriteOp,
@@ -77,6 +118,9 @@
// Only used if keep_local_log_files is true and log_type is kKafka.
KafkaLogOptions kafka_log_options;
+ // To specify AWS related extras: server IP:port, scheme, timeouts, proxy.
+ AwsOptions aws_options;
+
//
// If true, then sst files are stored locally and uploaded to the cloud in
// the background. On restart, all files from the cloud that are not present
And use it inside AwsEnv::AwsEnv()
--- ./server-10.4/storage/rocksdb/rocksdb-cloud_public_master_2018-11-13_01-57-33/cloud/aws/aws_env.cc.orig<--->2018-11-13 02:14:46.789526686 +0300
+++ ./server-10.4/storage/rocksdb/rocksdb-cloud_public_master_2018-11-13_01-57-33/cloud/aws/aws_env.cc<>2018-11-13 06:35:15.237283017 +0300
@@ -276,11 +276,56 @@
creds ? "[given]" : "[not given]");
.
base_env_ = underlying_env;
- Aws::InitAPI(Aws::SDKOptions());
- // create AWS S3 client with appropriate timeouts
+
+ // Get additional data for 'sdk_options' and 'config' from cloud_env_options.aws_options
+ // Check if the corresponded value is not usable - use defaults
+
+ Aws::SDKOptions sdk_options;
Aws::Client::ClientConfiguration config;
- config.connectTimeoutMs = 30000;
- config.requestTimeoutMs = 600000;
+
+ // General SDK options: logging
+ if (cloud_env_options.aws_options.logLevel != -1)
+ sdk_options.loggingOptions.logLevel = (Aws::Utils::Logging::LogLevel)cloud_env_options.aws_options.logLevel;
+ Aws::InitAPI(sdk_options);
+
+ // Timeout control
+ if (cloud_env_options.aws_options.connectTimeoutMs != -1)
+ config.connectTimeoutMs = cloud_env_options.aws_options.connectTimeoutMs;
+ else
+ config.connectTimeoutMs = 30000;
+
+ if (cloud_env_options.aws_options.requestTimeoutMs != -1)
+ config.requestTimeoutMs = cloud_env_options.aws_options.requestTimeoutMs;
+ else
+ config.requestTimeoutMs = 600000;
+
+ // Endpoint override
+ if (cloud_env_options.aws_options.endpointOverride != "")
+ config.endpointOverride = cloud_env_options.aws_options.endpointOverride.c_str();
+
+ if (cloud_env_options.aws_options.scheme != "")
+ config.scheme = Aws::Http::SchemeMapper::FromString(cloud_env_options.aws_options.scheme.c_str());
+
+ if (cloud_env_options.aws_options.verifySSL != -1)
+ config.verifySSL = (bool)cloud_env_options.aws_options.verifySSL;
+
+ // Proxy settings, not using by default and if proxyHost == ""
+ if (cloud_env_options.aws_options.proxyHost != "")
+ {
+ config.proxyHost = cloud_env_options.aws_options.proxyHost.c_str();
+
+ if (cloud_env_options.aws_options.proxyScheme != "")
+ config.proxyScheme = Aws::Http::SchemeMapper::FromString(cloud_env_options.aws_options.proxyScheme.c_str());
+
+ if (cloud_env_options.aws_options.proxyPort > 0)
+ config.proxyPort = cloud_env_options.aws_options.proxyPort;
+
+ if (cloud_env_options.aws_options.proxyUserName != "")
+ config.proxyUserName = cloud_env_options.aws_options.proxyUserName.c_str();
+
+ if (cloud_env_options.aws_options.proxyPassword != "")
+ config.proxyPassword = cloud_env_options.aws_options.proxyPassword.c_str();
+ }
// Setup how retries need to be done
config.retryStrategy =
Set corresponded fields of cloud_env_options.aws_options for AwsEnv::AwsEnv() call.
Worked fine for me using local Minio server.
Currently only RocksDB options are printer in the header of the LOG.
When we clone a rocksdb cloud instance, two separate instance of rocksdb are sharing the same set of sst files. Each instance has sst-file-deletions switched off. This means that we need a 'Purger' that can look at all rocksdb instance on that cloud storage and delete files that are not part of any of the rocksdb instances.
I run a simplified version of db_bench (only contains insert/read) with keep_local_sst_files=false and after around five minutes, it crashed silently. From the log, I can see it happened during the compaction between L0 to L1. I suspect it may be related to fetching from and writing to S3 at the same time which causes error in libcurl.
op=randomfill
key_size=16
value_size=5000
num_insertion=1000000
keep_local_sst_files=false
// Here I set smaller values to reproduce the error more quickly.
options.write_buffer_size = 8 << 20
options.max_bytes_for_level_base = 32 * 1048576
options.target_file_size_base = 16 * 1024 * 1024
Ubuntu 20.04LTS. (P.S. The problem also appears in my Ubuntu 18.04 EC2 instance)
Thread 5 "rocksdb:high0" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7ffff5326700 (LWP 234292)]
__libc_write (nbytes=31, buf=0x7fffe8037423, fd=10) at ../sysdeps/unix/sysv/linux/write.c:26
26 ../sysdeps/unix/sysv/linux/write.c: No such file or directory.
(gdb) bt
#0 __libc_write (nbytes=31, buf=0x7fffe8037423, fd=10) at ../sysdeps/unix/sysv/linux/write.c:26
#1 __libc_write (fd=10, buf=0x7fffe8037423, nbytes=31) at ../sysdeps/unix/sysv/linux/write.c:24
#2 0x00007ffff7198359 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#3 0x00007ffff719353e in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#4 0x00007ffff7192554 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#5 0x00007ffff7192a17 in BIO_write () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#6 0x00007ffff6f4cdde in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#7 0x00007ffff6f4dcd9 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#8 0x00007ffff6f5786e in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#9 0x00007ffff6f55a65 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#10 0x00007ffff6f60ea3 in SSL_shutdown () from /lib/x86_64-linux-gnu/libssl.so.1.1
#11 0x00007ffff7426235 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#12 0x00007ffff74262a5 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#13 0x00007ffff73e481e in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#14 0x00007ffff73e70e5 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#15 0x00007ffff73f9a5f in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#16 0x00007ffff73fa8c1 in curl_multi_perform () from /lib/x86_64-linux-gnu/libcurl.so.4
#17 0x00007ffff73f0d3b in curl_easy_perform () from /lib/x86_64-linux-gnu/libcurl.so.4
#18 0x00007ffff7b9bdf0 in Aws::Http::CurlHttpClient::MakeRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const ()
from /usr/local/lib/libaws-cpp-sdk-core.so
#19 0x00007ffff7bd9459 in Aws::Client::AWSClient::AttemptOneRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*) const ()
from /usr/local/lib/libaws-cpp-sdk-core.so
#20 0x00007ffff7bf780b in Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*) const ()
from /usr/local/lib/libaws-cpp-sdk-core.so
#21 0x00007ffff7bf9abd in Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*) const ()
from /usr/local/lib/libaws-cpp-sdk-core.so
#22 0x00007ffff7e6e373 in Aws::S3::S3Client::PutObject(Aws::S3::Model::PutObjectRequest const&) const ()
from /usr/local/lib/libaws-cpp-sdk-s3.so
#23 0x000055555589623d in rocksdb::AwsS3ClientWrapper::PutCloudObject (size_hint=4485000, request=...,
this=0x555555a41190) at /usr/include/c++/9/bits/shared_ptr_base.h:1020
#24 rocksdb::S3StorageProvider::DoPutCloudObject (this=0x555555a1bc20,
local_file="/tmp/bench_cloud/000084.sst-6d650822daa1fee9",
bucket_name="rockset.cloud-db-examples.alec",
object_path="/tmp/bench_cloud/000084.sst-6d650822daa1fee9", file_size=4485000)
at cloud/aws/aws_s3.cc:889
#25 0x00005555558a7229 in rocksdb::CloudStorageProviderImpl::PutCloudObject (this=0x555555a1bc20,
local_file="/tmp/bench_cloud/000084.sst-6d650822daa1fee9",
bucket_name="rockset.cloud-db-examples.alec",
object_path="/tmp/bench_cloud/000084.sst-6d650822daa1fee9") at cloud/cloud_storage_provider.cc:361
#26 0x00005555556dd16b in rocksdb::CloudEnvImpl::CopyLocalFileToDest (this=0x555555a1c640,
local_name="/tmp/bench_cloud/000084.sst-6d650822daa1fee9",
dest_name="/tmp/bench_cloud/000084.sst-6d650822daa1fee9") at /usr/include/c++/9/bits/basic_string.h:936
#27 0x00005555558a5949 in rocksdb::CloudStorageWritableFileImpl::Close (this=0x7fffe808a430)
at cloud/cloud_storage_provider.cc:175
#28 0x00005555555f8b55 in rocksdb::LegacyWritableFileWrapper::Close (this=<optimized out>)
at /usr/include/c++/9/bits/unique_ptr.h:360
#29 0x000055555561a1ea in rocksdb::WritableFileWriter::Close (this=this@entry=0x555555b1c460)
at ./include/rocksdb/status.h:348
#30 0x00005555558ad2b0 in rocksdb::BuildTable (dbname=..., env=<optimized out>, fs=<optimized out>,
--Type <RET> for more, q to quit, c to continue without paging--
ioptions=..., mutable_cf_options=..., file_options=..., table_cache=<optimized out>,
iter=<optimized out>, range_del_iters=..., meta=<optimized out>, internal_comparator=...,
int_tbl_prop_collector_factories=<optimized out>, column_family_id=<optimized out>,
column_family_name=..., snapshots=..., earliest_write_conflict_snapshot=<optimized out>,
snapshot_checker=<optimized out>, compression=<optimized out>, sample_for_compression=<optimized out>,
compression_opts=..., paranoid_file_checks=<optimized out>, internal_stats=<optimized out>,
reason=<optimized out>, event_logger=<optimized out>, job_id=<optimized out>,
io_priority=<optimized out>, table_properties=<optimized out>, level=<optimized out>,
creation_time=<optimized out>, oldest_key_time=<optimized out>, write_hint=<optimized out>,
file_creation_time=<optimized out>) at db/builder.cc:215
#31 0x00005555557a49c7 in rocksdb::FlushJob::WriteLevel0Table (this=0x7ffff5324140)
at ./db/column_family.h:328
#32 0x00005555557a6520 in rocksdb::FlushJob::Run (this=this@entry=0x7ffff5324140,
prep_tracker=prep_tracker@entry=0x555555b250f8, file_meta=file_meta@entry=0x7ffff5323c50)
at db/flush_job.cc:226
#33 0x0000555555744451 in rocksdb::DBImpl::FlushMemTableToOutputFile (this=0x555555b23df0,
cfd=0x555555b31430, mutable_cf_options=..., made_progress=0x7ffff5324d03, job_context=0x7ffff5324d40,
superversion_context=0x7fffe8000de0, snapshot_seqs=std::vector of length 0, capacity 0,
earliest_write_conflict_snapshot=72057594037927935, snapshot_checker=0x0, log_buffer=0x7ffff5324f10,
thread_pri=rocksdb::Env::HIGH) at db/db_impl/db_impl_compaction_flush.cc:191
#34 0x0000555555747bba in rocksdb::DBImpl::FlushMemTablesToOutputFiles (this=0x555555b23df0,
bg_flush_args=..., made_progress=0x7ffff5324d03, job_context=0x7ffff5324d40,
log_buffer=0x7ffff5324f10, thread_pri=rocksdb::Env::HIGH) at db/db_impl/db_impl_compaction_flush.cc:256
#35 0x000055555574905c in rocksdb::DBImpl::BackgroundFlush (this=<optimized out>,
made_progress=0x7ffff5324d03, job_context=0x7ffff5324d40, log_buffer=0x7ffff5324f10,
reason=0x7ffff5324d04, thread_pri=rocksdb::Env::HIGH) at db/db_impl/db_impl_compaction_flush.cc:2200
#36 0x0000555555752b5c in rocksdb::DBImpl::BackgroundCallFlush (this=0x555555b23df0,
thread_pri=rocksdb::Env::HIGH) at db/db_impl/db_impl_compaction_flush.cc:2237
#37 0x00005555556b636a in std::function<void ()>::operator()() const (this=0x7ffff5325880)
at /usr/include/c++/9/bits/std_function.h:683
#38 rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x555555a1b430, thread_id=0)
at util/threadpool_imp.cc:265
#39 0x00005555556b64fe in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x555555b20f30)
at util/threadpool_imp.cc:306
#40 0x00007ffff7883cb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#41 0x00007ffff7ad0609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#42 0x00007ffff7573103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
drwxr-xr-x 2 alec alec 4096 Sept 1 12:55 ./
drwxrwxrwt 34 root root 12288 Sept 1 15:16 ../
-rw-r--r-- 1 alec alec 8089317 Sept 1 12:55 000080.log
-rw-r--r-- 1 alec alec 1406175 Sept 1 12:55 000083.log
-rw-r--r-- 1 alec alec 4485000 Sept 1 12:55 000084.sst-6d650822daa1fee9
-rw-r--r-- 1 alec alec 34 Sept 1 12:44 CLOUDMANIFEST
-rw-r--r-- 1 alec alec 16 Sept 1 12:44 CURRENT
-rw-r--r-- 1 alec alec 37 Sept 1 12:44 IDENTITY
-rw-rw-r-- 1 alec alec 173288 Sept 1 12:55 LOG
-rw-r--r-- 1 alec alec 4131 Sept 1 12:55 MANIFEST-6d650822daa1fee9
-rw-r--r-- 1 alec alec 5270 Sept 1 12:44 OPTIONS-000005
2020/09/01-12:46:28.320495 7ffff5326700 [ush_job.cc:347] [default] [JOB 39] Level-0 flush table #81: started
2020/09/01-12:46:30.883438 7ffff5326700 [s3] PutCloudObject rockset.cloud-db-examples.alec//tmp/bench_cloud/000081.sst-6d650822daa1fee9, size 4493335, OK
2020/09/01-12:46:31.557440 7ffff5326700 EVENT_LOG_v1 {"time_micros": 1598935591557385, "cf_name": "default", "job": 39, "event": "table_file_creation", "file_number": 81, "file_size": 4493335, "table_properties": {"data_size": 4467761, "index_size": 44425, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 2117, "raw_key_size": 38496, "raw_average_key_size": 24, "raw_value_size": 8020000, "raw_average_value_size": 5000, "num_data_blocks": 1604, "num_entries": 1604, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "Snappy", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1598935585, "oldest_key_time": 1598935585, "file_creation_time": 1598935588}}
2020/09/01-12:46:31.557477 7ffff5326700 [ush_job.cc:394] [default] [JOB 39] Level-0 flush table #81: 4493335 bytes OK
2020/09/01-12:46:31.758168 7ffff5326700 [s3] PutCloudObject rockset.cloud-db-examples.alec//tmp/bench_cloud/MANIFEST-6d650822daa1fee9, size 3935, OK
2020/09/01-12:46:31.758266 7ffff5326700 [WARN] [lumn_family.cc:845] [default] Stopping writes because we have 37 level-0 files
2020/09/01-12:46:31.822738 7ffff5326700 (Original Log Time 2020/09/01-12:46:31.575978) [mtable_list.cc:445] [default] Level-0 commit table #81 started
2020/09/01-12:46:31.822750 7ffff5326700 (Original Log Time 2020/09/01-12:46:31.758230) [mtable_list.cc:501] [default] Level-0 commit table #81: memtable #1 done
2020/09/01-12:46:31.822754 7ffff5326700 (Original Log Time 2020/09/01-12:46:31.758258) EVENT_LOG_v1 {"time_micros": 1598935591758242, "job": 39, "event": "flush_finished", "output_compression": "Snappy", "lsm_state": [37, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0}
2020/09/01-12:46:31.822759 7ffff5326700 (Original Log Time 2020/09/01-12:46:31.758292) [_impl/db_impl_compaction_flush.cc:203] [default] Level summary: files[37 0 0 0 0 0 0] max score 8.25
2020/09/01-12:46:31.822779 7ffff5326700 [_impl/db_impl_files.cc:348] [JOB 39] Try to delete WAL files size 8076360, prev total WAL file size 8076360, number of live WAL files 2.
2020/09/01-12:54:15.305865 7fffdc7f8700 [_impl/db_impl.cc:849] ------- DUMPING STATS -------
2020/09/01-12:54:15.305988 7fffdc7f8700 [_impl/db_impl.cc:851]
** DB Stats **
Uptime(secs): 600.8 total, 600.0 interval
Cumulative writes: 59K writes, 59K keys, 59K commit groups, 1.0 writes per commit group, ingest: 0.28 GB, 0.47 MB/s
Cumulative WAL: 59K writes, 0 syncs, 59385.00 writes per sync, written: 0.28 GB, 0.47 MB/s
Cumulative stall: 00:02:9.577 H:M:S, 21.6 percent
Interval writes: 59K writes, 59K keys, 59K commit groups, 1.0 writes per commit group, ingest: 284.98 MB, 0.47 MB/s
Interval WAL: 59K writes, 0 syncs, 59385.00 writes per sync, written: 0.28 MB, 0.47 MB/s
Interval stall: 00:02:9.577 H:M:S, 21.6 percent
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 37/4 158.55 MB 8.2 0.0 0.0 0.0 0.2 0.2 0.0 1.0 0.0 1.3 123.63 0.00 37 3.341 0 0
Sum 37/4 158.55 MB 0.0 0.0 0.0 0.0 0.2 0.2 0.0 1.0 0.0 1.3 123.63 0.00 37 3.341 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.2 0.2 0.0 1.0 0.0 1.3 123.63 0.00 37 3.341 0 0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.2 0.2 0.0 0.0 0.0 1.3 123.63 0.00 37 3.341 0 0
Uptime(secs): 600.8 total, 600.0 interval
Flush(GB): cumulative 0.155, interval 0.155
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.15 GB write, 0.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 123.6 seconds
Interval compaction: 0.15 GB write, 0.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 123.6 seconds
Stalls(count): 16 level0_slowdown, 16 level0_slowdown_with_compaction, 2 level0_numfiles, 2 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 36 memtable_compaction, 0 memtable_slowdown, interval 54 total count
** File Read Latency Histogram By Level [default] **
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 37/4 158.55 MB 8.2 0.0 0.0 0.0 0.2 0.2 0.0 1.0 0.0 1.3 123.63 0.00 37 3.341 0 0
Sum 37/4 158.55 MB 0.0 0.0 0.0 0.0 0.2 0.2 0.0 1.0 0.0 1.3 123.63 0.00 37 3.341 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.2 0.2 0.0 0.0 0.0 1.3 123.63 0.00 37 3.341 0 0
Uptime(secs): 600.8 total, 0.0 interval
Flush(GB): cumulative 0.155, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.15 GB write, 0.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 123.6 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 16 level0_slowdown, 16 level0_slowdown_with_compaction, 2 level0_numfiles, 2 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 36 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [default] **
2020/09/01-12:54:45.349226 7ffff5b4a700 [s3] PutCloudObject rockset.cloud-db-examples.alec//tmp/bench_cloud/000018.sst-6d650822daa1fee9, size 16879170, OK
2020/09/01-12:54:45.354373 7ffff5b4a700 [mpaction/compaction_job.cc:1344] [default] [JOB 6] Generated table #18: 6025 keys, 16879170 bytes
2020/09/01-12:54:45.354447 7ffff5b4a700 EVENT_LOG_v1 {"time_micros": 1598936085354403, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 18, "file_size": 16879170, "table_properties": {"data_size": 16782104, "index_size": 170989, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 7621, "raw_key_size": 144600, "raw_average_key_size": 24, "raw_value_size": 30125000, "raw_average_value_size": 5000, "num_data_blocks": 6025, "num_entries": 6025, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "Snappy", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1598935455, "oldest_key_time": 0, "file_creation_time": 1598935484}}
2020/09/01-12:55:22.509997 7ffff5b4a700 [s3] PutCloudObject rockset.cloud-db-examples.alec//tmp/bench_cloud/000082.sst-6d650822daa1fee9, size 1051429, OK
2020/09/01-12:55:22.510547 7ffff5b4a700 [mpaction/compaction_job.cc:1344] [default] [JOB 6] Generated table #82: 375 keys, 1051429 bytes
2020/09/01-12:55:22.510615 7ffff5b4a700 EVENT_LOG_v1 {"time_micros": 1598936122510574, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 82, "file_size": 1051429, "table_properties": {"data_size": 1044539, "index_size": 10335, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9000, "raw_average_key_size": 24, "raw_value_size": 1875000, "raw_average_value_size": 5000, "num_data_blocks": 375, "num_entries": 375, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "Snappy", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1598935455, "oldest_key_time": 0, "file_creation_time": 1598936085}}
2020/09/01-12:55:23.528381 7ffff5b4a700 [mpaction/compaction_job.cc:1412] [default] [JOB 6] Compacted 4@0 files to L1 => 17930599 bytes
2020/09/01-12:55:23.710201 7ffff5b4a700 [s3] PutCloudObject rockset.cloud-db-examples.alec//tmp/bench_cloud/MANIFEST-6d650822daa1fee9, size 4131, OK
2020/09/01-12:55:23.710313 7ffff5b4a700 [WARN] [lumn_family.cc:888] [default] Stalling writes because we have 33 level-0 files rate 16777216
2020/09/01-12:55:23.710376 7ffff5b4a700 (Original Log Time 2020/09/01-12:55:23.710269) [mpaction/compaction_job.cc:743] [default] compacted to: files[33 2 0 0 0 0 0] max score 8.25, MB/sec: 0.0 rd, 0.0 wr, level 1, files in(4, 0) out(2) MB in(17.1, 0.0) out(17.1), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 6414, records dropped: 14 output_compression: Snappy
2020/09/01-12:55:23.710386 7ffff5b4a700 (Original Log Time 2020/09/01-12:55:23.710304) EVENT_LOG_v1 {"time_micros": 1598936123710288, "job": 6, "event": "compaction_finished", "compaction_time_micros": 638116028, "compaction_time_cpu_micros": 0, "output_level": 1, "num_output_files": 2, "total_output_size": 17930599, "num_input_records": 6414, "num_output_records": 6400, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [33, 2, 0, 0, 0, 0, 0]}
2020/09/01-12:55:23.710497 7ffff5b4a700 EVENT_LOG_v1 {"time_micros": 1598936123710494, "job": 6, "event": "table_file_deletion", "file_number": 14}
2020/09/01-12:55:23.710554 7ffff5b4a700 EVENT_LOG_v1 {"time_micros": 1598936123710552, "job": 6, "event": "table_file_deletion", "file_number": 12}
2020/09/01-12:55:23.710585 7ffff5b4a700 EVENT_LOG_v1 {"time_micros": 1598936123710583, "job": 6, "event": "table_file_deletion", "file_number": 10}
2020/09/01-12:55:23.710615 7ffff5b4a700 EVENT_LOG_v1 {"time_micros": 1598936123710613, "job": 6, "event": "table_file_deletion", "file_number": 7}
I am testing db_bench on aws, and when I run this:
./cloud/benchmarks/bulkload_seq.sh
I get this stack trace:
Load 500M keys of size 1 KB each sequentially into rocksdb-cloud.....
AwsEnv.src_bucket_prefix: dbbench.dhruba
AwsEnv.src_object_prefix:
AwsEnv.src_bucket_region: us-west-2
AwsEnv.dest_bucket_prefix: dbbench.dhruba
AwsEnv.dest_object_prefix:
AwsEnv.dest_bucket_region: us-west-2
AwsEnv.credentials: [given]
AwsEnv connection to endpoint in region: us-west-2
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
db_bench: cloud/cloud_env.cc:84: std::__cxx11::string rocksdb::CloudEnvImpl::RemapFilename(const string&) const: Assertion `cloud_manifest_' failed.
Received signal 6 (Aborted)
#0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7ff3191ee428] ?? ??:0
#1 /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7ff3191f002a] ?? ??:0
#2 /lib/x86_64-linux-gnu/libc.so.6(+0x2dbd7) [0x7ff3191e6bd7] ?? ??:0
#3 /lib/x86_64-linux-gnu/libc.so.6(+0x2dc82) [0x7ff3191e6c82] ?? ??:0
#4 ./db_bench() [0x868baa] rocksdb::CloudEnvImpl::RemapFilename(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const /data/home/dhruba/rocksdb-cloud/cloud/cloud_env.cc:84 (discriminator 1)
#5 ./db_bench() [0x840c5b] rocksdb::AwsEnv::NewWritableFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unique_ptr<rocksdb::WritableFile, std::default_delete<rocksdb::WritableFile> >*, rocksdb::EnvOptions const&) /data/home/dhruba/rocksdb-cloud/cloud/aws/aws_env.cc:611
#6 ./db_bench() [0x7550d5] rocksdb::NewWritableFile(rocksdb::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unique_ptr<rocksdb::WritableFile, std::default_delete<rocksdb::WritableFile> >*, rocksdb::EnvOptions const&) /data/home/dhruba/rocksdb-cloud/util/file_reader_writer.cc:653
#7 ./db_bench() [0x5d3917] rocksdb::DBImpl::NewDB() /data/home/dhruba/rocksdb-cloud/db/db_impl_open.cc:232 (discriminator 1)
#8 ./db_bench() [0x5d917c] rocksdb::Status::operator=(rocksdb::Status&&) /data/home/dhruba/rocksdb-cloud/./include/rocksdb/status.h:295
#9 ./db_bench() [0x5d9ece] rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<ro ksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool) /data/home/dhruba/rocksdb-cloud/db/db_impl_open.cc:1025 (discriminator 1)
#10 ./db_bench() [0x5db7d3] rocksdb::DB::Open(rocksdb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::DB**) /data/home/dhruba/rocksdb-cloud/db/db_impl_open.cc:961
#11 ./db_bench() [0x5293de] rocksdb::Status::operator=(rocksdb::Status&&) /data/home/dhruba/rocksdb-cloud/./include/rocksdb/status.h:295
#12 ./db_bench() [0x535dde] rocksdb::Options::~Options() /data/home/dhruba/rocksdb-cloud/./include/rocksdb/options.h:918
#13 ./db_bench() [0x537190] rocksdb::Benchmark::Run() /data/home/dhruba/rocksdb-cloud/tools/db_bench_tool.cc:2406
#14 ./db_bench() [0x5120d4] rocksdb::db_bench_tool(int, char**) /data/home/dhruba/rocksdb-cloud/tools/db_bench_tool.cc:5487
@igorcanadi : is this related to CloudManifest cf5e241
As of now, rocksdb-cloud only seems to support AWS-S3 as the backend for cloud backup. This request is to support Google Cloud Storage (GCS) as well. This can leverage the CloudStorageProvider refactoring being introduced by Issue #71.
but works in RocksDB.
my.cnf:
rocksdb_cloud_keep_local_sst_files=true
rocksdb_cloud_keep_local_log_files=true
rocksdb_use_direct_io_for_flush_and_compaction=0
rocksdb_use_direct_reads=0
rocksdb_allow_mmap_reads=1
rocksdb_allow_mmap_writes=1
rocksdb_flush_log_at_trx_commit=0
rocksdb_write_disable_wal=true
MyRocksCloud:
2019-04-16 3:56:28 0 [Note] /home/myrocks/Work/mariadb-git_2019-03-29/mariadb/sql/mysqld (mysqld 10.4.2-MariaDB) starting as process 6802 ...
2019-04-16 3:56:28 0 [Note] Starting MyRock Cloud init...
2019-04-16 3:56:28 0 [Note] RocksDB dir: /mnt/ssd2/mariadb-git_2019-03-29/rocksdata
2019-04-16 3:56:28 0 [Note] RocksDB original dir: /mnt/ssd2/mariadb-git_2019-03-29/rocksdata
2019-04-16 3:56:28 0 [Note] rocksdb::CloudEnv::NewAwsEnv
2019-04-16 3:56:28 0 [Note] [TODO - my commit #1 is needed] librocksdb is MODIFIED!
2019-04-16 3:56:28 0 [Note] SSTs are local and Cloud
2019-04-16 3:56:28 0 [Note] .log and MANIFEST are local only
2019-04-16 3:56:28 0 [Note] request_timeout_ms = 2000
2019-04-16 3:56:28 0 [Note] rocksdb::CloudEnv::NewAwsEnv is ok
2019-04-16 3:56:28 0 [Note] rocksdb::DBCloud::PreloadCloudManifest() is ok
2019-04-16 3:56:28 0 [ERROR] LibRocksDB:[s3] GetChildren /mnt/ssd2/mariadb-git_2019-03-29/rocksdata/trash error on local dir
2019-04-16 3:56:28 0 [Note] RocksDB: 2 column families found
2019-04-16 3:56:28 0 [Note] RocksDB: Column Families at start:
2019-04-16 3:56:28 0 [Note] cf=default
2019-04-16 3:56:28 0 [Note] write_buffer_size=1073741824
2019-04-16 3:56:28 0 [Note] target_file_size_base=1073741824
2019-04-16 3:56:28 0 [Note] cf=__system__
2019-04-16 3:56:28 0 [Note] write_buffer_size=1073741824
2019-04-16 3:56:28 0 [Note] target_file_size_base=1073741824
2019-04-16 3:56:28 0 [Note] rocksdb::DBCloud::Open
2019-04-16 3:56:28 0 [Note] no persistent cache
2019-04-16 3:56:28 0 [Note] Enter rocksdb::DBCloud::Open
2019-04-16 3:56:28 0 [Note] Leave rocksdb::DBCloud::Open
2019-04-16 3:56:28 0 [Note] DBCloud::Open is ok
2019-04-16 3:56:28 0 [Note] [?] Should we do cf_manager.init before or after TransactionDB::WrapDB
2019-04-16 3:56:28 0 [Note] TransactionDB::WrapDB is ok
2019-04-16 3:56:28 0 [Note] RocksDB Cloud init done
2019-04-16 3:56:28 0 [ERROR] RocksDB: Failed to write to WAL (dictionary), Status Code: 3, Status: Not implemented: SyncWAL() is not supported for this implementation of WAL file: try setting Options::allow_mmap_writes to false
2019-04-16 3:56:28 0 [ERROR] MyRocks: aborting on WAL write error.
190416 3:56:28 [ERROR] mysqld got signal 6 ;
Could you pls. add cmake build of librocksdb cloud with AWS and Kafka support?
I'm proposing the following change to implement it + add cloud files excluded from build, using custom (not system) librdkafka and aws-sdk:
--- ./server-10.4/storage/rocksdb/rocksdb-cloud_public_ master_2018-11-13_01-57-33/CMakeLists.txt.orig 2018-11-13 02:00:37.965539921 +0300
+++ ./server-10.4/storage/rocksdb/rocksdb-cloud_public_master_2018-11-13_01-57-33/CMakeLists.txt 2018-10-16 01:09:21.643306559 +0300
@@ -45,6 +45,46 @@
option(WITH_SNAPPY "build with SNAPPY" OFF)
option(WITH_LZ4 "build with lz4" OFF)
option(WITH_ZLIB "build with zlib" OFF)
+
+# Adding debug info
+set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g")
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g")
+set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -g")
+
+# Adding librdkafka++
+add_definitions(-DUSE_KAFKA)
+include_directories(/home/myrocks/Work/librdkafka/librdkafka-0.11.5/install/usr/local/include)
+list(APPEND THIRDPARTY_LIBS rdkafka rdkafka++)
+link_directories(/home/myrocks/Work/librdkafka/librdkafka-0.11.5/install/usr/local/lib)
+
+# Adding AWS libs
+add_definitions(-DUSE_AWS)
+include_directories(/home/myrocks/Work/aws/aws-sdk-cpp_master_2018-06-25/install/usr/local/include)
+list(APPEND THIRDPARTY_LIBS aws-cpp-sdk-s3 aws-cpp-sdk-kinesis aws-cpp-sdk-core)
+link_directories(/home/myrocks/Work/aws/aws-sdk-cpp_master_2018-06-25/install/usr/local/lib)
+
if(MSVC)
# Defaults currently different for GFLAGS.
# We will address find_package work a little later
@@ -617,6 +657,18 @@
utilities/ttl/db_ttl_impl.cc
utilities/write_batch_with_index/write_batch_with_index.cc
utilities/write_batch_with_index/write_batch_with_index_internal.cc
+ cloud/aws/aws_env.cc
+ cloud/aws/aws_kafka.cc
+ cloud/aws/aws_kinesis.cc
+ cloud/aws/aws_log.cc
+ cloud/aws/aws_retry.cc
+ cloud/aws/aws_s3.cc
+ cloud/db_cloud_impl.cc
+ cloud/cloud_env.cc
+ cloud/cloud_env_options.cc
+ cloud/manifest_reader.cc
+ cloud/purge.cc
+ cloud/cloud_manifest.cc
$<TARGET_OBJECTS:build_version>)
if(HAVE_SSE42 AND NOT FORCE_SSE42)
Enhance rocksdb-cloud to support cloning from a clone.
The current code allows creating a zero-copy-clone (zcc) of a rocksdb instance but it does not support creating a clone of an existing cloned instance.
Hello and thank you for RocksDB-Cloud,
In the CONTRIBUTING.md file in the master branch, the BSD License is mentioned and link (to LICENSE file) is broken.
https://github.com/rockset/rocksdb-cloud/blob/master/CONTRIBUTING.md#license
By contributing to RocksDB-Cloud, you agree that your contributions will be
licensed under the [BSD License](LICENSE).
I see the upstream commit that removed this for facebook#2590:
facebook@a7321fc#diff-6a3371457528722a734f3c51d9238c13
in the rocksdb-cloud master branch history:
https://github.com/rockset/rocksdb-cloud/commits/master/CONTRIBUTING.md
a7321fc#diff-6a3371457528722a734f3c51d9238c13
so the git history of commits does not match the content of CONTRIBUTING.md.
Thanks
RocksDB cloud has an option CloudEnvOptions.keep_local_sst_files that can be configure to keep all sst files in local storage or none at all (All files are also stored in cloud-storage S3).
However, files that are written in L0 and L1 are best kept locally because they are part of frequent and time-sensitive compaction jobs. It is best if we keep all the files in L0-Lk in local storage as well. Files that are in levels Lk - Lmax can reside only on cloud storage.
Change the name of the option keep_local_sst_files to keep_local_sst_files_upto_level. Also, write code to support this feature. Setting keep_local_sst_files_upto_level = -1 means that no files are stored on local storage.
I have installed all the dependencies, and git cloned the branch azure-blob
cd rocksdb-cloud
export EXTRA_CXXFLAGS="-I /usr/local/Cellar/openssl/1.0.2j/include" export EXTRA_LDFLAGS="-L/usr/local/Cellar/openssl/1.0.2j/lib -lboost_system -lssl -lcrypto -lcpprest -lazurestorage -lboost_system -lboost_thread -lboost_chrono"
USE_AWS=1
make -j8 static_lib
cd cloud/examples
make
the extra build flags are for azure, USE_AWS is for aws, but when i make in /cloud/examples, it won't pass compile.
the error says that rocksdb::CloudEnvOptions does not have member region
how to fix this?
Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev
When I run the following command
./db_bench
--env_uri="s3://" --aws_access_id=ID --aws_secret_key=KEY
--benchmarks="fillseq,readseq,fillrandom,levelstats"
-compression_type=none -histogram=1 -perf_level=5 --statistics
-threads=1 -max_background_compactions=8 -max_background_flushes=4
-max_bytes_for_level_base=$((51210241024)) -write_buffer_size=$((12810241024))
-cache_size=$((79210241024))
-seek_nexts=10
-level_compaction_dynamic_level_bytes=1 -target_file_size_base=$((810241024))
-bloom_bits=24 -use_block_based_filter=0
-key_size=16 -value_size=512
-db="/home/ubuntu/test/rocksdb-zyh311"
-num=10000000 | teedate +%Y%m%d%H%M%S.log
I can get the results of fillreq and readrandom. The process of executing fillrandom will suddenly exit without any prompt or error output in LOG.
Running the unit tests sometimes results in the following stack trace:
[ RUN ] DBTest.HiddenValuesAreRemoved
Received signal 11 (Segmentation fault)
#0 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZSt28_Rb_tree_rebalance_for_erasePSt18_Rb_tree_node_baseRS_+0x2b) [0x7fef8a28ce9b] ?? ??:0
#1 ./db_test() [0x8d5626] std::_Rb_tree<std::pair<std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::function<void ()> >, std::pair<std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration long, std::ratio<1l, 1000000000l> > >, std::function<void ()> >, std::_Identity<std::pair<std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::function<void ()> > >, rocksdb::detail::Co p, std::allocator<std::pair<std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::function<void ()> > > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<std::chrono::time_point<st:chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::function<void ()> > >) /usr/include/c++/5/bits/stl_tree.h:2248
#2 ./db_test() [0x8c7e13] __gthread_mutex_unlock /usr/include/x86_64-linux-gnu/c++/5/bits/gthr-default.h:778
#3 ./db_test() [0x8c83e5] std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<rocksdb::detail::JobHandle> >, std::allocato <std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<rocksdb::detail::JobHandle> > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allo ator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<tr, false, true> >::erase(std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<rocksdb::detail::JobHandle> >, false, true>) /usr/include/c++/5/bits/hashtable.h:745
#4 ./db_test() [0x8e6a9e] rocksdb::S3WritableFile::Close() /data/home/dhruba/rocksdb-cloud/cloud/aws/aws_s3.cc:295
#5 ./db_test() [0x7d77d9] rocksdb::Status::operator=(rocksdb::Status&&) /data/home/dhruba/rocksdb-cloud/./include/rocksdb/status.h:295
#6 ./db_test() [0x607c2f] rocksdb::Status::operator=(rocksdb::Status&&) /data/home/dhruba/rocksdb-cloud/./include/rocksdb/status.h:294
#7 ./db_test() [0x60b2db] rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*) /data/home/dhruba/rocksdb-cloud/db/compaction_job.cc:983
#8 ./db_test() [0x60cd5a] __gnu_cxx::__normal_iterator<std::thread*, std::vector<std::thread, std::allocator<std::thread> > >::__normal_iterator(std::thread* const&) /usr/include/c++/5/bits/stl_iterator.h:741
#9 ./db_test() [0x645053] rocksdb::Status::~Status() /data/home/dhruba/rocksdb-cloud/./include/rocksdb/status.h:29
#10 ./db_test() [0x647825] std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Alloc_hider::_Alloc_hider(char*, std::allocator<char> const&) /usr/include/c++/5/bits/basic_string.h:109
#11 ./db_test() [0x647e5d] rocksdb::DBImpl::BGWorkCompaction(void*) /data/home/dhruba/rocksdb-cloud/db/db_impl_compaction_flush.cc:1189
#12 ./db_test() [0x7e8263] std::_Function_base::~_Function_base() /usr/include/c++/5/functional:1829
#13 ./db_test() [0x7e847b] rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) /data/home/dhruba/rocksdb-cloud/util/threadpool_imp.cc:280
#14 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7fef8a2a0c80] ?? ??:0
#15 /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fef8b2006ba] ?? ??:0
#16 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fef89a0641d] ?? ??:0
Aws::Auth::InstanceProfileCredentialsProvider() should be used.
A few other case should be supported to, e.g. credentials in a file, anonymous access
Please find my CLA attached and signed.
what is the right way to delete or recreate a bucket containing the database? what function should I call for it?
The RocksDB-Cloud core engine does three things:
It will be great if compaction (3) can be offloaded into a separate cloud-compaction-tier. The rocksdb-cloud engine picks files for compaction, but instead of compacting them itself, it writes the compaction-job-parameters into a AWS-kinesis stream A. The job has details about the source and destination files for the compaction, as well as the sequence numbers that have snapshots.
A machine in the cloud-compaction-tier ( needs only CPU and is no state) tails stream A and extracts the job-parameters from the record. Then it downloads relevant sst files from AWS, does the compaction and uploads the new files into AWS. It writes into a separate kinesis stream B saying that the compaction job is done.
The original rocksdb-cloud instance tails stream B, and sees that the compaction is done. It downloads the relevant sst files from AWS and commits the compaction into its local DB.
There is an issue in "S3StorageProvider::DoPutCloudObject" method. Call to "UploadFile" method has wrong sequence.
-> Correct sequence is (bucket_name, object_path, local_file)
S3StorageProvider::DoPutCloudObject:
auto handle =
s3client_->UploadFile(ToAwsString(local_file), ToAwsString(bucket_name),
ToAwsString(object_path), file_size);
auto handle =
s3client_->UploadFile(ToAwsString(bucket_name),
ToAwsString(object_path), ToAwsString(local_file), file_size);
rocksdb-cloud/cloud/aws/aws_s3.cc
Line 868 in 770e0d1
I'm trying to run rockdb-cloud on my MacBook. Installed aws-sdk-cpp v1.4.30 using homebrew and I can see the libs in /usr/local/lib/libaws*. As instructed I'm using USE_AWS=1 environment variable and compile and run db_bench as stated in https://github.com/rockset/rocksdb-cloud/blob/master/cloud/README.md.
However, db_bench simply returns following error:
No Env registered for URI: s3://
It seems like I must be missing something trivial, but can't figure out what. Thanks!
I notice that there are some third-party language bindings for RocksDB-Cloud such Erlang and Java. Is there any plan for C language bindings support?
We would like to know how many times a MANIFEST file is uploaded to rocksdb. We would also like to know the latency histogram of this call.
The MANIFEST file is uploaded to the cloud every time it is updated and sycned
https://github.com/rockset/rocksdb-cloud/blob/master/cloud/aws/aws_s3.cc#L369
Also, our manifest file_size is set to 4 MB.
https://github.com/rockset/rocksdb-cloud/blob/master/cloud/db_cloud_impl.h#L43
We would like to create a include/rocksdb/cloud/cloud_statistics.h which is similar in structure to https://github.com/rockset/rocksdb-cloud/blob/master/include/rocksdb/statistics.h. It will have only one Ticker called NUM_WRITES_TO_MANIFEST.
Rocksdb-cloud is a StackableDB https://github.com/rockset/rocksdb-cloud/blob/master/include/rocksdb/cloud/db_cloud.h#L18, so overload the GetStatistics call to return both the base rocksdb statistics as well as the newly minted rocksdb-cloud statistics.
Do you have a list of changes compared to rocksdb? Are these chnages only touching an env or were the other parts of rocksdb edited?
when can we expect a new tagged release of rocksdb-cloud? It would help to make some release for our erlang binding :)
Hello! I see this error output repeatedly in my logs:
I0417 16:47:59.117947 17678 logger.hpp:79] [kinesis] error reading myprogram Rate exceeded for shard shardId-000000000000 in stream myprogram under account XXXXXXXXXXXX.
I0417 16:47:59.754103 17678 logger.hpp:79] [aws] Encountered S3 failure Rate exceeded for shard shardId-000000000000 in stream myprogram under account XXXXXXXXXXXX. (code 139, http 400). Exception ProvisionedThroughputExceededException. re
try attempt 0 max retries 10. Using default retry policy...
It seems like this is a reasonable error if I was writing or reading frequently with a large quantity of data. However, my values are all only about 60-70 bytes. I don't think I am performing any operation that should cause this behavior.
Some way to quantify how many shards we need, or the read/write throughput requirements.
I expect to see the Kinesis controller working without errors out of the box.
I see these errors all over the logs -- it does not look like there is any way to handle any production level of throughput
I started seeing the errors when I set:
cloud_env_options.log_type = LogType::kLogKinesis;
It looks as though I may need to rewrite some of the KinesisController in order to meet my use case.
I am perfectly happy to do this, but was hoping that I could have a conversation with somebody on the rocksdb-cloud team before doing so. I want to make sure that what I write might also be useful to the rocksdb-cloud project! I also want to make sure that I am not missing something!
Please let me know if we can set something up -- we could discuss on this thread if it is the proper place for the information :)
-T
Always succeeding compaction.
After
[db/flush_job.cc:299] [default] [JOB 12254] Level-0 flush table #288302: started
end up in
[db/flush_job.cc:339] [default] [JOB 12254] Level-0 flush table #288302: 2451124 bytes IO error: /data/tsdb_data/288302.sst-986775bdc1e78e51: No response body. Response code: 0
Then is not recovering from this state, continuously writing the same error in the logs.
I encountered the same pattern of error twice, roughly with a week interval between them.
I have hundredth of GB data, with a >GB/day continuous influx. Therefore I cannot describe steps to reproduce the same behavior.
I can copy the interesting parts of the log from the first occurence of the error.
cloud/LOG.old.1538471660279624
2018/09/18-09:22:05.677372 7f6b857fa700 [WARN] [aws] Encountered S3 failure No response body. Response code: 0 (code 100, http 0). Exception . retry attempt 0 max retries 10. Using default retry policy...
2018/09/18-09:22:05.679344 7f6b857fa700 [s3] PutObject rockset.prod.tiko.ch//data/tsdb_data/288302.sst-986775bdc1e78e51, size 2451124, status IO error: /data/tsdb_data/288302.sst-986775bdc1e78e51: No response body. Response code: 0
2018/09/18-09:22:05.679367 7f6b857fa700 [ERROR] [s3] S3WritableFile closing PutObject failed on local file /data/tsdb_data/288302.sst-986775bdc1e78e51
db/data_tsdb_data_LOG.old.1538471660354912
2018/09/18-09:21:53.254042 7f6b857fa700 [db/flush_job.cc:299] [default] [JOB 12254] Level-0 flush table #288302: started
2018/09/18-09:21:54.538795 7f6b843ed700 [db/db_impl_write.cc:1293] [default] New memtable created with log file: #288303. Immutable memtables: 2.
2018/09/18-09:21:54.538810 7f6b843ed700 [db/column_family.cc:762] [default] Increasing compaction threads because we have 7 level-0 files
2018/09/18-09:21:54.538977 7f6b87fff700 (Original Log Time 2018/09/18-09:21:54.538887) [db/db_impl_compaction_flush.cc:1582] Compaction nothing to do
2018/09/18-09:21:55.769982 7f6b849f3700 [db/db_impl_write.cc:1293] [default] New memtable created with log file: #288304. Immutable memtables: 3.
2018/09/18-09:21:55.770000 7f6b849f3700 [db/column_family.cc:762] [default] Increasing compaction threads because we have 7 level-0 files
2018/09/18-09:21:55.771992 7f6b86ffd700 [db/db_impl_compaction_flush.cc:50] [JOB 12257] Syncing log #288301
2018/09/18-09:21:55.772067 7f6bb57fa700 (Original Log Time 2018/09/18-09:21:55.772002) [db/db_impl_compaction_flush.cc:1582] Compaction nothing to do
2018/09/18-09:21:55.772860 7f6b86ffd700 [db/db_impl_compaction_flush.cc:50] [JOB 12257] Syncing log #288303
2018/09/18-09:21:55.772944 7f6b86ffd700 (Original Log Time 2018/09/18-09:21:55.771969) [db/db_impl_compaction_flush.cc:1265] Calling FlushMemTableToOutputFile with column family [default], flush slots available 4, compaction slots available 12, flush slots scheduled 2, compaction slots scheduled 2
...
2018/09/18-09:21:59.043948 7f6b867fc700 [db/flush_job.cc:339] [default] [JOB 12262] Level-0 flush table #288308: 2484196 bytes OK
2018/09/18-09:21:59.043972 7f6b867fc700 [WARN] [db/column_family.cc:656] [default] Stopping writes because we have 6 immutable memtables (waiting for flush), max_write_buffer_number is set to 6
2018/09/18-09:21:59.044066 7f6b867fc700 (Original Log Time 2018/09/18-09:21:59.043967) EVENT_LOG_v1 {"time_micros": 1537262519043962, "job": 12262, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [7, 5, 38, 340, 542, 0, 0], "immutable_memtables": 6}
2018/09/18-09:21:59.044071 7f6b867fc700 (Original Log Time 2018/09/18-09:21:59.043998) [db/db_impl_compaction_flush.cc:145] [default] Level summary: base level 1 max bytes base 1073741824 files[7 5 38 340 542 0 0] max score 2.00
2018/09/18-09:21:59.044123 7f6bb4ff9700 (Original Log Time 2018/09/18-09:21:59.044085) [db/db_impl_compaction_flush.cc:1582] Compaction nothing to do
2018/09/18-09:22:05.680323 7f6b857fa700 [db/flush_job.cc:339] [default] [JOB 12254] Level-0 flush table #288302: 2451124 bytes IO error: /data/tsdb_data/288302.sst-986775bdc1e78e51: No response body. Response code: 0
2018/09/18-09:22:05.681128 7f6b857fa700 [ERROR] [db/db_impl_compaction_flush.cc:1304] Waiting after background flush error: IO error: /data/tsdb_data/288302.sst-986775bdc1e78e51: No response body. Response code: 0Accumulated background error counts: 1
2018/09/18-09:22:05.681133 7f6b857fa700 (Original Log Time 2018/09/18-09:22:05.681108) EVENT_LOG_v1 {"time_micros": 1537262525681101, "job": 12254, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [7, 5, 38, 340, 542, 0, 0], "immutable_memtables": 6}
2018/09/18-09:22:06.643828 7f6bae7fc700 [db/compaction_job.cc:1171] [default] [JOB 12132] Generated table #288299: 8198406 keys, 339213185 bytes
2018/09/18-09:22:06.643861 7f6bae7fc700 EVENT_LOG_v1 {"time_micros": 1537262526643842, "cf_name": "default", "job": 12132, "event": "table_file_creation", "file_number": 288299, "file_size": 339213185, "table_properties": {"data_size": 335545209, "index_size": 3667034, "filter_size": 0, "raw_key_size": 255598900, "raw_average_key_size": 31, "raw_value_size": 236433701, "raw_average_value_size": 28, "num_data_blocks": 82657, "num_entries": 8198406, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "8198406"}}
....
2018/09/18-09:22:16.607308 7f6bae7fc700 [db/compaction_job.cc:1171] [default] [JOB 12132] Generated table #288309: 2429461 keys, 100298266 bytes
2018/09/18-09:22:16.607355 7f6bae7fc700 EVENT_LOG_v1 {"time_micros": 1537262536607337, "cf_name": "default", "job": 12132, "event": "table_file_creation", "file_number": 288309, "file_size": 100298266, "table_properties": {"data_size": 99243640, "index_size": 1053687, "filter_size": 0, "raw_key_size": 73123214, "raw_average_key_size": 30, "raw_value_size": 70064970, "raw_average_value_size": 28, "num_data_blocks": 24464, "num_entries": 2429461, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "2429461"}}
2018/09/18-09:22:16.607540 7f6bae7fc700 [db/compaction_job.cc:1245] [default] [JOB 12132] Compacted 3@0 + 5@1 files to L1 => 1164420646 bytes
2018/09/18-09:22:16.751714 7f6bae7fc700 [WARN] [db/column_family.cc:656] [default] Stopping writes because we have 6 immutable memtables (waiting for flush), max_write_buffer_number is set to 6
2018/09/18-09:22:16.751911 7f6b85ffb700 [ERROR] [db/db_impl_compaction_flush.cc:1304] Waiting after background flush error: IO error: /data/tsdb_data/288302.sst-986775bdc1e78e51: No response body. Response code: 0Accumulated background error counts: 2
2018/09/18-09:22:16.751944 7f6bb5ffb700 [ERROR] [db/db_impl_compaction_flush.cc:1384] Waiting after background compaction error: IO error: /data/tsdb_data/288302.sst-986775bdc1e78e51: No response body. Response code: 0, Accumulated background error counts: 3
2018/09/18-09:22:16.751955 7f6bae7fc700 (Original Log Time 2018/09/18-09:22:16.751672) [db/compaction_job.cc:640] [default] compacted to: base level 1 max bytes base 1073741824 files[4 5 38 340 542 0 0] max score 2.00, MB/sec: 9.3 rd, 9.3 wr, level 1, files in(3, 5) out(5) MB in(37.9, 1073.6) out(1110.5), read-write-amplify(58.7) write-amplify(29.3) OK, records in: 19651066, records dropped: 0 output_compression: NoCompression
2018/09/18-09:22:16.751963 7f6bae7fc700 (Original Log Time 2018/09/18-09:22:16.751704) EVENT_LOG_v1 {"time_micros": 1537262536751689, "job": 12132, "event": "compaction_finished", "compaction_time_micros": 125523798, "output_level": 1, "num_output_files": 5, "total_output_size": 1164420646, "num_input_records": 138029, "num_output_records": 19651066, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [4, 5, 38, 340, 542, 0, 0]}
With the (future) support for configuring an environment from strings/properties and the eminent addition of new CloudEnv types and storage providers, there needs to be some revamping of the NewAwsEnv and CloudEnvOptions. Here are a few of the issues as I see them:
My question is how to address this going forward. What should the API and general programming guidelines be?
As a starting point, I would propose the following:
Comments? Suggestions? Alternatives?
Since this affects the public API, it would be nice to do this all at once and get it right the first time :-)
Hi Everybody! I am encountering some behavior which I haven't been able to understand for several days. I am not 100% certain this is a bug -- it may be that there is a simple idea I am not aware of. It seems to me that the behavior is unexpected, and my configuration may be incorrect. At the moment, I am not sure what to look for.
After tailing the WAL from Kinesis and applying the record to the local fileystem with CloudLogController::Apply(const Slice& in), the KV pairs are available to read from the Reader DB
The WAL .log files are applied to the local filesystem of the Reader, but then issuing a Get from the Reader does not return the Value.
Note: I am not sure what I need to call in order to make the Reader see the changes in the WalCacheDir. Is it appropriate to point the reader's wal_dir to this directory?
(This directory is configured by the rocksdb-cloud library to be here: /tmp/ROCKSET/myprogram/[UID] )
Here is how my program is set up -- based on the readonly CLI flag, we either launch as a reader or as a writer. I want to be able to launch N readers, tailing from Kinesis, and 1 writer, writing to Kinesis. Is this pattern currently possible?
` rocksdb::Options opts;
opts.create_if_missing = true;
opts.paranoid_checks = true;
opts.compaction_style = kCompactionStyleLevel;
opts.write_buffer_size = 67108864; // 64MB
opts.max_write_buffer_number = 3;
opts.target_file_size_base = 67108864; // 64MB
opts.max_background_compactions = 4;
opts.max_background_flushes = 1;
opts.level0_file_num_compaction_trigger = 8;
opts.level0_slowdown_writes_trigger = 17;
opts.level0_stop_writes_trigger = 24;
opts.num_levels = 4;
opts.max_bytes_for_level_base = 536870912; // 512MB
opts.max_bytes_for_level_multiplier = 8;
opts.db_log_dir = "/tmp/db/logs";
opts.max_log_file_size = 1024 * 1024;
opts.log_file_time_to_roll = 60 * 60 * 24;
opts.keep_log_file_num = 1000;
std::shared_ptr<Logger> rocksdb_logger(new MyRocksLogger(ROCKSDB_LOG_COLOR));
opts.info_log = rocksdb_logger;
const std::string aws_access_key_id = myprog::env().aws_access_key_id();
const std::string aws_secret_access_key = myprog::env().aws_secret_access_key();
const std::string src_bucket_name = myprog::env().bucket_name();
const std::string& dst_bucket_name = src_bucket_name;
const std::string src_object_prefix = myprog::bucket_object_root().string();
const std::string& dst_object_prefix = src_object_prefix;
const std::string src_bucket_region = myprog::env().aws_region();
const std::string& dst_bucket_region = src_bucket_region;
CloudEnvOptions cloud_env_options;
cloud_env_options.credentials.access_key_id.assign(aws_access_key_id);
cloud_env_options.credentials.secret_key.assign(aws_secret_access_key);
cloud_env_options.keep_local_sst_files = true;
cloud_env_options.keep_local_log_files = false;
cloud_env_options.log_type = LogType::kLogKinesis;
std::shared_ptr<Logger> aws_logger(new MyRocksLogger(AWS_LOG_COLOR));
// Store a reference to a cloud env. A new cloud env object should be
// associated with every new cloud-db.
CloudEnv* cenv;
rocksdb::Status status = CloudEnv::NewAwsEnv(
Env::Default(),
src_bucket_name,
src_object_prefix,
src_bucket_region,
dst_bucket_name,
dst_object_prefix,
dst_bucket_region,
cloud_env_options,
aws_logger,
&cenv);
check_status(status,
"Unable to create cloud env in bucket " + dst_bucket_name + ". " + status.ToString());
opts.env = cenv;
const bool readonly = myprog::readonly();
if(readonly)
{
const std::string db_root = "/tmp/db/reader";
const std::string cache_dir = "/tmp/db/reader/cache";
const uint64_t cache_size = 2;
const std::string dst_bucket_name = myprog::bucket_name();
LOG(INFO) << "rocksdb::DBCloud::Open (readonly==true)" << std::endl;
opts.wal_dir = static_cast<rocksdb::AwsEnv*>(opts.env)->GetWALCacheDir(); //is this the correct way to do it?
rocksdb::Status status =
rocksdb::DBCloud::Open(
opts,
db_root,
cache_dir,
cache_size,
db,
readonly);
myprog::check_status(status,
"failed to open database at path " + db_root +
" with bucket: " + dst_bucket_name +
", status: " + status.ToString());
}else{
const std::string db_root = "/tmp/db/writer";
const std::string cache_dir = "/tmp/db/writer/cache";
const uint64_t cache_size = 2;
const std::string dst_bucket_name = myprog::bucket_name();
LOG(INFO) << "rocksdb::DBCloud::Open" << std::endl;
rocksdb::Status status =
rocksdb::DBCloud::Open(
opts,
db_root,
cache_dir,
cache_size,
db,
readonly);
myprog::check_status(status,
"failed to open database at path " +
db_root +
" with bucket: " +
dst_bucket_name +
", status: " +
status.ToString());
}
`
I am hopeful that somebody here will please help point me in the right direction! It feels to me like there might just be one simple thing I am missing!
Best,
-T
Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev
Be able to build AWS and rocksdb-cloud as static libraries.
Fails to link some AWS functions with SSL and CURL related libraries.
Take this Dockerfile and try to build with docker build .
FROM ubuntu:16.04
RUN apt-get update && \
apt-get install -y wget libevent-dev automake \
libtool flex bison pkg-config g++ ccache \
libssl-dev gawk make git subversion cmake \
libgflags-dev libsnappy-dev libcrypto++-dev \
libcurl4-openssl-dev uuid-dev zlib1g-dev libpulse-dev \
libbz2-dev unzip libboost-all-dev
env PATH /usr/lib/ccache:$PATH
env MAKEJ 5
env USE_AWS=1
env CMAKE_PREFIX_PATH=/usr/local/aws-sdk-cpp
RUN git clone https://github.com/aws/aws-sdk-cpp.git /usr/local/aws-sdk-cpp \
&& cd /usr/local/aws-sdk-cpp \
&& cmake -DCMAKE_BUILD_TYPE=Release '-DBUILD_ONLY=s3;kinesis' \
-DCUSTOM_MEMORY_MANAGEMENT=0 -DENABLE_TESTING=OFF -DBUILD_SHARED_LIBS=OFF -DSTATIC_LINKING=1 \
&& make -j 8 && make install
RUN git clone https://github.com/rockset/rocksdb-cloud.git /rocksdb-cloud \
&& cd /rocksdb-cloud \
&& make clean all db_bench \
&& make -j $MAKEJ static_lib \
&& make install
VOLUME /w
I get the following linker errors:
ar: creating librocksdb_debug.a
CC tools/db_bench.o
CC tools/db_bench_tool.o
CC util/testutil.o
CCLD db_bench
//usr/local/lib/libaws-cpp-sdk-core.a(CryptoImpl.cpp.o): In function `Aws::Utils::Crypto::SecureRandomBytes_OpenSSLImpl::GetBytes(unsigned char*, unsigned long)':
CryptoImpl.cpp:(.text+0xba): undefined reference to `RAND_bytes'
//usr/local/lib/libaws-cpp-sdk-core.a(CryptoImpl.cpp.o): In function `Aws::Utils::Crypto::Sha256OpenSSLImpl::Calculate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator
<char> > const&)':
CryptoImpl.cpp:(.text+0x154): undefined reference to `EVP_MD_CTX_create'
CryptoImpl.cpp:(.text+0x15c): undefined reference to `EVP_sha256'
CryptoImpl.cpp:(.text+0x169): undefined reference to `EVP_DigestInit_ex'
CryptoImpl.cpp:(.text+0x179): undefined reference to `EVP_DigestUpdate'
CryptoImpl.cpp:(.text+0x17e): undefined reference to `EVP_sha256'
...
CurlHttpClient.cpp:(.text+0xd8d): undefined reference to `curl_easy_setopt'
CurlHttpClient.cpp:(.text+0xebd): undefined reference to `curl_easy_setopt'
//usr/local/lib/libaws-cpp-sdk-core.a(CurlHttpClient.cpp.o): In function `Aws::Http::CurlHttpClient::InitGlobalState()':
CurlHttpClient.cpp:(.text+0x11c5): undefined reference to `curl_global_init'
//usr/local/lib/libaws-cpp-sdk-core.a(CurlHttpClient.cpp.o): In function `Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::
Utils::RateLimits::RateLimiterInterface*) const':
CurlHttpClient.cpp:(.text+0x2806): undefined reference to `curl_slist_append'
CurlHttpClient.cpp:(.text+0x284d): undefined reference to `curl_slist_append'
CurlHttpClient.cpp:(.text+0x2b37): undefined reference to `curl_easy_setopt'
CurlHttpClient.cpp:(.text+0x2c8d): undefined reference to `curl_easy_setopt'
CurlHttpClient.cpp:(.text+0x2ca3): undefined reference to `curl_easy_setopt'
CurlHttpClient.cpp:(.text+0x2cba): undefined reference to `curl_easy_setopt'
CurlHttpClient.cpp:(.text+0x2cd0): undefined reference to `curl_easy_setopt'
We have the AWS libraries in /usr/local/lib:
libaws-cpp-sdk-core.a libaws-cpp-sdk-kinesis.a libaws-cpp-sdk-s3.a
Also we have the following SSL libraries:
... apt list | grep libssl
libssl-dev/xenial-updates,xenial-security,now 1.0.2g-1ubuntu4.11 amd64 [installed]
libssl-doc/xenial-updates,xenial-security,now 1.0.2g-1ubuntu4.11 all [installed,automatic]
libssl-ocaml/xenial 0.5.2-1 amd64
libssl-ocaml-dev/xenial 0.5.2-1 amd64
libssl1.0.0/xenial-updates,xenial-security,now 1.0.2g-1ubuntu4.11 amd64 [installed,automatic]
libssl1.0.0-dbg/xenial-updates,xenial-security 1.0.2g-1ubuntu4.11 amd64
libsslcommon2/xenial 0.16-9ubuntu2 amd64
libsslcommon2-dev/xenial 0.16-9ubuntu2 amd64
Also libcurl with ssl:
libcurl4-openssl-dev/xenial-updates,xenial-security,now 7.47.0-1ubuntu2.7 amd64 [installed]
What I am doing wrong, or we have some issues with static libraries?
Will this fork ever be submitted upstream / merge upstream changes.
Hi Rockset,
I am wondering if the work on the Azure branch is abandoned or you guys are planing to not support Azure as of now ?
Also wondering is there any bindings to use something like HDFS ?
Thanks
I am trying out rocksdb-cloud. I have a toy application that pushes out a few GB of KV data (both keys and values about 20 bytes) and then goes into a mode where old data is deleted at the same rate as new one is inserted.
I have configured 2GB of persistent read cache in rocksdb.
I am getting Data Corruption errors with release build and an assert in debug build.
By code inspection it looks like this line is wrong (On branch v5.18.3):
rocksdb-cloud/cloud/aws/aws_s3.cc
Line 159 in 97d69b7
It logically makes sense for max_size to be >= kMaxVarint64Length to prevent the input buffer overrun, and the calling function sets it to something like 3*kMaxVarint64Length +1.
diff --git a/cloud/aws/aws_s3.cc b/cloud/aws/aws_s3.cc
index 90fa8c41f..a0688c52d 100644
--- a/cloud/aws/aws_s3.cc
+++ b/cloud/aws/aws_s3.cc
@@ -158,11 +158,14 @@ size_t S3ReadableFile::GetUniqueId(char* id, size_t max_size) const {
WalFileType log_type;
ParseFileName(RemoveEpoch(basename(fname_)), &file_number, &file_type,
&log_type);
With the following fix both assert and data corruption is gone in my local build.
If keep_local_sst_files = true, then we assume that all sst files are copied into local storage at the time of database open. This occurs in LoadTablehandlers:
https://github.com/rockset/rocksdb-cloud/blob/master/db/version_builder.cc#L370
The code tries to open all the sst file using a pool of background threads. It handles the case of file download&open error by doing this check:
if (file_meta->table_reader_handle != nullptr) {
// Load table_reader
file_meta->fd.table_reader = table_cache_->GetTableReaderFromHandle(
file_meta->table_reader_handle);
}
This means that it is possible that the entire set of sst files might not be downloaded at the time of db open. This happens if the S3 client throws error due to timeouts/network-congestion.
Now, in the course of normal reads to the db, the cloud code wants to open a sst file and keep_local_sst_files=true, then it assumes that the file is always on local storage and does not try to download it from S3. This code is here:
https://github.com/rockset/rocksdb-cloud/blob/master/cloud/aws/aws_env.cc#L518
This check was introduced by this commit:
78d4405
CC @igorcanadi
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.