bulk load about rocksdb

[dhruba]

If you look at https://github.com/facebook/rocksdb/wiki/Performance-Benchmarks#2-bulk-load-of-keys-in-random-order, you will find the command line options to load data in bulk-load fashion.

echo "Bulk load database into L0...."
bpl=10485760;overlap=10;mcz=2;del=300000000;levels=2;ctrig=10000000; delay=10000000; stop=10000000; wbn=30; mbc=20; mb=1073741824;wbs=268435456; dds=1; sync=0; r=1000000000; t=1; vs=800; bs=65536; cs=1048576; of=500000; si=1000000; ./db_bench --benchmarks=fillrandom --disable_seek_compaction=1 --mmap_read=0 --statistics=1 --histogram=1 --num=$r --threads=$t --value_size=$vs --block_size=$bs --cache_size=$cs --bloom_bits=10 --cache_numshardbits=4 --open_files=$of --verify_checksum=1 --db=/data/mysql/leveldb/test --sync=$sync --disable_wal=1 --compression_type=zlib --stats_interval=$si --compression_ratio=50 --disable_data_sync=$dds --write_buffer_size=$wbs --target_file_size_base=$mb --max_write_buffer_number=$wbn --max_background_compactions=$mbc --level0_file_num_compaction_trigger=$ctrig --level0_slowdown_writes_trigger=$delay --level0_stop_writes_trigger=$stop --num_levels=$levels --delete_obsolete_files_period_micros=$del --min_level_to_compress=$mcz --max_grandparent_overlap_factor=$overlap --stats_per_interval=1 --max_bytes_for_level_base=$bpl --memtablerep=vector --use_existing_db=0 --disable_auto_compactions=1 --source_compaction_factor=10000000
echo "Running manual compaction to do a global sort map-reduce style...."
bpl=10485760;overlap=10;mcz=2;del=300000000;levels=2;ctrig=10000000; delay=10000000; stop=10000000; wbn=30; mbc=20; mb=1073741824;wbs=268435456; dds=1; sync=0; r=1000000000; t=1; vs=800; bs=65536; cs=1048576; of=500000; si=1000000; ./db_bench --benchmarks=compact --disable_seek_compaction=1 --mmap_read=0 --statistics=1 --histogram=1 --num=$r --threads=$t --value_size=$vs --block_size=$bs --cache_size=$cs --bloom_bits=10 --cache_numshardbits=4 --open_files=$of --verify_checksum=1 --db=/data/mysql/leveldb/test --sync=$sync --disable_wal=1 --compression_type=zlib --stats_interval=$si --compression_ratio=50 --disable_data_sync=$dds --write_buffer_size=$wbs --target_file_size_base=$mb --max_write_buffer_number=$wbn --max_background_compactions=$mbc --level0_file_num_compaction_trigger=$ctrig --level0_slowdown_writes_trigger=$delay --level0_stop_writes_trigger=$stop --num_levels=$levels --delete_obsolete_files_period_micros=$del --min_level_to_compress=$mcz --max_grandparent_overlap_factor=$overlap --stats_per_interval=1 --max_bytes_for_level_base=$bpl --memtablerep=vector --use_existing_db=1 --disable_auto_compactions=1 --source_compaction_factor=10000000
du -s -k test
504730832 test

[mdcallag]

My guess is that these are the key values as they prevent compaction from
L0->L1 during the load...
ctrig=10000000; delay=10000000; stop=10000000

On Fri, Nov 29, 2013 at 6:07 PM, dhruba borthakur
[email protected]:

If you look at
https://github.com/facebook/rocksdb/wiki/Performance-Benchmarks#2-bulk-load-of-keys-in-random-order,
you will find the command line options to load data in bulk-load fashion.

echo "Bulk load database into L0...."
bpl=10485760;overlap=10;mcz=2;del=300000000;levels=2;ctrig=10000000;
delay=10000000; stop=10000000; wbn=30; mbc=20; mb=1073741824;wbs=268435456;
dds=1; sync=0; r=1000000000; t=1; vs=800; bs=65536; cs=1048576; of=500000;
si=1000000; ./db_bench --benchmarks=fillrandom --disable_seek_compaction=1
--mmap_read=0 --statistics=1 --histogram=1 --num=$r --threads=$t
--value_size=$vs --block_size=$bs --cache_size=$cs --bloom_bits=10
--cache_numshardbits=4 --open_files=$of --verify_checksum=1
--db=/data/mysql/leveldb/test --sync=$sync --disable_wal=1
--compression_type=zlib --stats_interval=$si --compression_ratio=50
--disable_data_sync=$dds --write_buffer_size=$wbs
--target_file_size_base=$mb --max_write_buffer_number=$wbn
--max_background_compactions=$mbc
--level0_file_num_compaction_trigger=$ctrig
--level0_slowdown_writes_trigger=$delay --level0_stop_writes_trigger=$stop
--num_levels=$levels --delete_obsolete_files_period_micros=$del
--min_level_to_compress=$mcz --max_grandparent_overlap_fac tor=$overlap
--stats_per_interval=1 --max_bytes_for_level_base=$bpl --memtablerep=vector
--use_existing_db=0 --disable_auto_compactions=1
--source_compaction_factor=10000000
echo "Running manual compaction to do a global sort map-reduce style...."
bpl=10485760;overlap=10;mcz=2;del=300000000;levels=2;ctrig=10000000;
delay=10000000; stop=10000000; wbn=30; mbc=20; mb=1073741824;wbs=268435456;
dds=1; sync=0; r=1000000000; t=1; vs=800; bs=65536; cs=1048576; of=500000;
si=1000000; ./db_bench --benchmarks=compact --disable_seek_compaction=1
--mmap_read=0 --statistics=1 --histogram=1 --num=$r --threads=$t
--value_size=$vs --block_size=$bs --cache_size=$cs --bloom_bits=10
--cache_numshardbits=4 --open_files=$of --verify_checksum=1
--db=/data/mysql/leveldb/test --sync=$sync --disable_wal=1
--compression_type=zlib --stats_interval=$si --compression_ratio=50
--disable_data_sync=$dds --write_buffer_size=$wbs
--target_file_size_base=$mb --max_write_buffer_number=$wbn
--max_background_compactions=$mbc
--level0_file_num_compaction_trigger=$ctrig
--level0_slowdown_writes_trigger=$delay --level0_stop_writes_trigger=$stop
--num_levels=$levels --delete_obsolete_files_period_micros=$del
--min_level_to_compress=$mcz --max_grandparent_overlap_factor =$overlap
--stats_per_interval=1 --max_bytes_for_level_base=$bpl --memtablerep=vector
--use_existing_db=1 --disable_auto_compactions=1
--source_compaction_factor=10000000
du -s -k test
504730832 test

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/21#issuecomment-29544041
.

Mark Callaghan
[email protected]

[dhruba]

The above comments show how to do bulk load which should solve your problem