Comments (6)
I see. We use jemalloc since the RSS with glibc malloc tends to be high for RocksDB-based applications (example: https://smalldatum.blogspot.com/2015/10/myrocks-versus-allocators-glibc.html). if there's actually a memory leak that won't help, but just fyi.
Hi ajkr, thanks for your infomation and help.
I did the tests of glibc and jemalloc, here are some findings from my experiments:
when I use jemalloc the memory increases as well. But after reaching the threshold set by the block cache, it stops increasing. This is different from the performance of glibc's malloc. After glibc reaches the point set by the cache, the memory will still continue to increase, but the extent of the increase will be reduced.
By the way, I use jeprof to draw the call stacks diagram of rocksdb. From the call stack diagram, only one NewDataBlockIterator can be seen, which is inconsistent with the previous iterator assumption of resource occupancy.
The main method of using memory is: rocksdb::AllocateBlock. I'm not sure if this makes sense but it should be certain that the memory increase is not caused by iterators.
from rocksdb.
In the online environment, when there is a lot of reading during peak periods, the memory will always increase, and once it increases, it will not fall back. The more rocksdb reads, the more resources it applies for, and the more memory and swap are consumed, which will not decrease.
My rocksdb size is 200GB. The RSS is 64GB, swap is 32GB.
from rocksdb.
I want to konw rocksdb itself close the iterators or not? I found the memory grow unlimitly and never decrese. I didn't use any iterators in my code and I just use the Get() interface only, so I guess rocksdb use lots of iterators by Get(), and not close it.
see the similar issue: #3216
I want to konw there is any method that I can release the iterator, make memory reclaim
from rocksdb.
I want to konw rocksdb itself close the iterators or not? I found the memory grow unlimitly and never decrese. I didn't use any iterators in my code and I just use the Get() interface only, so I guess rocksdb use lots of iterators by Get(), and not close it.
see the similar issue: #3216I want to konw there is any method that I can release the iterator, make memory reclaim
There is no API to release the iterators you described since they are not known to exist.
Can you build with ASAN enabled, and then run your program for a bounded duration to check for memory leaks?
What allocator do you use? Can you provide a heap profile, and/or allocator stats? If you use jemalloc, the latter can be printed to the LOG file by setting dump_malloc_stats
and stats_dump_period_sec
.
from rocksdb.
I want to konw rocksdb itself close the iterators or not? I found the memory grow unlimitly and never decrese. I didn't use any iterators in my code and I just use the Get() interface only, so I guess rocksdb use lots of iterators by Get(), and not close it.
see the similar issue: #3216I want to konw there is any method that I can release the iterator, make memory reclaim
There is no API to release the iterators you described since they are not known to exist.
Can you build with ASAN enabled, and then run your program for a bounded duration to check for memory leaks?
What allocator do you use? Can you provide a heap profile, and/or allocator stats? If you use jemalloc, the latter can be printed to the LOG file by setting
dump_malloc_stats
andstats_dump_period_sec
.
OK, Thanks. I will try it. I currently use glibc as allocator.
from rocksdb.
I see. We use jemalloc since the RSS with glibc malloc tends to be high for RocksDB-based applications (example: https://smalldatum.blogspot.com/2015/10/myrocks-versus-allocators-glibc.html). if there's actually a memory leak that won't help, but just fyi.
from rocksdb.
Related Issues (20)
- TransactionDB->CreateColumnFamilyWithImport doesn't create valid column family handles HOT 1
- Cache dumper could exit early
- rocksdb 9.0.0 fails to build on GCC 13.2.1 with `-march=x86-64-v3`
- Cache Dump all keys without filter
- checkpoint directory is empty when db is empty HOT 2
- [Java] In read-only mode can't get data from blob only if there is just one checkpoint with one entry HOT 11
- Solution to the periodic slowdown of GetUpdatesSnce
- Heading typo on wiki docs: PlainTable Format HOT 1
- does rocksdb provide any monitoring metrics? HOT 1
- Java release for 9.1 HOT 2
- segFault while write large data on multiple thread HOT 3
- Feature request: rate limit compaction triggered by periodic compaction seconds/ ttl only HOT 5
- The value of 'micros/op' is not equal to 1,000,000 divided by the value of 'ops/sec' HOT 4
- Question about CompactRange behavior with option atomic_flush=true
- Segfault During compaction using FIFO Compaction style for a single CF
- Safer shutdown behaviour by deafult HOT 2
- StdLogger truncating last letter in some cases HOT 1
- High Memory Usage/ LRU cache size is not being respected HOT 1
- rocksdb abnormal exit
- Feature request: log to stderr logger + LOG file
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rocksdb.