Comments (7)
You are right, I missed this one.. Actually, I think replacing the uint32_t ref
with atomic_uint ref
should be enough to fix this issue since only one thread will decrement the ref count to zero. Do you have the option to try this fix?
from siridb-server.
I am testing #159 - basically the same thing. I'll let it run overnight.
from siridb-server.
@joente I had some time to investigate the core dump a little bit more. It looks like some kind of memory corruption.
If you look at the iteration at:
siridb-server/src/siri/db/listener.c
Lines 5758 to 5776 in 0447d82
The series
struct looks like a garbage memory:
The culprit, in this case, is name_len
that (I think) makes no sense. Although the memory looks corrupted, it's properly allocated. The pcre2_match
it's just slow going over a string that is 22kb long. Looks like all this is not synced to the disk because server restart helps for a couple of hours.
from siridb-server.
@joente Could you please give me any pointers on where to start investigating? This is plaguing most of our instances. I can definitely allocate time to fix this, but I am not sure where to start looking for possible corruption.
We don't do anything fancy, 90% of our queries are inserts, the rest is select mean()/min()/max()
with explicit series or regexp. There doesn't seem to be an easy repro case.
from siridb-server.
@ubnt-michals, I'm not entirely sure where it goes wrong but I will look into this issue and hope to give you an answer soon. Do you know if the issue disappears once you eliminate regexp queries and stick to fixed or tagged series?
from siridb-server.
@joente @luznicky
Some more observations:
The corruption probably doesn't happen by deleting a series.
In this case q_wrapper->vec->data[6000]
is the same pointer as siridb->series_map->vec->data[6000]
. Deleting a series would cause the siridb->series_map->vec
to be set to NULL
. Also both vec
s have the same len
.
In this core dump, a total of 3 series indexed 5998, 5999, 6000 looks corrupt. The data in them are not totally random.
For example, name_len is 22032
in all of them.
Also, the fact that they are allocated next to each other points me to some buffer overrun.
from siridb-server.
I think we have found a potential culprit of this memory corruption.
From the looks of it, the reference counting is not thread-safe.
The same locking that is for example here:
siridb-server/src/siri/db/shard.c
Lines 1419 to 1423 in ff98940
Is not here around siridb_series_decref
siridb-server/src/siri/db/shard.c
Line 1469 in ff98940
The "optimize" task runs in a different thread, and siridb_series_incref
and siridb_series_decref
are not atomic. Therefore they should create a race condition and use after free
would explain what we see in the core dumps.
@joente can you confirm that that's the case? Maybe we are missing something and the use of siridb_series_incref
/siridb_series_decref
is not a problem.
from siridb-server.
Related Issues (20)
- Is there any data compression method? HOT 5
- Configuration variable `server_name` should use the default port when omitted HOT 1
- Kubernetes compatibility HOT 1
- Change SIRIDB_DEFAULT_DB_PATH to SIRIDB_DB_PATH HOT 1
- Server stuck in libuv HOT 5
- Question. Time series model and query. HOT 2
- HTTP Pipelining issue in embedded api service HOT 17
- Confused about database duration and shards expiration HOT 4
- Is there any detailed document for setup pool(s)? HOT 4
- Improve the internal mapping to use less memory HOT 1
- Writes being blocked during complex query HOT 2
- Update siridb to work with libcleri 1.x
- Simple insert HOT 4
- Drop dulicate values
- Error when adding to pool HOT 2
- Disk Failure HOT 2
- Support for Encrypted Connections HOT 1
- Freezing of DB HOT 2
- DB resilience when there is a power loss HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from siridb-server.