Possible memory corruption about siridb-server

[joente]

You are right, I missed this one.. Actually, I think replacing the uint32_t ref with atomic_uint ref should be enough to fix this issue since only one thread will decrement the ref count to zero. Do you have the option to try this fix?

[ubnt-michals]

I am testing #159 - basically the same thing. I'll let it run overnight.

[ubnt-michals]

@joente I had some time to investigate the core dump a little bit more. It looks like some kind of memory corruption.

If you look at the iteration at:

siridb-server/src/siri/db/listener.c

Lines 5758 to 5776 in 0447d82

	for (; q_wrapper->vec_index < index_end; q_wrapper->vec_index++)
	{
	series = (siridb_series_t *)
	q_wrapper->vec->data[q_wrapper->vec_index];
	pcre_exec_ret = pcre2_match(
	q_wrapper->regex,
	(PCRE2_SPTR8) series->name,
	series->name_len,
	0, /* start looking at this point */
	0, /* OPTIONS */
	q_wrapper->match_data,
	0); /* length of sub_str_vec */
	if ( pcre_exec_ret < 0 ||
	imap_add(q_wrapper->series_tmp, series->id, series))
	{
	siridb_series_decref(series);
	}
	}

The series struct looks like a garbage memory:

The culprit, in this case, is name_len that (I think) makes no sense. Although the memory looks corrupted, it's properly allocated. The pcre2_match it's just slow going over a string that is 22kb long. Looks like all this is not synced to the disk because server restart helps for a couple of hours.

[ubnt-michals]

@joente Could you please give me any pointers on where to start investigating? This is plaguing most of our instances. I can definitely allocate time to fix this, but I am not sure where to start looking for possible corruption.

We don't do anything fancy, 90% of our queries are inserts, the rest is select mean()/min()/max() with explicit series or regexp. There doesn't seem to be an easy repro case.

[joente]

@ubnt-michals, I'm not entirely sure where it goes wrong but I will look into this issue and hope to give you an answer soon. Do you know if the issue disappears once you eliminate regexp queries and stick to fixed or tagged series?

[ubnt-michals]

@joente @luznicky
Some more observations:

The corruption probably doesn't happen by deleting a series.

In this case q_wrapper->vec->data[6000] is the same pointer as siridb->series_map->vec->data[6000]. Deleting a series would cause the siridb->series_map->vec to be set to NULL. Also both vecs have the same len.

In this core dump, a total of 3 series indexed 5998, 5999, 6000 looks corrupt. The data in them are not totally random.

For example, name_len is 22032 in all of them.

Also, the fact that they are allocated next to each other points me to some buffer overrun.

[ubnt-michals]

I think we have found a potential culprit of this memory corruption.

From the looks of it, the reference counting is not thread-safe.

The same locking that is for example here:

siridb-server/src/siri/db/shard.c

Lines 1419 to 1423 in ff98940

	uv_mutex_lock(&siridb->series_mutex);
	vec_t * vec = imap_2vec_ref(siridb->series_map);
	uv_mutex_unlock(&siridb->series_mutex);

Is not here around siridb_series_decref

siridb-server/src/siri/db/shard.c

Line 1469 in ff98940

siridb_series_decref(series);

The "optimize" task runs in a different thread, and siridb_series_incref and siridb_series_decref are not atomic. Therefore they should create a race condition and use after free would explain what we see in the core dumps.

@joente can you confirm that that's the case? Maybe we are missing something and the use of siridb_series_incref/siridb_series_decref is not a problem.