boost.mutation_writer_test.test_token_group_based_splitting_mutation_writer fails due to bad-alloc about scylladb

[raphaelsc]

turns out output was very useful, by reusing same random seed, I see that each mutation in a complex schema could grow often above 2M, and with 300 partitions, that could easily exahust memory. A possible fix is to allow generate_random_mutations() to generate less partitions than asked to, if footprint grows beyond some threshold (e.g. 100M). @denesb thoughts?

random-seed=1848215131
INFO  2024-02-06 09:34:37,394 [shard 0:main] testlog - Before creating random schema: free memory: 704266240
INFO  2024-02-06 09:34:37,395 [shard 0:main] testlog - After creating random schema: free memory: 703557632
INFO  2024-02-06 09:34:37,395 [shard 0:main] testlog - Random schema:
CREATE TYPE test_token_group_based_splitting_mutation_writer.udt428 (
    f0 text,
    f1 timeuuid,
    f2 frozen<set<blob>>,
    f3 inet
);
CREATE TYPE test_token_group_based_splitting_mutation_writer.udt76 (
    f0 inet,
    f1 timeuuid,
    f2 frozen<list<smallint>>,
    f3 smallint
);
CREATE TABLE test_token_group_based_splitting_mutation_writer.table694 (
	pk0 text,
	pk1 frozen<tuple<frozen<map<date, varint>>, double>>,
	pk2 timeuuid,
	ck0 timestamp,
	ck1 frozen<udt428>,
	ck2 ascii,
	ck3 timestamp,
	v0 time,
	v1 timeuuid,
	v2 blob,
	v3 list<frozen<set<frozen<list<frozen<map<frozen<map<timestamp, frozen<list<text>>>>, tinyint>>>>>>>,
	v4 boolean,
	s0 ascii static,
	s1 varint static,
	s2 inet static,
	s3 duration static,
	s4 frozen<tuple<frozen<set<frozen<udt76>>>, bigint>> static,
	s5 frozen<tuple<frozen<set<blob>>, boolean, frozen<map<inet, date>>>> static
	PRIMARY KEY ((pk0, pk1, pk2), ck0, ck1, ck2, ck3))
INFO  2024-02-06 09:34:37,657 [shard 0:main] testlog - Muts: 1, Memory usage: 1374583, Last mut's memory usage: 1374583, Free memory: 695107584
INFO  2024-02-06 09:34:38,172 [shard 0:main] testlog - Muts: 2, Memory usage: 3455502, Last mut's memory usage: 2080919, Free memory: 692129792
INFO  2024-02-06 09:34:38,351 [shard 0:main] testlog - Muts: 3, Memory usage: 4262138, Last mut's memory usage: 806636, Free memory: 691298304
INFO  2024-02-06 09:34:38,569 [shard 0:main] testlog - Muts: 4, Memory usage: 5175237, Last mut's memory usage: 913099, Free memory: 690171904
INFO  2024-02-06 09:34:38,641 [shard 0:main] testlog - Muts: 5, Memory usage: 5470427, Last mut's memory usage: 295190, Free memory: 690053120
INFO  2024-02-06 09:34:39,281 [shard 0:main] testlog - Muts: 6, Memory usage: 7895482, Last mut's memory usage: 2425055, Free memory: 686268416
INFO  2024-02-06 09:34:39,842 [shard 0:main] testlog - Muts: 7, Memory usage: 10649063, Last mut's memory usage: 2753581, Free memory: 681201664
INFO  2024-02-06 09:34:40,024 [shard 0:main] testlog - Muts: 8, Memory usage: 11684608, Last mut's memory usage: 1035545, Free memory: 680415232
INFO  2024-02-06 09:34:40,446 [shard 0:main] testlog - Muts: 9, Memory usage: 13345857, Last mut's memory usage: 1661249, Free memory: 677691392
INFO  2024-02-06 09:34:40,599 [shard 0:main] testlog - Muts: 10, Memory usage: 13902792, Last mut's memory usage: 556935, Free memory: 677257216
...
INFO  2024-02-06 09:36:39,504 [shard 0:main] testlog - Muts: 290, Memory usage: 447803617, Last mut's memory usage: 727445, Free memory: 50786304
INFO  2024-02-06 09:36:39,949 [shard 0:main] testlog - Muts: 291, Memory usage: 449978968, Last mut's memory usage: 2175351, Free memory: 47718400
INFO  2024-02-06 09:36:40,527 [shard 0:main] testlog - Muts: 292, Memory usage: 452378167, Last mut's memory usage: 2399199, Free memory: 43986944
INFO  2024-02-06 09:36:40,651 [shard 0:main] testlog - Muts: 293, Memory usage: 452841584, Last mut's memory usage: 463417, Free memory: 43806720
INFO  2024-02-06 09:36:41,174 [shard 0:main] testlog - Muts: 294, Memory usage: 454963894, Last mut's memory usage: 2122310, Free memory: 39526400
INFO  2024-02-06 09:36:41,523 [shard 0:main] testlog - Muts: 295, Memory usage: 456475296, Last mut's memory usage: 1511402, Free memory: 37834752
INFO  2024-02-06 09:36:42,142 [shard 0:main] testlog - Muts: 296, Memory usage: 458832584, Last mut's memory usage: 2357288, Free memory: 34955264
INFO  2024-02-06 09:36:42,430 [shard 0:main] testlog - Muts: 297, Memory usage: 459938281, Last mut's memory usage: 1105697, Free memory: 33738752
INFO  2024-02-06 09:36:42,795 [shard 0:main] testlog - Muts: 298, Memory usage: 461917513, Last mut's memory usage: 1979232, Free memory: 30736384
INFO  2024-02-06 09:36:43,236 [shard 0:main] testlog - Muts: 299, Memory usage: 464338414, Last mut's memory usage: 2420901, Free memory: 26525696
INFO  2024-02-06 09:36:43,431 [shard 0:main] testlog - Muts: 300, Memory usage: 465329174, Last mut's memory usage: 990760, Free memory: 25653248
INFO  2024-02-06 09:36:43,438 [shard 0:main] testlog - After generating random mutations: Memory usage: 465329174
INFO  2024-02-06 09:36:43,438 [shard 0:main] testlog - segregating
unknown location(0): fatal error: in "test_token_group_based_splitting_mutation_writer": std::bad_alloc: std::bad_alloc

[denesb]

@raphaelsc generate_random_mutations() has controls for how many rows/partitions to generate. You can reduce the number of rows to a small amount, because what you want to stress here is the token-based splitting anyway.

[denesb]

You can also reduce the number of columns to generate for the schema in the first place, to make rows smaller.

[raphaelsc]

You can also reduce the number of columns to generate for the schema in the first place, to make rows smaller.

I see test_timestamp_based_splitting_mutation_writer uses a similar schema, but no way it has the same partition count. Will send a patch. thanks!