Comments (5)
@erez-speedb please do a performance test on this branch. tnx!
from speedb.
as far i can see the wf fill up an d overwrite performance is very good and many flush request is being made but i can see that l0/l1/l2 have many many files and the compactions/ flush is not working well...
i can see from log
EVENT_LOG_v1 {"time_micros": 1657494329348817, "job": 1800, "event": "compacti
on_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [7669, 7656, 7643, 7630, 7617, 7603, 7590, 75
78, 7565, 7551, 7538, 7526, 7515, 7504, 7493, 7482, 7471, 7461, 7450, 7438, 7427, 7416, 7405, 7394, 7383, 7372
, 7361, 7349, 7338, 7327, 7316, 7305, 7293, 7282, 7271, 7260, 7249, 7238, 7227, 7215, 7204, 7192, 7181, 7169,
7158, 7146, 7134, 7122, 7110, 7098, 7086, 7074, 7062, 7054, 7052, 7046, 7037, 7028, 7019, 7009, 7000, 6991, 69
82, 6972, 6963, 6954, 6944, 6935, 6926, 6916, 6907, 6898, 6888, 6879, 6869, 6860, 6851, 6842, 6832, 6823, 6814
, 6804, 6794, 6785, 6776, 6766, 6757, 6747, 6737, 6728, 6718, 6709, 6700, 6690, 6681, 6671, 6661, 6651, 6640,
6630, 6620, 6610, 6599, 6589, 6579, 6568, 6557, 6546, 6536, 6525, 6515, 6504, 6493, 6483, 6472, 6461, 6451, 64
40, 6430, 6419, 6409, 6398, 6387, 6377, 6366, 6355, 6345, 6334, 6324, 6313, 6302, 6292, 6281, 6270, 6260, 6249
, 6239, 6228, 6217, 6207, 6196, 6185, 6175, 6164, 6153, 6142, 6132, 6121, 6110, 6099, 6089, 6079, 6068, 6057,
6046, 6036, 6025, 6014, 6003, 5993, 5982, 5971, 5961, 5950, 5939, 5929, 5919], "files_L1": [5920, 5921, 5923,
5924, 5925, 5927, 5930, 5932, 5933, 5935, 5936, 5937, 5941, 5942, 5943, 5944, 5945, 5946, 5947, 5951, 5952, 59
53, 5954, 5955, 5956, 5957, 5959, 5963, 5965, 5966, 5967, 5968, 5974, 5975, 5976, 5977, 5978, 5979, 5980, 5984
, 5985, 5986, 5987, 5989, 5996, 5997, 5998, 5999, 6000, 6001, 6004, 6006, 6007, 6009, 6010, 6011, 6015, 6016,
6018, 6020, 6022, 6023, 6028, 6030, 6031, 6032, 6033, 6034, 6037, 6039, 6040, 6041, 6043, 6047, 6048, 6050, 60
52, 6053, 6054, 6055, 6059, 6060, 6061, 6065, 6066, 6069, 6070, 6071, 6072, 6073, 6074, 6076, 6077, 6083, 6084
, 6086, 6087, 6090, 6091, 6093, 6094, 6096, 6097, 6101, 6102, 6103, 6104, 6106, 6107, 6108, 6112, 6113, 6114,
6115, 6117, 6118, 6122, 6123, 6124, 6126, 6127, 6128, 6129, 6133, 6134, 6136, 6137, 6138, 6140, 6143, 6145, 61
46, 6148, 6149, 6150, 6151, 6155, 6156, 6158, 6159, 6160, 6161, 6162, 6165, 6166, 6167, 6168, 6170, 6171, 6172
, 6173, 6176, 6177, 6178, 6179, 6180, 6181, 6182, 6183, 6186, 6187, 6188, 6190, 6192, 6193, 6194, 6198, 6202,
6203, 6204, 6205, 6208, 6209, 6210, 6211, 6212, 6214, 6215, 6218, 6219, 6220, 6221, 6222, 6223, 6224, 6225, 62
26, 6230, 6231, 6234, 6235, 6236, 6237, 6241, 6242, 6243, 6245, 6246, 6247, 6250, 6251, 6252, 6253, 6254, 6255
, 6256, 6258, 6261, 6262, 6264, 6265, 6266, 6268, 6271, 6272, 6273, 6274, 6276, 6277, 6278, 6286, 6287, 6288,
6289, 6290, 6294, 6295, 6296, 6297, 6298, 6299, 6303, 6305, 6306, 6308, 6309, 6310, 6311, 6314, 6316, 6317, 63
18, 6319, 6321, 6326, 6327, 6328, 6329, 6332, 6335, 6336, 6338, 6339, 6340, 6341, 6342, 6343, 6346, 6347, 6349
, 6350, 6352, 6353, 6356, 6357, 6358, 6359, 6360, 6362, 6364, 6367, 6371, 6373, 6374, 6378, 6379, 6380, 6381,
6383, 6384, 6385, 6389, 6391, 6392, 6394, 6396, 6399, 6401, 6402, 6404, 6405, 6406, 6407, 6410, 6411, 6412, 64
13, 6414, 6415, 6416, 6417, 6421, 6422, 6424, 6425, 6426, 6431, 6433, 6434, 6435, 6436, 6437, 6438, 6443, 6444
, 6445, 6446, 6447, 6448, 6449, 6453, 6454, 6455, 6458, 6462, 6466, 6467, 6468, 6469, 6470, 6473, 6474, 6475,
6476, 6478, 6479, 6480, 6484, 6485, 6486, 6487, 6489, 6490, 6491, 6494, 6496, 6497, 6499, 6500, 6501, 6505, 65
06, 6508, 6510, 6511, 6513, 6516, 6518, 6519, 6521, 6523, 6526, 6528, 6529, 6532, 6533, 6537, 6538, 6540, 6541
, 6549, 6550, 6552, 6554, 6555, 6558, 6560, 6561, 6562, 6563, 6564, 6565, 6569, 6570, 6572, 6573, 6575, 6576,
6577, 6581, 6582, 6584, 6586, 6587, 6591, 6592, 6594, 6595, 6597, 6600, 6602, 6604, 6605, 6607, 6608, 6612, 66
13, 6614, 6615, 6617, 6618, 6621, 6623, 6624, 6625, 6627, 6631, 6632, 6633, 6635, 6636, 6637, 6642, 6644, 6646
, 6647, 6648, 6652, 6653, 6656, 6657, 6659, 6662, 6664, 6665, 6667, 6668, 6669, 6672, 6673, 6674, 6676, 6677,
6682, 6684, 6685, 6687, 6688, 6692, 6694, 6695, 6696, 6697, 6698, 6701, 6702, 6703, 6704, 6705, 6706, 6707, 67
....... L0 IS HUGE!
from speedb.
running the same code on the git repository works fine.
from speedb.
added commit to set the default use_spdb_writes to true
from speedb.
new write flow code was created
from speedb.
Related Issues (20)
- trigger check license and history workflow on push and pr HOT 1
- Unit Tests: Investigate & Fix WriteBufferManagerFlushInitiationTest.RegisteringLate (write_buffer_manager_test.cc)
- Workaround for a Cmake bug that happens when cross-compiling using Cmake on macOS x86_64 CPUs with target CPU arm64
- ZLIB 1.3: Complete the update of the url-s in Makefile / CMakeLists.txt HOT 1
- Force adding names to threads
- Proactive Flushes:A proactive flush may be initiated prematurely after a flush due to "Write Buffer Full"
- stress test error: running with ts resulted in inconsistent values HOT 1
- 2.8.0 release speedbjni JAR is not available in maven repositories HOT 2
- Enhance memory reporting to include more information about memtable
- set check license and history workflow as a separate test
- add jtest to CI
- Add the speedb_get_version api to the version
- check license and history - bad object on a PR from fork
- test pr from local and remote repo
- check license and history
- check license and history - bad object on PR from fork
- Decrease the default bucket number in hash_spd to 400,000 HOT 3
- db_stress gets stuck HOT 1
- Iterator gets verification errors
- min_background_io_size and max_background_io_size from DBOptions are not copied in BuildDBOptions()
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from speedb.