creatstar / rtrans Goto Github PK

1. To start a new data manager, need parameters
data_dir: the dir to store the data and metadata
format_version: the version of index and data file
size_per_index: size in byte per index
index_num_per_file: number of index per index file
max_size_data_file: maximum size per data file
data_blk_size: block size for data file


data_dir +
         + metadata [bin file] #generated by data manager
         + transid [dir] #transid for every backend
         + 0...n [dir]
            + index.0
            + data.0

metadata structrue: 
    [format_version, size_per_index, index_num_per_file, index_file_num_per_dir,max_size_data_file, data_blk_size, need_compress], 
    size: 2+4+4+4+8+4+2 = 28 bytes
index structrue: 
    [transid, type_no, dir_no, file_no, data_blk_no, data_len, checksum], 
    size: 8+4+4+4+8+4+4 = 36
data structure: 
    [transid, checksum, raw_datapack]


format_version: 16-bits unsigned integer (S)
size_per_index: 32-bits unsigned integer (L)
index_num_per_file: 32-bits unsigned integer (L)
index_file_num_per_dir: 32-bits unsigned integer (L)
max_size_data_file: 64-bits unsigned integer (Q)
data_blk_size: 32-bits unsigned integer (L)
need_compress: 16-bits unsigned integer (S)

transid: 64-bits unsigned integer (Q)
type_no: 32-bits unsigned integer (L)
dir_no: 32-bits unsigned integer (L)
file_no: 32-bits unsigned integer (L)
data_blk_no: 64-bits unsigned integer (Q)
data_len: 32-bits unsigned integer(L)
check_sum: 32-bits unsigned integer(L) CRC check sum

NOTICE:
1. transid start from 1, 0 stands for nil