• xdb/index/src/index_errors.rs

  Line 27 in 609ed02

  pub fn btree_index_from_bytes_delete_key_not_found() -> Error {

• xdb/index/src/index_errors.rs

  Line 35 in 609ed02

  pub fn btree_index_from_bytes_delete_key_or_value_not_found() -> Error {

• xdb/index/src/index_errors.rs

  Line 51 in 609ed02

  pub fn unique_btree_index_from_bytes_delete_key_not_found() -> Error {

• xdb/index/src/index_errors.rs

  Line 59 in 609ed02

  pub fn unique_btree_index_from_bytes_insert_key_or_value_not_found() -> Error {

• xdb/index/src/index_errors.rs

  Line 67 in 609ed02

  pub fn unique_btree_index_from_bytes_remove_key_or_value_not_found() -> Error {

• xdb/index/src/index_errors.rs

  Line 75 in 609ed02

  pub fn hash_map_index_from_bytes_delete_key_not_found() -> Error {

• xdb/index/src/index_errors.rs

  Line 83 in 609ed02

  pub fn hash_map_index_from_bytes_delete_key_or_value_not_found() -> Error {

• xdb/index/src/index_errors.rs

  Line 91 in 609ed02

  pub fn hash_map_index_from_bytes_remove_key_or_value_not_found() -> Error {

• xdb/index/src/index_errors.rs

  Line 99 in 609ed02

  pub fn unique_hash_map_index_from_bytes_delete_key_not_found() -> Error {

• xdb/index/src/index_errors.rs

  Line 107 in 609ed02

  pub fn unique_hash_map_index_from_bytes_insert_key_or_value_not_found() -> Error {

• xdb/index/src/index_errors.rs

  Line 115 in 609ed02

  pub fn unique_hash_map_index_from_bytes_remove_key_or_value_not_found() -> Error {

rust-clippy-analyze
CodeQL Action v1 will be deprecated on December 7th, 2022. Please upgrade to v2. For more information, see https://github.blog/changelog/2022-04-27-code-scanning-deprecation-of-codeql-action-v1/

Check here

• use u32::to_le_bytes
  

  xdb/util/src/lib.rs

  Line 18 in 14cd92d

  pub fn u32_to_bytes(n: u32) -> ([u8; 4]) {

• use u32::from_le_bytes
  

  xdb/util/src/lib.rs

  Line 29 in 14cd92d

  pub fn bytes_to_u32(bytes: &[u8]) -> u32 {

• update all implementations of u32_to_bytes, bytes_to_u32

To Do:

read_block(
    &mut self,
    block_index: BlockIndex,
    data_start_index: Option<BlockLength>,
    data_end_index: Option<BlockLength>
) {
   // ...
}

Why needed?

Make log traversal more efficient. No need to read full segment payload while traversing.

Errors in the code base are not standardised so far.

All Errors should be collected in a single file for each sub-package

Error {
  code: 500, // HTTP like error code
  message: format!(
    "{}/{} - {}",
    MODULE_NAME,
    "<error_brief>",
    "<error_description>"
  )
}

Current status (5c9792ae0edfe8233987fea622c94084747883e4):

Existing structure of project

We are using Cargo workspace to manage multiple packages(library or binary) in this repo.

[workspace]

members = [
    "util",       # Dir: /util, Description: Reusable functions and structs, such as custom error struct
    "storage",    # Dir: /storage, Description: Module to manage storage file with blocks of fixed size. ie, the primitive storage engine
    "logchain",   # Dir: /logchain, Description: Abstract function for Storage struct, to store data as a chain of blocks.
    "index",      # Dir: /index, Description: Inbuild HashMapIndex, BtreeIndex, etc... logic buit for logchain, along with kv store(Manage CRUD for keyvalue store).
]

Existing naming used

• block: A segment of fixed size on storage file.

• block_index: The serial index of a block on storage file.

• block_data: The data of a block on storage file.

• block_header: The header of a block on storage file.

• block_len: The length of a block on storage file.

• log / logchain: A chain of blocks. (This one makes me cry)

• kv: module to manage Key-value store on block storage.

• tupple: A tuple in Key-value store.

Issues

Issues with current naming

1. block is a segment of fixed size on storage file (we have tell this docs). But a block can be any entity of any size.
2. logchain This is a crazy meaningless name.

Issues with existing structure

1. Now, we see that this project can be used to build multiple database binaries. Not just one graph DB as intended intially. So, it makes more sence to segregate storage engine libraries and buildable packages.
2. In index module, we have kv module, which is a has primitive function to manage crud for KV store. So, kv module deserves to be a separate library, and should not be restructed to inbuilt index data structues present in index module.
3. In Error framework, We have and error type Happens (which denotes to expected errors which can be part of the logic). This brings in requirement to perform error handling, which may not suit dependency injection as good practice. We have used this extensively in index module, so we must reconsider this.

Resolutions

New naming

• page: A segment of fixed size on storage file. (Rename from block)

• page_index: The serial index of a page on storage file.

• page_payload: The data intended to be stored in a page.

• page_header: Initial bytes of a page to store little-endian representation for size of page_payload in bytes.

• page_padding: Unused space of a page.

• page_body: All the bytes inside a page. page_header + page_payload + page_padding

• pagestore: Module to manage storage file with pages of fixed size. ie, the primitive storage engine. (Rename from storage)

• pagelist: A chain of pages. (Rename from logchain)

New structure

Follow convention

More about pages
About firebirdDB

[workspace]

members = [
    "util",                         # {Error, Logger, Cursor}
    "storage_engine/page_store",    # {PagePayload, PageStore, PageStoreHeader, PageIndex}
    "storage_engine/page_list",     # {create, append, delete}
    "storage_engine/kv_serializer", # {serialize_kv, deserialize_kv, serialize_kv_list, deserialize_kv_list}
    "storage_engine/index_traits",  # {IndexOpration, UnqiueIndexOpration, IndexSerializer, IndexClone}
    
    # More in future
    "kv_db_lite",
    "kv_db",
    "document_db_lite",
    "document_db",
    "graph_db",
]

• Verify if Resolution works
• Implement the changes

Need to close following issues before or along side

• #27
• #12
• #3

PickUpAfter #33 is merged

New Items in util

pub const ENV: Env = Env::Dev;

pub mod byte_cursor;

pub mod error;

pub mod test_util;

pub fn make_chunks(data: &[u8], chunk_len: usize) -> (usize, std::slice::Chunks<u8>)

/// Returns data: Vec<u8>
pub fn read_log(
   storage: &mut Storage,
   start_segment_block_index: BlockIndex
) -> Result<Vec<u8>, Error> {
   // traverse log
   // return log data as Vec<u8>
}

• define read_log
• integration tests for read_log in logchain/tests/logchain_storage_flows.rs

Tracking issue for:

• https://github.com/rishavbhowmik/xdb/security/code-scanning/43

store and test set of indexes in Storage as logs.

• xdb/logchain/README.md

  Line 3 in bde1210

  An abstraction of build on Storage Engine (`storage` module) to store appendable data in a chain of blocks.

• xdb/storage/README.md

  Line 7 in bde1210

  Mantain a storage file, segmented in blocks of size fixed `BLOCK_LEN`.

Create a abstract function to return chunks, chunk_length and blocks_required

fn make_chunks(data, chunk_len) -> (blocks_required, chunks);

• define function
• add unit tests

Originally posted by @rishavbhowmik in #5 (comment)