kirov7 / couloydb

CouloyDB is a storage engine that strikes a balance between performance and storage costs.

"We can only see a short distance ahead, but we can see plenty there that needs to be done." - Alan Turing

• 🌱 I’m currently learning distributed system and the data storage engine
• 👯 I’m looking to collaborate on CouloyDB and other projects
• 💬 Ask me about CQUT CFC Studio
• 📫 How to reach me: [email protected]

couloydb's Issues

Refactor Kuloy to call CouloyDB's transaction interface for operations.

The original Kuloy implemented data structures and supported commands at the server level. However, now that the storage engine layer (CouloyDB) natively supports data structures, it requires modification to utilize the interfaces provided by the storage engine layer for command implementation.

原本的 Kuloy 实现数据结构以及支持命令的方式是在 Server 层进行实现的，而现在存储引擎层（CouloyDB）已经原生支持数据结构，所以需要进行更改，用存储引擎层提供的接口来实现命令。

Implement ZSet data type

Implement the ZSet data structure at the storage engine layer.You can refer to the implementation method of the Hash data structure.

在存储引擎层实现ZSet数据结构。可以参考Hash数据结构的实现方法。

Implement Set data type

Implement the Set data type at the storage engine level.

First, you need to enable the storage engine to support the Set data type.The tasks include designing and adding index for the Set data type, creating Set indexes upon database startup, updating Set indexes after transaction commits, and so on. For the specific implementation, you can refer to the Hash type(This task is already assigned to @yoshihiro-shu, so the next phase of the task may have to wait for him to complete).

Afterward, there are some apis to implement. The specific apis to be implemented are listed in the table below.Please also api test cases for each instruction.

API	Explanation	assigned	done
SADD key member [member ...]	Add one or more members to a set.	@yoshihiro-shu	√
SCARD key	Get the number of members in a set.	@yoshihiro-shu	√
SMEMBERS key	Get all the members in a set.	@yoshihiro-shu	√
SREM key member [member ...]	Remove one or more members from a set.	@yoshihiro-shu	√
SRANDMEMBER key [count]	Get one or more random members from a set.
SPOP key [count]	Remove and return one or more random members from a set.
SMOVE source destination member	Move a member from one set to another.
SDIFF key [key ...]	Get the difference between sets.
SINTER key [key ...]	Get the intersection of sets.
SUNION key [key ...]	Get the union of sets.
SUNIONSTORE destination key [key ...]	Store the union of sets into a new set.

---

首先，你需要让存储引擎支持Set数据类型。任务包括为Set数据类型设计和添加索引，在数据库启动时创建Set索引，在事务提交后更新Set索引等。对于具体的实现，你可以参考Hash类型（此任务已经分配给yoshihiro shu，所以下一阶段的任务可能要等他完成）。

然后，接下来还有一些指令需要实现。具体需要实现的指令见下表。每个指令也请完成测试用例的实现。

接口	命令解释	承接人	是否完成
SADD key member [member ...]	向集合中添加一个或多个成员。	@yoshihiro-shu	√
SCARD key	获取集合中的成员数量。	@yoshihiro-shu	√
SMEMBERS key	获取集合中的所有成员。	@yoshihiro-shu	√
SISMEMBER key member	检查成员是否存在于集合中。	@yoshihiro-shu	√
SREM key member [member ...]	从集合中移除一个或多个成员。
SRANDMEMBER key [count]	从集合中随机获取一个或多个成员。
SPOP key [count]	随机移除并返回集合中的一个或多个成员。
SMOVE source destination member	将成员从一个集合移动到另一个集合。
SDIFF key [key ...]	返回给定集合之间的差集。
SINTER key [key ...]	返回给定集合的交集。
SUNION key [key ...]	返回给定集合的并集。
SUNIONSTORE destination key [key ...]	将给定集合的并集存储到一个新的集合中。

Hope to build CouloyDB together

Currently, the development teams for CouloyDB and Kuloy are relatively small. We hope to have more people participate in the project's development, collaboratively improving it to achieve a production-level standard.

We will publish the upcoming work plans in the form of issues. If any interested individuals wish to participate, they can claim ownership of the relevant issues. This approach helps prevent duplication of efforts.

There are various areas where contributions are possible, such as development in the storage engine layer, service layer, enhancing test cases, and adding documentation or comments. Everyone is welcome to submit pull requests or open issues to share ideas.We welcome all forms of contributions!

Friends with ideas or questions are welcome to join our WeChat group. Feel free to engage in discussions and exchange thoughts with everyone.

---

目前，CouloyDB和Kuloy的开发团队人数相对较少。我们希望能够有更多的朋友能够参与到项目的开发中，共同改进并达到生产级水平。

我们将以issue的形式发布未来的工作计划。如果有任何感兴趣的人希望参与，可以认领相关的issue。这样的方式有助于避免重复工作。

有许多方面可以进行贡献，比如存储引擎层的开发、服务层的开发、完善测试用例以及添加文档或者注释。大家如果有自己的好想法也可以发起pr或打开一个 issue。我们非常欢迎各种形式的贡献！

拥有想法或者问题的朋友，欢迎加入我们的WeChat群。随时参与讨论并与大家交流想法。

Implement List data structure

Implement the List data structure at the storage engine layer.You can refer to the implementation method of the Hash data structure.

在存储引擎层实现List数据结构。可以参考Hash数据结构的实现方法。

`

There may be some concurrency issues.

Get 可能会出现的问题：

1. 可能无法读取到最新的数据

	线程1 Get	线程2 Put
1	从 MemTable 中获取键 key 的 DataPos
2		将键 key 的新值写入 DataFile 中
3		更新 MemTable
4	根据刚刚获取的 DataPos 从 DataFile 中取值

2. 可能会读到已经被删除的数据的旧值

	线程1 Get	线程2 Del
1	从 MemTable 中获取键 key 的 DataPos
2		将键 key 的删除条目写入 DataFile 中
3		更新 MemTable，删除对应的 DataPos
4	根据刚刚获取的 DataPos 从 DataFile 中取值

Del 可能会出现的问题：

1. 覆盖掉刚插入的新值

	线程1 Del	线程2 Put
1	从 MemTable 中判断 key 是存在的
2		将键 key 的新值写入 DataFile 中
3		更新 MemTable
4	将键 key 的删除条目写入 DataFile 中
5	更新 MemTable，删除对应的 DataPos

不知道我的理解是不是有问题，如果会出现以上情况的话那么一把大锁应该是无法避免的。

Implement more Hash apis.

The Hash structure already supports some commonly used APIs, like HSet, HGet, etc. You can implement more apis for it.

API	Explanation	assigned	done
HSET key field value	Set the value of a field in a hash.	@bigboss2063	✅
HGET key field	Get the value of a field in a hash.	@bigboss2063	✅
HMSET key field value [field value ...]	Set multiple fields and values in a hash.	@bigboss2063	✅
HMGET key field [field ...]	Get the values of one or more fields in a hash.	@bigboss2063	✅
HDEL key field [field ...]	Delete one or more fields from a hash.	@bigboss2063	✅
HEXISTS key field	Check if a field exists in a hash.	@bigboss2063	✅
HGetAll key	Get all the fileds and the values in a hash.	@bigboss2063	✅
HLEN key	Get the number of fields in a hash.	@99HU	✅
HKEYS key	Get all the field names in a hash.	@99HU	✅
HVALS key	Get all the values in a hash.	@99HU	✅
HStrLen key field	Get the length of value of a field in a hash	@99HU	✅

Anyone who wants to implement other api is welcome to comment below!

现在 Hash 数据结构已经实现了一些常用的接口，比如 HSet、HGet 等等。可以为它实现更多的接口。

指令	解释	承接人	是否完成
HSET key field value	设置哈希表中字段的值。	@bigboss2063	✅
HGET key field	获取哈希表中字段的值。	@bigboss2063	✅
HMSET key field value [field value ...]	同时设置哈希表中多个字段的值。	@bigboss2063	✅
HMGET key field [field ...]	获取哈希表中一个或多个字段的值。	@bigboss2063	✅
HDEL key field [field ...]	删除哈希表中一个或多个字段。	@bigboss2063	✅
HEXISTS key field	检查哈希表中是否存在指定的字段。	@bigboss2063	✅
HGetAll key	获取哈希表中的所有字段和值。	@bigboss2063	✅
HLEN key	获取哈希表中字段的数量	@99HU	✅
HKEYS key	获取哈希表中所有字段。	@99HU	✅
HVALS key	获取哈希表中所有字段的值	@99HU	✅
HStrLen key field	获取哈希表中字段的值对应的长度	@99HU	✅

还有想实现其他接口的朋友欢迎在下面评论出来！

不清晰的lock逻辑，iterator文件未被使用

您好，我最近在阅读代码时，关于数据库的lock逻辑中，有的地方让我困惑。

根据我阅读代码时的理解，db.mu的用途是对磁盘文件读写时加锁，索引的数据结构有单独的mu。

例如，在文件db.go的第112行，db.mu在Get函数中对索引加了锁，而在Put/Delete db.mu没有对索引加锁。
在文件db.go的第182行，db.mu在Fold函数中也对索引加了锁。

还有个问题是，iterator文件未被使用。

我相信您对这个项目有更深入的了解，能够解答我的困惑。我希望这个问题反馈能对您有所帮助，期待您的回复。

感谢您的时间和努力，祝您的项目成功！