pengubco / guiid

The Globally Unique Integer Identifier (GUIID) Service generates globally unique, increasing 64 bits integer IDs. GUUID runs on a a cluster of servers.

GUIID service

The Globally Unique Integer Identifier (GUIID) Service generates globally unique, increasing 64 bits integer IDs. GUUID runs on a a cluster of servers. Each server generates ID using the snowflake ID algorithm. By default, the service uses 10 bits to identify the server and thus the service supports 1024 servers. Each server can generate 4,096 ids per millisecond.

If your use case requires the following, GUIID service is a good fit.

1. An increasing 64 bits integer ID.
2. Globally unique. No collision of IDs.
3. Highly available. The ID is on the critical path and must be fault tolerant of single node failure.
4. Highly scalable.
5. Low latency. P99 lower than 1ms.

Does GUIID defy the CAP theorem?

No. GUIID service is a distributed system so it can guarantee only two out of three properties of CAP. GUIID chooses consistency and partition tolerance. High availability is achieved by running on a cluster of servers coordinated by etcd cluster. As long as more than half of the etcd nodes are healthy and one GUIID server is healthy, the GUIID service is available.

See design for how we coordinate a collection of stateless servers to generate globally unique and increasing IDs.

How to run the code?

Servers are stateless and do not depend on persisted storage. Servers do not talk with each other. Instead, servers coordinate through etcd. Each server needs read and write permission of a prefix on etcd. In order to run locally, follow the Appendix 1 to setup a etcd server on the localhost. Also install a gRPC client that can get snowflake ID from the GUIID service. Insomnia or grpc-client-cli would work.

Run one GUIID server as a Go program

make build-go
# assume the config file is ./example/config/localhost.json
./build/guiid -config_from_file=./example/config/localhost.json
# make gRPC call to get a new ID
echo '{}' | grpc-client-cli --proto ./api/v1/snowflake.proto --service=SnowflakeID --method=nextID localhost:7669
echo '2' | grpc-client-cli --proto ./api/v1/snowflake.proto --service=SnowflakeID --method=nextMultipleIDs localhost:7669

Run one GUIID server as a Docker container

The latest build image is published on DockerHub pengubco/guiid.

make build-docker
# assume the config file is ./example/config/docker.json
docker run --rm --name guiid --mount type=bind,source="$(pwd)"/example/config,target=/app-config -p 7669:7669 guiid -config_from_file=/app-config/docker.json
# make gRPC call to get ID
echo '{}' | grpc-client-cli --proto ./api/v1/snowflake.proto --service=SnowflakeID --method=nextID localhost:7669
echo '2' | grpc-client-cli --proto ./api/v1/snowflake.proto --service=SnowflakeID --method=nextMultipleIDs localhost:7669

Run a cluster of 5 GUIID servers in Kubernetes.

The setup is for demo purpose.

# Step 1. create a namespace to demo
kubectl create namespace guiid
# Step 2. create a pod running etcd and expose it through NodePort Service.
kubectl create -f ./example/pod-etcd.yaml 
# Step 3. setup user, role in etcd. Execute commands in Appendix 1 inside the etcd pod. 
# Step 4. create a configmap to store the json config
kubectl create -f ./example/configmap.yaml 
# Step 5. deploy the guiid service
kubectl create -f ./example/deployment.yaml 

# make gRPC call to get snowflake ID. 
echo '{}' | grpc-client-cli --proto ./api/v1/snowflake.proto --service=SnowflakeID --method=nextID [hostname-or-ip-of-node]:30001
echo '2' | grpc-client-cli --proto ./api/v1/snowflake.proto --service=SnowflakeID --method=nextMultipleIDs [hostname-or-ip-of-node]:30001

Contribute

Thank you for considering contributing to GUIID! We welcome contributions of all kinds, whether you're an experienced developer or just starting out. Feel free to ask questions, report issues and create pull requests.

Appendix

1. Prepare a local etcd server.

Step 1. Install the etcd 3.5 and start it. Step 2. Setup user, role and permission. Learn more about etcd RBAC.

etcdctl --user root:root user add guiid --new-user-password="passwd"
etcdctl --user root:root role add guiid
etcdctl --user root:root role grant-permission --prefix=true guiid readwrite /guiid/ 
etcdctl --user root:root user grant guiid guiid
# verify read and write
etcdctl --user guiid:passwd put /guiid/k1 v1
etcdctl --user guiid:passwd get /guiid/k1 
etcdctl --user guiid:passwd del /guiid/k1 

guiid's Issues

last_reported_timestamp was reset to 0 when 1). a new worker join the cluster and 2). the worker has not received request before report timestamp

Across time and across workers, the timestamp associated with a worker ID should always be monotonic increasing. Resetting it back to 0 is wrong.

Add design.md

The readme does not have details of how the ID service work. Include this info in a a separate design.md.

Support return multiple IDs in one call

There are cases where the application would like to get X, e.g. 100, 200, at a time. The Snowflak-ID supports 4K IDs in one milliseconds, so this is an easy and should not incur noticeable longer blocking compared to request of a single ID.

Add linter

efficient iteration of worker IDs

Right now, when a worker starts, it iterates through all possible worker IDs, starting from 0, all the way up to the max worker.
If the only available worker id is 1023. Iterating from 0 to 1022 takes time and are unnecessary.

Fix: list used worker IDs from etcd. Then only iterate through worker IDs that are not returned from etcd.

add gRPC client module

Create a client module for customers that talk with the snowflake-id service.

Add tests for worker