maestre3d / alexandria

The Alexandria Project is an open-source platform where people can share their knowledge through books, podcasts, docs and videos.

A. Ruiz
Software Engineer
Site-Reliability Enginner

A. Ruiz is a software engineer and site-reliability engineer with over 4 years of professional experience building cloud-native systems using programming languages such as Go, Java, Javascript and C#.

Actively working at Aplazo (Full-time / Staff Sofware Engineer) and Neutrino Corporation (Founder / Part-time / Lead Engineer).

Technology Stack

See A. Ruiz's full technology stack here.

Contact

• Twitter: @aruizea
• Email: [email protected]
• Neutrino Corporation Twitter: @NeutrinoCorp

alexandria's Issues

Connection refused by zipkins service and Kafka client has run out of available brokers

Hi!
Let me congratulate you for the work done.

In order to understand all the technologies I just ran docker compose after git cloning the repo.

Found that most of the services are not running due to zipkins. Can you please help me on this.

Zipkins logs say it is running out of memory.

OS: macOS Big Sur
RAM:16GB

Logs of author service:

2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:02 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused

Log of blob service:

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
2021/02/18 18:22:45 starting event service
2021/02/18 18:22:45 starting http service
2021/02/18 18:23:30 stopping services
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
2021/02/19 11:26:17 starting http service
2021/02/19 11:26:17 starting event service
2021/02/19 11:38:56 failed to send the request: Post "http://zipkin:9411/api/v2/spans": EOF
2021/02/19 11:38:56 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp: lookup zipkin on 127.0.0.11:53: no such host
2021/02/19 11:38:57 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:38:58 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:38:59 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:00 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:36 failed to send the request: Post "http://zipkin:9411/api/v2/spans": EOF
2021/02/19 11:51:37 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:37 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:38 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:40 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:41 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:42 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:43 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused

Log of Identity service:

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/identity-service/cmd/alexandria-server/main.go:22 +0x4cd
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/identity-service/cmd/alexandria-server/main.go:22 +0x4cd
2021/02/19 11:26:17 starting event service
2021/02/19 11:38:56 failed to send the request: Post "http://zipkin:9411/api/v2/spans": EOF
2021/02/19 11:38:57 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:38:57 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused

log of media service:

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/media-service/cmd/alexandria-server/main.go:24 +0xa67
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/media-service/cmd/alexandria-server/main.go:24 +0xa67
2021/02/19 11:26:10 starting http service
2021/02/19 11:26:10 starting event service
2021/02/19 11:26:10 starting grpc service
2021/02/19 11:38:56 failed to send the request: Post "http://zipkin:9411/api/v2/spans": read tcp 172.18.0.3:48376->172.18.0.5:9411: read: connection reset by peer
2021/02/19 11:38:56 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused

log of Kafka:

[2021-02-19 11:26:17,403] INFO [GroupCoordinator 1001]: Dynamic Member with unknown member id joins group BLOB in Stable state. Created a new member id sarama-0393cc0d-256f-4ffd-b223-f6a8f5c2e2c4 for this member and add to the group. (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:17,404] INFO [GroupCoordinator 1001]: Preparing to rebalance group BLOB in state PreparingRebalance with old generation 4 (__consumer_offsets-7) (reason: Adding new member sarama-0393cc0d-256f-4ffd-b223-f6a8f5c2e2c4 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,219] INFO [GroupCoordinator 1001]: Member sarama-822e38f0-a93f-4b75-8c26-691e7b6f36eb in group BLOB has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,228] INFO [GroupCoordinator 1001]: Stabilized group BLOB generation 5 (__consumer_offsets-7) (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,239] INFO [GroupCoordinator 1001]: Member sarama-843fec96-1f1c-4cfa-82b4-c1bb3741cf5b in group MEDIA has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,240] INFO [GroupCoordinator 1001]: Assignment received from leader for group BLOB for generation 5 (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,240] INFO [GroupCoordinator 1001]: Member sarama-be35b581-6c0c-4035-b966-9722e590d926 in group MEDIA has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,241] INFO [GroupCoordinator 1001]: Member sarama-86bfb6b4-30b7-4ee9-8700-03cd6e8e761d in group MEDIA has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,242] INFO [GroupCoordinator 1001]: Member sarama-69889ea2-f1e6-4df6-a4c0-7223b4aaae28 in group MEDIA has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,243] INFO [GroupCoordinator 1001]: Member sarama-ec49ff0b-f652-4bd2-88c9-ef04fd2a7142 in group MEDIA has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,245] INFO [GroupCoordinator 1001]: Member sarama-7ec8ad41-81c2-488a-908b-4d1960e4461a in group MEDIA has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,247] INFO [GroupCoordinator 1001]: Stabilized group MEDIA generation 4 (__consumer_offsets-22) (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,256] INFO [GroupCoordinator 1001]: Assignment received from leader for group MEDIA for generation 4 (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,257] INFO [GroupCoordinator 1001]: Member sarama-e7612267-93fd-45f3-a6d6-61566fca8d9b in group AUTHOR has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,259] INFO [GroupCoordinator 1001]: Member sarama-a2e0a140-5409-433b-8d20-5c23af0ac6e9 in group AUTHOR has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,261] INFO [GroupCoordinator 1001]: Member sarama-d6a7d600-5dd1-483e-b97b-ca20b7b8521b in group AUTHOR has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,265] INFO [GroupCoordinator 1001]: Member sarama-0fa018de-82b3-4662-9f9c-97f0f09aa8fa in group AUTHOR has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,271] INFO [GroupCoordinator 1001]: Member sarama-f6735991-37a1-411e-82ea-f34cf67cc129 in group AUTHOR has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,275] INFO [GroupCoordinator 1001]: Stabilized group AUTHOR generation 3 (__consumer_offsets-17) (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,284] INFO [GroupCoordinator 1001]: Assignment received from leader for group AUTHOR for generation 3 (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,286] WARN [GroupCoordinator 1001]: Setting empty assignments for members Set(sarama-c656e792-e201-4965-8382-e38e8bcd9668) of AUTHOR for generation 3 (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,287] INFO [GroupCoordinator 1001]: Member sarama-4fc614db-133a-4f46-8e76-305cb8f443b6 in group IDENTITY has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,288] INFO [GroupCoordinator 1001]: Member sarama-bb1f58a2-9cc5-400c-a1e9-877a801c2fdc in group IDENTITY has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,288] INFO [GroupCoordinator 1001]: Member sarama-aa250f0d-0b6c-49eb-b9b8-567b55f0494b in group IDENTITY has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,288] INFO [GroupCoordinator 1001]: Stabilized group IDENTITY generation 4 (__consumer_offsets-36) (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,331] WARN [GroupCoordinator 1001]: Sending empty assignment to member sarama-c656e792-e201-4965-8382-e38e8bcd9668 of AUTHOR for generation 3 with no errors (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,333] INFO [GroupCoordinator 1001]: Assignment received from leader for group IDENTITY for generation 4 (kafka.coordinator.group.GroupCoordinator)

log of zipkin:

:: version 2.23.2 :: commit 7bf3aab ::

2021-02-19 11:39:02.035  INFO [/] 1 --- [oss-http-*:9411] c.l.a.s.Server                           : Serving HTTP at /0.0.0.0:9411 - http://127.0.0.1:9411/
Terminating due to java.lang.OutOfMemoryError: Java heap space

Change "backup" field name to

Describe the bug
Currently, the transaction body contains a field called "backup". That is a incorrect naming convention.

Expected behavior
Rename transaction's body/payload's "backup" field to "snapshot" to keep event-sourcing naming conventions.

Change default zap logging for gokit's logging

We're currently using our own logging abstraction, nothing bad with it but we shall be using gokit's since it's implementation is already open to any kind of logger, from fmt to zap logger or logrus.

Cron job for entities with status pending

Is your feature request related to a problem? Please describe.
Whenever a new entity is created, in most of the cases it will start a transaction.
The problem here is sometimes there will be leftovers if some problem occurred.

Describe the solution you'd like
It would be better if we implemented a cron job inside every service in a goroutine/thread that will run every X time (from min to days), where we could remove (hard-remove) every entity with the "STATUS_PENDING" state.

Describe alternatives you've considered
Any.

Additional context
No

Apache Solr for distributed indexing

Is your feature request related to a problem? Please describe.
Yes, right now distributed data indexation is impossible and we still need to implement complex query building patterns in order to satisfy our indexation standards.

Describe the solution you'd like
Currently, we use Redis as cache and PostgreSQL as main RDBMS, yet for full-text searching we need something more powerful and easy to implement. Using Apache Solr as distributed indexing database for the job is one of the best choices we have, even more if we are already using Apache Zookeeper (required by Solr) and Apache Kafka.

Describe alternatives you've considered
Since we might use EKS (e.g. Elasticsearch) for other inquiries, we should consider using the ElasticStack.

Additional context
No

No information while transaction is being executed at client-side

Describe the bug
Whenever we start a distributed transaction (SAGAs mostly), the client is not able to see what is the current state in real-time and even worst, if something fails, there is no way to know what happened.

To Reproduce
Steps to reproduce the behavior:

1. Make a POST HTTP request or a gRPC action call to any endpoint with dependencies (almost all).
2. Wait for response, you shall see the created entity with state set to PENDING.
3. You need to execute a short-polling retry policy if you want "real-time" state. Aside of that technique, there's no actual way to know the entity's state explicitly (if failed, then be able to see what happened)

Expected behavior
We must have a way to see entity's state in real-time while a distributed transaction is being executed along with following error information if a failing scenario happened.

Additional context
No.

Use ElasticStack for distributed logging and monitoring

We shall consider using the Elastic stack as the following scenario shows:

• Beats: Data crawling and gathering
• Logstash: Data input
• Elasticsearch: Data indexing and querying
• Kibana: Data visualization

There are docker compose configurations and k8's helm charts available to be used anytime.

gRPC for Author service

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
We shall be using RPC over plain HTTP/1.1 protocol, it's using HTTP/2 and endpoints are very handy.

Describe alternatives you've considered
None

Additional context
No

Distributed tracing and metrics with Kafka

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
We already have distributed tracing on the sync transport layer, yet we lack of this important feature at the async transport layer.
I'd like an implementation with either OpenCensus or OpenTracing, also metrics could be using either Prometheus or OpenMetrics.

Describe alternatives you've considered
By now, we lack of async instrumentation.

Additional context
No

Change Media service architecture for Go kit's

We're currently using our own approach, but it lacks of tracing/metrics and also nice logging. In addition, it also depends on the transport layer too.

Blob API/Service

Is your feature request related to a problem? Please describe.
There's no way to upload profile and author pictures and media content by now.

Describe the solution you'd like
A new service which contains all the required tooling for file handling (from multipart to newer formats as webp or avif).
Communication could be done with either domain or integration (SAGA) events.

Describe alternatives you've considered
Add an endpoint or modify the update one in each service for multipart uploading.
Although this lacks of the SOLID principles and the need to modify the domain layer, it's still an option.

Additional context
Consider following same approaches from GCP or MS Azure Blob APIs.

Category API

Is your feature request related to a problem? Please describe.
There is no way to fetch important data like authors and media with their respective categories. Categorizing them would be the best way to achieve our current milestones and strategic objectives.

Describe the solution you'd like
Along with Apache Solr or Elasticsearch, we could develop a powerful category API so anyone can fetch their authors and media with a better experience.

In addition, the Category API could be used as a hashtag module like in popular social media. Thus, the following Trending API would be easier to implement.

Describe alternatives you've considered
None.

Additional context
No.

gRPC support for Media service

Add RPC support for media service

Add validator to domain entities

Is your feature request related to a problem? Please describe.
Validation can be exhausting and a complete mess.

Describe the solution you'd like
We could use a idiomatic go validator like HapiJS joi.

Describe alternatives you've considered
We could use Go Playground's validator or Asaskevich's govalidator.

Additional context
None.

OpenCensus and Prometheus integration

Add OpenCensus and Prometheus integration to visualize metrics

Change transaction's "backup" field name to "snapshot"

Describe the bug
Currently, the transaction body contains a field called "backup". That is a incorrect naming convention.

Expected behavior
Rename transaction's body/payload's "backup" field to "snapshot" to keep event-sourcing naming conventions.

API Gateway and more using Netflix Zuul

Is your feature request related to a problem? Please describe.
Currently, we don't have any API Gateway implementation for our services.
It is not a good strategy to not use an API Gateway pattern.

Describe the solution you'd like
Using an edge server such as Netflix Zuul to manage our services with load-balancing, security and reverse proxying.

Describe alternatives you've considered
Making our own implementation with raw http clients with k8s + istio.

Additional context
Zuul needs Eureka service discovery system by default but this can be disabled in it's config.

Add Create Media Saga transaction

Is your feature request related to a problem? Please describe.
No.

Describe the solution you'd like
Currently, we're using a non-transaction approach when creating a new media entity, yet it's a must to do a saga-like transaction to validate media's user ID and author ID.

Describe alternatives you've considered
A saga transaction using the message broker in turn with help of domain and integration events.

Additional context
Here's a saga's explanation

Add author service