dev-mastery / comments-api

DevMastery Comments Microservice API

To manage comments on various Dev Mastery properties.

Using Clean Architecture for Microservice APIs in Node.js with MongoDB and Express

In this video we talk about Bob Martin's Clean Architecture model and I will show you how we can apply it to a Microservice built in node.js with MongoDB and Express JS - @arcdev1

Features

• XSS Protection (via sanitize-html)
• Flags Spam (via Akismet)
• Flags rude or inappropriate language (English only via Content Moderator)
• Flags personally identifiable information (English only via Content Moderator)

Running Locally

Prerequisites

• Git
• Node JS
• Mongo DB (To use the Mongo DB interface as shown in the Mastery Monday youtube video, you need to install Mongo Compass)
• Azure Content Moderator account (free)
• Akismet Developer account (free)

1. Clone the repo and install dependencies

git clone 
cd comments-api
npm i

2. Modify the .env file

Save sampledotenv as .env and then add your database and Content Moderator + Akismet API details.

3. Startup your MongoDB

Usually this is just: mongod on the command line.

4. Start the server

To run in production mode where code is transpiled by Babel into a dist folder and run directly in node:

npm start

To run in development mode where code is run by babel-node via nodemon and re-transpiled any time there is a change:

npm run dev

comments-api's Issues

Implementing logging

Hi, how do we go about implementing logging? Do we just import a logger function anywhere in our use cases or pass it down from the frameworks & drivers layer?

Is Id considered to be in the framework layer?

I see that the factory builders for entities like comment accept Id as an argument. Is the implementation of Id in the Id folder considered part of the "framework" layer in clean architecture because it's relying on an external dependency (cuid) directly?

How should I be thinking about this?

race condition

Next code from use case add-comment has race condition error in design.

If two the same requests will be received at the same time,
then if (exists) will be false for each of them
and two comments with same hash will be inserted

...
    const exists = await commentsDb.findByHash({ hash: comment.getHash() })
    if (exists) {
      return exists
    }

    const moderated = await handleModeration({ comment })
    const commentSource = moderated.getSource()
    return commentsDb.insert({
...

This example app has no business rules.
Crud methods can be mapped to controller without need of use cases.
If I need to add real business logic this error will be critical e.g:

• user should not be able to post more than 1 comment per hour

How can I avoid this error in clean architecture?

Nested lists within a comment

Hi, first of all thank you very much fore sharing the code and producing such a great video. This is very valuable and I can imaging it helps a lot of developers including me.

I try to think a step further and it seams like I can not get my head around the following use-case. Basically how would I handle a list which will be accessed through the root. Bear with me while I try to come up with an example that might be related to comments-api to explain it a little bit better.

How would I handle something like tagging a comment for example with cheers, likes, applause or emojis. Or tag it with labels as question, helpful etc.
Basically when I create a comment it contains the comment text like before. But I might have an empty list of tagables.
I can then assign a certain tag (whatever it might be) to a comment and also count them up if there are multiple of the same.
Would there be something like makeTagList and makeTag? Or would it be enough to just have the makeTag function injected into makeComment? Or how would I handle the nesting.
I hope may example makes sense and kind of transports what I am trying to ask.

It would be great, if you find the time to look at this and share your thoughts.
Looking forward to it.

Error handling in "Repositories"

I know you don't have a formal Repository file to abstract away your data access concerns, but how do you handle errors with MongoDB in your add-comment Use Case and Repo?

Errors having to do with validation or duplicate emails, for example, are, to me, a data layer concern (at least, the specific error message from the DB Driver should be transformed to something more useful to the domain layer).

Thanks.

VS Code debugging

Hi Bill

How can I debug/step through the code in VS Code?

Regards

Preload Connection and Data

I received the following question from a Dev Mastery newsletter subscriber who graciously allowed me to share it here:

Got some questions which might be related to it that trouble me and I can find a way through them..

1. Lets say that there is a precondition to your application that it must have valid mongoDB connection before it can listen on port. ( start up).
2. What happen if we want to preload data(from ext source) that our application must work with before we can start listening to requests. where do you do all that ?

All ur use cases looked like requirements that are business logic, but this is more of an architecture design.

who is responsible to satisfy these conditions, and how can we do it in a clean way

Why you use IIFE in db access file ?

Missing DM_ENV

Hi, the sampledotenv doesn't contain a DM_ENV,

Before the build the index.js code looks like this:
if (process.env.DM_ENV === 'dev') {

After build, the code looks like this:
if ((process && process.env && process.env.DM_ENV || "dev") === 'dev') {
which I think should equate to "dev" === 'dev' but somehow doesn't.

so index.js never starts listening:
app.listen(3000, () => {

until I add DM_ENV=dev to my .env file.

Which is kind of fundamental to you all running this code so are you all just adding DM_ENV and then not mentioning it because it's so obvious, or is something else going on that I just cannot see?

can one use case be dependent on another usecase.

Thank you @arcdev1 for the great article and repo on clean architecture.
I have a few questions related to use case creation in clean architecture.

I am working on an application which moves mongodb records to elasticsearch and I am trying to develop it using clean architecture approach.
At high level, I could think of only one use case for this application which is :- moveDatafromDbToEs.

But it will involve several steps like :-
1)Fetch data from db in a batch
2)prepare/Format the records to be pushed to elasticsearch
3)Bulk Push data to elasticsearch
4)Filter failed and successfully pushed records.
5)Create a final response with how many records were pushed successfully and how many records failed.

Usecase creation approach:-
Method 1:- Create one big use case with all the above steps as small functions within it.
Method 2:- Break the big use case into small use case For eg:- a use case that handles elasticsearch push related operations 1) Format data, 2) push data to elasticsearch, 3) filter which records successfully pushed and which failed ) and inject this usecase as a dependency in main use case (moveDatafromDbToEs). This approach is similar to handlemoderation usecase of your application.

Question 1 :- If I follow the method 1 Won’t it make the use case look very big and break srp.
Question 2:- If I go with method 2, is it ok to have one usecase dependent on other usecases
Question 3 (in general ):- Is it ok to have a usecase having more than 3-4 dependencies.

Awaiting your response. Thanks

What is your opinion of Awilix for DI?

Hi Bill @arcdev1,

I just discovered Awilix.

Was interested in your opinion being that dependency injection is a big part of your comments-api example and video, and I respect your opinion.

I like the way is wires up the arguments for you.

Regards,

Grant.

Wouldn't it be ok to directly import well-known functions, such as pipe, in our code instead of injecting it?

About the is-questionable.js file, I wonder if you are not really taking dependency injection to far.

While I agree that we should only have dependencies from the outer circles to the inner ones and not the other way around, I believe that this rule could be relaxed for well known functions that probably won't change or break unexpectedly.

For example, pipe is a well-known function composition utility. While it has several implementations (and you also rolled your own), their behavior is the same (or almost). querystring is a native Node.js module, so you probably wouldn't want to roll out your own API to handle query strings.

Would it be so bad if we had something like:

import pipe from '@dev-master/pipe';
import querystring from 'querystring';

export default function makeIsQuestionable ({
  issueHttpRequest,
}) {
  // ....
}

Also, in the index.js file in the same folder, you are just assigning issueHttpRequest to axios, querystring to Node's native querystring module, etc., which means your is-questionable input ports are actually the APIs from axios and querystring, so the dependency direction here is actually the opposite it should be.

axios might be a case apart because you can say:

Oh, I do this because of testing. It's easier to not make actual HTTP request doing so.

Well, you could actually achieve that using something like axios-mock-adapter.

And if you ever drop axios in favor of request, you'd be required to write an adapter from the request API to the axios API.

Furthermore, when talking about querystring (which is pretty much a module holding pure functions) and pipe (a pure function), I don't see any actual gain in injecting it the way you're doing here.

So, in the end, simply writing:

import pipe from '@dev-master/pipe';
import querystring from 'querystring';
import request from 'axios`;

export default async function isQuestionable //...

Would not be less compliant to the Clean Architecture definition as your current version already is.

I understand this repository was created for didactic reasons and tries to follow the architecture definition as far as JavaScript allows it, but this is a really cool discussion I've been willing to have for a while and couldn't find someone to do that, so forgive my daring 😅.

sampledotenv needs "DM_API_ROOT"

Hi, thank you for the great work + tutorial.

I think that DM_API_ROOT=/api should be added to the sampledotenv file as it would otherwise not work currently.

Design patterns

Is is possible to use your structure As a code for a full project or is this code style only good for micro services and if I am to add design patterns for a more complex project based on this architecture is this a good or bad idea ?

Dependency injection

Hello @arcdev1 and thank you so much for the sample code and the amazing videos.
However I would like to discuss about the dependency injection topic.

As you can see from a sample project I made I usually enforce the dependency injection in order to have a great flexibility in testing as follows:

Connect to the database => pass the connection to the repository => pass the repository to the service => pass the service to the controller => pass di controller to the router => pass the router to the server

1. I would like to hear your thoughts about it

I'm also trying to figure it out if:

2. the clean pattern used in this project (comments-api) gives me the same testing flexibility that I explained above

3. how would you change my project in order to respect the clean pattern

4. I think that in order to better understand the pattern and the testing and future changes flexibility it would be great to add another entity which interacts with the main one and see in action integration testing with mocks as well

5. Does this mapping make sense(?):

• my repository : your data-access
• my services : your use-cases

6. Your post-comment controller is using the add-comment use-case.
   The add-comment use-case needs the db and is using the comment entity.
   You can unit test from the inside (entity) to the outside (use-case and so on) but (e.g) if I want to run a test from the very outside changing the implementation of one of the libs used by the entity my feeling is that I wont be able to do that

7. Does it makes sense as soon as I've more entities to have a central (e.g where I create the server) point where I:

• create the db connection
• instantiate the two entities data-access passing to their builders the db connection
• pass the above to the use-cases builder
• pass the use-cases to the controllers
• pass the controllers to the routes
• create the server and start it

Sorry for the long post, I hope that all my questions and examples let you understand my difficulties in approaching the clean architecture.
Maybe I'm also intending testing in the wrong way so I'm happy to improve myself and learn how to properly do it :)

How to support api error codes and subcodes

Hi, first of all thanks a lot for your video explaining clean architecture and the example service. It really was of great help. I have a question on my mind, how to support api error responses with attached error codes and subcodes without allowing the usecase to know which http status and error codes | subcodes returned through the api. The first option that came to my mind is to create custom errors for the usecases to throw and in the controller layer I would map those errors to http status codes and service error codes. The second option is to store the mapping within the custom error.

If the first option is viable should the mapping be based on error messages or error types.
Thank you.

Why dependency injection here?

comments-api/src/controllers/get-comments.js

Line 1 in e55a2a9

export default function makeGetComments ({ listComments }) {

Hey Bill, thanks for the awesome video! Just a question...

Why are you injecting an use-case in a controller, which is at a lower level compared to an use-case? Wouldn't be a source-code dependency in this case more "right"?

Shouldn't data-access methods return entities ?

comments-api/src/data-access/comments-db.js

Line 18 in e55a2a9

return (await result.toArray()).map(({ _id: id, ...found }) => ({

I was wondering if data-access methods should be returning entities. In this case, we could call makeComment on each result to have a list of entities rather than a list a plain javascript objects.

In javascript we can't see the difference, but in a typed language (like Typescript) we would.

Question about the controllers

Why do you inject the add-comment in the controller post-comment. Should inner layers not be imported into outer layers usually? I can not wrap my head around this. On all other layers you imported the inner ones and injected the outer layers.

How should I include Transactions, without breaking this source code architecture?

Hello,
First of all thanks for a wonderful tutorial!!

I am working on an Express-based API that uses MySQL for data persistence. I have to implement a confirm order functionality

• I have an already recorded order with ordered item list in my DB (like order in the pending state)
• Now on confirming order action I need to subtract each item quantity from each some super-item present in the DB record, with a check that no value turns negative (i.e. no ordered qty is more than present qty)
• If it is done for all the items under a "transaction cover" then I have to commit the transaction
• Else revert back to the previous state (i.e rollback)

I know how to run Mysql specific transactions and all the things, but with a lot of coupling and poor source code architecture. (If I do it that way, then DB won't be like plugin anymore)

I am not able to understand how to do this while maintaining the architecture and at what layer to implement this transaction thing, use-case/data-access, or what?
Thanks in advance

PS: also suggest to me, if it is an automation logic or business logic?

VSCode IntelliSense

Hi @arcdev1,

This is not an issue with your code, only the tools, so feel free to close this issue.

After watching your video on Clean Architecture related to this repo I decided to try the pattern out.
There is, in my opinion, a large disadvantage to the dependency injection.

Visual Studio Code, the text editor I am using, looses IntelliSense on the module dependencies.

I tried changing the dependency injection from:

function ({ dep1, dep2, dep3 }) {}

to:

function (dep1, dep2, dep3) {}

hoping that it would improve the situation. It did not.

I'm using pure JavaScript, not TypeScript. I imagine I could fix the IntelliSense by adding jsDocs with type details to the functions however that seems rather redundant.

I know IntelliSense is not required however from a productivity point of view it will slow down development. For example If you don't remember the exact name of the function you are calling on a dependence you will need to look it up.

Suggestions welcome.

Thanks for the great insight this subject has given me.

Question regarding e2e tests

Hi, thank you for a great video and for sharing the code. I have a question regarding the end to end tests that jest runs:

Does the below line depend on the response from a call to a third party service? I am just trying to learn and understand the code.

comments-api/__test__/e2e.spec.js

Line 36 in 0d119f7

expect(doc.published).toBe(true)

What I mean is, I can see makeFakeComment sets published to true on the comment, but it comes back as false in the response. I think I have answered my own question: I do not have Askimet or Content Moderator set up correctly?

Thanks again.

Implment Abstracts for model and collection

Hi Bill,
First, thanks a lot for sharing this code and the YouTube explanation, (we are still waiting for more : ).
It changes the way i think about architecture.

• 1: How would you implement Abstract model and abstract collection ?
  Say you want to add cache to all your db queries of collection,

• 2: Say you have around 300 entities in your app (real case), all have 1 or more fields with same logic
  (e.g strings field which need to be parsed, int field which you want to validate, date fields which you want to relate them to specific timezone)
  How would you make code which can be adapt to all of them
  would that be a DB concern or model concern ? or would you import at the top or extend with prototype the each model?

• 3: Did you ever use this architecture at this scale ?

Concurrent database writes

Hey, first of all thanks a lot for sharing this code & your episode, it really had a huge impact on arch. style in my projects. But I have ran into an issue that I see is also present in your repository, and that is: let's say you have 2 concurrent updates on a single comment entity - that's not a problem if you have only text fields, but it might become one when you have i.e. an array inside a document - one operation will overwrite the other one (if there were 2 pushes, only one will be performed even though both should be merged somehow) - my question is, how do I deal with that, and most importantly - should I even care?

Works out of the box?

Hello Bill,

I just wanted to try and mess around with the app similar to how you did it in mastery Monday by doing a POST with a body of {
"author":"Wayne",
"text":"Party time",
"postId":"cjt65art5350vy000hm1rp3s9"
}

However I only get 404 errors. Do I need to do something additional to get the routes working?

Validation in entity

Hi Bill!
Thanks a lot for your code - it was really useful for me.
I have the next question - how correct is it to use validation in entities? Is it still within business rules?
Maybe we should use validation in other layers of the application?

Wrong import when injecting is-questionable in handleModeration

comments-api/src/use-cases/index.js

Line 7 in 7c93f3d

import isQuestionable from '../is-questionable/is-questionable'

Shouldn't the import be

import isQuestionable from '../is-questionable'

?

Thank you for your awesome videos and examples btw !

authentication and authorization

Thanks for your excellent example of applying Clean Architecture to a Node/JavaScript app. I have been working on refactoring an existing app to Clean Architecture and I have arrived at a solution quite similar to yours. As an aside, I was surprised at how the code trends to a very functional programming feel. Anyway, I have hit a bit of a wall trying to integrate authentication and authorization into Clean Architecture.

Suppose, as in many real world cases, your app has administrative use cases that are restricted to users with admin privileges. Also suppose, as Uncle Bob describes repeatedly, that your app has a web API that a web app GUI consume, as well as console API (which I might actually implement) that might authenticate based on the OS user or maybe even get an authentication token and store it in the user's home directory. Where would you place the logic that authenticates the user invoking the use case? When using Express to build a web API, I'm sure you're aware the common solution is to use middleware that extracts an authentication token from HTTP headers and uses the token to look up the associated session and user in some session store and user database. With respect to this, here some questions that have caused me to stumble.

1. Would you implement the HTTP header token in the adapters layer?
2. How would you implement that to account for both web and console interfaces?
3. Where would you place the logic to check for administrative privileges as well as other authorization checks on a normal requesting user's access? I started by putting these checks in the use case by declaring a specific interface for an authorization check, e.g.,

   interface CommentsPrivilegeService {
     currentUserCanEdit(comment: Comment): Promise<void>
     currentUserCanDelete(comment: Comment): Promise<void>
   }

and injecting an implementation of that service into the use case. That's where I ran into trouble, though, because during an HTTP request that, there is no way to track the user from the request headers through to the use case without actually passing the user or the user ID explicitly to the use case, then passing the user to the CommentsPrivilegeService methods.

So, I see three options.

1. I can leave the privilege checking in the use case by injecting a privilege checking interface as above, but modify the use case and the privilege checking service methods to accept a user or user ID parameter.
2. I can create an instance of a use case for every incoming request instead of only one at startup which will take a fresh instance of the privilege checking service that is bound to the requesting user, something like

   async handleEditComment(httpRequest: CommentsAppHttpRequest): CommentsAppHttpResponse  {
     // this was injected into the controller
     const userId: UserId = userIdFromSessionTokenInHttpRequest(httpRequest)
     const privileges: CommentsPrivilegeService = makePrivilegeServiceForCurrentUser(userId)
     const editComment = makeEditComment(privileges)
     const editCommentArgs = argsFromRequest(httpRequest)
     const result = await editComment(editCommentArgs)
     return CommentsAppHttpResponse.from(result)
   }

3. Perform the privilege check in the controller instead of in the use case. This means I would have to test that controllers for all the different interfaces (web, console) make the proper privilege checks, instead of testing that only at the use case level.

I am sure there are best practices for authentication and authorization in clean architecture and domain driven design, but I am having trouble coming up with anything concrete in the context of Node/JavaScript researching on the web. I have seen examples in Java and .Net that use AOP frameworks to authenticate and authorize outside the use cases, while the use cases assume the calling user is authorized, but I like the idea of writing the use cases to explicitly call out the authorizations they require.

I am very interested to know your thoughts.

.env variable undefined at the time of running data-access module

Hi @arcdev1, first of all thank you so much for this repo and the lesson about clean architecture on Youtube. I learnt a lot from it and now I'm using both as a guideline for my first own node app.

In data-access/index.js there's something I had to do slightly different because it wasn't working the way you did it, which is here:

In my case, the variable declared in .env (URI in my case) is undefined because (as I found out) the whole data-access module gets executed even before src/index.js does, which contains dotenv.config(),. I solved this by moving URI and the instantiation of MongoClient inside the makeDb function, which gets executed only once the server is started.

I wonder what happened in my case would be expected and if there is any drawback with my solution?

Cheers.

Classes and encapsulation

So you are using factories to implement dependency injection, taking advantage of closures. Is this a known pattern, does it have a name? I can only assume that is used in FP style languages? Is it used in Java?

Trying to use it with an inner class instead of inner function.

function makeComment({Id, md5, sanitize, makeSource}) {
  return class Comment {
    constructor(
      author,
      createdOn = Date.now(),
      id = Id.makeId(),
      source,
      modifiedOn = Date.now(),
      postId,
      published = false,
      replyToId,
      text
    ) {
      ....
    }
  }
}

The fact that it is using dependencies from outer scope makes me feel like breaking encapsulation (but we know that a javascript class is a function constructor). Maybe because of the habit of passing all needed dependencies to the constructor.

How to define a repository interface

Hello there Bill.

First I would like to thank you for this awesome video and repo you came with, it opened my mind in ways I can't even express and I'm learning a lot with it. Now to the question.

I'm building a more complex app using a very similar approach but using typescript.
Simplifying the whole thing, I have an entity User which declares a dependency on a UserRepository that will be used to check if the given email is unique across the database on creation (which is an enterprise business rule, therefore needs to be here). This UserRepository interface has a findByEmail method that returns a simple user data structure (name, email and password).

So far so good, but I have one use case ShowUser which will be used to display the user information with some extra data. This use case should use the same UserRepository (defined in the entity), but the return value of findByEmail now should return a slightly different user data structure, because now I want it to include the user boards, which you can think of as a join of 2 tables.

Now comes the whole problem, with a lot of solutions which seem bad to me.

Solution 1: Easily fix the problem by going to the User entity and altering the user data structure to include a boards field. But now I modified the data structure with an application business rule in a place that should only be concerned with enterprise business rules.

Solution 2: Create another method in the interface called maybe findByEmailWithBoards, which sounds horrible just by the name (what if I need more fields?) and also has the same problem as solution 1.

Solution 3: Make the repository simpler, returning a simple user data structure and creating a new BoardsRepository to create a "manual join" of both data structures. This sounds a bit better but I completely lose the speed and convenience of joining multiple tables (or even having embedded documents in mongodb) that databases offer, leading me to have to make multiple queries (which can grow a lot) instead of 1, which will probably have a huge impact on performance in certain cases.

Solution 4: Having one UserRepository interface in the User entity and another for the use case, which then the implementation would implement both, satisfying both needs. But that sounds very weird for me, especially on how would I name them to differentiate between them.

There are probably more solutions that I came about but all of them seem to have a big drawback like these, so I'm not sure if this is gonna have to be a trade off or maybe I'm missing something.

This approach is a bit different than yours since you don't use the repository/database inside your entities, but there's definitely gonna be a point where you'll have to if you want to have more complex business rules.

So what to do when your application business rules conflict with your enterprise ones like here?

Thanks again and sorry for the long question! I hope you have the time to discuss this because I'm kinda out of ideas for now and I'm really hopping to get to something solid.

How should the add-comment use-case be if using classes in js?

class AddUser{ constructor(db){ this.db = db; } async addUser(){ // logic } }
like this?

Point to clarify, request for guidance.

First, I'd like to give you credit, this repo and the related video are amazing and have helped me grasp a lot of Clean Architecture concepts I've been having issues with. One thing I'm still having an issue with is making the switch from my reliance on using Mongoose models as the Business Entities to placing Mongoose on the outter circle.

I've seen your comments on the youtube video, suggesting we use Mongoose as just another framework to inject into the Comments-db. But, would that free us from writing the code that's in comments-db.js? That file seems like it covers a lot of what Mongoose does (e.g. findbyid.. findall..). Which, for my applications that tend to not be monolithic MEAN stack apps, it seems a bit excessive to do for each Entity.

Could you suggest whether you think that, for a monolithic API, could we save on boilerplate code and leverage Mongoose or should we just give up on mongoose and build a robust '-db.js' dbo file for each entity?

thanks!

Typo in src/index.js: app.put instead of app.patch

The title says it all.

comments-api/src/index.js

Line 26 in 2b79772

app.patch(`${apiRoot}/comments/:id`, makeCallback(patchComment))

comments-api/src/index.js

Line 27 in 2b79772

app.patch(`${apiRoot}/comments`, makeCallback(patchComment))

What's the point

comments-api/src/data-access/comments-db.js

Line 17 in a4e7189

const result = await db.collection('comments').find(query)

Hi man, first of all: thank you for your contribution for the community !

I was looking at this piece of code, and I don't understand why you import the DB and inside this code you use logic technology-specific... what's the point ?

I mean... If I need to change the data base I need to change makeDb origin, and ALSO this technology specific logic.

Am I right?

Thanks !

Build and Make functions

Hi Bill, I am confused about the build and make methods and was wondering if you could provide some clarity on them.

When looking at comment.js it seems to me that buildMakeComment() is used to inject dependencies and makeComment() is used to construct the comment entity. However in other files you use the word make as a prefix for functions that take in dependencies. So I don't think that is their purpose.

I am really interested in Clean Architecture and your video was amazing, your response will help get me closer to implementing something similar in my applications.

Many thanks in advance,

Andy

typescript version

It would be nice if you can make typescript version of this, thank you!

Content Moderator Timeout Issue

Hello @arcdev1 and thank you so much for the sample code and the amazing videos.

I am facing signup issue while creating this reference url "https://contentmoderator.cognitive.microsoft.com" which you given.

I got this below error always, even I have good internet connection.

Can you suggest me, how to resolve on this same. Thanks in advance sir

Classes vs. factory functions?

Hi,

Firstly, thanks for the tutorial; I’ve spent the weekend refactoring my node js monolith and it is so much cleaner, and unit testing has never been so easy!

To my question: what is the difference between using classes vs factory functions in this architecture? Is there any difference or is it just semantics? Performance implications? I’m using TypeScript, do you think classes are a better fit in this case?