just-hms / large-scale-multistructure-db

large scale 2022/23 project

large-scale-multi-structure-DB

Requirements

• docker
• docker compose plugin

deploy

# first time only command
make docker-contexts
# add ssh key to remotes
# rename .env.template to .env
# add secrets
make down
make deploy
# when done
make deploy-down

dev

docker

# rename .env.template to .env
# add secrets
make down
make dev

# get the doc in json and html :)
curl http://localhost:8080/api/swagger/doc.json
curl http://localhost:7000/api/swagger/index.html

locally

# rename .env.template to .env
# add secrets
make down
make infrastructure
go run ./...

test

docker

# rename .env.template to .env
# add secrets
make down
make test

locally

# rename .env.template to .env
# add secrets
make down
make infrastructure
go test ./...

benchmark

make infrastructure
# run the importer
cd be/internal/usecase/repo
go test -run=^$ -bench=.

google docs

• https://docs.google.com/document/d/18_51BtU6xor5OmC4hQTUU8mCdDXr_RVSLhZaOjpiBwo

presentations

• https://docs.google.com/presentation/d/1h7jxOkOvwBJSB7aGfDJR2bifR1MJf_eIA3p5B8Vb1QY/edit?usp=sharing

large-scale-multistructure-db's Issues

Review fixes

A couple of things need addressing:

• Add missing autogenerated Doc
• Fix votes applying to all reviews in a shop

Add description to barbershop

Add missing ShopName to Appointment from Importer

Software Architecture Preliminary Idea

Fix multiple Up/DownVotes by the same User in the Importer

Use-case diagram

Redis geocoding

add the geocoding informations inside redis

• do some benchmarks against mongo
• use the fastest inside the Find function (which will not returns all the barbershop data)

REST API

• complete the todo
• complete empty {} in the reqs

Testing locally

• create a .ENV file inside be
• get the db options from the .ENV file
• add a make an infrastructure-up and down
• call them inside testing

Add redundancies inside `slot`

• add shop.Employees to make HolidayUseacase.Set() function access only the slots

• read more into the AppointmentUseCase.Book()

  func (uc *AppointmentUseCase) Book(ctx context.Context, appointment *entity.Appointment) error {
  
      us, err := uc.userRepo.GetByID(ctx, appointment.UserID)
      if err != nil {
    	  return err
      }
  
      if us.CurrentAppointment != nil {
    	  return errors.New("cannot book two appointments")
      }
  
      shop, err := uc.shopRepo.GetByID(ctx, appointment.BarbershopID)
      if err != nil {
    	  return err
      }
  
      minus := 0
      slot, err := uc.cache.Get(ctx, appointment.BarbershopID, appointment.Start)
      if err == nil {
    	  minus = slot.BookedAppoIntments + slot.UnavailableEmployees
      }
  
      if shop.Employees-minus <= 0 {
    	  return errors.New("cannot book because this slot is full")
      }
  
      err = uc.cache.Book(ctx, appointment)
      if err != nil {
    	  return err
      }
  
      err = uc.appointmentRepo.Book(ctx, appointment)
      return err
  }

Data scraping

Deploy building frontend statically

Error processing query inside find

payload:

{
    "latitude": 43.857105,
    "longitude": 10.430389,
    "radius": 10
}

response error

{"error":"(NoQueryExecutionPlans) error processing query: ns=barber-deploy.barbershopsTree: GEONEAR  field=location maxdist=10 isNearSphere=0\nSort: {}\nProj: {}\n planner returned error :: caused by :: unable to find index for $geoNear query"}

Investigate null ShopName in ShopRanking aggregation

Refine Admin analytics

• Use Names instead of IDs whenever possible
• Add comments to relevant aggregations

Don't give a User an Appointment if it wouldn't be pending

Fix LonLat test

fix calendar time format

Frontend project structure

• load default folder and shit
• README (npm run dev and shit)

I dont understand how to handle an api_key uncle pear

semmai domani/stasera guardiamo insieme

Add Analytics to the Backend

Here is a list of the Analytics we have to implement. An Analytic can be used either by a Barber or by an Admin. Each Analytics is grouped per-month.

Barber Analytics:

• View/Appointment ratio
• Appointment cancellation ratio
• Number of Appointments, Views and Reviews
• Average rating weighted on each Review's Freshness and Up/DownVotes
• Number of Up/DownVotes
• Users that previously booked an Appointment, are still active (Views/Appointments), but no longer book on the shop

Admin Analytics:

• Number of new Users
• Number of Appointments, Views and Reviews
• Shop ranking weighted on number of Appointments and Reviews engagement
• User and Shop ranking based on number of Appointment's cancellations

After the Analytics have been implemented, their Routes must also be implemented in the Controller:

• Implement Barber Analytics route
• Implement Admin Analytics route

Backend project structure

• go mod
• folder structure
• README

Env and migrations improvement

• create a top level config.go file that handles the env variables
• create a migrate.go file that handles migrations

Implement `Review` `Up/Down-votes`

Create `ADD` and `DELETE` barber endpoint inside `barbershop`

• don't change the db structure
• change the endpoints from a PUT inside the user to a POST and DELETE inside barbershop

this is the current state of the repo/user.go

func (r *UserRepo) EditShopsByIDs(ctx context.Context, user *entity.User, IDs []string) error {

	ownedShops := []*entity.BarberShop{}

	// TODO:
	//	- this is slow af
	//	- maybe receieve all the data in the modify user request

	for _, barbershopID := range IDs {

		barbershop := &entity.BarberShop{}
		err := r.DB.Collection("barbershops").FindOne(ctx, bson.M{"_id": barbershopID}).Decode(&barbershop)

		if err != nil {
			return err
		}

		ownedShops = append(
			ownedShops,
			&entity.BarberShop{
				ID:   barbershopID,
				Name: barbershop.Name,
			},
		)
	}

	userType := entity.USER
	if len(ownedShops) > 0 {
		userType = entity.BARBER
	}

	filter := bson.M{"_id": user.ID}
	update := bson.M{"$set": bson.M{"ownedShops": ownedShops, "type": userType}}

	_, err := r.DB.Collection("users").UpdateOne(ctx, filter, update)
	return err
}

Improvments

• The procedure is easier to execute and makes more sense
• There is no additional reads inside the db to access the barbershops data (a barbershop is added or removed once per time)

Problems

• Breaking change

Bring Importer up to speed with changes

The Imported Database is currently not up to speed with the changes made to the Testing Database during development.
We have to address the following issues:

• Rename id fields
• Salt and hash stored passwords
• Add Mongo index to enable Geolocation
• Revert location info to the Mongo Geolocation format
• Rename the Barbershop's "location" field to "address"
• Add "status" field to Appointement ("pending"/"completed"/"canceled")
• Remove Review reporting
• Round up an Appointment's "startDate" to the nearest half hour

Implement `Calendar` `Hours` `Appointment` `Holidays`

Add Local Replication

Add Database replication to the local Test DBs, then later export it to the deploy machines IP

Remove index creation at runtime from the Backend

Remove duplicates from InactiveUsers Aggregation

Implement frontend requests for this REST_API

this are the implemented API

// TODO :
// - fix trailing /
// - don't return hash password
// - return the ID

// link the path to the routes
users := router.Group("/user")
{
  users.POST("/", ur.Register)
  users.POST("/login", ur.Login)
  users.GET("/self", mr.RequireAuth, mr.MarkWithAuthID, ur.Show)
  users.DELETE("/self", mr.RequireAuth, mr.MarkWithAuthID, ur.Delete)
}

admin := router.Group("/admin")
admin.Use(mr.RequireAdmin)
{
  admin.GET("/user", ur.ShowAll)
  admin.GET("/user/:id", ur.Show)
  admin.DELETE("/user/:id", ur.Delete)
  admin.PUT("/user/:id", ur.Modify) // NOT TESTED
}

Refactor `Comments` to `Reviews` everywhere

Revise Slot functionality

Slot functionality has to be rewritten in order to prevent accessing both MongoDB and Redis each time.
Ideally, Redis and Mongo should only be accessed together when:

• Creating a new Appointment
• Completing or Deleting an Appointment

Redis should be accessed by itself when:

• Reading an appointment list for a BarberShop

In order to achieve this a couple of changes have to be implemented:

• SetHoliday is removed, the Employees field of a BarberShop is now what gets updated
• Slot now has available employees, instead of Unavailable
• Revise what happens on Book

Importer fixes

Fix non-positive ratings

Address Backend Chores

The linked PR aims to address several "smaller" fixes needed in the Backend. Here are the individual fixes/chores performed:

• Revise the Barber's ownedShops field
• Revise the Appointment Entity and its DB representation
• Add Username input capability to User
• Fix Shop Modify description not working and docs
• Add "status" field to Appointment entity

documentation

Make Docker files work remotely

General doubts

• How to retrieve different data based on the same collection at this moment

  ex: barberShop data is different based on the logged user

• General code review

• How to implement Reviews and Appointments pagination

Fix scraper

Add a couple of fixes to the scrapers

Error 404: Password reset endpoint

• write test
• fix any found issue

Migrate Views, Appointments and Reviews into their own Mongo Collections

This change was made necessary due to a couple of reasons:

• Analytics were hard to implement without the use of the $unwind operator
• Whenever a list of Shops was sent over the network, a lot of unnecessary data was included as well

In order to address these issues, the aforementioned entities need to be decoupled from the Barbershop entity into their own Collection.

• Migrate Views and test
• Migrate Reviews and test
• Migrate Votes and test
• Migrate Appointments and test
• Reflect the changes in the Importer

Add importer

Add an importer script that populates the Databases with scraped data

Requirements handling

• decide which db to use for each requirements

Implement `barber_shop` rest api

Fix swapped Lon Lat

graph plotting fix and comments

Fix Slot Book

Fix an off-by-one error in Slotrepo's Book

UML

fix fe build