h593286 / fireguard

ADA504 project

Fireguard

An ADA504 group-project

By Fredrik Fluge, Daniel K.Gunleiksrud, Magnus Noreide and Eilert Skram

Architecture

The project is a monorepo micro-service (bearly). Each component is split into its respective folder and the broker aspect of the architecture is separated out.

GitHub Structure

src/

api (API layer):

• authentication: Contains server logic for handling authentication
  • auth.py: contains api guard methods that are used for verifying user tokens
  • User.py: contains definition of the User object
• requesthandler: Contains the api implementation
  • api_logic.py`: Contains functions with the logic thats being executed when the endpoints are called
  • api_server.py: Endpoint hosting
  • buildFireguardApi.py: Init and builds the API

broker (API layer):

• publisher.py: The TTF publisher, publishes TTF to HiveMQ. Runs through the list of cities and publishes their respective TTF at a set time-interval (i.e once a day)
• runner.py: Runs the publisher, maintains uptime
• api_client.py: Enables communication with the api for the broker

client (client layer):

• clientprototype.py: The webclient prototype, now no longer functional for Get req to the api (Does not have auth), but acts as a subscriber. Displays the subscribed city.

data (data layer):

• apihandler: The apihandler and the respective api clients for external api-servies, i.e MET and FROST.
  • apihandler.py: common api class for using both MET and FROST APIs
  • frostClient.py: api client for using the FROST API
  • METClient.py: api client for using the MET API
• databasehandler: the databasehandlers for storage and collection of existing data
  • databaseHandler.py: contains the abstract class for interfacing with a storage medium. This class is solely used as a interface
  • mongodb.py: implementation of the databasehandler interface for storing forecasts and observations on a mongodb server
• dataextractor: Component for extracting data from weather servies into correct format.
  • dataExtractor.py: implementation of the extractors for getting data out of observations and forecasts returned by MET and FROST
• datatypes.py: data transfer objects that are used for returning data through the api, storing data in the database and for computing TTF

helpers (helper methods)

• annotations.py: helper annotations for use in the project

service (Service layer):

• datacollector: The component responsible for collecting data
  • dataCollector.py: implementation of the data collector, utilizes the database handler and api clients for retrieving data
• TTFmodel: Model for calculating TTF as provided by the professor
  • compute.py: main methods for finding the TTF, based on the temperature, humidity and wind speed
  • parameters.py: parameters used by the algorithm for finding TTF
  • preprocess.py: methods for preparing the weather data before applying the algorithm
  • utils.py: helper methods used by the algorithm
• frcapi.py: the applications business logic.

static/

the static folder contains frontend files for client usage. These files can either be served by running the flask application in src/client or by using the docker compose files. The files are served using the https://hub.docker.com/r/lipanski/docker-static-website image

test/

the test folder contains unittests for the project. The folder structure follows the same pattern as the src folder(to an extent), this helps with locating what tests are associated with a part of the application.

Instructions

To replicate the project and to run it on your own device, please follow the instructions below.

Pre-req:

• HiveMQ: https://console.hivemq.cloud/ (Create subscriber and publisher)
• Poetry
• Docker
• MongoDB

1. Clone GitHub project

Clone the GitHub project.

If you are unsure on how to do this, follow this link.

2. Add the .env files to the project

add a .env file in the root folder of the repository and add the following values, There is an .env.template you can copy that contains these env variables.

FROST_CLIENT_ID            ='your client id'
FROST_CLIENT_SECRET        ='your client secret'
KEYCLOAK_ADMIN             ='admin'
KEYCLOAK_PASSWORD          ='password'
KEYCLOAK_REALM_NAME        = fireguard
MONGO_INITDB_ROOT_USERNAME ='username'
KEYCLOAK_SERVER_URL        = http://localhost:8090
FIREGUARD_API_URL          = http://localhost:8080
MONGO_INITDB_ROOT_PASSWORD ='password'
MONGO_DB_CONNECTION_STRING ='mongodb://username:password@fireguard-database:27017/admin?retryWrites=true&loadBalanced=false&connectTimeoutMS=10000&authSource=admin&authMechanism=SCRAM-SHA-1'
BROKER_URL                 = 'c406645d204a4c93919e442f4c8bcc09.s1.eu.hivemq.cloud'
BROKER_PORT                = '8883'
PUBLISHER_USERNAME         = 'your hivemq publisher username'
PUBLISHER_PASSWORD         = 'your hivemq publisher password'
CLIENT_ID                  = 'fireguard_client'
USER_ID                    = 'admin'
USER_PASSWORD              = 'password'

3. Spin up development application

For running the application in development mode, you'll need to include both of the docker-compose files in the docker compose call. docker-compose.yml is the main file and spins up everything the application needs to run, while docker-compose.dev.yml assigns the src folder as the volume for the broker and api, while also specifying that the application runs in Development mode.

docker compose -f docker-compose.yml -f docker-compose.development.yml up

Local url for the API server http://127.0.0.1:8080/

Local url for the Authorization server http://127.0.0.1:8090

Local url for the Frontend http://127.0.0.1:3000

Get bearer token for protected endpoints:

for auth token, use postman, web_api.http or shell to make an request against the auth server:

export token=$(\
curl -X POST http://localhost:8090/realms/fireguard/protocol/openid-connect/token \
-d 'client_id=fireguard_client' \
-d 'username=admin&password=password&grant_type=password'| jq --raw-output '.access_token' \
)

This will return an jwt token that can be used for authorizing further requests against the api.

then, use token on one of the protected endpoints (postman or shell):

curl http://localhost:8080/fire_risk/city/Bergen?ts_from=2024-04-30T21:59:59Z&ts_to=2024-05-09T18:59:59Z -H "Authorization: Bearer "$token

Endpoints

GET cities/{city:string}

gets information for the specified city.

Response object

{
    "city": "string", 
    "lat": "string", 
    "lng": "string", 
    "country": "string", 
    "iso2": "string", 
    "admin_name": "string", 
    "capital": "string", 
    "population": "string", 
    "population_proper": "string"
}, 

GET cities/{latitude:float}/{longitude:float}

Returns the city closest to the provided point

Response object

{
    "city": "string", 
    "lat": "string", 
    "lng": "string", 
    "country": "string", 
    "iso2": "string", 
    "admin_name": "string", 
    "capital": "string", 
    "population": "string", 
    "population_proper": "string"
}, 

GET fire_risk/city/{city:string}?ts_from=datetime&ts_to=datetime

returns ttf for the specified city and time range.

Response object:

{
  "location": {
    "latitude": "float",
    "longitude": "float"
  },
  "firerisks": [
    {
      "timestamp": "datetime",
      "ttf": "float"
    }
  ]
}

GET /fire_risk/{latitude:float}/{longitude:float}?ts_from=datetime&ts_to=datetime

Returns ttf for the specified coordinates in the given time range

Response object:

{
  "location": {
    "latitude": "float",
    "longitude": "float"
  },
  "firerisks": [
    {
      "timestamp": "datetime",
      "ttf": "float"
    }
  ]
}

fireguard's Issues

APIlayer - FastAPI

Peer Review - Group 1

README Walkthrough

We have tested implementing the Project on three different computers, and the following comments are valid for all of the testers:

• Should add a line to remember to start up Docker Desktop before running the docker-compose line.
• Running the docker-compose directly created an error with the keycloak module (not finding ./bin/kc.sh)
• When following the README file, 2 of 3 modules are running (the one not running being the keycloak module). This causes 127.0.0.1:8080 to run as expected, but 127.0.0.1:8090 does not run as expected due to the keycloak module.

Summarized – For a programmer that has never seen the project code, or has familiarity with the project, the README file describes the steps needed to set up the project, however from a fresh perspective where users do not have familiarity with e.g. Docker, Postman and Keycloak respectively, the README file is not detailed enough to run the application based on the README file alone.

Checklist Review

Source Code Management

• Are all components of the fire risk system under version control?
  Yes

• What repository layout is used: mono-repo or poly-repo?
  The repository layout used is a mono-repo layout.
  In terms of well-documented code, an argument can be made for the code being well-documented for those who are familiar with the code. For a new contributor, the code needs to be commented better.

• What workflow is used in the repo? Trunk-based vs. GitHub-flow vs. Git-flow? Does commit-messages refer to feature/bug-tickets? Are tags or branches used to identify features or stable versions?
  The workflow used in the repo is GitHub-flow.
  Branches are structured well in terms of following the feature names.

Continuous Integration and Deployment

• Are there test cases for the most central functionality?
  There are several tests for the central functionality of the project. There are no test coverage limits set in the CI-pipelines (e.g. 80-90% Code completion). This could improve the project.

• Are there CI-pipelines for each component that run the tests automatically?
  Yes – There Is essentially only one big component in this project.

• Are there Dockerfiles for each component that create container images?
  Yes – There are 2 components (keycloak & app) both of them have a Dockerfile

• Do the CI-pipelines contain a step for building the Docker image and additionally pushing it to a container registry?
  No.

Functionality

• Does the system comprise a component to automatically fetch weather data from MET?
  No – None that we could find in the source code.

• Does the system offer a REST API for retrieving the fire risk at a given location?
  Yes

• Does the system comprise the fire risk model from the frcm project?
  The project is based on the original fire risk model from the FRCM project, but the code has been refactored and implemented to their use case. The credits file from the original FRCM project seems to be missing, so the GNU-license is not being followed properly.

• Is there functionality that automatically observes the fire risk at given location (i.e. fetches weather data in the background and updates the fire risk continuously)?
  No

• Perform a system test of all components together: i.e. “Does it work?”
  No, it does not work for us.

Non-functional Requirements

• Does the system use messaging for the communication of two or more components?
  No.

• Does the system store some information (at least historical weather data) persistently? What type of storage is used (file/block storage vs. object storage vs. database: relational vs. non-relational)?
  There is a database set up, using mongodb. However, everything in the code seems to be storing locally.

• Is information encrypted in-transfer and/or at-rest? How?
  Time limit reached 2 hours – skipped.

• Is authentication and authorization implemented for specific system functionality? How?
  It is implemented using KeyCloak.

API Layer - REST API (FASTAPI)

tests for api layer

Interface service- apilayer

Prototype client

Service Layer - Data Collector

Service Layer - TTF

Data Layer - Frostclient

tests for service layer

GitHub Actions

Data Layer - Database handler skeletoncode

API Layer - Authentication

Data Layer - APIhandler

Data Layer - Data extractor

Docker setup