joshniemela / ku-courses

A better version of KU's course catalogue

KU-Courses

The entire application is governed through the docker-compose.yml file and is built with docker compose:

Starting the application

1. Install docker and docker-compose, this may need a restart of your system since Docker is a very low level program.
2. Run docker compose up --build as either a user with permissions to docker, or with sudo/doas, the build flag is required if the backend or frontend code has been changed, additionally -d will make it detach from the terminal.
3. Wait for the scraper in the backend to complete scraping pages, this may take about 15 minutes.
4. Run docker compose restart, this is required so that the parser will run and so that the vector store can create new embeddings.
5. ???
6. PROFIT!!!

db-manager

The backend is built with Clojure, a functional programmering language based on Lisp which runs on the Java Virtual Machine.
This part serves multiple purposes, it is responsible for scraping the course pages from KU as well as the statistics from STADS.
The backend also serves the frontend and contains the "datascript" database and is responsible for refreshing and various services occasionally (this feature is partially broken at the moment).

vector_store

This service is responsible for the semantic searches used in the get_course_overviews route, instead of using trigrams or full-text, we decided to use vector searches for the lower latency.

rust_parser

This service is the parser that takes the scraped course pages and parses them into a format we can use in the database for searching and for serving to the frontend.

frontend

Frontend is built in Svelte/Typescript. This is a highly responsible SPA that shows the courses in the form of cards which can be clicked into to get a more detailed view of the course.

Credits

• Thanks to Jákup Lützen for creating the original course parser in Python.
• Thanks to Kristian Pedersen for creating the original frontend, and help in designing the architecture and first database schema.
• Thanks to Zander Bournonville for creating the statistics parser.

ku-courses's Issues

As the dev-team I'd like to be able to have all of the ports, database config, etc all be centralised in one config file in root (S)

Currently, all the various components of the project, refer to DATA_DIR separately, their ports are defined locally in the files, and other config options are all done within the code, can we use either .envrc or some other thing to put everything into environment variables and grab the information from there instead?

As a user i would like to be able to have a quick overview of all the available courses (L)

Filtering exams removes other exams

Advanced Computational Geophysics for instance has both an oral exam and a written assignment, if filtered on oral exam, all exams will be shown with only oral exams, the secondary exam is filtered away, this happens in the SQL code of db-manager

TODO:

• test if manually querying postgresql causes the same issue
• find fix?

Back button Z level

Back button on the course pages is below the stats graph and cannot be clicked

A complete rewrite of the python parser (Urgent)

The parser has to be completely rewritten due to being completely impossible to navigate and the fact that newer courses generated by KU no longer parse due to the panel-bodies being restructured differently (the previous statement is wrong, due to an acute case of stupidity I forgot to check if the HTMLs that were being scraped were acutally HTML files (they were not))

Weird formatting of lists on some courses

(The image is from the course "mad" )

The P tags which is the direct descendant of a li tag needs to be removed and their content inserted into the li tag

Margins

The pages need left-right margin to look slightly better for the user

Nested itemised lists in HTML

Parser or json grammar used by the frontend cannot represent nested itemised lists, example: Machine Learning A, it seems that the li tags get merged into one
disku link: https://disku.jniemela.dk/course/NDAK22000U
ku link: https://kurser.ku.dk/course/ndak22000u/2023-2024

Parser missing features (M):

These issues are in decreasing order of priority

• It would be very wise to start writing unit tests to ensure how the parser is behaving and if its doing the expected of the below, or use property based testing
• Almost everything is a list that contains 1 element, we need to figure out if they can be turned into key-vals or if they appear as larger items often enough to warrant being lists
• workload is currently a list of the shape [..., "exams", 40, "total", 60, ...] this needs to be a list of key vals: [..., "exams": 40, "total": 60", ...], the first two values of category and type can be entirely omitted since we already implicitly know what they should be
• WARNING DIV is a placeholder that needs to be removed once we know the parser is doing stuff as intended with divs
• Some courses contain escape codes for danish letters instead of the actual danish letters: \uXXXXXinstead of æøå, is this reproducible?
• Some keys have not been localised to english, etc Engelsk titel $\rightarrow$ english title
• Schedule possibly still squishes div's together by calling .text too early, confirm?
• Nulls appear in one of the fields in LBIK10214U, why does this happen and does it appear in other courses too?
• duration & placement should match for the numbering using regex so we can save them as INT(1) in the database
• credit extract the number of credits with regex so we can save them as FLOAT in the database
• language change the keys so they are en or dk so we can save them as CHAR(2)
• level change so we just save the abbreviations of the titles, it can therefore be CHAR(3)

As a user i would like to have consistent styling across browsers