relatable-course-visualization / relatable

Relatable is an interactive course detailer for the University of Saskatchewan. A user searches for a course, and the application displays its prerequisites, dependencies, and other useful information.

Product Description

Relatable is an interactive course detailer for the University of Saskatchewan. A user searches for a course, and the application displays its prerequisites, dependencies, and other useful information. Data is captured from the UofS Course Catalogue.

• Technology used: Python, Django, React.js, MySQL, and HTML/CSS.
• Method: We scrape the data from the course catalogue using beautiful soup. Then, we clean the data and store it in our MySQL database. Finally, users search and interact with courses on the front-end, making API calls to the server to capture relevant data from the database.

Preview

Installation Steps

All done in the terminal

Client

• Note: MAKE SURE TO PERFORM THIS STEPS IN THE CLIENT FOLDER

• Node.js - Install node.js version 19.1.0 from https://nodejs.org/en/

• React.js

  • Install react using npx create-react-app
  • Install all the package.json dependencies using npm install
    • npm install @material-ui/core
    • npm install @mui/icons-material
    • npx json-server -p 3000 -w data/db.json
    • npm install axios npm install --save react-helmet
    • npm install @material-ui/core
    • npm install @mui/icons-material
    • npx json-server -p 3000 -w data/db.json
    • npm install axios
    • npm install --save react-helmet

Server

• Note: MAKE SURE TO PERFORM THIS STEPS IN THE SERVER FOLDER

• Django Dependencies

  • Install Django

    • pip install django

    • pip3 install djangorestframework-jsonapi

    • pip install mysqlclient

    • pip3 install djangorestframework

• Non-Django (Webscraping) Dependencies

  • Note: MAKE SURE TO DO THIS IN THE DIRECTORY relatable/server/webscraping
  • pip install bs4
  • pip install requests
  • pip install regex
  • pip install nltk

Database

• Installing MySQL

  • Install the community version from https://www.mysql.com/downloads/ (Go to Community addition at the bottom)
  • Download the “MySQL Installer for Windows”
    • Download the second one
  • In the MySQL installer
    • Select the developer version
    • Ignore the warnings
    • Install everything by clicking execute
    • Click next through everything
    • Create a root password; root is the main user
  • Click next through everything and execute
  • In the “Connect To Server” page, type in your password
  • Install the MySQL Workbench for a GUI. You can use the terminal to interact with MySQL if sought; in this case, add MySQL to your device's PATH variable (tutorial)

• Setting up the database

  • Create a new user: CREATE USER ‘username’@‘localhost’
  • Create a new database (keep track of the name): CREATE DATABASE 'name'

• Connecting django to the server

  • Add database details in the corresponding server .env file (see below)

Setting up .env files

• There are three .env files to populate with corresponding data
  • Create a .env file in Relatable/server/webscraping directory. Set the variable SERVER_URL={the url of the server running}
  • Create a .env file in the directory Relatable/server/server. Set the corresponding variables:
    • SECRET_KEY={the secret key of the database}
    • DATABASE_NAME={the name of the database}
    • DATABASE_USER={the user of the database}
    • DATABASE_PASSWORD={the password of the database}
    • DATABASE_HOST=localhost
    • DATABASE_PORT={whatever port you choose to use}
• Create a .env file in Relatable/client. Set the variable REACT_APP_SERVER_ENDPOINT={the url of the server running; must coincide with the one used in the webscraping folder}
• Restart your IDE

Running the application

• Running the server
  • CD to the server directory, then use the command python manage.py runserver
  • Note: The first you make changes to models.py or settings.py, you would have to do python manage.py makemigrations and then python manage.py migrate
• Scraping data and updating the database
  • CD to the server directory
  • Run (i.e., python path_to_file) the following files inside the webscraping directory (note, the first two files will take 10-20 minutes each as they pull 4609 courses from UofS's webpages):
    • initializeCourse.py
    • initializeDependency.py
    • initializePrerequisite.py
• Running the client
  • CD to the client directory
  • Use the command npm start

RESTART YOUR IDE BEFORE RUNNING OR IF YOU'VE DONE ALL THE STEPS CURRENTLY AND IT DOESN'T STILL WORK

relatable's Issues

Develop about us page

This should be based on the about us page design

ENVE 432 Bug

The parsing was incorrect with the "299"

We can either just change this manually, or create a more appropriate solution for this use case.

seq2seq prereq conversion method

Apply sequence to sequence method to convert prerequisite strings into properly formatted strings.

Configure routing

Set up routing for the system using React Router

navigation for courses not in catalogue

add feature to be able to jump back to last search when you click on a course that is not in course catalogue

clean up repository visuals

• Clean up the read me to be human-readable and visually appealing
  • make Product Description easier to understand
  • Update preview photos
  • remove installation steps
• make GitHub About section easier to understand

Course Card

Use MUI for easy card styling.

Some complexity with hover; should be a common use case for expanding a card (probably tutorials online)

• Consider attaching an event handler on the component for hover which runs a function to display additional data
• Consider conditionally rendering two versions of the components (closed and expanded) because on event handler onHover

Once done, we can create a new branch to add this component inside the .map that renders from the search bar component.

term and time for courses

This will be a big feature to add and will involve a complicated web scrape of through paws.

Set up local databases

For @kyle-boughen and @Inioluwa21

Develop footer

This should be based on the footer design

deployment

Currently, just at a research stage, but very soon we will want to put fully into action

special course card for non-catalogue courses

In #38, courses not in the catalogue were added to the CourseTable2023 table using a new column not_in_catalogue. 0 indicates that the course exists; 1 indicates that it does not.

The course card component should include conditional rendering to render a course not in catalogue UI if the given course is not in the catalogue.

The logic for this from the endpoint should be complete from #36 . This issue simply focuses on creating the conditional UI render.

Blocker: #36

Designs

• Finalize designs
• Upload them onto a "design" folder in the main root directory
• Create a .md file for typography & color: colors (primary and secondary), font-style, font-sizes (for headers, text, etc.)

Complete_DB_Functions

Need to create the following functions and post/put functions:

1. Clean_Preq
2. Marked_Preq
3. Dependencies

It will likely be helpful (and necessary) to create helper get function from DB.

sequence labeling with CRFs

footer covers show more button

There is a "show more" button on the search component. The Search limits the number of shown courses to 10 and allows repeated display of 10 more every time the show more button is clicked. However, the footer seems to be hiding this button.

Screenshots

At the bottom of the page but the button is not visible.

Integrate Course Card with Search

The course card component is currently developed as an independent component. Integrate it within Search to render when the user search. Also, integrate logic to handle clickable-UI for courses. Checking for non_in_catalogue should be done, but the UI does not need to be; this is done in #43.

Blockers: #30

Courses not in Catalogue

As we discussed on Wednesday, some preq strings reference courses that are no longer offered at USask. We would like to add these courses into DB, however, they must have a clear flag. The file 'postDependentCourses.py' has a notInCatalogue list
which stores all course codes that do not have a matching row entry in the db (and therefore are not in the course catalogue). We would like to add these to the db and create a flag for them. Create the flag as you see fit, however, if you add another column (which you likely will need to), you will have to edit models and all other post/put functions. Big picture:

1. Decide how to flag courses and make this update in all the affected files
2. In post dependency, post all courses not in catalogue to DB with the flag

contact us page

Create a contact us page.

Little bit more complex due to form submission (not just a UI ticket). If we want to spend time elsewhere and add this at a later sprint once deployed, we can definitely do that.

Use MUI for all components and styling--do not have an additional CSS page. Reference Marmik's new MUI code here: https://github.com/marmikpatel19/marmikpatel19.github.io/tree/version-2

Tasks

• Link the new page to the navbar
• Add a brief . Write whatever seems appropriate. Consider adding a point about reporting bugs through the form.
• Add a MUI form. Add whatever fields seem relevant.
• Add a place to store submission
  • Ideas:
    • New db table with new API endpoint
    • A google spreadsheet (example: https://github.com/Infokidz-Academy/Infokidz-Academy-Website/blob/main/client/src/components/contact/contactform.jsx)
• Add a way to notify team members about a new submission
  • Ideas:
    • Trigger email
    • Not do this and leave it for team members to manually check

Update Database Documentation

Due to database refactoring, the Course table has changed and the Prerequisite and Dependency tables have deprecated. Update these tables in the documentation to reflect the current state. Wait until #38 is complete to add that change as well (if there is a change).

Maintain the current documentation as V1 and add new documentation at the top as V2.

Blockers: #38

refactorDB

There are a variety of subtasks needed to complete this including:

1. Decide on how we will store newly formatted strings
2. Parse formatted strings so that they are in a proper data structure to post
3. Refactor existing POST functions
4. Refactor dependencies to include string and weak
5. Rescrape AND retest form conversion as course catalogue has been updated

add USask labelling somewhere

Need to identify somewhere that the courses are for USask

Server down

404: NOT_FOUND
Code: NOT_FOUND
ID: cle1::lkxjr-1695705142303-ad45a5a52054

Consider move the deployment to other provider / ask Greg for hosting it somewhere in usask?

Restructure Course Table

Reference the discussion had here: meeting

Develop navbar

This should be based on the navbar design

refactor footer

For smaller mobile size, the vertical footer takes up too much space, leading to poor UX. Reduce the spacing between components to mitigate this issue.

For larger screen sizes, it is also too big. See second screenshot

Screenshots:

update course button UI

Current the buttons rendered for prerequisites and dependencies are the default MUI buttons.

Improve the UI so that they

• Aren't too large like they are right now
• Match Relatable color scheme
• Look less like traditional buttons

Screenshots:

Blockers: #36

Search Bar

Create search bar for the home page. Leverage MUI for for easy styling. Make use of old logic as much as possible as that correct hit the endpoint for collecting all the courses and correctly maps them with searchability.

About-Us-Header

About us needs header and footer, should be a quick fix.

Make low fidelity sketches

You can use the 10+10 design method talked about in CMPT 381 (due Tuesday 1:00pm)

update favicon

Update favicon

Create test data for machine learning

Today or tomorrow I will create a text document that contains source prerequisite strings for us to manual convert to formatted test strings.
Here is an example of how the text will look initially:
Fraser
CMPT 145 or CMPT 115 or CMPT 117.
CMPT 214.3 and one of MATH 104.3, MATH 110.3, MATH 123.3, MATH 125.3, MATH 133.4, MATH 176.3, or STAT 245.3 (or equivalent).
...
Marmik
....
Kyle
...
Ini
...

Convert the strings under your name. See the document on one drive for the rules for this. Separate the original string from the converted string with a tab.

Refactor Navbar

Address the comments made in #56

• Reduce the max-width of the navbar
• Fix the alignment on mobile screen