compserv / hknweb

The new HKN website (using Django, hopefully at hkn.eecs.berkeley.edu soon)

hknweb

Welcome! This is the in-progress website redesign for the IEEE-Eta Kappa Nu (HKN) University of California, Berkeley Mu Chapter, built with Django, Django REST framework, and Vue.js.

Setup and Development

1. Fork and Clone the Repository: Go to https://github.com/compserv/hknweb and click on the "Fork" button in the top left corner to fork the repository to your GitHub account. Clone the repository to your local machine.

2. Install Python 3.9 and Poetry: Ensure you have Python 3.9 installed on your system. You can download it from python.org. Then, install Pipx if not already installed. Finally, install Poetry:

   pipx install poetry

3. Install Dependencies: Navigate to the cloned repository directory (hknweb) in your terminal and run the following command to install project dependencies using Poetry:

   poetry install

4. Activate Virtual Environment: Run the following command to activate the virtual environment created by Poetry:

   poetry shell

5. Apply Migrations: Apply all database changes using the following command:

   python manage.py migrate

6. Run the Server: Finally, run the development server using the following command:

   python manage.py runserver

   You should now be able to access the website locally at http://localhost:8000.

   In order to access the admin interface, run

   python manage.py createsuperuser

Complete the following prompts to create a local admin user, which can then be used to access the admin interface by logging in at http://localhost:8000/admin.

Deployment

The deployment pipeline pulls from compserv/master to OCF servers using fabfile.py.

# Activate our dev environment
poetry shell

# Depending on your ssh setup, this may or may not work for you:
HKNWEB_MODE="prod" fab deploy

# If your SSH key to the apphost requires a password to unlock:
HKNWEB_MODE="prod" fab --prompt-for-passphrase deploy

# If you have no SSH key to the apphost, and require password authentication:
HKNWEB_MODE="prod" fab --prompt-for-login-password deploy

Contributing

If you'd like to contribute to this project, feel free to fork the repository, make your changes, and submit a pull request.

License

This project is licensed under the MIT License. Feel free to use, modify, and distribute the code as per the terms of the license.

hknweb's Issues

Exams revamp

Exam archives were implemented in #14. I think the functionality works as-is, but there are some things which are still desired:

• Uploads are managed through the django admin interface. Requires admin logins (which we would like to restrict to compserv), and
• We'd like to be able to indicate whether professors refused to release their exams for that semester. This is very beneficial to all parties: we visibly demonstrate that we respect professors' requests to avoid uploading their exams, and students will (hopefully) not hold the misconception that we are slacking on exam uploads. This also leaves a paper trail for future tutoring officers, who (in the absence of other records) can heuristically determine if professors do not want their exams released. (We may still want to ask every semester, however.) Alternatively, we may also want an internal-only indicator that professors have permanently opted out of releasing their exams.
• Departments are hard-coded. If we decide to support other "departments", especially "EECS", we should convert this to a ForeignKey.
• IDs are hard-coded as an IntegerField. We'd like to convert this to an implicit AutoField (to pseudo-guarantee uniqueness) by removing this field from the Course and CourseSemester models.
• Semesters are length-255 CharField fields. This may not be desirable. The current encoding scheme consists of naming semesters by "YYYYS", where YYYY is the year and S=1 in the spring, S=3 in the fall. (S=2 is ostensibly reserved for summer sessions.) We might want to encode semesters as models, or perform some sort of database validation to ensure consistency.
• Instructors are encoded as strings. We'd rather encode them as Instructor objects, which can be fixed for typos once (across the whole database), added independently, and verified with other validation checks. (This would also let us cross-link all exams from a single instructor easily.) This should be converted to a ForeignKey.
• Exam categories are hard-coded as Midterm 1, 2, 3, and Exam. This works well for the table format, but I'm unsure if this accommodates courses that use "quests" (quiz tests). (This might just be an optional thing.)
• Making exams harder to delete. (I haven't checked the implementation in hknweb right now, but in the hkn-rails site the button to delete is right next to the button to view, if you're logged in with permissions to delete.)

Fix CSS for certain pages

Pages with CSS messed up include: alumni, alumni/search, alumni/form, alumni/detail, cand, cand/candreq (this one is particularly nasty), cand/challengeconfirm, cand/detail, and possibly more

Docker / container instead of Vagrant VM

Vagrant is rather heavyweight for a dev environment, if familiar. Could we get away with running Docker / containerd containers, and use that instead?

Metrics:

• User-friendliness: cannot be any harder for newcomers than Vagrant
• Download size: slower downloads = more time waiting to start developing
• Performance: same order-of-magnitude as VM (X% slower / faster is acceptable)
• Dev / prod similarity: must be able to run same MySQL / Python stack as prod

This would have the (theoretical) advantage of being able to plug into (potential) future containerized OCF infrastructure.

Cannot return to home page when dropdown menu covers main logo

Sending emails

https://docs.djangoproject.com/en/2.1/howto/deployment/checklist/#email-backend-and-related-settings

Spurious / misnamed hknweb/courses/templates/login.html

ModuleNotFoundError: No module named 'social_django'

Was receiving the following error:

ModuleNotFoundError: No module named 'social_django'
Makefile:32: recipe for target 'migrate' failed

Perhaps it is the way that my computer was setup before, but upon searching I found that pip install social-auth-app-django should be run. I ran pipenv run pip install social-auth-app-django and this fixed the problem, but not sure if this is the correct way to go about things or not (maybe add to Pipfile).

#37 references getting rid of pipenv so this solution may not be relevant with changes to come, but wanted to mention it here so either setup can be fixed or setup documentation can mention the extra command.

Create VP admin page for rsvp confirmations

Better testing / Travis CI

Testing is pretty much non-existent right now.

The current Travis-CI, configured with .travis.yml, does three things on a freshly installed Ubuntu 16.04 OS:

• Install dependencies, with pipenv
• Run SQL migrations (python manage.py migrate)
• Run checks (python manage.py check)

And that's it. No tests on if anything actually works, no tests if urls hit the right pages, no checks if methods work, or really anything.

A good first start on this will be route tests: checking if a GET request to a specific url returns the right page. Beyond this we can check for db functionality on POST / GET requests, etc.

Relevant docs pages:

• Django reference on testing
• Django tutorial on testing
• Testing tools

Bad social_django migration on prod server

Current workaround is by faking the social_django migrations:

https://docs.djangoproject.com/en/2.2/ref/django-admin/#cmdoption-migrate-fake

This needs to be fixed if we want to get OAuth login working.

Elections revamp

Tangentially related to #52.

We have an older candidate project, #23, which added an elections page. I think this app needs some refactoring to accomodate committees, committee officer counts, and positions which may vary over time.

Concretely, this means that you would have to:

• Create a Committee model (ideally, reuse the one from #21)
• Adjust the code / forms to accomodate a varying number of officers (since this changes semester-to-semester, it would be impractical to encode a limit / number anywhere in the code, including in the Committee model, instead allowing any number of officers to be elected any election)
• Create a Position model (idk, you should see if this works or not) which refers to the position people are elected to: officer, assistant officer, committee. Currently in the hkn-rails site this is a string, but strings are easy to mess up. A link to a database object as a ForeignKey would be harder to mess up.
• (optional) create an Committeeship model, tied to a specific semester and a committee, so that a single person would have multiple Committeeship (i.e. the Committeeship model should have a ForeignKey to a user), and use that to codify people's election to a position
• Rewrite the election form with all of these changes
• (optional) create an election log, noting who was nominated to each position in addition to who was elected.

I'm judging this as an easy project due to the limited amount of design work for this; code-wise this is a highly constrained project (you're refactoring code, if the code works as before you're doing it right) but does need you to learn how Django models and forms work.

Pretty minimal UI experience needed.

Make alumni create regular user accounts

Because Slack is an unreliable info dump:

Models

Priorities

• Course model
• User model (+officer model)
• Events model

Implement event waitlisting (not just cap)

Officers / people revamp

#16 and #21 are involved in creating an officers page: see hkn.eecs.berkeley.edu/about/officers.

I'd say this is feature-incomplete.

This will need a broader revamp, which will entail:

• Figuring out how to model officer history, which consists of 'officerships' that only take place over a single semester
• Displaying officer pages for each semester
• Giving assistant officers first-class support on the site (as opposed the current site, where assistant officers must be added through the rails console)
• Being able to add / remove committees / officerships from people (probably automatically implemented through the django admin interface, rather than through the regular site)
• Figuring out how to tie committees to the django permissions for groups
• Figuring out a good url scheme for all the people pages (officers, assistant officers
• Integrating this with the elections page (optional)
• Building the same thing for the committee member page (or just integrate into the main page?)

Admittedly, most of these can be ported from the old site at https://github.com/compserv/hkn-rails.

Time zone fixes

I’m pretty sure the only thing we need to do is change the TIME_ZONE setting in settings/common.py to America/Los_Angeles.

https://docs.djangoproject.com/en/2.2/ref/settings/#std:setting-TIME_ZONE

Officer portal for officer challenges

Make the portal where an officer can view all challenges requested from them, including ones reviewed and not reviewed

Fabfile breakage

Oh boy.

-- Migrating tables
Operations to perform:
  Apply all migrations: admin, alumni, auth, contenttypes, events, hknweb, markdown_pages, sessions, shortlinks, tutoring
Running migrations:
  Applying alumni.0001_initial... OK
  Applying alumni.0002_auto_20181105_0632...
Traceback (most recent call last):
  File "./manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/core/management/base.py", line 316, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/core/management/base.py", line 353, in execute
    output = self.handle(*args, **options)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 203, in handle
    fake_initial=fake_initial,
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/db/migrations/operations/fields.py", line 216, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 523, in alter_field
    old_db_params, new_db_params, strict)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 627, in _alter_field
    new_default = self.effective_default(new_field)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 239, in effective_default
    return field.get_db_prep_save(default, self.connection)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 790, in get_db_prep_save
    return self.get_db_prep_value(value, connection=connection, prepared=False)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 785, in get_db_prep_value
    value = self.get_prep_value(value)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 1018, in get_prep_value
    return self.to_python(value)
  File "/home/h/hk/hkn/hknweb/prod/shared/venv/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 1011, in to_python
    params={'value': value},
django.core.exceptions.ValidationError: ["'' value must be either True or False."]

Add GitHub note on front page

We can encourage would-be contributors by adding a github link on the front page of both dev-hkn.eecs and hkn.eecs; lowering the barrier to entry for new contributors (helping them find the repo) or encouraging technically-inclined visitors to contribute is worthwhile.

Also so we can get bug reports from visitors.

User-facing messages

Django has a built-in messages contrib module, for displaying user-facing messages, i.e.

• Login successful!
• Login failed: unrecognized username / password.
• Signup failed: bad password / ...

Briana implemented an error message for failed user signup, but because there is no corresponding code in the template to view it, it is silent.

This should probably be implemented in the base.html.

Pin dependency versions exactly (without ranges)

We should probably pin our dependencies exactly in requirements.txt and Pipfile:

requirements.txt
---
dependency=='1.4.5'

Pipfile
---
dependency = '==1.4.5'

as opposed to version ranges like:

dependency = "~=1.4.5"
dependency = "*"

Why? Version ranges create inconsistent dependency versions on different machines, which:

• May cause dependency API breakage, in turn breaking our site (even in spite of SemVer guarantees)
• Causes the Pipfile.lock lock file to update, and when people git commit -a or git add -A this creates merge conflicts in Pipfile.lock, which does not update in patches cleanly.
• Slows down dependency installation, by forcing a Pipfile.lock dependency recalculation (aka pipenv lock)

Recommended reading: https://nvie.com/posts/pin-your-packages/, https://nvie.com/posts/better-package-management/

Preventing bot accounts (captchas)

Preventing bot accounts is a concern for the site.

Bot accounts take up username and storage space, and slow down account creation by requiring human intervention (compserv account approval). If access permissions are not done correctly, they may have access to private data.

There are various options we have for this:

• Do nothing: Simplest for users, allows all bots. Requires extensive manual intervention.
• Email filtering: Allow only emails with *.berkeley.edu domains (in particular, cs.berkeley.edu, ee.berkeley.edu and hkn.eecs.berkeley.edu must also be allowed). This would require email verification (also desirable, perhaps) to avoid being defeated by simply generating a random email.
• Google reCAPTCHA v2, currently used: Offers state-of-the-art bot detection and ease-of-use, but has concerns about privacy (user tracking). That said, we also use Google Analytics anyways, so if we're already making a deal with the devil this will offer significant benefits. Seems to allow some bots through on our current site, though, so it may depend on our settings.
• Honeypot: have an invisible form that robots will try to fill out (i.e. CSS display:none or visibility:hidden), or a form with an obvious human answer (but no obvious robot answer). Can be brute forced if obvious human answer, sometimes bypassed if robots detect invisible form.
• Google OAuth signin (aka OpenID Connect). Useful for existing members with @hkn.eecs.berkeley.edu accounts. Enabling this for @berkeley.edu will require interfacing with CalNet, which is quite complex.
• Custom / open-source CAPTCHA: tend to be difficult to use.
• Time-based detection: detecting bots by form population speed (time from load to submit). May be fooled by intelligent bots (false negative) or flag legitimate users with auto-form-fill (false positive).
• Symbolic math: generating a random math problem, ideally symbolic to frustrate simple bots, but may require numbers. idk pull some cs70 or cs61a for this, but not everyone's taken cs70.

At the end of it all, there are three major priorities which must be balanced:

1. Preventing bot account creation. This saves compserv time, and keeps data more secure.
2. Keeping human account creation simple. We don't want to frustrate users who can't get past our captcha, which may include those with visual disabilities.
3. Implementation complexity. A solution which is too complex will not be maintainable, problematic given the rate of bot and AI evolution.

The best solution may involve a combination (allowing Google OAuth signin + Google reCAPTCHA for new accounts + email filtering, in my opinion).

Shortlink edit permissions

Shortlinks (hkn.eecs.berkeley.edu/shortlinks) currently have a major flaw: they can only be edited by their creator or by superusers (compserv).

There are two potential solutions to this:

1. Allow a committee (group) to edit a shortlink. This has the upside of having limited edit scope, with slightly more user input required (a dropdown menu to select committee / group) and a more complex model. (How can we let anyone edit a shortlink? Is there a group containing all users? Maybe a members group?)
2. Allow anyone to edit any shortlink. This has the upside of being simple and surviving member graduation / committee, but has greater potential for abuse / malicious edits / mistakes. This might be mitigated by storing edit history (which may also be desirable, but at the cost of greater complexity).
3. Allow all officers to edit any shortlink. (Potentially assistant officers / committee members as well.)

Course map revamp

Note: this is separate from #30, which deals with reducing the render time of the existing course map.

This is the master issue for the course map redesign. Next to exams and course surveys, the course map is probably the most visible and important part of the site for the department, professors, and students for long-term course planning and the evolution of EECS demographics and interests.

This is important. Thousands of students will plan and have planned their career at Berkeley based in part on this course map. This will affect course enrollments and departmental funding.

With that out of the way, let's dive in.

You should get access to the hkn-rails source repo at https://github.com/compserv/hkn-rails.

The front-end design of the course map is detailed at https://github.com/compserv/hknweb/wiki/Course-Map.

Internally, the course map is represented as a directed graph, with nodes representing courses and arrows representing prerequisites. It is stored as an adjacency graph, with nodes containing a list of prerequisites and arrows containing a parent / child node.

The most important changes which need to occur is:

1. Merging the CS and EE graphs.
   Visually, they should remain discrete, but logically they should be rendered as a single graph.
   We will need cross-departmental links between such pairings as CS189 and EE126/127, and potentially

jasdkfjaklsdjfajskldf
i should finish writing this

Password security (for username accounts)

In short, we should try to encourage better passwords, either through length requirements (8-10 char min), a password heuristic (see zxcvbn below), or disallowing weak passwords (like the 10,000 most common or stupid simple passwords.)

Inspired by various blog posts on password security:

• Jeff Atwood, co-founder of Stack Overflow: Your Password is Too Short, Hacker, Hack Thyself, Password Rules are Bullshit
• Dropbox's password strength estimation, and zxcvbn, accompanying JS library
• Everything you ever wanted to know about building a secure password reset feature, from Troy Hunt, Microsoft security expert
• The God Login, Atwood's blog post on making login pages that are actually usable

There's a few key points to consider about improving user password security, beyond server-side security (salting + hashing, using memory-hard key derivation functions like bcrypt over ASIC-brute-forceable ones, etc.):

• Make password rules simple. None of those "must include at least one upper-case, five lower-case, 2 number" composition rules. (Source: NIST 2016 recommendations.) It might artificially make passwords more secure, but less usable and actually easier to brute force (as shown by XKCD).
• Encourage long passwords, not necessarily complex passwords. See the above XKCD link. Discourse, the open-source forum software Atwood also founded, checks for (x) unique characters from at least (y) total characters.
• Allow password manager copy-and-paste. This involves using standard HTML password fields, avoiding JS that interferes with copy+paste, and testing with password managers (bitwarden, lastpass, etc.) on multiple browsers. (In practice, as long as we don't do anything fancy to the login page this should work.)
• Disallow very weak passwords. Discourse also uses a minimum password length of 10 characters, with admins / moderators required to use 15 characters. It also checks passwords against the 10,000 most common passwords (used by 30% of users), and for passwords equal to their username and email address.
• Use someone else's login. The easy way out is to let Google handle security, and use Google OpenID Connect (OAuth) login. We should allow this for members, but candidates won't have hkn emails by then, so username accounts must be allowed.

And finally, with credits to the OCF, we could try hacking our own users, by checking common passwords or brute-forcing short passwords. This is do-able without actually logging in as them (we can just compare password hashes), but this is probably more effort than we can normally spare.

Add data migration for initial (seed) groups in database

https://docs.djangoproject.com/en/2.2/topics/migrations/#data-migrations

We'll need to add the following Groups for permissions to work:

• officer
• candidate

(not plural)

This is required for new development instances (i.e. new developers), who initialize new (blank) databases and thus will have any functionality involving candidates or officers broken until they manually add candidate and officer groups.

Improve time / date entry for events

Static page markdown transfer

Pull request #15 added markdown pages: static pages (with no dynamic js content), which can be created and edited by regular users in Markdown. (You may be more familiar with Github Flavored Markdown). If you've ever written a comment on Reddit, Stack Overflow, or Github, chances are you've unwittingly been using Markdown.

Many of the pages on the old site are either hard-coded in HTML or uploaded under our static page feature in HTML. This project will entail converting these pages to simplified Markdown text, so we can re-upload these in the new website.

You can do this either manually or automatically with cleanup; a quick search on doing this task with pandoc suggested this stack overflow link with the following snippet:

pandoc -f html -t commonmark myfile.html >myfile.md

As for the exact pages to convert, you'll have to hunt them down on the current website (hkn.eecs.berkeley.edu). I've found a few examples below:

• Service
• Junior EECS Day
• Indrel

Of note, if you'd like to rewrite these pages, you should definitely check first with the corresponding committee - simpler is better, but not if it omits important information. Also, in some cases these committees may want the extra formatting HTML can offer.

We'll give you compserv permissions on the site for you to access the copy-paste HTML, without the header / footer cruft.

If you take on this task, you should ask us to add you to the org so you can access the old hkn-rails site and the html files.

Of note, these will eventually need to be stored as text fields in the database. (This functionality is already implemented in the markdown-pages app.) But since we don't have a stable database / backups, we won't be uploading these to the SQL database in the OCF apphost, but instead saving these text files in the Github somewhere for the time being. (Preferably somewhere sensible like hknweb/static/md, but idk.)

Django secret key regeneration

At some point, we're going to have to regenerate the secret key in settings.py.

This would be ideally set with some environment variable on the OCF apphost server, as suggested by the Django docs.

Server-side course-map rendering (d3.js)

Rendering times (excluding loading times) for the current course map are quite long (>5s): https://hkn.eecs.berkeley.edu/courseguides

This is due to the graph being rendered client-side as a d3.js force graph (into an SVG image), which simulates a spring-loaded network graph in time steps until it converges.

This might be mitigated by rendering it once, on the server, caching the resulting SVG (maybe using some hashing system to detect changes?), and sending the SVG as a static asset.

https://mango-is.com/blog/engineering/pre-render-d3-js-charts-at-server-side/

The client-side physics script is at app/assets/javascripts/coursesurveys.js.erb.

The data for the graph is a JSON downloaded from https://hkn.eecs.berkeley.edu/coursesurveys/chartinfo.

Shortlink url breakage

Currently, all shortlinks are naked, in the sense that these urls are resolved without additional prefixes:

https://hkn.eecs.berkeley.edu/<shortlink>
https://hkn.mu/<shortlink>

As currently implemented in hknweb, however, shortlinks are namespaced behind s/. In other words, all shortlink urls are only accessible at:

https://hkn.eecs.berkeley.edu/s/<shortlink>
https://hkn.mu/s/<shortlink>

This will break many existing urls to our website (decal, member signup, serv, studrel, etc.), even if namespacing is desirable. The fix should involve routing shortlinks in the root hknweb/urls.py, passing on all urls that match no other urls to the shortlinks app.

Update Vagrant VM to match prod env

Partially spurred by #27, which broke the mysql config in the vagrant.

Static file delivery on ocfweb

#31 introduces deploys to the apphost, but static files on dev-hkn.eecs.berkeley.edu are not being served.

To quote @jvperrin, we'll need "something to serve the static files since nginx won't do that on apphosting)."

There's a number of things this will need in order to function correctly:

• Setting STATIC_ROOT and MEDIA_ROOT
• Running python3 manage.py collectstatic during deploys (with all the pipenv machinery)
• Making the static files directorie under ~/public_html/hknweb/static, and the media files directory under ~/public_html/hknweb/media (separating will be good).

This will cause the static files to be served at https://www.ocf.berkeley.edu/~hkn/ (so the STATIC_URL should probably be set here).

Relevant OCF Django docs: https://github.com/ocf/ocfweb/blob/f23a2ea41db1da54e51c70af07b20275fc3d829d/ocfweb/docs/docs/services/web/django.md

Relevant docs pages for Django:

• Static files
• Deploy checklist
• User-uploaded files
• STATIC_ROOT
• MEDIA_ROOT

Tutoring schedule algorithm

Currently tutoring hours are assigned by a black-box algorithm: given each tutor's expressed preference (+1, 0, -infinity) for each tutoring slot (hour, location), it somehow optimizes the total happiness of the assignment over all tutors.

I say "black-box", because it is just that: a black box which I have no idea how it works. (@jvperrin may be more familiar.)

Your mission, should you choose to accept it, is to decipher how this algorithm works by delving through the old hkn-rails code, and re-implementing it in the new site.

We'd also love to have:

• Subshifting (for people swapping shifts for a week or more permanently)
• Disabling tutoring for a week (or indefinite amount of time), for instance over the summer or during spring / winter break
• Course preferences (reflected in the hkn-rails site).

Write setup.sh for environment setup

Historically I’ve never liked setup.sh scripts, but we might as well use one:

• It’s already written in the Vagrantfile for provisioning
• We can point new developers to run setup.sh if they don’t want a VM
• We can solve this HKNWEB_MODE madness by setting it to dev in setup.sh, and just not using setup.sh in prod
• We can check that people are using the virtualenv with VIRTUAL_ENV
• Add pre-commit hooks for everyone

If we want to steal some code, CSM has a decent example which covers lots of edge cases:

https://github.com/csmberkeley/csm_web/blob/dev/setup.sh

Fix navbar account dropdown

@Matthew-Signorotti

Deploy cannot connect to mysql

Fabric attempts to connect through the default socket file: /var/run/mysql/mysql.sock. However this is non-existent on the OCF apphost.

TODO: research how the default mysql command connects to the mariadb server, and setup django to use that instead of the socket. (Or find the socket file.)

Searching

Currently, Django doesn’t support fuzzy searching beyond that of a SQL query. This is problematic if we want to have, say, regex matching of names, match slightly misspelled names, or search arbitrary fields without a rigid SQL query.

In hkn-rails, we used Solr as an indexing engine. It’s a production-grade, heavyweight search indexer and fuzzy matched, but this also means it hogs a lot of ram (it nearly outweighs all other programs on the app host, with ~200 MB or so of memory), and synchronizing its startup and shutdown with rails has been problematic lately.

With that said, I’d like to propose we find alternatives. I recently stumbled across Sonic, a Rust-based search indexer with under 30 MB of ram usage. It should have good Python interop because Rust has C API compatibility, but I’m not sure of the specifics. It uses a non-standard open-source license though, this will require investigation.

Make deploys less jank

The deploy script was implemented in #31. As a first attempt, it appears to be working surprisingly well, but is extremely brittle and not user-friendly:

• Force-pushing branches breaks the git fetch on the apphost
• Migrations are not atomic, in the sense that not all migrations are rolled back if any one fails. (Individual migrations are atomic.) In particular, if database models are coded incorrectly and cause errors, if baked into migrations that fail, this requires manual migration rollback.
• Old releases are never cleaned up; ideally we'd keep a maximum of 5 or 10 releases (I'd prefer 10), and delete the oldest ones when new ones are deployed
• Bad releases are never cleaned up; while we don't want to clean up the most recent failed release (for debugging), if we deploy several bad releases in a row we still want to keep the last good release, while cleaning up bad releases.
• Configuration is done by editing the fabfile.py, adding a sub-dict to a config dict. This requires knowledge of the structure of the configuration, and how it is intended to be configured. We cannot, at the moment, declare multiple deploy targets and select one from the command-line at runtime (although we can specify a release rollback number and commit hash, but untested).
• Clobbering static files (which happens on every release, right now) requires user confirmation.

And on, and on. In addition, I'd prefer to either move all of the code into fabfile.py, or move all the standard functionality beyond configuration into the deploy/ folder, for ease of packaging.

@jameslzhu will be your point of contact for this, as the original author of the deploy fabfile and as an (kinda) experienced user of Capistrano, whose design this deploy script is based upon.

Add rollbar error reporting

Rollbar is an online error reporting service. It aggregates errors by type and traceback (location) and emails us when they occur (at order-of-magnitude intervals: first time, 10th, 100th, 1000th, etc.). We use it with the rails site, as it’s easier than scrolling through the log files.

Integrating this with the Django website would be very helpful for tracking down errors on the live site, especially with bad urls.

The Python library for interfacing with rollbar is at https://github.com/rollbar/pyrollbar.

Usertastrophe (or migrationcalypse?)

This could either not be a problem, or be a ticking time bomb that will only get worse with time.

Source: Substituting a custom User model

Changing AUTH_USER_MODEL after you’ve created database tables is significantly more difficult since it affects foreign keys and many-to-many relationships, for example.
This change can’t be done automatically and requires manually fixing your schema, moving your data from the old user table, and possibly manually reapplying some migrations. See #25313 for an outline of the steps.
Due to limitations of Django’s dynamic dependency feature for swappable models, the model referenced by AUTH_USER_MODEL must be created in the first migration of its app (usually called 0001_initial); otherwise, you’ll have dependency issues.

If we'd like to integrate Google OAuth login (aka OpenID Connect) in the future, this might*** require customizing User with a custom subclass:

from django.contrib.auth.models import AbstractUser

class User(AbstractUser):
    pass

Unfortunately, since this has to be done as the very first app migration, this might require us to nuke the SQL migrations in every app tied to User and regenerate them from scratch.

---

There is hope on the horizon; glancing at Google Django integrations like social-auth-app-django, they appear to use completely separate auth models, and do not seem to require modifying AUTH_USER_MODEL.

This will require investigation into whether is is the time bomb I worry it is.

Member directory

Similar to the officers / people page, but instead of listing the officers for this semester, list all of the members who have initiated in HKN.

I think a potential design for this would involve making a user detail page for each user, and then a single, paginated (i.e. 50 per page), table of everyone who is in the 'members' group.

It is important to distinguish members from all users: absent a 100% effective bot blocking solution, we only want members to be able to view the directory, and not Russian bots. Adding users to the 'members' group should only be doable with human (compserv / vp) approval. Users with insufficient permissions should be unable to view the members page. (Please check the Django docs for page view permissions.)

(Separately, making a candidate page would also be very useful.)

Replace pipenv?

At this point I'm starting to be convinced pipenv is not the right solution for deploys, for a few reasons:

1. It does not respect existing venvs well. We'd prefer to have a cached venv for the production instance, and unless we specifically source the venv before running pipenv, or symlink a .venv link to the shared/venv folder and set the PIPENV_VENV_IN_PROJECT env variable, it will create a different venv for each timestamped release, because they are at different paths.
2. It is under active development. Occasionally this has caused breaking changes (for me, personally) that cause pipenv itself to fail to run, and require upgrading pipenv.
3. dependency = "*" by default. This wreaks havoc on dependency pinning, in particular because if left unpinned before another person installs the dependency, version mismatches will ensue in the Pipfile.lock, with the resulting merge conflict requiring a brand-new pipenv lock to fix or manual intervention. (Admittedly, this can be fixed by pinning a version at the start, but pip does this with pip freeze automatically.)

During development, pipenv install will install versions beyond those specified in the lockfile, even when not prompted to upgrade versions.

With that said, there are major features of pipenv that remove pain points in development:

1. Automatic venv installation with dependency installation. This cuts out all of the venv / virtualenv / source bin/activate/ / pip install -r requirements.txt nonsense, and installs everything with pipenv install.
2. Dependency resolution. Pip's dependency resolution is apparently not very sophisticated; but pipenv is not the only dependency resolver (see poetry below).
3. Automatic lockfiles. Pip versioning typically requires a requirements.txt with full versioning and dependencies, and a requirements-minimal.txt with the minimal requirements (roots of the graph). Pipenv saves the generated dependency list with hashes (a+ for security), usable with pipenv install --deploy.
4. Editor integration. It integrates with Sublime, VS Code, PyCharm, and a couple other services (Github vulnerabilities).

Alternatives might include poetry, the leading competitor to pipenv.

Google logins

Users currently login using the default Django auth module (guide, API reference).

We'd like to add an optional alternative login, using Google OpenID Connect (sometimes known as OAuth login). In particular, we'd like to support logins with our hkn.eecs.berkeley.edu domain accounts, only. (I'm against allowing logins from other domains, like @gmail.com or @berkeley.edu, in particular because we don't control them and because @berkeley.edu requires CalNet authentication, apparently a pain.)

I think the major library to do this is called python-social-auth, with the specific backend for Django, social-auth-app-django. The authentication pipeline docs can be found here.

While I wouldn't recommend implementing this login manually, there are various OpenID / OAuth pages which may be helpful for understanding the protocol:

• OAuth 2.0, the base protocol for authenticated communications. Note this is 2.0, not 1.0.
• OpenID Connect: the login spec's website, built on top of OAuth
• Google OpenID Connect: Google's implementation of login using OAuth / OpenID
• Google Sign In: Google's solution on top of OpenID Connect, lets you put a "Sign in with Google" button on the link. Probably the simplest.

Upgrade to Django 2.2 LTS

Django 2.2 will be released soon. The release process indicates this will be a LTS (long-term service) release, with security and bug fixes for 3 years, until 2022.

Django has a site dedicated to upgrading versions. Upgrading to 2.2 will include replacing deprecated functions, hopefully minimal since we started the project on Django 2.0.

Send reminders in X days

@alexander-zw

Events calendar gcal integration

Figure out how to import / export from Google Calendar to the events calendar.

I have no idea how to do this: libraries, design, or Google APIs. Research required.

Color-code events by type

Perhaps add a specific RGB (#FFFFFF) color field to each event type?

Remove @csrf_exempt annotations

hknweb/hknweb/events/views.py

Line 45 in cf0ed9b

@csrf_exempt #doing this for now bc idk how to make csrf work

Officer challenge confirmation fixes

When someone requests an officer challenge, anyone can confirm/deny the challenge if given the link, not just the officer that gave the challenge. Need authentication.

If the challenge is already reviewed, should have some page that reminds officer it is already reviewed (are you sure you want to edit?). This is not super necessary though.