code-dig-ger / backend

This is the Backend open-source API developed to provide data to our Frontend React Website. Codedigger aims at accumulating the data of competitive programming platforms into one platform.

Codedigger Backend API

Table of Contents

1. Introduction
2. Getting Started
   1. Fork, clone locally and create a branch
   2. Setting Environment First Time
      1. Basic Requirements
      2. Creating Virtual Environment
      3. Installing Requirements
      4. Setting up Database and Environment File
      5. Migrating Database
      6. Create Superuser
   3. Starting Development Server
   4. Leaving the virtual environment
   5. Update requirements file
   6. Update Database
3. Code of Conduct
4. Contributing
5. License
6. How to Get Help

Introduction

This is the backdoor REST API developed to do complex calclations, provide and store data for Codedigger Website.

Codedigger aims at accumulating the data of competitive programming platforms (Codeforces, Codechef, Atcoder, SPOJ and UVA Online Judge) into one platform. We used their publicly available APIs and introduced several unique features into our application such as friends, mentors, ladders, upsolve, problem and contest filter. This can be the ultimate stop for everyone practicing competitive programming.

User can practice topicwise list to expertise any topic, levelwise list to practice based on rating and difficulty, or can create their own list and follow that. They can filter problems based on difficulty, tag, or solved by mentor from all the sites combined at one place.

Upsolving means to solve those problems, you're not able to solve during the contest. With the help of codedigger, you will be able to upsolve codeforces, codechef, and atcoder contests in an easy and better way.

Getting Started

Fork, clone locally & create a branch

Fork Codedigger Backend repository and clone at your local

git clone https://github.com/<yourGithubUsername>/Backend.git
cd Backend/

Create a branch with a descriptive name. A good branch name would be :

git checkout -b feature/AmazingFeature

git checkout -b bugfix/user

Steps to Follow (Recommended) -

If you're working on a issue or issue is assigned to you -

• Create a branch for issue by commenting /cib or /cib 2-3 word desc. on the issue.
• This will create a branch with descriptive name.
• You can push all your code related to issue in this branch.
• When you're done, create a merge PR to master branch.
• After review, your branch will be merged.

Setting Environment First Time

Basic Requirements

1. Python
2. pip

Creating Virtual Environment

A virtual environment is a tool that helps keep dependencies required and the project isolated. If you wish to install a new library and write

pip install name_of_library

on the terminal without activating an environment, all the packages will be installed globally which is not a good practice if you’re working with different projects on your computer.

If this sounds a bit complicated, don’t worry so much because a virtual environment is just a directory that will contain all the necessary files for our project to run.

Installing venv (required once)

Windows

py -m pip install --user virtualenv
py -m venv env

Linux

python3 -m pip install --user virtualenv
python3 -m venv env

You have to start virtual environment everytime you start new terminal -

Windows

Using gitbash

. env/Scripts/activate

Using Powershell

. env\Scripts\activate

Linux

source env/bin/activate

Installing Requirements

Windows

pip install -r requirements.txt

Linux

pip3 install -r requirements.txt

Setting up Database and Environment File

Requirement- MySQL

You can download MySQL from here or download XAMPP.

Start MySQL server and create a database.

Configuring Environment Variables

cd to codedigger directory, this is our working directory. For now we will run all command in this directory only -

cd codedigger/

Make environment file in codedigger directory by copying the example file -

cp codedigger/.env.example codedigger/.env

Database Settings

DB_NAME = codedigger
DB_USER = root
DB_PASSWORD = 
DB_HOST = 127.0.0.1
DB_PORT = 3306

GMAIL Settings

You need to turn on less secure app access for sending mails.

EMAIL_HOST_USER = your_mail_id
EMAIL_HOST_PASSWORD = your_password

Migrating Database

Windows

py manage.py migrate

Linux

python3 manage.py migrate

Create Superuser

Windows

py manage.py createsupeser

Linux

python3 manage.py createsupeser

Starting Development Server

Windows

py manage.py runserver

Linux

python3 manage.py runserver

Leaving the virtual environment

deactivate

Update requirements file (Critical)

If you have installed new dependency, the pip freeze command lists the third-party packages and versions installed in the environment.

Windows

pip freeze > requirements.txt

Linux

pip3 freeze > requirements.txt

Update Database

Everytime you change db models, you need to run makemigrations and migrate to update on database.

Windows

py manage.py makemigrations
py manage.py migrate

Linux

python3 manage.py makemigrations
python3 manage.py migrate

Code of Conduct

This project and everyone participating in it is governed by the Codedigger Code of Conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to [email protected].

Contributing

We encourage you to participate in this open source project. We love Pull Requests, Bug Reports, ideas, (security) code reviews or any other kind of positive contribution. Please review CONTRIBUTING.md for details on how to file an issue or pr.

License

Copyright 2021 Codedigger

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

How to Get Help

Email us - [email protected]

Join! our Discord Community

Back to top

backend's Issues

Implement user_rating function of Codeforces API.

In the codedigger/codeforces/api.py, implement user_rating function which calls user.rating method of Codeforces API.

Also, add a test case in codedigger/codeforces/tests/test_api.py to test the same.

feat: Editor Table to store which user has access to edit which lists

User can add their friends and give them edit access.

This way many students can edit a list and store the problems in that common list.

Editor Model

User: Foreign Key
List: Foreign Key

Routes to add/ modify -

• Owner of the list can add members/ other users to edit a list
• Now added Users have the right to add problems to that list.
  Route : /lists​/userlist​/add #36
• Users can view their own lists and shared lists.
  Route : /lists/user/:username #38

bugfix: Problem Filter

If anyone is filtering Problems based on Rating,

First check Problem difficulty shouldn't be None.

Also in response serializer, if Problem Difficulty is None, make rating None too.

This is to save random ratings not to come after the filter.

[Refactor]: User Views File

In the codedigger/user/views.py -

Many times we used hardcoded BAD REQUEST response, instead, we should use exceptions for that -

Example -

return Response({'status': 'Failed','error' :'Email has not been provided'}, status=status.HTTP_400_BAD_REQUEST)

a better way to do this

raise ValidationException('Email has not been provided')

Both are one-liners, but later one has some meaning and is nice looking.

There are almost 38 lines in this view which has to be updated like this.

Route to get Codeforces Contest Submissions

Create a new Route to get Codeforces Contest Submissions

/contest/codeforces/{handle}/{contestId}/submissions

Return contest Object with Submissions of that contest.

The same dictionary that was returned by Codeforces API.

refactor: Atcoder

Initialize a new app for Atcoder.

Move all the scrapers and API calls to the new app Atcoder.

All the routes and functions related to Atcoder would move to Atcoder App.

This is to minimize errors and can extend the functionality of the current project.

Site certificate seems to have expired

Please do the needful.

Change Route name for checking Problem in Contest

Change route name - /contest/codeforces/{handle}/{probId}.

Update in production.

feat: team

Initialize a new app for Team formation.

Create Models based on ER below -

Many to many relationships between User Model and Team Model.

Routes to be created -

• CRUD Operation for Team. Create, Read, Update and Delete a team.
• User can invite friends to join. Send a mail with an invitation link.
• User can join the team by invitation code/ link.

Important function -

• Get a list of users in a team where is_invited = False i.e. they have joined the team.
• Get a list of invited users where is_invited = True.

feat: a route to get all lists of a user

Route -
/lists/user/:username

With the help of this route, GET all the public lists of a user whose username = provided username

If provided username = authenticated user:
i.e. a user is accessing his own lists - all public/ private lists of that user will be sent in the response.

Use Reference :
​
/lists​/userlist​/

codedigger.lists.views.UserlistGetView

bug: Codeforces Updater for List.

Create a new route to update complete Codefores submissions and save them in Solved Model.

Check if we are doing the same for other platforms as well, if yes include them too.

refactor: SPOJ

Refactoring SPOJ related code.

• Refactor all scrapers and paste them into the scraper.py file.
• Refactor all scheduled functions and paste them into the cron.py file.
• Refactor list updater for SPOJ and move it into the SPOJ app.

Some points -

1. scraper.py file must contain code to scrap the SPOJ website and get the content from it.
2. Divide code for update and save the database in other files e.g. model_utils in Codeforces app.
3. cron.py must contain all the functions that we want to schedule.

codeforces wrong submission

Currently, if the submission verdict is not OK, we are considering that submission to be wrong, add one more case the submission should pass the first test case to be counted in wrong submissions. Codeforces don't count submissions who were wrong on the first TC for the penalty.

File - codedigger/codeforces/api_utils.py
Function - is_verdict_wrong

feat: Add a route to Search Codeforces User from the database.

Would like to add a route where we can get all the list of users matching the given query.

Example route -

/codeforces/search?q=tou

GET request to this route would provide me a list of matching Codeforces users (atmost 5 most similar)

Example response -

{
    "status": "OK",
    "result": [
        {
            "name": "Gennady",
            "handle": "tourist",
            "profile" : "https://userpic.codeforces.org/422/title/50a270ed4a722867.jpg"
        },
        {
            "name": "Tourist",
            "handle": "touristt",
            "profile" : "https://userpic.codeforces.org/no-title.jpg"
        }
    ]
}

Search users from our database, not by using API.
We are storing all the codeforces users in our database only codedigger.codeforces.models.user.
Query to the database to get matching users.

Add a test case to test your route.

feat: Add description field while storing problem in a user list.

While we are storing a problem to a problem list.

Currently we POST request to /lists/userlist/add url to add a problem to a particular list.

Now, this accepts only - prob_id, platform, and a slug of the list.

add another field description, so that we can store the description also along with ProblemID and ListID.

Function to be update - codedigger.lists.view.UserlistAddProblemView

Enhancement: Rating Change Reminder Email HTML CSS

Codedigger sends an email whenever participants participate in a Codeforces contest and rating changes.

It would be good if we can pretty the email using CSS or some animations. More ideas are welcomed.

Current Email Format -

Sample Context Data (to be feed to HTML)-
See Doc see how to use templates in Django.

{
    "rating_change": {
        "contestId": 1540,
        "contestName": "Codeforces Round #728 (Div. 1)",
        "handle": "shivamsinghal1012",
        "rank": 20,
        "ratingUpdateTimeSeconds": 1624643400,
        "oldRating": 1800,
        "newRating": 2000,
        "oldRank": "expert",
        "newRank": "candidate master",
        "oldcolor": "user-blue",
        "newcolor": "user-violet",
        "isoldlegendary": False,
        "isnewlegendary": False,
    },
    "cdata": {
        "contest": {
            "name": "Codeforces Round #728 (Div. 1)",
            "contestId": 1540,
            "duration": 8100,
            "startTime": 1624635300,
            "Type": "R",
            "participants": 1500,
        },
        "worldRank": 20,
        "countryRank": 2,
        "organizationRank": 1,
        "totalCountryParticipants": 100,
        "totalOrganizationParticipants": 20,
    },
}

You can assume, you will get the data, but only need to update codedigger/templates/codeforces/rating_reminder.html.

Requied - HTML, CSS knowledge.

To run your code at local -

Pull feat/codeforces/rating_reminder branch.

Run server as mentioned in the README.md file.

Visit http://127.0.0.1:8000/codeforces/rating-change-email

The codedigger/templates/codeforces/rating_reminder.html template is rendered from the codedigger/codeforces/views.py file rating_change_email function.

EMAIL Best Practice

1. Make sure your HTML email is responsive for different screen sizes and devices.
2. Make sure your styling works for different email clients.
3. Be conscious of how long your HTML emails take to load.

Documentation: search user route

Create a documentation page for the search-user route.

Write Contributing Guidelines in CONTRIBUTING.md File

A CONTRIBUTING file tells our audience how to participate in your project.
For example, you might include information on:

• How to file a bug report (try using issue and pull request templates)
• How to suggest a new feature
• How to set up your environment and run tests

In addition to technical details, a CONTRIBUTING file is an opportunity to communicate your expectations for contributions, such as:

• The types of contributions you’re looking for
• Your roadmap or vision for the project
• How contributors should (or should not) get in touch with you

Using a warm, friendly tone and offering specific suggestions for contributions (such as writing documentation, or making a website) can go a long way in making newcomers feel welcomed and excited to participate.

In the earliest stages of your project, your CONTRIBUTING file can be simple. You should always explain how to report bugs or file issues, and any technical requirements (like tests) to make a contribution.

Template Contributing File
More about -

• 1
• 2

Example

Docs and Test Case for Codechef Upsolve Route

Do not store Codechef User as it is not required.

• Write needed test case for CodeChef Upsolve route.
• Write Documentation for CodeChef Upsolve route

Optimize Github workflow autoyapf

Introduction

autoyapf is a GitHub workflow used to automatically reformatting the python code to PEP8 format using yapf whenever a pull request to master is initiated.

Current Behaviour

Currently, it checks and reformats all the '.py' files in the repository but this can be optimize by just checking which file has been updated and only reformat those files.

Expected Behaviour

Optimize autoyapf.yml file so that it only reformat those files in PR which is updated by the contributor in the following PR.

Assign Problems to Codeforces Contest

Assign Problems to Codeforces Contest for Extension.

Based on User Rating
Select 5 Problems -

Steps -
Filter Problems of Codeforces and Regular Contest

Filter and Exclude Problems that were solved by User

Select Problems in range +-200.

feat: create a function to filter problems

Create a function -
if we give param "COOK", "LONG", "LUNCH" or "START" to that function
it should return problems to a particular contest.

E.g. - Given "COOK" and "LONG" to function - it should return Problems Model Object List containing problems to Long Challenge and Cook-Off.

Steps -
Filter Contests using some filter.
Filter Problems that are in that contests.

feat : GET stats to a list

Route -
/lists/:listSlug/stats

With the help of this route, GET all the stats of list whose listSlug = provided listSlug

The provided listSlug must be public if authenticated user != owner
Stats are visible on public lists only.

1. Standing Page Route /lists/:listSlug/stats/standing - Standing based on the Problem Solved of this list, you will get a rank.
2. Friends Standing Page Route /lists/:listSlug/stats/standing?friend=true - Friends Standing and Problems Solved by Friends.

change url pattern for search codeforces user.

Current url -

path('search', views.SearchUser.as_view(), name="search-user")

to

path('users', views.SearchUser.as_view(), name="codeforces-users")

feat: Codechef Routes

Developing Following Routes for Codechef App -

• Get the list of Contests based on a certain filter
  Example - Future, Present, Practice, Past or Starter, Long, CookOff, etc.
• For a ContestID, Get Problem List in that Contest.
• For a CodechefHandle, Get the complete Profile of that User.
• For a CodechefHandle, Get the contests given by the user.
• For a CodechefHandle, Get the problems solved by that user.
• For a CodechefHandle, ProblemCode, Get the submission result.
• For a CodechefHandle, Recent Submission of a User provided in Profile Page.

bugfix: User is able to add problems to other user list.

Description
I as a user am able to add problems to any other list provided slug and problemID.

To Reproduce
Authenticate yourself.

Send a response to url /lists/userlist/add with POST data

{
  "prob_id": "TEST",
  "slug": "to-do-2505",
  "platform": "C"
}

Expected behavior
If I am the owner of the list, this problem is successfully added, but if I am not the owner of this list, it should throw an error.

Actual behavior
Problem is added to the list irrespective of the owner. Users can add problems to others lists.

Function Responsible for this - codedigger.lists.views.UserlistAddProblemView

feat: Enrolled Model to store user has enrolled to which lists

Users can enroll themselves in any public lists and solve the problems from that list and can track their progress.

Enrolled Model

User: Foreign Key
List: Foreign Key

Routes :

• User can enroll in any Public List/ Self List.
• User can view all the enrolled lists.

feat: More filter in Problem Filter

Some more filter Ideas Problem can be filter in -

• Based on Problem Index
• Based on Contest
• Based on Wrong Submission ( Problem that has a wrong submission and not solved yet )

feat: a route to view all problems of a public list

Route -
/lists/:listSlug/problems

With the help of this route, GET all the problems with pagination (similar to admin practice list) of a list whose listSlug = provided listSlug

The provided listSlug must be public if authenticated user != owner

If provided listSlug Owner = authenticated user:
i.e. a user is accessing his own list - then private lists are allowed. (User has permission to view).

Use Reference :
​
GET /lists​/userlist​/edit​/{slug}

codedigger.lists.views.EditUserlistView.get

or

codedigger.lists.views.LevelwiseRetrieveView

feat: Problem Filter Ideas

Problems to Upsolve (i.e. Problems not solved in Contest)

UVa - Filter Problems from CP3 Book.

Important Tags

Atcoder Contest - Begineer, Grand, Regular, Educational DP

SPOJ - Classical, Challenge, Partial, Tutorial, Riddle, Basics

Improvements -

Set per_page minimum bound 5.

[Refactor]: Upsolve Functions (Problem View)

As you know we are providing upsolve contests for 3 sites - Codeforces, Codechef, Atcoder. We hardcoded return responses for all 3 differently again and again (3 times). We want to refactor and make a function that can do the same work by calling the function.
Codeforces Response

    return Response({
            'status' : 'OK' , 
            'result' : UpsolveContestSerializer(c, many =True, context = data).data, 
            'links' : {
                'first' :  path + 'page=1',
                'last' :  path + 'page='+str(NumPage),
                'prev' :  Prev,
                'next' :  Next
            },
            'meta' : {
                'current_page' : page,
                'from' : (page-1)*10 + 1,
                'last_page' : NumPage,
                'path' : request.build_absolute_uri('/problems/upsolve/codeforces'),
                'per_page' :  10,
                'to' :  page*10,
                'total' :  total
            }
        })

Codechef Response

    return Response({
            'status' :  'OK' , 
            'result' :  user_contest_details ,
            'links' : {
                'first' : path + 'page=1',
                'last' :  path + 'page='+str(NumPage),
                'prev' :  Prev,
                'next' :  Next
            },

            'meta' : {
                'current_page' :  page,
                'from' : (page-1)*10 + 1,
                'last_page' : NumPage,
                'path' : request.build_absolute_uri('/problems/upsolve/codechef'),
                'per_page' : 10,
                'to' : page*10,
                'total' :  total
            }
        
        })

Same Atcoder.

Make a function in codedigger/problem/utils.py file which can do the same.
See #7 commit for reference.

Sample example -

def get_upsolve_response_dict(some_param):
    # do some calculation 
    # return response same for all 3 
    return Response({
            'status' :  'OK' , 
            'result' :  user_contest_details ,
            'links' : {
                'first' : path + 'page=1',
                'last' :  path + 'page='+str(NumPage),
                'prev' :  Prev,
                'next' :  Next
            },

            'meta' : {
                'current_page' :  page,
                'from' : (page-1)*10 + 1,
                'last_page' : NumPage,
                'path' : request.build_absolute_uri('/problems/upsolve/<different>'),
                'per_page' : 10,
                'to' : page*10,
                'total' :  total
            }
        
        })

feat: route to view all public lists with suitable search parameters

Route: GET lists

Root route to GET all public lists provided the search parameters.

Example - lists?q=dp this will search all lists whose name has "dp" as a subsequence.

Sample Response -

{
  "status" : "OK",
  "result"  :  
  [
    {
      "name" : "My DP Problem List",
      "slug" : "my-dp-problem-list-46",
      "description" : "This List contains dp problems",
      "owner" : "owner-46" 
    },
    {
      "name" : "Best DP Problems",
      "slug" : "best-dp-problems-56",
      "description" : "This List contains best dp problems",
      "owner" : "owner-56" 
    }
  ]
}

Docs: List Standing route documentation

Create a doc for list standing route.

feat: Codeforces Profile Stat

Additional Codeforces Stat

From User status API

• Problem Solved in Each Rating ( Regular Contest )
• Problem Solved in Each Index i.e. A, B, C ( Regular Contest )
• Problem Solved in Each Index i.e. A, B, C ( GYM Contest )
• Number of Problem Upsolved
• Problem Solved in Contest
• Number of Correct Submission
• Number of Incorrect Submission
  IMP - Do not count submission that fails on TC1 or Compilation Error. Tc > 1.
  Do not count the submission of that problem who have not submitted correctly.
  That will be the accuracy or an average number of attempts the user made to submit a problem.

Codechef Problem Scraper and Profile Scraper

In 2021, Codechef introduced Div3 and a new contest and long challenge are not rated for everyone.

This is causing a problem in our scraping the contest and problems which is a major issue in upsolve section of CodeChef.
New contests are not available in CodeChef Upsolving due to this issue.

Review the problem scraper codedigger/problem/scraper/codechef.py and how we are currently storing the CodeChef problems. Think of a better solution to this.

Also review the codechef profile card - API Response and Frontend

If the profile card needs some new change, do the needful changes in the codedigger/user/profile.py file.

Expectation -
Correctly store all the problems with known ContestId of each problem.
For example, if a problem is available on Div1, Div2, and Div3 - we should store that in our database. And use wherever needed (mainly in upsolve).

Is the repo dead? not able to configure it on my local

Describe the bug
I am trying to run the installation commands in README but not able to configure it on my local machine.

To Reproduce
installed dependencies from requirements.txt but getting error from deps like djangorestframework-simplejwt, ruamel.yaml==0.16.12, ruamel.yaml.clib==0.2.2 and typed-ast==1.4.2

• for simplejwt it was incorrect version so updated that and others I removed it from requirement txt

• after that everything got installed but when I was running the project it was not mentioned we need to create DJANGO_SETTINGS_MODULE I tried to create but getting error as django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

Expected behavior
I was expected it to run smoothly but still not working

Additional context

• also for mysql_config it is not mentioned in readme that we need add libmysqlclient-dev. I added that and resolved the issue with mysql config
• for this command to cp codedigger/.env.example codedigger/.env there's no such file provided in the repo.
• Also in readme there's typo correct command to create super user is python manage.py createsuperuser

test: Owner can change their private lists to public.

Route : PUT /lists/userlist/edit/{slug}

Testing all requirements and the needful changes it required -

• Given a slug, the Owner is successfully able to change their Private list to Public.
• Restrict owner to change public lists to private
• Only the owner is able to edit this list, no other user can change the details of the list.

Code Path - codedigger.lists.views.EditUserlistView.put

[Refactor]: Efficient Use of Codeforces API

We are using Codeforces API. Currently, they are hardcoded, but this is not a good practice.

Now you know all the places where we are requesting Codeforces API. We want to make a separate file and functions in that file that we can import anywhere we want.

Current Situation -
In the file codedigger/lists/solved_update.py

url = 'https://codeforces.com/api/user.status?handle=' + str(cf_handle)
res = requests.get(url)
if res.status_code != 200 : 
    return 
req = res.json()
if req['status'] != 'OK':
    return

Again we are using that code two times in codedigger/problem/views.py

res = requests.get("https://codeforces.com/api/user.status?handle="+student)
if res.status_code!=200:
    return Response({'status' : 'FAILED' , 'error' : 'Codeforces API not working'},status = status.HTTP_400_BAD_REQUEST)
res=res.json() 
if res['status']!="OK":
    return  Response({'status' : 'FAILED' , 'error' : 'Codeforces API not working'},status = status.HTTP_400_BAD_REQUEST) 

Also in codedigger/problem/utils.py

url = "https://codeforces.com/api/user.status?handle="+handle
res = requests.get(url)
if res.status_code != 200:
	return (RContest , VContest , SolvedInContest , Upsolved , Wrong)
data = res.json()
if data['status'] != 'OK' : 
	return (RContest , VContest , SolvedInContest , Upsolved , Wrong)

Instead of all these, we want only one function which does this task.

Steps to follow -

1. Make a file codedigger/codeforces/api_utils.py
   In this file we are going to store, all the required functions for Codeforces API.

2. Make a function user_status(handle)
   This function will get data from Codeforces API. Submission of User or User Status and return.

Best way to define this function -

def user_status(handle):
	url = "https://codeforces.com/api/user.status?handle="+str(handle)
	res = requests.get(url)
	if res.status_code != 200:
		raise ValidationException('Codeforces API is not working')
	data = res.json()
	if data['status'] != 'OK' : 
		raise ValidationException('Codeforces API is not working')
	return data

Now to use this function, need to call using try exception

try: 
	data = user_status(handle)
except ValidationException: 
	# Do whatever is required when we get the exception
	# return or return some value etc 

This way we design functions for all the routes we use of Codeforces API