codalab / codalab-competitions

CodaLab Competitions

What is CodaLab?

CodaLab is an open-source web-based platform that enables researchers, developers, and data scientists to collaborate, with the goal of advancing research fields where machine learning and advanced computation is used. CodaLab helps to solve many common problems in the arena of data-oriented research through its online community where people can share worksheets and participate in competitions.

To see Codalab Competition's in action, visit codalab.lisn.fr.

Codabench, the next-gen of CodaLab Competitions, is out. Try it out!

Documentation

• CodaLab Wiki

Community

The CodaLab community forum is hosted on Google Groups.

• CodaLab Competitions Google Groups Forum

Quick installation (for Linux!)

To participate in competitions, or even organize your own competition, you don't need to install anything, you just need to sign in an instance of the platform (e.g. this one). If you wish to configure your own instance of CodaLab competitions, here are the instructions:

Install docker and add your user to the docker group, if you haven't already

$ wget -qO- https://get.docker.com/ | sh
$ sudo usermod -aG docker $USER

Clone this repo and get the default environment setup

$ git clone https://github.com/codalab/codalab-competitions
$ cd codalab-competitions
$ cp .env_sample .env
$ pip install docker-compose
$ docker-compose up -d

Now you should be able to access http://localhost/

More details on how to configure your own instance:

• Configure Codalab from scratch
• Set up data storage

License

Copyright (c) 2013-2015, The Outercurve Foundation. Copyright (c) 2016-2021, Université Paris-Saclay. This software is released under the Apache License 2.0 (the "License"); you may not use the software except in compliance with the License.

The text of the Apache License 2.0 can be found online at: http://www.opensource.org/licenses/apache2.0.php

Cite CodaLab Competitions in your research

@article{codalab_competitions_JMLR,
  author  = {Adrien Pavao and Isabelle Guyon and Anne-Catherine Letournel and Dinh-Tuan Tran and Xavier Baro and Hugo Jair Escalante and Sergio Escalera and Tyler Thomas and Zhen Xu},
  title   = {CodaLab Competitions: An Open Source Platform to Organize Scientific Challenges},
  journal = {Journal of Machine Learning Research},
  year    = {2023},
  volume  = {24},
  number  = {198},
  pages   = {1--6},
  url     = {http://jmlr.org/papers/v24/21-1436.html}
}

codalab-competitions's Issues

About Page: Glossary: Competition section – “…score these redictions…” should be “predictions.”

<53524>

About Page: Glossary: Competition section – “…score these redictions…” should be “predictions.”

Participant: Password Reset “Please contact us…” should be a link

<53501>

Participant: Password Reset “Please contact us…” should be a link

The Menu invoked when browser page is resized does not present menu items when clicked

<53508>

The Menu invoked when browser page is resized does not present menu items when clicked

Remove Bundles menu item from main navigation menu

The /bundle view should not be exposed in the main menu bar. Bundles are not an end-user facing concept. There is on-going work to define how experiments, datasets, programs and macros are exposed to end-users.

Competitions index returns 500 Internal Server Error

Full output:
"GET /competitions/partials/indexpage?%22page=1&per_page=6%22&=1374607422748 HTTP/1.1" 500 12843

It appears the URL structure is incorrect. Should there be a _partials folder ?

Git commit hash in app for reporting

We need to have a way to specify the version (git commit hash) of the application that a bug or feature request or failure corresponds to.

For auto-reporting/feedback we should have it configured during deployment, initially just a link on the page with the url/hashref is good enough.

Script to seed web site with objects and data

This is a two step process:

1. Need to fix current method asap:

   python .\manage.py loaddata .\fixtures\initial.json
   IntegrityError: Problem installing fixtures: The row in table 'web_contententity
   ' with primary key '1' has an invalid foreign key: web_contententity.container_i
   d contains a value '1' that does not have a corresponding value in web_contentco
   ntainer.id.

2. Need to provide a more flexible way to import data. E.g write a Python script to insert object instances instead of having to manually create objects in the admin view before exporting them to JSON. This task is to define and provide the flexible mechanism for inserting data. Once we have the mechanism others can help fill-up the script (e.g import sample competition; import datasets, programs or more generically bundles).

About Page: Chrome: Navigation list appears on the right side (instead of left)

<53516>

About Page: Chrome: Navigation list appears on the right side (instead of left)

ul.headerNavigation li.active a underbar doesn't work for each <li>

Design and implement footer

Finalize design of footer at the base of every web page and implement it.

Participate: Competitions: filter is not present for competitions list, as per F#

<53665>

Filter option for Competitions list is not present, as per F# prototype.

Create a competition

Need to update the competition creation to use foundation

Command to run tests

Provide basic test harness, instructions to run it and instructions to add a test.

Participant: User Name not immediately visible after sign in

<53389>

Participant: User Name not immediately visible after sign in

Footer: Feedback link is not working

<53544>

Footer: Feedback link is not working. Click the link in Footer and nothing happens.

Document progressive registration flow

Start a wiki page to document the intended registration flow.

For now the registration form will remain as is with e-mail, username and password as required field, and the requirement that username and e-mail must be unique in the system.

Footer: Terms of Use: Codalab should be CodaLab

<53564>

Codalab should be CodaLab (camel casing needed)

Participate: Sign in: Register <link> is missing, as per F#

<53667>

Register is missing in Python and present in F#.

Footer: Help : Create a competition page not showing content

http://screencast.com/t/4LruCo6YXlT

<53543>

Footer: Help page is missing. Click Help in Footer and nothing happens.

Footer: Notices: Codalab should be CodaLab

<53545>

Footer: Notices: Codalab should be CodaLab

Update to cl-test1-ubuntu.cloudapp.net fails

Update to cl-test1-ubuntu.cloudapp.net fails

(venv)azureuser@cl-test1-ubuntu:/codalab$ python codalab/manage.py syncdb
Traceback (most recent call last):
File "codalab/manage.py", line 11, in
from configurations.management import execute_from_command_line
File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/configurations/management.py", line 3, in
importer.install()
File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/configurations/importer.py", line 76, in install
commands = management.get_commands()
File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/django/core/management/init.py", line 109, in get_commands
apps = settings.INSTALLED_APPS
File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/django/conf/init.py", line 53, in getattr
self._setup(name)
File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/django/conf/init.py", line 48, in _setup
self._wrapped = Settings(settings_module)
File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/django/conf/init.py", line 134, in init
raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'codalab.settings' (Is it on sys.path?): No module named djcelery
(venv)azureuser@cl-test1-ubuntu:/codalab$

Participate: Home Page: Competitions and Experiments icons are not links.

<53386>

Home Page Competitions and Experiments icons are not links to their respective areas on the site.

Participant: Reset Password should be a pop out dialog

<53394>

Reset Password should be a pop out dialog. Currently occupies full width of browser.

Participant: Sign in: cursor focus should be on first required field of entry

<53392>

For Sign in, cursor focus should be on first required field of entry, with border on text field, as per F#.

Finish converting competition editor

Document branch management

Document process for maintaining branches targeted at active development and at upcoming releases.

Participate: Competition states are not clear, at-a-glance

<53387>

Participate: Competition states are not clear, at-a-glance when viewing the list

Notifications

We need to understand notifications in CodaLab. We need notifications for things:

1. User requested participation in a Competition
2. User's request was processed
3. New Data for a competition was posted
4. A phase for a competition became active (or inactive)
5. New competitions or experiments have been created

There are a set of signals that it makes sense to think of as "notifications". We should figure out the notification architecture once, then grind through the different ways we want to use it.

This will make the system feel richer and tie together the entire system into a single experience.

Deleting a competition doesn't have referential integrity

Deleting a competition from the delete button works.
When creating a new competition (after deleting one), the new competition id collides with the deleted competition id for all associated data that was not deleted when the competition was.

We need to ensure we delete all associated data when we delete the competition.

Steps to reproduce:

• Load up a fresh codalab following the instructions
• delete the competition via the ui
• run the competition.py script to recreate a competition (it will fail currently and shouldn't fail)

There should be real test for this that can be automated.

Search Competitions

We want the ability to have the global search bar back and have it search competitions.

https://trello.com/c/qWZKLXpb/18-browse-competitions

About Page: link to Get Help with creating a competition” goes to Page not found”

<53518>

About Page: link to Get Help with creating a competition” goes to Page not found”

Footer: Terms and Conditions AND Terms of Use seem to not both be necessary.

<53565>

Footer: Terms and Conditions AND Terms of Use seem to not both be necessary.

Participant: Sign in info occupies full width of browser

<53391>

Sign in info occupies full width of browser

Document process to deploy to Windows Azure

Current documentation needs to move to the GitHub project site and needs to include the setup for a SQL Azure database.

Participant: Sign in: error messaging for required fields is insufficiently discernible.

<53393>
Participant: Sign in: error messaging for required fields is insufficiently discernible, in Python version.

Complete sign-out process

This will make the completion of the sign-out process automatic (that is no need to click a button to confirm and as a result no need to style the page that is shown today).

Participate: Competition Timeline could indicate “current phase” more clearly

<44137>

Competition Timeline could indicate “current phase” more clearly
Note: A prompt to user would be good if they may not have noticed timeline Phase details.

About Page: Glossary: Experiment section – “An experimentcontains…” needs a space.

<53529>

About Page: Glossary: Experiment section – “An experimentcontains…” needs a space.

Participant: Password Reset text should read “Forgot your password?” instead of “forgotten…”

<53487>

Participant: Password Reset text should read “Forgot your password?” instead of “forgotten…”

Password Reset *required field error is insufficiently conspicuous

<53504>

Password Reset *required field error is insufficiently conspicuous

Finish converting competition delete from prototype

First Competition Django Script

I'm going make a new issue to specifically write a django scripting script to build out our first competition so that I can capture all the competition information in one place (and also see if it works for populating data).

Create User Profile

A user should have a minimal profile.

This is where a user can see their personal information (link a gravatar, social account, etc).

This is also where we can allow users to view information about other participants, the last item in this story: https://trello.com/c/romJnmlT/19-view-competitions

We should be able to view profiles and edit your own (very little right now) information.

List competitions

Update competition list for foundation

Participant: Signed in: “Profile” purpose is unclear and does nothing when clicked

<53390>

“Profile” purpose is unclear and does nothing when clicked

Permissions Review

We need to go through all of the actions and endpoints and make sure we're enforcing the access controls we intend.

Participate in a competition

Update participate in a competition for foundation

About Page: Glossary: Experiment section – IPython links to 404 error

<53537>

About Page: Glossary: Experiment section – IPython links to 404 error

Filter Competitions

We want to have the filter competitions functionality back on the competitions page.

https://trello.com/c/qWZKLXpb/18-browse-competitions

unable to update cl-test1-ubuntu...

update cl-test1-ubuntu.cloudapp.net fails

(venv)azureuser@cl-test1-ubuntu:/codalab$ python codalab/manage.py syncdb
Traceback (most recent call last):
File "codalab/manage.py", line 11, in
from configurations.management import execute_from_command_line
File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/configurations/management.py", line 3, in
importer.install()
File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/configurations/importer.py", line 76, in install
commands = management.get_commands()
File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/django/core/management/init.py", line 109, in get_commands
apps = settings.INSTALLED_APPS
File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/django/conf/init.py", line 53, in getattr
self._setup(name)
File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/django/conf/init.py", line 48, in _setup
self._wrapped = Settings(settings_module)
File "/home/azureuser/codalab/venv/local/lib/python2.7/site-packages/django/conf/init.py", line 134, in init
raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'codalab.settings' (Is it on sys.path?): No module named djcelery
(venv)azureuser@cl-test1-ubuntu:/codalab$