royceniu / challenge-howto

Challenge HOW-TO

How To Participate

Read the important information

• Read the information from our official website MMSys Website carefully.

Training your model

• You should design an algorithm to predict the bandwidth. We provide a Gym to you and here is a reinforcement learning example to demonstrate how to design a bandwidth estimator model by the Gym

Prepare your submission

You should submit your model and paper to participate the challenge. Please refer the MMSys Website.

Model submission

• You should convert your model or algorithm for AlphaRTC to a PyInfer instance. Here is a tiny example of acceptable submission. Meanwhile, you can verify the validation of your model following this section. You implementation will run in the Challenge-Environment that we pre-installed some popular third-parties library in this environment.

• You should compress all materials of your bandwidth estimator as a zip package. Here is an valid submission example.

• Submit your materials into our OpenNetLab platform. We provide online evaluation before the deadline.

Notes

• Please use the first author's email address to login this system.
• We only store your last submission for evaluation.
• We limit the number of submissions for each participant to 3 times per day.

Paper submission

Please refer the MMSys Website.

Evaluation System

We will provide two stages of the evaluation on OpenNetLab, online evaluation and offline evaluation. The goals of the evaluations are different.

Online evaluation

When a participant submit a zip, we check the basic functions of the submissions to ensure every submission can work in the offline evaluation. Each submission is scheduled to one pair of the servers randomly. The scores from the online evaluation are only for references.

Offline evaluation

We will start the offline evaluation after the final deadline. All submissions will be tested in different scenarios (e.g. networks and locations). Then we calculate the final cores and rank the submissions.

Contact

• Google Group
• Slack

Important Resources

• AlphaRTC: https://github.com/OpenNetLab/AlphaRTC
• AlphaRTC Gym: https://github.com/OpenNetLab/gym
• AlphaRTC Gym-example: https://github.com/OpenNetLab/gym-example
• Challenge Runtime Environment: https://github.com/OpenNetLab/Challenge-Environment
• Challenge Submission Example: https://github.com/OpenNetLab/Challenge-Example