| layout | page | |||
|---|---|---|---|---|
| title | Challenges | |||
| description | Big Code Challenges | |||
| tags |
|
|||
| comments | false | |||
| share | false |
The goal of challenges is for the research community to define important problems on which multiple solutions should appear. Most challenges already include some system as a baseline and the goal is to produce a better working (e.g. more precise) system. Challenges should be self-contained -- it should take a few hours for a newcomer to implement a naive solution to a challenge.
For a problem to become a challenge, the following is needed:
- The challenge must define a task and a metric to optimize.
- The metric to optimize is challenging: the state-of-the-art does not fully solve the problem for all practical purposes and improvements are needed.
- A training dataset and an evaluation dataset is publicly available. The data should be accessible: using proprietary tools, writing emails or signing-up for services just to obtain the challenge or the data is not acceptable.
- A work in a research paper tried to solve (a variant of) the problem defined in the challenge.
Ideally a challenge is defined by its description, training and evaluation data. In case reading the data requires some special parser (or a database), this parser should be freely available. If you want to add a challenge, create a pull request here.
At this time, no challenge is ready to satisfy all criteria. Have a look at the challenges that are not yet well defined.
Some challenge candidates that do not yet satisfy the requirements