Skip to content

Commit 3398e3f

Browse files
committed
Update website.
1 parent db5e473 commit 3398e3f

File tree

1 file changed

+2
-5
lines changed

1 file changed

+2
-5
lines changed

challenges/index.md

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -21,17 +21,14 @@ Typically a challenge consists of a dataset and a task that should be performed
2121
For a problem to become a challenge, the following in needed:
2222

2323
* The challenge must define a <b>task</b> and a <b>metric</b> that is optimized.
24+
* The metric to optimize is <b>challenging</b>: the state-of-the-art does not solve the problem for all practical purposes and improvements are needed.
2425
* A <b>training dataset</b> is publicly available.
2526
* An <b>evaluation dataset</b> is publicly available.
27+
* The data should be <b>accessible</b>: using proprietary tools, writing emails or signing-up for services just to obtain the challenge or the data is not acceptable.
2628
* A work in a research paper tried to solve (a variant of) the problem defined in the challenge.
27-
* It should be possible to create a simple stupid baseline solution without downloading some existing tools, virtual machines or other research prototype systems solving the problem in the task.
28-
* The metric to optimize is <b>challenging</b>: the state-of-the-art does not solve the problem for all practical purposes and improvements are needed.
2929

3030
Ideally a challenge is defined by its description, training and evaluation data. In case reading the data requires some special parser (or a database), this parser should be freely available. If you want to add a challenge, create a pull request <a href="https://github.com/learnbigcode/learnbigcode.github.io/tree/master/challenges">here</a>.
3131

32-
List of current challenges:
33-
34-
3532
<div class="highlightitem">
3633
<h2>Candidates</h2>
3734

0 commit comments

Comments
 (0)