Submissions

Submission Runs

Each group that has registered for the shared task is allowed to submit up to 3 different runs per subtask. Groups that registered for both tasks therefore can submit up to 6 runs. Submitting multiple runs, however, is not mandatory. It is perfectly sufficient to just submit one run per subtask.

 

Filename Format of Submission Runs

The names of the files to represent your different runs should follow strict naming conventions. Each filename should consist of three components. The format is:

 

<group-id>_<subtask>_<run-number>.txt

 

<group-id>: we only accept alphabetic characters from the English alphabet. Do not use any special characters (not even Umlaute or similar characters.). No spaces are allowed as part of the filename. (For the sake of better readability you may use uppercase and lowercase letters.)

Note that this id will be used in the official documents publishing the results of the shared task, so we encourage you to think of an id by which one can easily identify the affiliation of your group.

 

<subtask>: either coarse for the coarse-grained task and fine for the fine-grained task. Please use exactly this spelling. It must be lowercase!

 

<run-number>: a single digit to indicate your run number, i.e. 1, 2 or 3.

 

Here is an example for a valid file format:

 

unisaar_coarse_1.txt

unisaar_coarse_2.txt

unisaar_coarse_3.txt

unisaar_fine_1.txt

unisaar_fine_2.txt

unisaar_fine_3.txt

 

File Format

The files should have the same format as that of the labeled data files (i.e. trial data, training data). For further details, please go to the Format section.

 

How to Submit the Submission Files of Your Runs

Send your file(s) to the organizers:

We will confirm any submission that we receive. If you do not obtain any confirmation email upon your submission within one day, please contact the organizers again. We encourage you to send this enquiry without any attachment.

As a further safety precaution, we will publish a list of the names of the runs we received by the submission deadline. You should double check that your runs are included on that list.

We will only consider submission files that conform to the specifications of the shared task. Please check in advance whether the format of your file is correct.

We recommend you use the given evaluation tool (it runs format checks prior to the computation of the evaluation scores).

Simply create a dummy gold standard file that uses randomly generated labels.

 

Important — Please Read!

Even if you submit runs that comply with the format restrictions, in order to appear in the official publication of the results, you will have to complete a survey and submit a participation paper.
You can find some information on the survey here.
Further information on the format of participation paper and its submission can be found here.

We have some limited quality restrictions regarding the participation paper. If you do not follow them, we will reject your submitted participation paper. This would also mean that we will not include you in the official publication of the results