Recent Updates


  • Jan 25, 2023 Evaluation phase is extended by 24 hours. This phase will end on February 1st 11:59 pm AOE.
  • Jan 25, 2023 Evaluation phase site with test data is available in Codalab.
  • Jan 18, 2023 Practice phase site on Codalab is open now.

Important Dates

* All deadlines are calculated at 11:59 pm
UTC-12 hours

Trial Data Ready Jul 15 (Fri), 2022
Training Data Ready Sep 30 (Fri), 2022
Evaluation Start Jan 25 (Wed), 2023
Evaluation End Feb 1 (Wed), 2023
System Description Paper Submission Due Feb 28 (Tue), 2023
Notification to Authors Mar 31 (Fri), 2023
Camera-ready Due Apr 21 (Fri), 2023
Workshop 13-14 July 2023 co-located with ACL

1. How to Participate

2. Training Data Format

Click here to download a small set of trial data in English.

We will follow the CoNLL format for the datasets. Here is an example data sample from the trial data.

.

In a data file, samples are separated by blank lines. Each data instance is tokenized and each line contains a single token with the associated label in the last (4th) column. Second and third columns (_) are ignored. Entities are labeled using the BIO scheme. That means, a token tagged as O is not part of an entity, B-X means the token is the first token of an X entity, I-X means the token is in the boundary (but not the first token) of an X type entity having multiple tokens. In the given example, the input text is:

the original ferrari daytona replica driven by don johnson in miami vice

The following image shows the entities as annotated. .

Here are some examples from the other languages.

3. Evaluation

In this shared task, we provide train/dev/test data for 12 languages. This codalab competition is in practice phase, where you are allowed to submit prediction file for dev sets. The evaluation framework is divided in two broad tracks.

  1. Multi-lingual: In this track, the participants have to train a single multi-lingual NER model using training data for all the languages. This model should be used to generate prediction files for each of the 12 languages’ evaluation (dev/test) set. That means the model should be able to handle monolingual data from any of the languages.
    Predictions from any mono-lingual model is not allowed in this track. Therefore, please do not submit predictions from mono-lingual models in this track..

  2. Mono-lingual: In this track, the participants have to train a model that works for only one language. For each language, there will be one dev/test set that contains examples for that particular language. Participants have to train a mono-lingual model for the language of their interest and use that to create prediction file for the evaluation set of that language.
    Predictions from any multi-lingual model is not allowed in this track.

4. Submission Instructions

The evaluation script is based on conlleval.pl.

4.1. Format of prediction file

The prediction file should follow CoNLL format but only contain tags. That means, each line contains only the predicted tags of the tokens and sentences are separated by a blank line. Make sure your tags in your prediction file are exactly aligned with the provide dev/test sets. For example,

4.2. Prepare submission files

Follow the below instructions to submit your prediction files for a track. Codalab requires all submissions in zip format

Some Resources for the Beginners in NLP

Communication