Registration & Communication

We ask all interested parties, including prospective participants, to subscribe to our spam-protected mailing list, where we will post updates a little more frequently than on the general task web site.  Access to the task data requires prior registration to this list, where we will make available a licensing template and download instructions.

Schedule (Revised in December 2014)

• Sunday, June 1, 2014: trial data available
• Tuesday, August 3, 2014: training data available
• Thursday, December 18, 2014: test data available
• Monday, December 22, 2014: last date to schedule the testing period
• Thursday, January 15, 2015: submission of system results

Access to the test data is limited to a period of six (6) days prior to the upload of system results.  We ask each team to let us know in advance when you want to start your individual five-day test period.  In other words, please send us a date between December 18, 2014 and January 10, 2015 on which you would like to gain access to the test data.

System Submissions

Submissions of system outputs must be made to the ‘official’ SemEval submission system (which will be opened to participants in December 2014).  This server is accessed through an individual, per-team upload address, which you will receive in email from the task organizers towards the end of the five-day evaulation period for your team (note that the latest possible submission deadline for this task is Thursday, January 15, 2015).

Our task has three target representations (DM, PAS, PSD), three tracks (closed, open, and gold), and two optional sub-tasks (predicate disambiguation and cross-linguistic variations).  Participants are expected to submit results for all three target representations, in-domain and out-of-domain, for English, but are free to submit to any of the three tracks, or to all of them, dependending on whether or not any data or tools were used in additional to the training semantic dependency graphs provided for this task.  Furthermore, participants are allowed to submit up to two runs for each target representation, language, and track, for example reflecting different parameterizations of their system.  For details on the difference between the three tracks and definition of different runs, please see the evaluation page.

Given the above parameters, each submission can contain between six and forty-four result files, all in the official tab-separated SDP file format as is documented on the data page.  To not have to deal with a large number of individual files, we ask that the complete submission be ‘packacked up’ in a single compressed zip archive before upload to the SemEval server.

We ask that you name result files uniformly using the scheme language‘.’domain‘.’track‘.’format‘.’run‘.sdp’, e.g. ‘en.ood.closed.dm.2.sdp’ for the second run using the DM target representation in the closed track for the English out-of-domain test data, or ‘cz.id.gold.pas.1.sdp’ for the first run with gold-track Chinese in-domain PSD outputs.

In addition to the ‘.sdp’ result files, each submission archive must included a README file that provides the following information:

• Team identifier (provided by SemEval organizers);
• Team member name(s) and affiliation(s);
• Designated contact person and email address;
• Inventory of results files included in the archive;
• System characteristics, including (if applicable):
  • core approach;
  • important features;
  • critical tools used;
  • data pre- or post-processing;
  • additional data used.
• Bibliographic references (if applicable).

To make our task in receiving and evaluating submisisons as smooth as possible, please be careful in implementing the above requirements and naming schemes.  The submission system makes it possible to re-submit results, and only the most recent file (submitted within the six-day evaluation period for each team) will be considered for evaluation.  As always, please do not hesitate to contact the task organizers (at the email address indicated in the right column) in case you require additional information or clarification.