Registration & Communication

We ask all interested parties, including prospective participants, to subscribe to our spam-protected mailing list, where we will post updates a little more frequently than on the general task web site.  Access to the task data requires prior registration to this list, where we will make available a licensing template and download instructions.

Schedule (Tentative for 2014)

• Monday, November 4, 2013: trial data available;
• Friday, December 13, 2013: training data available;
• Monday, March 24, 2014: release of test data;
• Sunday, March 30, 2014: submission of system results;
• Thursday, May 15, 2014: paper submission;
• Tuesday, June 10, 2014: reviewing results;
• Monday, June 30, 2014 (tbc): camera-ready submission.

System Submissions

Submissions of system outputs must be made to the ‘official’ SemEval submission system, i.e. through ftp(1) upload to the server at ‘alt.qcri.org’.  This server requires a user name and password to connect; all registered participants have received this information directly from the SemEval 2014 organizers (Preslav Nakov and Torsten Zesch).  If you do not have this information, please contact the SemEval 2014 organizers immediately (and before the final submission deadline for this task, which is Sunday, March 30, 2014).

Our task has three target formats (DM, PAS, PCEDT) and two tracks (closed and open); participants are expected to submit results for all three target formats, but are free to submit to either of the two tracks, or to both of them (dependencing on whether or not any data or tools were used in additional to the training semantic dependency graphs provided for this task).  Furthermore, participants are allowed to submit up to two runs for each target format and task, for example reflecting different parameterizations of their system.  For details on the difference between the two tracks and definition of different runs, please see the evaluation page.

Given the above parameters, each submission can contain between three and twelve (3 × 2 × 2) result files, all in the official tab-separated SDP file format as is documented on the data page.  To not have to deal with a large number of individual files, we ask that the complete submission be ‘packacked up’ in a single compressed archive (using ‘.zip’ or ‘.tgz’ archivers) before upload to the SemEval server; we are aware that this request deviates slightly from the instructions you have received from the SemEval organizers, but we have obtained their approval to implement a task-specific simpler scheme, i.e. using a single archive submitted for each participating team.  We ask that you name result files uniformly, using the following scheme track ‘.’ format ‘.’ run ‘.sdp’, e.g. ‘closed.dm.2.sdp’ for the second run using the DM target format in the closed track, or ‘open.pcedt.1.sdp’ for the first run with open-track PCEDT outputs.

The name of the compressed archive that is uploaded should contain the team identifier, as assigned to you in the confirmation email of your registration from the SemEval organizers.  If possible, please do not make multiple submissions for the same team, but in case you have to re-upload a set of system results (within the submission deadline), please add a suffix to the archive name: if your team identifier were GT, for example, and you had already uploaded a first set of results as ‘gt.zip’, then please use an archive name like ‘gt_1.zip’ for the re-submission.

In addition to the ‘.sdp’ result files, each submission archive must included a README file that provides the following information:

1. Team identifier (provided by SemEval organizers);
2. Team member name(s) and affiliation(s);
3. Designated contact person and email address;
4. Inventory of results files included in the archive;
5. System characteristics, including (if applicable):
   • core approach;
   • important features;
   • critical tools used;
   • data pre- or post-processing;
   • additional data used.
6. Bibliographic references (if applicable).

To make our task in receiving and evaluating submisisons as smooth as possible, please be careful in implementing the above requirements and naming schemes.  As always, please do not hesitate to contact the task organizers (at the email address indicated in the right column) in case you require additional information or clarification.