Broad-Coverage Semantic Dependency Parsing

Motivation

Syntactic dependency parsing has seen great advances in the past decade, in part owing to relatively broad consensus on target representations, and in part reflecting the successful execution of a series of CoNLL shared tasks.  From this very active research area accurate and efficient syntactic parsers have developed for a wide range of natural languages.  However, the predominant target representation in dependency parsing to date are trees, in the formal sense that every node in the dependency graph is reachable from a distinguished root node by exactly one directed path.  This assumption is an essential prerequisite for both the parsing algorithms and the machine learning methods in state-of-the-art syntactic dependency parsers.  Unfortunately, this means that these parsers are ill-suited for producing meaning representations, i.e. moving from the analysis of grammatical structure to sentence semantics.  Even if syntactic parsing arguably can be limited to tree structures, this is obviously not the case in semantic analysis, where a node will often be the argument of multiple predicates (i.e. have more than one incoming arc), and it will often be desirable to leave some nodes unattached (with no incoming arcs), for semantically vacuous classes as, for example, particles, complementizers, or relative pronouns.

Thus, this task seeks to stimulate the dependency parsing community to move towards more general graph processing, to thus enable semantic dependency parsing, i.e. a more direct analysis of ‘who did what to whom’.  We attach four sample semantic dependency graphs (as a PDF file), demonstrating the target representations we want to employ for the WSJ sentence:

A similar technique is almost impossible to apply to other crops, such as cotton, soybeans, and rice.

Here, ‘technique’ for example, is the argument of at least the determiner (as the quantificational locus), the modifier ‘similar’, and the predicate ‘apply’.  Conversely, the predicative copula, infinitival ‘to’, and the particle marking the deep object of ‘apply’ arguably have no semantic contribution of their own.  Besides calling for node re-entrancies and partial connectivity, an adequate representation of semantic dependencies will likely also exhibit higher degrees of non-projectivity than typical syntactic dependency trees.

Besides the relation to syntactic dependency parsing, the proposed task also has some overlap with Semantic Role Labeling (SRL).  In much previous work, however, target representations typically draw on resources like PropBank and NomBank, which are limited to argument identification and labeling for verbal and nominal predicates.  A plethora of semantic phenomena, e.g. negation and other scopal embedding, comparatives, possessives, various types of modification, and even conjunction, typically remain unanalyzed in SRL.  Thus, target representations are partial to a degree that can prohibit semantic downstream processing, for example inference-based techniques.  In this task, we require parsers to identify all semantic dependencies, i.e. compute a representation that integrates all content words in one structure.  Nevertheless, we anticipate that relatively straightforward adaptations of existing SRL approaches can be applied to yield broad-coverage semantic dependency parsing.

In recent years, we see beginning research into parsing with graph-structured representations, for example Sagae & Tsujii (2008), Das, et al. (2010), Jones, et al. (2013), Chiang, et al. (2013), and Henderson, et al. (2013).  However, some of these studies are purely theoretical, others limited to smaller, non-standard data sets.  We anticipate an increase in interest for this line of research, as well as emerging resources that can mature into broadly accepted target representations of semantic dependencies.

For these reasons, we expect that a SemEval 2014 task would be a good vehicle to pull together, better understand, and make more widely accessible candidate target annotations, as well as to energize and synchronize emerging work on algorithms and statistical models for parsing into these types of more semantic representations.

Training and Testing Data

For English, we are aware of three independent annotations over the venerable WSJ text underlying the Penn Treebank (PTB) that have the formal and linguistic properties we are looking for:

These resources constitute parallel semantic annotations over the same common text, but to date they have not been related to each other and, actually, have hardly been used for training and testing of data-driven analyzers.  As is evident in our running example, showing the DM, PAS, and PCEDT semantic dependency graphs for the sentence above, there are contentful differences among these annotations, and there is of course not one obvious (or even objective) truth.  For this task, we will synchronize  these resources at the sentence and tokenization levels (making sure they all annotate the exact same text), for approximately 750,000 annotated tokens in the WSJ domain.  More background on the linguistic characterization of these representations as well as on the task data format is available through separate pages.

Contact Info

Organizers

  • Dan Flickinger
  • Jan Hajič
  • Marco Kuhlmann
  • Yusuke Miyao
  • Stephan Oepen
  • Yi Zhang
  • Daniel Zeman

sdp-organizers@emmtee.net

Other Info

Announcements

[22-apr-14] Complete results (system submissions and official scores) as well as the gold-standard test data are now available for public download.

[31-mar-14] We have received submissions from nine teams; a draft summary of evaluation results has been emailed to participating teams.

[25-mar-14] We have posted some additional, task-specific instructions for how to submit system results to the SemEval evaluation; please make sure to follow these requirements carefully.

[22-mar-14] The test data (and corresponding ‘companion’ syntactic analyses, for use in the open track) are now available to registered participants; please see the task mailing list for details.

[08-mar-14] We have released a minor update to the companion archive, adding a handful of missing dependencies and fixing a problem in the file format.

[05-feb-14] We have posted the description of a baseline approach and experimental results on the suggested development sub-set of our training data (Section 20) on the evaluation page; on the same page, we have further specified the mechanics of submitting results to the evaluation.

[17-jan-14] Version 1.0 of the ‘companion’ data for the open track is now available, providing syntactic analyses (in phrase structure and bi-lexical dependency form) as overlays to our training data.  Please see the file README.txt in the companion archive for details.

[13-jan-14] We are releasing an update to the training data today, making a number of minor improvements to the DM and PCEDT graphs; also, we are now providing an on-line interface to search and explore visually the target representations for this task.  For details, please see our task-specific mailing list.

[12-dec-13] Some 750,000 tokens of WSJ text, annotated in our three semantic dependency formats will become available for download tomorrow.  To obtain the data, prospective participants need to enter a no-cost evaluation license with the Linguistic Data Consortium (LDC).  For access to the license form, please subscribe to our spam-protected mailing list.  Next, we are working to prepare our syntactic ‘companion’ data (to serve as optional input in the open track), which we expect to release in early January.

[24-nov-13] Version 1.1. of the trial data is now available, adding missing lemma values and streamlining argument labels in the DM format, removing a handful of items that used to have empty graphs in PAS, and generally aligning all items at the level of individual tokens (leaving 189 sentences in our trial data).  This last move means that all three formats now uniformly use single-character Unicode glyphs for quote marks, dashes, ellipses, and apostrophes (rather than multi-character LaTeX-style approxmiations, as were used in the original ASCII release of the text).  Furthermore, we encourage all interested parties, including prospective participants, to subscribe to our spam-protected mailing list, where we will post updates a little more frequently than on the general task web site.

[07-nov-13] We have clarified the interpretation of the top column (and renamed it from the earlier root) and elaborated the discussion of graph properties in the various formats.  We will continue to extend and revise the documentation on our three types of dependency graphs, but only announce such incremental changes here when they affect the data format.

[04-nov-13] A 198-sentence subset of what will be the training data has been released as trial data, to exemplify the file format and type of annotations available.  Please do get in touch, in case you see anything suprising!

[28-oct-13] We are in the process of finalizing the task description, posting some example dependencies, and making available some trial data.  For the time being, please consider these pages very much a work in progress, i.e. contents and form will be subject to refinement over the next few days

.