Results

Following are the official evaluation scores for all system submissions, runs, and tracks. In case the table below does not display properly for you, the same information is also available as a spreadsheet in Open Document Format (or as an on-line spreadsheet).

 

Primary Metrics: Labeled Dependencies Including TOP Nodes
                                 
      DM PAS PCEDT    
      LP LR LF LM LP LR LF LM LP LR LF LM average rank
closed Peking 1 0.9027 0.8854 0.8940 0.2671 0.9344 0.9069 0.9204 0.3813 0.7875 0.7396 0.7628 0.1105 0.8591 1
closed Peking 2 0.9069 0.8792 0.8928 0.2745 0.9295 0.9064 0.9178 0.3605 0.7828 0.7436 0.7627 0.1083 0.8578  
closed Priberam 1 0.8882 0.8735 0.8808 0.2240 0.9195 0.8992 0.9093 0.3264 0.7880 0.7470 0.7670 0.0942 0.8524 2
closed Copenhagen-Malmö 2 0.8478 0.8404 0.8441 0.2033 0.8769 0.8837 0.8803 0.1016 0.7115 0.6865 0.6988 0.0801 0.8077 3
closed Copenhagen-Malmö 1 0.8720 0.8020 0.8356 0.0000 0.9130 0.8131 0.8601 0.0000 0.7276 0.6284 0.6744 0.0000 0.7900  
closed Potsdam 2 0.7936 0.7934 0.7935 0.0757 0.8815 0.8160 0.8475 0.0653 0.6968 0.6625 0.6792 0.0519 0.7734 4
closed Alpage 2 0.7942 0.7724 0.7832 0.0972 0.8565 0.8271 0.8416 0.1795 0.7053 0.6528 0.6781 0.0682 0.7676 5
closed Alpage 1 0.7960 0.7732 0.7844 0.0890 0.8407 0.8087 0.8244 0.1276 0.6881 0.6687 0.6783 0.0690 0.7623  
closed Potsdam 1 0.8339 0.7288 0.7778 0.0423 0.8818 0.7400 0.8047 0.0178 0.7225 0.6710 0.6958 0.0564 0.7594  
closed Linköping 1 0.7854 0.7805 0.7829 0.0608 0.7616 0.7555 0.7585 0.0119 0.6066 0.6435 0.6245 0.0401 0.7220 6
average 0.8421 0.8129 0.8269 0.1334 0.8795 0.8357 0.8565 0.1572 0.7217 0.6844 0.7021 0.0679    
                                 
open Priberam 1 0.9023 0.8811 0.8916 0.2685 0.9256 0.9097 0.9176 0.3783 0.8014 0.7579 0.7790 0.1068 0.8627 1
open CMU 1 0.8446 0.8348 0.8397 0.0875 0.9078 0.8851 0.8963 0.2604 0.7681 0.7072 0.7364 0.0712 0.8242 2
open Turku 2 0.8094 0.8214 0.8153 0.0823 0.8733 0.8776 0.8754 0.1721 0.7242 0.7237 0.7240 0.0682 0.8049 3
open Turku 1 0.7989 0.8245 0.8115 0.0757 0.8742 0.8801 0.8771 0.1780 0.7223 0.7298 0.7260 0.0668 0.8049  
open Potsdam 2 0.8132 0.8091 0.8111 0.0905 0.8941 0.8261 0.8588 0.0749 0.7035 0.6733 0.6880 0.0542 0.7860 4
open Alpage 1 0.8346 0.7955 0.8146 0.1076 0.8723 0.8282 0.8497 0.1543 0.7098 0.6751 0.6920 0.0660 0.7854 5
open Alpage 2 0.8577 0.7446 0.7971 0.0660 0.8860 0.8276 0.8558 0.1091 0.7233 0.6704 0.6958 0.0697 0.7829  
open Potsdam 1 0.8454 0.7380 0.7880 0.0415 0.8972 0.7508 0.8175 0.0178 0.7252 0.6733 0.6983 0.0564 0.7679  
open In-House 1 0.9258 0.9234 0.9246 0.4807 0.9209 0.9202 0.9206 0.4384 0.4089 0.4567 0.4315 0.0030 0.7589 6
average 0.8480 0.8192 0.8326 0.1445 0.8946 0.8561 0.8743 0.1982 0.6985 0.6742 0.6857 0.0625    
                                 
                                 
                                 
Unlabeled Dependencies Including TOP Nodes
                                 
      DM PAS PCEDT    
      UP UR UF UM UP UR UF UM UP UR UF UM average rank
closed Peking 1 0.9170 0.8995 0.9082 0.2997 0.9455 0.9176 0.9313 0.3961 0.9202 0.8642 0.8913 0.3316 0.9103 1
closed Peking 2 0.9207 0.8926 0.9064 0.3064 0.9412 0.9177 0.9293 0.3739 0.9154 0.8696 0.8919 0.3353 0.9092  
closed Priberam 1 0.9014 0.8865 0.8939 0.2478 0.9318 0.9112 0.9214 0.3435 0.9021 0.8551 0.8780 0.2619 0.8977 2
closed Copenhagen-Malmö 2 0.8676 0.8600 0.8638 0.2366 0.8909 0.8978 0.8943 0.1091 0.8476 0.8177 0.8324 0.2582 0.8635 3
closed Copenhagen-Malmö 1 0.8916 0.8200 0.8543 0.0000 0.9261 0.8248 0.8725 0.0000 0.8815 0.7614 0.8171 0.0000 0.8480  
closed Alpage 2 0.8300 0.8072 0.8185 0.1239 0.8762 0.8461 0.8609 0.1944 0.8450 0.7821 0.8123 0.1669 0.8306 4
closed Potsdam 2 0.8156 0.8154 0.8155 0.0846 0.8962 0.8296 0.8616 0.0668 0.8337 0.7927 0.8127 0.1484 0.8299 5
closed Alpage 1 0.8207 0.7972 0.8088 0.1046 0.8609 0.8281 0.8441 0.1335 0.8203 0.7973 0.8087 0.1966 0.8205  
closed Potsdam 1 0.8547 0.7470 0.7972 0.0467 0.8970 0.7528 0.8186 0.0200 0.8636 0.8021 0.8317 0.1647 0.8158  
closed Linköping 1 0.8145 0.8094 0.8120 0.0727 0.7814 0.7752 0.7783 0.0126 0.7367 0.7815 0.7584 0.1246 0.7829 6
average 0.8634 0.8335 0.8479 0.1523 0.8947 0.8501 0.8712 0.1650 0.8566 0.8124 0.8334 0.1988    
                                 
open Priberam 1 0.9141 0.8926 0.9032 0.2990 0.9362 0.9202 0.9281 0.3924 0.9158 0.8661 0.8903 0.3071 0.9072 1
open CMU 1 0.8603 0.8503 0.8553 0.0979 0.9179 0.8950 0.9063 0.2715 0.8885 0.8179 0.8517 0.1706 0.8711 2
open Turku 2 0.8287 0.8410 0.8348 0.0964 0.8876 0.8918 0.8897 0.1803 0.8589 0.8584 0.8586 0.2033 0.8610 3
open Turku 1 0.8182 0.8444 0.8311 0.0838 0.8885 0.8945 0.8915 0.1869 0.8550 0.8639 0.8594 0.2055 0.8607  
open SDP 1 0.9361 0.9337 0.9349 0.5230 0.9320 0.9314 0.9317 0.4429 0.6557 0.7323 0.6919 0.0148 0.8528 4
open Alpage 2 0.8819 0.7656 0.8197 0.0779 0.9005 0.8412 0.8699 0.1150 0.8814 0.8170 0.8480 0.2018 0.8458 5
open Alpage 1 0.8574 0.8172 0.8368 0.1194 0.8895 0.8445 0.8664 0.1662 0.8480 0.8066 0.8268 0.2211 0.8433  
open Potsdam 2 0.8337 0.8295 0.8316 0.1031 0.9078 0.8387 0.8719 0.0779 0.8446 0.8083 0.8260 0.1528 0.8432 6
open Potsdam 1 0.8643 0.7545 0.8057 0.0497 0.9099 0.7614 0.8291 0.0208 0.8732 0.8107 0.8408 0.1677 0.8252  
average 0.8661 0.8365 0.8503 0.1611 0.9078 0.8687 0.8872 0.2060 0.8468 0.8201 0.8326 0.1827    
                                 
                                 
                                 
Labeled Dependencies Excluding TOP Nodes
                                 
      DM PAS PCEDT    
      LP LR LF LM LP LR LF LM LP LR LF LM average rank
closed Peking 1 0.9020 0.8842 0.8930 0.2730 0.9330 0.9049 0.9187 0.3828 0.7740 0.7254 0.7489 0.1105 0.8536 1
closed Peking 2 0.9066 0.8780 0.8921 0.2789 0.9282 0.9041 0.9160 0.3605 0.7690 0.7290 0.7485 0.1091 0.8522  
closed Priberam 1 0.8881 0.8727 0.8803 0.2300 0.9187 0.8975 0.9080 0.3301 0.7768 0.7325 0.7540 0.0942 0.8474 2
closed Copenhagen 1 0.8720 0.8514 0.8616 0.2129 0.9130 0.8507 0.8808 0.1165 0.7276 0.6799 0.7029 0.0853 0.8151 3
closed Copenhagen 2 0.8543 0.8463 0.8503 0.2248 0.8791 0.8862 0.8826 0.1053 0.7076 0.6877 0.6975 0.0883 0.8102  
closed Potsdam 2 0.7956 0.8013 0.7985 0.1202 0.8886 0.8207 0.8533 0.0957 0.6896 0.6535 0.6710 0.0727 0.7743 4
closed Alpage 2 0.7990 0.7829 0.7909 0.1306 0.8591 0.8323 0.8455 0.2010 0.6954 0.6578 0.6761 0.0727 0.7708 5
closed Alpage 1 0.8032 0.7798 0.7913 0.1009 0.8464 0.8137 0.8297 0.1410 0.6812 0.6675 0.6743 0.0779 0.7651  
closed Potsdam 1 0.8387 0.7327 0.7821 0.0697 0.8897 0.7412 0.8087 0.0297 0.7173 0.6627 0.6889 0.0786 0.7599  
closed Linköping 1 0.7964 0.7865 0.7914 0.0838 0.7759 0.7538 0.7647 0.0141 0.6352 0.6218 0.6284 0.0423 0.7282 6
average 0.8456 0.8216 0.8332 0.1725 0.8832 0.8405 0.8608 0.1777 0.7174 0.6818 0.6991 0.0832    
                                 
open Priberam 1 0.9029 0.8805 0.8916 0.2745 0.9247 0.9081 0.9163 0.3806 0.7891 0.7437 0.7657 0.1083 0.8579 1
open CMU 1 0.8542 0.8436 0.8488 0.1083 0.9104 0.8866 0.8983 0.2774 0.7564 0.6990 0.7266 0.0742 0.8246 2
open Turku 1 0.7991 0.8262 0.8124 0.0772 0.8720 0.8781 0.8750 0.1788 0.7096 0.7146 0.7121 0.0675 0.7998 3
open Turku 2 0.8102 0.8228 0.8165 0.0861 0.8711 0.8755 0.8733 0.1728 0.7112 0.7079 0.7096 0.0690 0.7998  
open Potsdam 2 0.8163 0.8179 0.8171 0.1499 0.9018 0.8313 0.8651 0.1083 0.6969 0.6651 0.6807 0.0720 0.7876 4
open Alpage 1 0.8387 0.7991 0.8185 0.1224 0.8764 0.8307 0.8529 0.1632 0.7034 0.6740 0.6884 0.0749 0.7866 5
open Alpage 2 0.8586 0.7620 0.8074 0.1083 0.8880 0.8293 0.8576 0.1246 0.7121 0.6737 0.6924 0.0794 0.7858  
open Potsdam 1 0.8510 0.7424 0.7930 0.0712 0.9059 0.7525 0.8221 0.0326 0.7203 0.6652 0.6917 0.0786 0.7689  
open In-House 1 0.9248 0.9224 0.9236 0.4822 0.9192 0.9186 0.9189 0.4392 0.4049 0.4273 0.4158 0.0134 0.7527 6
average 0.8506 0.8241 0.8365 0.1644 0.8966 0.8567 0.8755 0.2086 0.6893 0.6634 0.6759 0.0708    
                                 
                                 
                                 
Unlabeled Dependencies Excluding TOP Nodes
                                 
      DM PAS PCEDT    
      UP UR UF UM UP UR UF UM UP UR UF UM average rank
closed Peking 1 0.9172 0.8992 0.9081 0.3086 0.9446 0.9161 0.9302 0.3984 0.9179 0.8602 0.8881 0.3338 0.9088 1
closed Peking 2 0.9214 0.8923 0.9066 0.3138 0.9404 0.9160 0.9281 0.3746 0.9128 0.8653 0.8884 0.3361 0.9077  
closed Priberam 1 0.9021 0.8864 0.8942 0.2552 0.9315 0.9100 0.9206 0.3479 0.9008 0.8494 0.8744 0.2626 0.8964 2
closed Copenhagen 1 0.8916 0.8705 0.8809 0.2507 0.9261 0.8630 0.8934 0.1231 0.8815 0.8237 0.8516 0.2515 0.8753 3
closed Copenhagen 2 0.8754 0.8671 0.8712 0.2641 0.8937 0.9010 0.8973 0.1180 0.8538 0.8298 0.8416 0.2826 0.8700  
closed Alpage 2 0.8367 0.8199 0.8282 0.1662 0.8796 0.8521 0.8657 0.2196 0.8432 0.7977 0.8198 0.1966 0.8379 4
closed Potsdam 2 0.8188 0.8246 0.8217 0.1417 0.9040 0.8350 0.8681 0.1009 0.8382 0.7943 0.8157 0.2047 0.8352 5
closed Alpage 1 0.8295 0.8053 0.8172 0.1202 0.8675 0.8340 0.8504 0.1491 0.8232 0.8066 0.8148 0.2166 0.8275  
closed Potsdam 1 0.8608 0.7520 0.8027 0.0809 0.9057 0.7546 0.8232 0.0341 0.8709 0.8045 0.8364 0.2329 0.8208  
closed Linköping 1 0.8274 0.8172 0.8223 0.1061 0.7971 0.7744 0.7856 0.0148 0.7877 0.7711 0.7793 0.1269 0.7957 6
average 0.8681 0.8435 0.8553 0.2007 0.8990 0.8556 0.8763 0.1881 0.8630 0.8203 0.8410 0.2444    
                                 
open Priberam 1 0.9155 0.8927 0.9039 0.3056 0.9359 0.9190 0.9274 0.3961 0.9134 0.8607 0.8863 0.3093 0.9059 1
open CMU 1 0.8708 0.8600 0.8654 0.1202 0.9209 0.8968 0.9087 0.2893 0.8861 0.8189 0.8512 0.1855 0.8751 2
open Turku 2 0.8307 0.8437 0.8371 0.1016 0.8860 0.8905 0.8882 0.1818 0.8576 0.8536 0.8556 0.2040 0.8603 3
open Turku 1 0.8195 0.8473 0.8332 0.0868 0.8870 0.8932 0.8901 0.1892 0.8537 0.8597 0.8567 0.2062 0.8600  
open SDP 1 0.9358 0.9333 0.9345 0.5245 0.9309 0.9302 0.9306 0.4458 0.6874 0.7254 0.7059 0.1358 0.8570 4
open Alpage 2 0.8837 0.7843 0.8310 0.1239 0.9032 0.8435 0.8723 0.1313 0.8798 0.8323 0.8554 0.2522 0.8529 5
open Potsdam 2 0.8379 0.8396 0.8388 0.1751 0.9161 0.8445 0.8788 0.1120 0.8500 0.8112 0.8302 0.2203 0.8493  
open Alpage 1 0.8629 0.8222 0.8420 0.1395 0.8944 0.8478 0.8705 0.1751 0.8518 0.8162 0.8336 0.2418 0.8487 6
open Potsdam 1 0.8711 0.7600 0.8118 0.0853 0.9193 0.7636 0.8342 0.0371 0.8813 0.8138 0.8462 0.2470 0.8307  
average 0.8698 0.8426 0.8553 0.1847 0.9104 0.8699 0.8890 0.2175 0.8512 0.8213 0.8357 0.2225    

 

Contact Info

Organizers

  • Dan Flickinger
  • Jan Hajič
  • Marco Kuhlmann
  • Yusuke Miyao
  • Stephan Oepen
  • Yi Zhang
  • Daniel Zeman

sdp-organizers@emmtee.net

Other Info

Announcements

[22-apr-14] Complete results (system submissions and official scores) as well as the gold-standard test data are now available for public download.

[31-mar-14] We have received submissions from nine teams; a draft summary of evaluation results has been emailed to participating teams.

[25-mar-14] We have posted some additional, task-specific instructions for how to submit system results to the SemEval evaluation; please make sure to follow these requirements carefully.

[22-mar-14] The test data (and corresponding ‘companion’ syntactic analyses, for use in the open track) are now available to registered participants; please see the task mailing list for details.

[08-mar-14] We have released a minor update to the companion archive, adding a handful of missing dependencies and fixing a problem in the file format.

[05-feb-14] We have posted the description of a baseline approach and experimental results on the suggested development sub-set of our training data (Section 20) on the evaluation page; on the same page, we have further specified the mechanics of submitting results to the evaluation.

[17-jan-14] Version 1.0 of the ‘companion’ data for the open track is now available, providing syntactic analyses (in phrase structure and bi-lexical dependency form) as overlays to our training data.  Please see the file README.txt in the companion archive for details.

[13-jan-14] We are releasing an update to the training data today, making a number of minor improvements to the DM and PCEDT graphs; also, we are now providing an on-line interface to search and explore visually the target representations for this task.  For details, please see our task-specific mailing list.

[12-dec-13] Some 750,000 tokens of WSJ text, annotated in our three semantic dependency formats will become available for download tomorrow.  To obtain the data, prospective participants need to enter a no-cost evaluation license with the Linguistic Data Consortium (LDC).  For access to the license form, please subscribe to our spam-protected mailing list.  Next, we are working to prepare our syntactic ‘companion’ data (to serve as optional input in the open track), which we expect to release in early January.

[24-nov-13] Version 1.1. of the trial data is now available, adding missing lemma values and streamlining argument labels in the DM format, removing a handful of items that used to have empty graphs in PAS, and generally aligning all items at the level of individual tokens (leaving 189 sentences in our trial data).  This last move means that all three formats now uniformly use single-character Unicode glyphs for quote marks, dashes, ellipses, and apostrophes (rather than multi-character LaTeX-style approxmiations, as were used in the original ASCII release of the text).  Furthermore, we encourage all interested parties, including prospective participants, to subscribe to our spam-protected mailing list, where we will post updates a little more frequently than on the general task web site.

[07-nov-13] We have clarified the interpretation of the top column (and renamed it from the earlier root) and elaborated the discussion of graph properties in the various formats.  We will continue to extend and revise the documentation on our three types of dependency graphs, but only announce such incremental changes here when they affect the data format.

[04-nov-13] A 198-sentence subset of what will be the training data has been released as trial data, to exemplify the file format and type of annotations available.  Please do get in touch, in case you see anything suprising!

[28-oct-13] We are in the process of finalizing the task description, posting some example dependencies, and making available some trial data.  For the time being, please consider these pages very much a work in progress, i.e. contents and form will be subject to refinement over the next few days

.
Copyright 2015 - SemEval-2014 Task 8. All Right Reserved

Designed by BhMad Studio  |  Powered by GetSimple