DSTC4
Fourth Dialog State Tracking Challenge @ IWSDS2016
Official Evaluation Results
Summary of Main Task Results
| |
Schedule 1 |
Schedule 2 |
| Team |
Entry |
Accuracy |
Precision |
Recall |
F-measure |
Accuracy |
Precision |
Recall |
F-measure |
| BSL |
0 |
0.0374 |
0.3589 |
0.1925 |
0.2506 |
0.0488 |
0.3750 |
0.2519 |
0.3014 |
| 1 |
0 |
0.0456 |
0.3876 |
0.3344 |
0.3591 |
0.0584 |
0.4384 |
0.3377 |
0.3815 |
| 1 |
1 |
0.0374 |
0.4214 |
0.2762 |
0.3336 |
0.0584 |
0.4384 |
0.3377 |
0.3815 |
| 1 |
2 |
0.0372 |
0.4173 |
0.2767 |
0.3328 |
0.0575 |
0.4362 |
0.3377 |
0.3807 |
| 1 |
3 |
0.0371 |
0.4179 |
0.2804 |
0.3356 |
0.0584 |
0.4384 |
0.3426 |
0.3846 |
| 2 |
0 |
0.0487 |
0.4079 |
0.2626 |
0.3195 |
0.0671 |
0.4280 |
0.3257 |
0.3699 |
| 2 |
1 |
0.0467 |
0.4481 |
0.2655 |
0.3335 |
0.0671 |
0.4674 |
0.3275 |
0.3851 |
| 2 |
2 |
0.0478 |
0.4523 |
0.2623 |
0.3320 |
0.0706 |
0.4679 |
0.3226 |
0.3819 |
| 2 |
3 |
0.0489 |
0.4440 |
0.2703 |
0.3361 |
0.0697 |
0.4634 |
0.3335 |
0.3878 |
| 3 |
0 |
0.1212 |
0.5393 |
0.4980 |
0.5178 |
0.1500 |
0.5569 |
0.5808 |
0.5686 |
| 3 |
1 |
0.1210 |
0.5449 |
0.4964 |
0.5196 |
0.1500 |
0.5619 |
0.5787 |
0.5702 |
| 3 |
2 |
0.1092 |
0.5304 |
0.5031 |
0.5164 |
0.1316 |
0.5437 |
0.5875 |
0.5648 |
| 3 |
3 |
0.1183 |
0.5780 |
0.4904 |
0.5306 |
0.1473 |
0.5898 |
0.5678 |
0.5786 |
| 4 |
0 |
0.0887 |
0.5280 |
0.3595 |
0.4278 |
0.1072 |
0.5354 |
0.4273 |
0.4753 |
| 4 |
1 |
0.0910 |
0.5314 |
0.3122 |
0.3933 |
0.1055 |
0.5325 |
0.3623 |
0.4312 |
| 4 |
2 |
0.1009 |
0.5583 |
0.3698 |
0.4449 |
0.1264 |
0.5666 |
0.4455 |
0.4988 |
| 4 |
3 |
0.1002 |
0.5545 |
0.3760 |
0.4481 |
0.1212 |
0.5642 |
0.4540 |
0.5031 |
| 5 |
0 |
0.0309 |
0.2980 |
0.2559 |
0.2754 |
0.0392 |
0.3344 |
0.2547 |
0.2892 |
| 5 |
1 |
0.0268 |
0.3405 |
0.2014 |
0.2531 |
0.0401 |
0.3584 |
0.2632 |
0.3035 |
| 5 |
2 |
0.0309 |
0.3039 |
0.2659 |
0.2836 |
0.0392 |
0.3398 |
0.2639 |
0.2971 |
| 6 |
0 |
0.0421 |
0.4175 |
0.2142 |
0.2831 |
0.0541 |
0.4380 |
0.2656 |
0.3307 |
| 6 |
1 |
0.0478 |
0.5516 |
0.2180 |
0.3125 |
0.0654 |
0.5857 |
0.2702 |
0.3698 |
| 6 |
2 |
0.0486 |
0.5623 |
0.2314 |
0.3279 |
0.0645 |
0.5941 |
0.2850 |
0.3852 |
| 7 |
0 |
0.0286 |
0.2768 |
0.1826 |
0.2200 |
0.0323 |
0.3054 |
0.2410 |
0.2694 |
| 7 |
1 |
0.0044 |
0.0085 |
0.0629 |
0.0150 |
0.0061 |
0.0109 |
0.0840 |
0.0194 |
Summary of Pilot Task Results
| SLU-GUIDE |
Speech Act |
Semantic Tag |
| Team |
Entry |
Precision |
Recall |
F-measure |
Precision |
Recall |
F-measure |
| 3 |
1 |
0.6287 |
0.5191 |
0.5687 |
0.5646 |
0.4886 |
0.5239 |
| 3 |
2 |
0.6330 |
0.5227 |
0.5726 |
0.5646 |
0.4886 |
0.5239 |
| 3 |
3 |
0.7451 |
0.6153 |
0.6740 |
0.5646 |
0.4886 |
0.5239 |
| 3 |
4 |
0.6314 |
0.5214 |
0.5712 |
0.5646 |
0.4886 |
0.5239 |
| 3 |
5 |
0.6762 |
0.5584 |
0.6117 |
0.5646 |
0.4886 |
0.5239 |
| SLU-TOURIST |
Speech Act |
Semantic Tag |
| Team |
Entry |
Precision |
Recall |
F-measure |
Precision |
Recall |
F-measure |
| 3 |
1 |
0.3583 |
0.2977 |
0.3252 |
0.5741 |
0.4764 |
0.5207 |
| 3 |
2 |
0.2931 |
0.2435 |
0.2660 |
0.5741 |
0.4764 |
0.5207 |
| 3 |
3 |
0.5627 |
0.4675 |
0.5107 |
0.5741 |
0.4764 |
0.5207 |
| 3 |
4 |
0.2939 |
0.2442 |
0.2668 |
0.5741 |
0.4764 |
0.5207 |
| 3 |
5 |
0.5736 |
0.4766 |
0.5206 |
0.5741 |
0.4764 |
0.5207 |