Fourth Dialog State Tracking Challenge
@ IWSDS2016

Task Descriptions

Main task: Dialog State Tracking at Sub-dialog Level

The goal of the main task of the challenge is to track dialog states for sub-dialog segments. For each turn in a given sub-dialog, the tracker should fill out a frame of slot-value pairs considering all dialog history prior to the turn. The performance of a tracker will be evaluated by comparing its outputs with reference annotations (see an example of reference annotations). Weighted accuracy will be used as evaluation metric to give higher score when a correct frame structure is completed in an earlier turn.

In the development phase, participants will be provided with a training set of dialogs with manual annotations over frame structures. In the test phase, each tracker will be evaluated on the results generated for a test set of unlabeled dialogs. A baseline system and evaluation script will be provided along with the training data. Participation in the main track is mandatory for all teams and/or individuals registered in the DSTC4.

Pilot tasks: four optional pilot tasks are available in DSTC4

* Spoken language understanding: Tag a given utterance with speech acts and semantic slots.
* Speech act prediction: Predict the speech act of the next turn imitating the policy of one speaker.
* Spoken language generation: Generate a response utterance for one of the participants.
* End-to-end system: Develop an end-to-end system playing the part of a guide or a tourist. This task will be conducted only if at least one team and/or individual registers for each of the pilot tasks above.

Open track: an optional open track is available in DSTC4

DSTC4 registered teams and/or individuals are free to work and report results on any proposed task of their interest over the provided dataset.