The goal of the main task of the challenge is to track dialog states for sub-dialog segments. For each turn in a given sub-dialog, the tracker should fill out a frame of slot-value pairs considering all dialog history prior to the turn. The performance of a tracker will be evaluated by comparing its outputs with reference annotations (see an example of reference annotations). Weighted accuracy will be used as evaluation metric to give higher score when a correct frame structure is completed in an earlier turn. In the development phase, participants will be provided with a training set of dialogs with manual annotations over frame structures. In the test phase, each tracker will be evaluated on the results generated for a test set of unlabeled dialogs. A baseline system and evaluation script will be provided along with the training data. Participation in the main track is mandatory for all teams and/or individuals registered in the DSTC4.
* Spoken language understanding: Tag a given utterance with speech acts and semantic slots. * Speech act prediction: Predict the speech act of the next turn imitating the policy of one speaker. * Spoken language generation: Generate a response utterance for one of the participants. * End-to-end system: Develop an end-to-end system playing the part of a guide or a tourist. This task will be conducted only if at least one team and/or individual registers for each of the pilot tasks above.
DSTC4 registered teams and/or individuals are free to work and report results on any proposed task of their interest over the provided dataset.