Measuring the differences between human-human and human-machine dialogs

• GRIOL, David ^[1] ; MOLINA, José ^[1]
  1. [1] Universidad Carlos III de Madrid

     Universidad Carlos III de Madrid

     Madrid, España

     
• Localización: ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal, ISSN-e 2255-2863, Vol. 4, Nº. 2, 2015, págs. 99-112
• Idioma: inglés
• DOI: 10.14201/ADCAIJ20154299112
• Enlaces
  • Texto completo
• Resumen

  • In this paper, we assess the applicability of user simulation techniques to generate dialogs which are similar to real human-machine spoken interactions.To do so, we present the results of the comparison between three corpora acquired by means of different techniques. The first corpus was acquired with real users.A statistical user simulation technique has been applied to the same task to acquire the second corpus. In this technique, the next user answer is selected by means of a classification process that takes into account the previous dialog history, the lexical information in the clause, and the subtask of the dialog to which it contributes. Finally, a dialog simulation technique has been developed for the acquisition of the third corpus. This technique uses a random selection of the user and system turns, defining stop conditions for automatically deciding if the simulated dialog is successful or not. We use several evaluation measures proposed in previous research to compare between our three acquired corpora, and then discuss the similarities and differences with regard to these measures.

• Referencias bibliográficas
  • Ai, H. and Litman, D., 2006. Comparing Real-Real, Simulated-Simulated, and Simulated-Real Spoken Dialogue Corpora. In Procs. of AAAI Workshop...
  • Ai, H. and Litman, D., 2007. Knowledge Consistent User Simulations for Dialog Systems. In Proc. of Interspeech’07, pages 2697–2700. Antwerp,...
  • Ai, H., Raux, A., Bohus, D., Eskenazi, M., and Litman, D., 2007a. Comparing Spoken Dialog Corpora Collected with Recruited Subjects versus...
  • Ai, H., Tetreault, J., and Litman, D., 2007b. Comparing User Simulation Models For Dialog Strategy Learning. In Proc. of NAACL HLT’07, pages...
  • A.L. Ballinas, A. M. and Rangel, A., 2011. Multiagent System Applied to the Modeling and Simulation of Pedestrian Traffic in Counterflow....
  • Angheluta, R., Busser, R. D., and Moens, M., 2002. The use of topic segmentation for automatic summarization. In Proc. ACL Workshop on Automatic...
  • Bailly, G., Raidt, S., and Elisei, F., 2010. Gaze, conversational agents and face-to-face communication. Speech Communication, 52(6):598–612.
  • Balmer, M. and Nagel, K., 2006. Innovations in Design & Decision Support Systems in Architecture and Urban Planning, chapter Shape Morphing...
  • Bandini, S., Celada, F., Manzoni, S., Puzone, R., and Vizzari, G., 2006. Modelling the Immune System with Situated Agents. Lecture Notes in...
  • Bandini, S., Manzoni, S., and Vizzari, G., 2009. Agent Based Modeling and Simulation: An Informatics Perspective. Journal of Artificial Societies...
  • Bangalore, S., Fabbrizio, G. D., and Stent, A., 2008. Learning the Structure of Task-driven Human-Human Dialogs. IEEE Trans Audio Speech Lang...
  • Bohus, D., Grau, S., Huggins-Daines, D., Keri, V., Krishna, G., Kumar, R., Raux, A., and Tomko, S., 2007. Conquest - An Open-Source Dialog...
  • Bohus, D. and Rudnicky, A., 2002. LARRI: A Language-Based Maintenance and Repair Assistant. In Proc. of Multi-Modal Dialogue in Mobile Environments...
  • Brahnam, S., 2009. Building Character for Artificial Conversational Agents: Ethos, Ethics, Believability, and Credibility. PsychNology Journal,...
  • Chai, J. and Jin, R., 2004. Discourse structure for context question answering. In Proc. HLT-NAACL Workshop on Pragmatics of Question Answering,...
  • Chung, G., 2004. Developing a flexible spoken dialog system using simulation. In Proc. of the 42nd Annual Meeting of the Association for Computational...
  • Doran, C., Aberdeen, J., Damianos, L., and Hirschman., L., 2001. Comparing several aspects of humancomputer and human-human dialogues. In...
  • Garcia, F., Hurtado, L., E.Sanchis, and Segarra, E., 2003. The incorporation of Confidence Measures to Language Understanding. In Proc. of...
  • Glass, J., Flammia, G., Goodine, D., Phillips, M., Polifroni, J., Sakai, S., Seneff, S., and Zue, V., 1995. Multilingual spoken-language understanding...
  • Griol, D., Callejas, Z., López-Cózar, R., and Riccardi, G., 2014. A domain-independent statistical methodology for dialog management in spoken...
  • Griol, D., Molina, J., and Callejas, Z., 2012. Bringing together commercial and academic perspectives for the development of intelligent AmI...
  • Hearst, M., 1994. Multi-paragraph segmentation of expository text. In Proc. ACL, pages 9–16.
  • Heath, B., Hill, R., and Ciarallo, F., 2009. A Survey of Agent-Based Modeling Practices (January 1998 to July 2008). Journal of Artificial...
  • Heinroth, T. and Minker, W., 2012. Introducing Spoken Dialogue Systems into Intelligent Environments. Kluwer Academic PublishersSpringer-Verlag.
  • Lin, B. and Lee, L., 2001. Computer aided analysis and design for spoken dialogue systems based on quantitative simulations. IEEE Trans. Speech...
  • Lopez-Cozar, R., la Torre, A. D., Segura, J., Rubio, A., and Sanchez, V., 2003. Assessment of dialogue systems by means of a new simulation...
  • Macal, C. and North, M., 2010. Tutorial on agent-based modelling and simulation. Journal of Simulation, 4:151–162.
  • Melin, H., Sandell, A., and Ihse, M., 2001. CTT-bank: A speech controlled telephone banking system - an initial evaluation. In TMH Quarterly...
  • Menezes, P., Lerasle, F., Dias, J., and Germa, T., 2007. Humanoid Robots, Human-like Machines, chapter Towards an Interactive Humanoid Companion...
  • Moller, S., Englert, R., Engelbrecht, K., Hafner, V., Jameson, A., Oulasvirta, A., Raake, A., and Reithinger, N., 2006. MeMo: towards automatic...
  • Navarro, L., Flacher, F., and Corruble, V., 2011. Dynamic Level of Detail for Large Scale Agent-Based Urban Simulations. In Proc. of the 10th...
  • Passoneau, R. and Litman, D., 1997. Discourse segmentation by human and automated means. Computational Linguistics, 23:103–139.
  • Pavón, J., Sansores, C., Gómez, J., and Wang, F., 2008. Modelling and simulation of social systems with INGENIAS. Int. Journal of Agent-Oriented...
  • Pérez-Marín, D. and Pascual-Nieto, I., 2011. Conversational Agents and Natural Language Interaction: Techniques and Effective Practices. IGI...
  • Pieraccini, R., 2012. The Voice in the Machine: Building Computers that Understand Speech. The MIT Press.
  • Ponte, J. and Croft, W., 1997. Text segmentation by topic. In Proc. ECDL, pages 120–129.
  • Schatzmann, J., Georgila, K., and Young, S., 2005. Quantitative Evaluation of User Simulation Techniques for
  • Spoken Dialogue Systems. In Proc. of the 6th SIGdial Workshop on Discourse and Dialogue, pages 45–54. Lisbon, Portugal.
  • Schatzmann, J., Thomson, B., and Young, S., 2007. Error Simulation for Training Statistical Dialogue Systems. In Proc. of ASRU’07, pages 526–531.
  • Schatzmann, J., Weilhammer, K., Stuttle, M., and Young, S., 2006. A Survey of Statistical User Simulation Techniques for Reinforcement-Learning...
  • Stepanov, E., Riccardi, G., and Bayer, A., 2014. The Development of the Multilingual LUNA Corpus for Spoken Language System Porting. In Proc....
  • Turunen, M., Hakulinen, J., and Kainulainen, A., 2006. Evaluation of a Spoken Dialogue System with Usability Tests and Long-term Pilot Studies:...
  • Vaquero, C., Saz, O., Lleida, E., Marcos, J., and Canal?s, C., 2006. VOCALIZA: An application for computeraided speech therapy in spanish...
  • Walker, M. A., 1998. Centering, anaphora resolution, and discourse structure, pages 401–435. Oxford University Press.
  • Weng, F., Varges, S., Raghunathan, B., Ratiu, F., Pon-Barry, H., Lathrop, B., Zhang, Q., Scheideck, T., Bratt, H.,
  • Xu, K., Purver, M., Mishra, R., Raya, M., Peters, S., Meng, Y., Cavedon, L., and Shriberg, L., 2006. CHAT: A Conversational Helper for Automotive...
  • Weyns, D., Boucké, N., and Holvoet, T., 2006. Gradient Field-Based Task Assignment in an AGV Transportation System. In Proc. of the 5th Int....
  • Windrum, P., Fagiolo, G., and Moneta, A., 2007. Empirical Validation of Agent-Based Models: Alternatives and Prospects. Journal of Artificial...
  • Yamron, J., 1998. Topic detection and tracking segmentation task. In Proc. Broadcast News Transcription and Understanding Workshop.
  • Zue, V., Seneff, S., Glass, J., Polifroni, J., Pao, C., Hazen, T., and Hetherington, L., 2000. JUPITER: A telephone-based conversational interface...