Large Language Models for in Situ Knowledge Documentation and Access With Augmented Reality

• Izquierdo-Domenech, Juan ^[1] ; Linares-Pellicer, Jordi ^[1] ; Ferri-Molla, Isabel ^[1]
  1. [1] Universidad Politécnica de Valencia

     Universidad Politécnica de Valencia

     Valencia, España

     
• Localización: IJIMAI, ISSN-e 1989-1660, Vol. 9, Nº. 3, 2025, págs. 19-29
• Idioma: inglés
• DOI: 10.9781/ijimai.2023.09.002
• Enlaces
  • Texto completo
• Resumen

  • Augmented reality (AR) has become a powerful tool for assisting operators in complex environments, such as shop floors, laboratories, and industrial settings. By displaying synthetic visual elements anchored in real environments and providing information for specific tasks, AR helps to improve efficiency and accuracy. However, a common bottleneck in these environments is introducing all necessary information, which often requires predefined structured formats and needs more ability for multimodal and Natural Language (NL) interaction. This work proposes a new method for dynamically documenting complex environments using AR in a multimodal, non-structured, and interactive manner. Our method employs Large Language Models (LLMs) to allow experts to describe elements from the real environment in NL and select corresponding AR elements in a dynamic and iterative process. This enables a more natural and flexible way of introducing information, allowing experts to describe the environment in their own words rather than being constrained by a predetermined structure. Any operator can then ask about any aspect of the environment in NL to receive a response and visual guidance from the AR system, thus allowing for a more natural and flexible way of introducing and retrieving information. These capabilities ultimately improve the effectiveness and efficiency of tasks in complex environments.

• Referencias bibliográficas
  • H. Kagermann, J. Helbig, A. Hellinger, W. Wahlster, Recommendations for implementing the strategic initiative INDUSTRIE 4.0: Securing the...
  • R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, B. MacIntyre, “Recent advances in augmented reality,” IEEE Computer Graphics and Applications,...
  • M. Billinghurst, A. Clark, G. Lee, “A survey of augmented reality,” Foundations and Trends in Human- Computer Interaction, vol. 8, no. 2-3, pp....
  • T. Caudell, D. Mizell, “Augmented reality: an application of heads-up display technology to manual manufacturing processes,” in Proceedings of...
  • C. H. Chu, L. Wang, S. Liu, Y. Zhang, M. Menozzi, “Augmented reality in smart manufacturing: Enabling collaboration between humans and artificial...
  • S. Jaschke, “Mobile learning applications for technical vocational and engineering education: The use of competence snippets in laboratory courses...
  • E. Marino, L. Barbieri, B. Colacino, A. K. Fleri, F. Bruno, “An Augmented Reality inspection tool to support workers in Industry 4.0 environments,” Computers...
  • M. Gattullo, G. W. Scurati, M. Fiorentino, A. E. Uva, F. Ferrise, M. Bordegoni, “Towards augmented reality manuals for industry 4.0: A methodology,”...
  • T. Masood, J. Egger, “Augmented reality in support of Industry 4.0— Implementation challenges and success factors,” Robotics and Computer- Integrated...
  • X. Xu, Y. Lu, B. Vogel-Heuser, L. Wang, “Industry 4.0 and Industry 5.0— Inception, conception and perception,” Journal of Manufacturing Systems, vol....
  • L. E. Garza, G. Pantoja, P. Ramírez, H. Ramírez, N. Rodríguez, E. González, R. Quintal, J. A. Pérez, “Augmented reality application for the...
  • J. Izquierdo-Domenech, J. Linares-Pellicer, J. Orta- Lopez, “Towards achieving a high degree of situational awareness and multimodal interaction...
  • J. Lave, E. Wenger, Situated learning: Legitimate peripheral participation. Cambridge university press, 1991. doi: 10.1017/CBO9780511815355.
  • C. A. Ventura, “Why switch from paper to electronic manuals?,” in Proceedings of the ACM conference on Document processing systems, 2000, pp....
  • F. Quint, F. Loch, “Using smart glasses to document maintenance processes,” Mensch und Computer 2015–Workshopband, pp. 203–208, 2015, doi:...
  • C. Kollatsch, P. Klimant, “Efficient integration process of production data into Augmented Reality based maintenance of machine tools,” Production Engineering,...
  • M. Hermann, T. Pentek, B. Otto, “Design principles for industrie 4.0 scenarios,” in Proceedings of the Annual Hawaii International Conference on...
  • T. P. Carvalho, F. A. Soares, R. Vita, R. d. P. Francisco, J. P. Basto, S. G. Alcalá, “A systematic literature review of machine learning...
  • J. Ribeiro, R. Lima, T. Eckhardt, S. Paiva, “Robotic Process Automation and Artificial Intelligence in J. Sääski, T. Salonen, M. Hakkarainen,...
  • T. Salonen, J. Sääski, C. Woodward, O. Korkalo, I. Marstio, K. Rainio, “Data pipeline from CAD to AR based assembly instructions,” in Proceedings of...
  • M. Fiorentino, G. Monno, A. E. Uva, “Tangible digital master for product lifecycle management in augmented reality,” International Journal...
  • M. Fiorentino, R. Radkowski, C. Stritzke, A. E. Uva, G. Monno, “Design review of CAD assemblies using bimanual natural interface,” International Journal...
  • L. Hou, X. Wang, L. Bernold, P. E. D. Love, “Using Animated Augmented Reality to Cognitively Guide Assembly,” Journal of Computing in Civil Engineering,...
  • L. Hou, X. Wang, M. Truijens, “Using Augmented Reality to Facilitate Piping Assembly: An Experiment-Based Evaluation,” Journal of Computing...
  • X. Wang, S. K. Ong, A. Y. Nee, “A comprehensive survey of augmented reality assembly research,” Advances in Manufacturing, vol. 4, pp. 1–22,...
  • Industry 4.0 - A Literature review,” in Procedia Computer Science, vol. 181, 2021, pp. 51–58, Elsevier B.V. doi: 10.1016/j.procs.2021.01.104.
  • A. Bécue, I. Praça, J. Gama, “Artificial intelligence, cyber-threats and Industry 4.0: challenges and opportunities,” Artificial Intelligence...
  • A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, “Attention Is All You Need,” in Advances...
  • G. W. Scurati, M. Gattullo, M. Fiorentino, F. Ferrise, M. Bordegoni, A. E. Uva, “Converting maintenance actions into standard symbols for Augmented...
  • D. K. Baroroh, C. H. Chu, L. Wang, “Systematic literature review on augmented reality in smart manufacturing: Collaboration between human...
  • P. Wang, X. Bai, M. Billinghurst, S. Zhang, X. Zhang, S. Wang, W. He, Y. Yan, H. Ji, “AR/MR Remote Collaboration on Physical Tasks: A Review,” Robotics...
  • M. Sereno, X. Wang, L. Besancon, M. J. McGuffin, T. Isenberg, “Collaborative Work in Augmented Reality: A Survey,” IEEE Transactions on Visualization...
  • B. Marques, S. Silva, J. Alves, T. Araujo, P. Dias, B. S. Santos, “A Conceptual Model and Taxonomy for Collaborative Augmented Reality,” IEEE...
  • B. Marques, S. Silva, J. Alves, A. Rocha, P. Dias, B. S. Santos, “Remote collaboration in maintenance contexts using augmented reality: insights from...
  • V. Elia, M. G. Gnoni, A. Lanzilotto, “Evaluating the application of augmented reality devices in manufacturing from a process point of view:...
  • M. L. Yuan, S. K. Ong, A. Y. Nee, “Augmented reality for assembly guidance using a virtual interactive tool,” International Journal of Production Research,...
  • S. K. Ong, Z. B. Wang, “Augmented assembly technologies based on 3D bare-hand interaction,” CIRP Annals - Manufacturing Technology, vol. 60, no....
  • D. Mourtzis, V. Siatras, J. Angelopoulos, “Real-time remote maintenance support based on augmented reality (AR),” Applied Sciences (Switzerland), vol....
  • A. Gilchrist, “Introducing Industry 4.0,” in Industry 4.0, Springer, 2016, ch. 13, pp. 195–215, doi: 10.1007/978-1-4842-2047-4.
  • Z. Ziaei, A. Hahto, J. Mattila, M. Siuko, L. Semeraro, “Real-time markerless Augmented Reality for Remote Handling system in bad viewing conditions,”...
  • D. Tatić, B. Tešić, “The application of augmented reality technologies for the improvement of occupational safety in an industrial environment,” Computers...
  • A. Syberfeldt, O. Danielsson, M. Holm, L. Wang, “Dynamic Operator Instructions Based on Augmented Reality and Rule-based Expert Systems,”...
  • R. Palmarini, I. Fernández, D. Amo, D. Ariansyah, S. Khan, J. A. Erkoyuncu, R. Roy, “Fast Augmented Reality Authoring: Fast Creation of AR...
  • T. Zonta, C. A. da Costa, R. da Rosa Righi, M. J. de Lima, E. S. da Trindade, G. P. Li, “Predictive maintenance in the Industry 4.0: A systematic literature...
  • M. Casillo, F. Colace, L. Fabbri, M. Lombardi, A. Romano, D. Santaniello, “Chatbot in industry 4.0: An approach for training new employees,” in...
  • N. U. Moroff, E. Kurt, J. Kamphues, “Machine Learning and Statistics: A Study for assessing innovative Demand Forecasting Models,” Procedia...
  • B. Maschler, M. Weyrich, “Deep Transfer Learning for Industrial Automation: A Review and Discussion of New Techniques for Data- Driven Machine...
  • J. Redmon, A. Farhadi, “YOLOv3: An Incremental Improvement,” arXiv preprint arXiv:1804.02767, 2018, doi: 10.48550/arXiv.1804.02767.
  • S. Chidambaram, H. Huang, F. He, X. Qian, A. M. Villanueva, T. S. Redick, W. Stuerzlinger, K. Ramani, “ProcessAR: An augmented reality-based tool...
  • R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, “On the Opportunities and Risks of Foundation Models,” arXiv preprint arXiv:2108.07258,...
  • R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High- Resolution Image Synthesis with Latent Diffusion Models,” in Proceedings of...
  • A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, “Improving Language Understanding by Generative Pre-Training,” OpenAI, 2018.
  • J. M. Rožanec, I. Novalija, P. Zajec, K. Kenda, H. Tavakoli Ghinani, S. Suh, E. Veliou, D. Papamartzivanos, T. Giannetsos, S. A. Menesidou,...
  • A. Akundi, D. Euresti, S. Luna, W. Ankobiah, A. Lopes, I. Edinbarough, “State of Industry 5.0—Analysis and Identification of Current Research Trends,”...
  • O. Hardt, K. Nader, L. Nadel, “Decay happens: The role of active forgetting in memory,” Trends in Cognitive Sciences, vol. 17, pp. 111–120,...
  • O. O. Adesope, D. A. Trevisan, N. Sundararajan, “Rethinking the Use of Tests: A Meta-Analysis of Practice Testing,” Review of Educational Research,...
  • J. D. Karpicke, H. L. Roediger, “The critical importance of retrieval for learning,” Science, vol. 319, pp. 966–968, 2 2008, doi: 10.1126/ science.1152408.
  • D. Bissig, C. Lustig, “Who benefits from memory training?,” Psychological Science, vol. 18, pp. 720–726, 8 2007, doi: 10.1111/j.1467-9280.2007.01966.x.
  • M. Wolf, M. Kleindienst, C. Ramsauer, C. Zierler, E. Winter, “Current and future industrial challenges: demographic change and measures for...
  • N. J. Cepeda, H. Pashler, E. Vul, J. T. Wixted, D. Rohrer, “Distributed practice in verbal recall tasks: A review and quantitative synthesis,” Psychological...
  • S. K. Carpenter, N. J. Cepeda, D. Rohrer, S. H. Kang, H. Pashler, “Using Spacing to Enhance Diverse Forms of Learning: Review of Recent Research...
  • Y. Elazar, N. Kassner, S. Ravfogel, A. Ravichander, E. Hovy, H. Schütze, Y. Goldberg, “Measuring and Improving Consistency in Pretrained Language...
  • V. Sanh, L. Debut, J. Chaumond, T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108,...
  • Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, “RoBERTa: A Robustly Optimized BERT Pretraining...
  • C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, “Exploring the Limits of Transfer Learning with...
  • T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, “BLOOM: A 176B-Parameter Open-Access Multilingual Language Model,” arXiv preprint...
  • S. Arroni, Y. Galán, X. Guzmán-Guzmán, E. R. Nuñez-Valdez, A. Gómez, “Sentiment Analysis and Classification of Hotel Opinions in Twitter With...
  • Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, H. S. Zheng, D. Zhou, N. Houlsby, D. Metzler,...
  • Y. Tay, J. Wei, H. W. Chung, V. Q. Tran, D. R. So, S. Shakeri, X. Garcia, H. S. Zheng, J. Rao, Chowdhery, D. Zhou, D. Metzler, S. Petrov,...
  • Together, “GPT-JT,” 2022. [Online]. Available: https://huggingface.co/ togethercomputer/GPT-JT-6B-v1.
  • A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” arXiv preprint...