Hi Philippe,
Very nice presentation and information on weather training and AR. Are there any links to the actual AR items used as seen in figures 1,2,and 3? I am interested in seeing how they operate and function for learning.
Thank you, Dr. Garry. The AR content was implemented in a testing application for Beta testing. The AR content will be made publicly available after the evaluations have been completed. I could share videos of the content in the next round of presentations if I made it. Thank you, Philippe.
danielliddle
1 year ago
I was wondering if you could add some more context about interactive print. Is this a printed sheet of paper with these same safety guidelines on them? Do the trainees read the printed sheet first then engage with the AR? Or does the printed text work simultaneously? Or is there no printed text at all? I understand what AR is, but I don’t understand how it was combined with print in this specific case to be considered interactive print.
Thank you for the question. The materials are considered interactive print because they are text-based materials overlayed with interactive AR activities. Students read the text-based materials until they see an AR marker. When they see the AR marker, they use the AR smartphone app to scan the AR marker and view the AR learning content through their smartphone screen. This process is similar to how a QR code works, using actual images in the text as markers for the AR content. Students read the instructions for the activity in the text, complete the learning activity with the 3D AR content, and then record their answers. After they have finished the AR activity, they close the application and continue reading the text. They repeat the process at each AR marker. The AR markers are placed into the document at specific points to provide students with context-appropriate weather information in their learning environment.
Jessica Campbell
1 year ago
I think semiotics plays a role in your study, specifically, how multisemiotic experiences support learning and cognition.
Thank you for the comment. I agree. Students use information from multiple modes to learn about weather. In this work, students are provided visual information (the thunderstorm model) to learn about thunderstorms. The visual information is an asset because it helps them learn about the visual appearances of weather phenomena and visual weather cues.
Jessica Campbell
1 year ago
What methods did you use to perform the usability study? How did you evaluate usability besides the pre-post summative knowledge testing? I am wondering if the participants had less of a learning curve since they were already familiar with the material.
Thank you for the questions and the comment. I respond to them below:
What methods did you use to perform the usability study?
The preliminary evaluation of the thunderstorm model was conducted with the methods of subject matter expert review (Pace & Sheehan, 2002) and an AR usability study for education (as described in Dey et al., 2018).
How did you evaluate usability besides the pre-post summative knowledge testing?
The usability of the AR activities was evaluated with measures of task time, task correctness, task load, system usability, and positives and improvements. Overall, the user evaluation of the AR learning activities included the following measures:
Factual knowledge (quiz score in percent) measure pre- and post-training
Visual knowledge (quiz score in percent) measured pre- and post-training
Task time (time in seconds) measured for every learning task
Task correctness (correctness in percent) measured for every learning task
Task load (NASA Task Load Index, scale 0-20) measured after every learning task (Hart & Staveland, 1988)
Content appropriateness (scale 1-5) measured after every task
Motivation (Reduced Instructional Materials Motivation Scale, scale 1-5) measured after the training (Loorbach et al., 2015)
System usability (System Usability Scale, scale 1-5) measured after the training (Lewis & Sauro, 2009)
Positive and improvements (statements) written after the training
I am wondering if the participants had less of a learning curve since they were already familiar with the material.
Student pilots who just started their training may not have had previous weather education. Compared to new student pilots, pilots with flight hours have experienced the required weather education and possibly more weather education. Further analysis based on the number of flight hours could help answer your question. Thank you for the question.
Dey, A., Billinghurst, M., Lindeman, R. W., & Swan, J. E. (2018). A systematic review of 10 Years of Augmented Reality usability studies: 2005 to 2014. Frontiers Robotics AI, 5(APR).
Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results from Empirical and Theoretical Research. Advances in Psychology., 52, 139–185.
Lewis, J. R., & Sauro, J. (2009). The factor structure of the system usability scale. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5619 LNCS, 94–103.
Loorbach, N., Peters, O., Karreman, J., & Steehouder, M. (2015). Validation of the Instructional Materials Motivation Survey (IMMS) in a self-directed instructional setting aimed at working with technology. British Journal of Educational Technology, 46(1), 204–218.
Pace, D. K., & Sheehan, J. (2002). Subject Matter Expert ( SME )/ Peer Use in M&S V&V. Proc. Workshop on Foundations for Modeling and Simulation (M&S) Verification and Validation (V&V) in the 21st Century.
Thank you for describing those methods and metrics and for providing the sources.
slee15
1 year ago
Hi Philipp, thank you for sharing this study. I am just wondering how you selected your participants. Any contextual factors that might have affected their interactive AR learning process? Did you select or screen samples with a certain consideration? Thank you!
Thank you for the questions. GA pilots with fewer than 1000 flight hours are at higher risk of weather-related accidents (Major et al., 2017). The pilots who may benefit most from the education are student pilots and pilots with less than 250 total flight hours. The studies included participants who were a GA student pilot or a pilot with fewer than 250 flight hours. The study protocol asked participants about their age, gender, flight certifications, total flight hours, instrument flight hours, flight hours in the past 6 months, previous weather courses, and hours of experience with AR. The information collected was used to describe the participant group.
Major, W. L., Carney, T., Keller, J. C., Xie, A., Price, M., Duncan, J. C., … Beaudin-Seiler, B. M. (2017). VFR-into-IMC Accident Trends: Perceptions of Deficiencies in Training. Journal of Aviation Technology and Engineering, 7(1), 50–57. https://doi.org/10.7771/2159-6670.1153
Hi Philippe, I have similar questions (sorry if it’s too close to the deadline to answer these): How did you recruit your participants- is there a pilot listserv or something you could access? Also, did you ask them if they had any prior AR experience? Thanks!
Thank you for the question. The study was advertised in courses at universities with aviation programs, in emails to flight schools, and campus-wide emails. After a participant signed up, I verified that they were a GA student pilot or a pilot with fewer than 250 flight hours. Once they began the study, the study protocol asked them to write their number of hours of experience with AR and describe their experiences with AR. In the evaluation of the AR activities, the 18 participants had an average of M = 29.8 (SD = 54.9) hours of experience using AR/VR, and they reported that it was primarily using VR for flight simulation.
Brandon Strubberg
1 year ago
Hi, Philippe. Thanks for sharing your work evaluating the use of AR in GA pilot weather training. You’ve already answered some of the questions I had in your responses to other commenters, so I’ll ask about other work in training and AR. In your research, what kinds of other such training, whether with passenger airline pilots and weather or other industries altogether, have you come across that was particularly insightful?
Thank you for the question. I find the work of investigating the impacts of integrating AR applications into classrooms to be particularly insighful. This review article is an example: Bacca, J., Baldiris, S., Fabregat, R., Graf, S., & Kinshuk. (2014). Augmented reality trends in education: A systematic review of research and applications. Educational Technology and Society, 17(4), 133–149.
Hi Philippe,
Very nice presentation and information on weather training and AR. Are there any links to the actual AR items used as seen in figures 1,2,and 3? I am interested in seeing how they operate and function for learning.
Thank you,
Steven P. Garry, PhD
Thank you, Dr. Garry. The AR content was implemented in a testing application for Beta testing. The AR content will be made publicly available after the evaluations have been completed. I could share videos of the content in the next round of presentations if I made it. Thank you, Philippe.
I was wondering if you could add some more context about interactive print. Is this a printed sheet of paper with these same safety guidelines on them? Do the trainees read the printed sheet first then engage with the AR? Or does the printed text work simultaneously? Or is there no printed text at all? I understand what AR is, but I don’t understand how it was combined with print in this specific case to be considered interactive print.
Thank you for the question. The materials are considered interactive print because they are text-based materials overlayed with interactive AR activities. Students read the text-based materials until they see an AR marker. When they see the AR marker, they use the AR smartphone app to scan the AR marker and view the AR learning content through their smartphone screen. This process is similar to how a QR code works, using actual images in the text as markers for the AR content. Students read the instructions for the activity in the text, complete the learning activity with the 3D AR content, and then record their answers. After they have finished the AR activity, they close the application and continue reading the text. They repeat the process at each AR marker. The AR markers are placed into the document at specific points to provide students with context-appropriate weather information in their learning environment.
I think semiotics plays a role in your study, specifically, how multisemiotic experiences support learning and cognition.
Thank you for the comment. I agree. Students use information from multiple modes to learn about weather. In this work, students are provided visual information (the thunderstorm model) to learn about thunderstorms. The visual information is an asset because it helps them learn about the visual appearances of weather phenomena and visual weather cues.
What methods did you use to perform the usability study? How did you evaluate usability besides the pre-post summative knowledge testing? I am wondering if the participants had less of a learning curve since they were already familiar with the material.
Thank you for the questions and the comment. I respond to them below:
What methods did you use to perform the usability study?
The preliminary evaluation of the thunderstorm model was conducted with the methods of subject matter expert review (Pace & Sheehan, 2002) and an AR usability study for education (as described in Dey et al., 2018).
How did you evaluate usability besides the pre-post summative knowledge testing?
The usability of the AR activities was evaluated with measures of task time, task correctness, task load, system usability, and positives and improvements. Overall, the user evaluation of the AR learning activities included the following measures:
I am wondering if the participants had less of a learning curve since they were already familiar with the material.
Student pilots who just started their training may not have had previous weather education. Compared to new student pilots, pilots with flight hours have experienced the required weather education and possibly more weather education. Further analysis based on the number of flight hours could help answer your question. Thank you for the question.
Dey, A., Billinghurst, M., Lindeman, R. W., & Swan, J. E. (2018). A systematic review of 10 Years of Augmented Reality usability studies: 2005 to 2014. Frontiers Robotics AI, 5(APR).
Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results from Empirical and Theoretical Research. Advances in Psychology., 52, 139–185.
Lewis, J. R., & Sauro, J. (2009). The factor structure of the system usability scale. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5619 LNCS, 94–103.
Loorbach, N., Peters, O., Karreman, J., & Steehouder, M. (2015). Validation of the Instructional Materials Motivation Survey (IMMS) in a self-directed instructional setting aimed at working with technology. British Journal of Educational Technology, 46(1), 204–218.
Pace, D. K., & Sheehan, J. (2002). Subject Matter Expert ( SME )/ Peer Use in M&S V&V. Proc. Workshop on Foundations for Modeling and Simulation (M&S) Verification and Validation (V&V) in the 21st Century.
Thank you for describing those methods and metrics and for providing the sources.
Hi Philipp, thank you for sharing this study. I am just wondering how you selected your participants. Any contextual factors that might have affected their interactive AR learning process? Did you select or screen samples with a certain consideration? Thank you!
Thank you for the questions. GA pilots with fewer than 1000 flight hours are at higher risk of weather-related accidents (Major et al., 2017). The pilots who may benefit most from the education are student pilots and pilots with less than 250 total flight hours. The studies included participants who were a GA student pilot or a pilot with fewer than 250 flight hours. The study protocol asked participants about their age, gender, flight certifications, total flight hours, instrument flight hours, flight hours in the past 6 months, previous weather courses, and hours of experience with AR. The information collected was used to describe the participant group.
Major, W. L., Carney, T., Keller, J. C., Xie, A., Price, M., Duncan, J. C., … Beaudin-Seiler, B. M. (2017). VFR-into-IMC Accident Trends: Perceptions of Deficiencies in Training. Journal of Aviation Technology and Engineering, 7(1), 50–57. https://doi.org/10.7771/2159-6670.1153
Hi Philippe, I have similar questions (sorry if it’s too close to the deadline to answer these): How did you recruit your participants- is there a pilot listserv or something you could access? Also, did you ask them if they had any prior AR experience? Thanks!
Thank you for the question. The study was advertised in courses at universities with aviation programs, in emails to flight schools, and campus-wide emails. After a participant signed up, I verified that they were a GA student pilot or a pilot with fewer than 250 flight hours. Once they began the study, the study protocol asked them to write their number of hours of experience with AR and describe their experiences with AR. In the evaluation of the AR activities, the 18 participants had an average of M = 29.8 (SD = 54.9) hours of experience using AR/VR, and they reported that it was primarily using VR for flight simulation.
Hi, Philippe. Thanks for sharing your work evaluating the use of AR in GA pilot weather training. You’ve already answered some of the questions I had in your responses to other commenters, so I’ll ask about other work in training and AR. In your research, what kinds of other such training, whether with passenger airline pilots and weather or other industries altogether, have you come across that was particularly insightful?
Thank you for the question. I find the work of investigating the impacts of integrating AR applications into classrooms to be particularly insighful. This review article is an example: Bacca, J., Baldiris, S., Fabregat, R., Graf, S., & Kinshuk. (2014). Augmented reality trends in education: A systematic review of research and applications. Educational Technology and Society, 17(4), 133–149.