Investigating Sensory Extensions as Input for Interactive Simulations

Chris Hill*, Casey Hunt*, Sammie Crowder, Brett L. Fielder, Emily B. Moore, Ann Eisenberg

(The first two authors equally contributed to the publication)


Sensory extensions enhance our awareness by transforming variations in stimuli normally undetectable by human senses into perceivable outputs. Similarly, interactive simulations for learning promote an understanding of abstract phenomena. Multimodal experiences combining sensory extension devices with interactive simulations give users the novel opportunity to connect their sensory experiences in the physical world to computer-simulated concepts. We explore this opportunity by designing a suite of wearable sensory extension devices that interface with a uniquely inclusive PhET Simulation, Ratio and Proportion. In this simulation, two hands can be moved on-screen to various values, representing different mathematical ratios. Users explore changing hand heights to find and maintain ratios through visual and auditory feedback. Our sensory extension devices translate force, distance, sound frequency, and magnetic field strength to quantitative values in order to control individual hands in the computer simulation.