Immersive audio visual content is rapidly climbing the popularity ladder. From movies to games, sights and sounds are supplemented by other senses for a more immersive experience. The D-BOX motion system provides haptic feedback in theatre seats, while the KOR-FX vest processes and converts desirable audio portions from a game or movie into haptic effects that can translate into the heartthrob of your virtual game avatar. With immersive content pervading all forms of media, can smartphone be left far behind?
Apple championed this initiative and was recently awarded a patent (US 9,083,821) that describes methods to convert audio data into haptic vibrations in mobile devices. The invention talks of parsing an audio signal into high and low frequencies, shifting audio frequency ranges and converting audio data into haptic data to convey audible information to a hearing-impaired mobile user.
A hearing impaired person might be deaf to certain high frequencies, hence limiting the ability to fully realize information contained in an audio data set. For example, listening to music rich in high frequencies may be less enjoyable to a person who is partially or completely deaf to some or all of those high frequencies.
Mobile devices typically include a mechanism for providing haptic feedback. For example, a mobile phone may include a piezoelectric motor for instance to provide haptic feedback to a user.
The patented method involves converting one portion of the range of audio frequencies into haptic data, shifting a second portion of the range of audio frequencies to a lower range, and presenting the converted first portion and the shifted second portion to a user. The converted first portion – i.e. the haptic data -can be provided to the user with haptic feedback such as vibration, temperature variation, or electric stimulus. Presenting the shifted second portion may comprise providing the user with sounds corresponding to the shifted second portion via an audio output mechanism. The first portion and the second portion may be mutually exclusive or may partly overlap.
One of the main motivations of the invention includes enhanced audible data delivery to a hearing-impaired mobile user. By shifting the range of high frequencies that the user cannot hear into an audible frequency range, and converting it to a vibration pattern to convey sound effect information can provide the user a sensation of enjoying the original information in the audio.
However, Apple notes that the invention’s augmented listening experience is not limited to hearing-impaired users alone. One interesting portion of the invention might enable users to set up listening preferences that would automatically convert certain frequencies into haptic vibrations. For example, a user listening to a concert in rich high frequency can shift it to a different frequency or convert the sound into a preset vibration pattern. Such conversion can be done in real-time and conveyed to the user through any suitable audio reproduction device and/or haptic mechanism.
The invention can further be extrapolated to assign unique haptic patterns to real world everyday events. Fire alarms, emergency vehicles, car horns, screams, dog barks, environmental noise, phone rings and knocks on the door are a couple of instances where this technology can be leveraged. In response to detecting an ambient noise corresponding to these, a unique vibration pattern can be actuated by a haptic mechanism within a mobile device to augment the auditory information with haptic feedback. This might also prove to be life-saving!
Finally, among other embodiments, the patent also teaches that a hearing impaired user can correct his tune by playing a musical instrument and concurrently receiving haptic feedback informing whether he is playing the instrument in the correct tune.
(Featured image source: https://pixabay.com/p-560318/?no_redirect)