This question fuels the research of Dr. Razib Iqbal, associate professor of computer science at Missouri State University. Much of Iqbal’s research revolves around multimedia and the Internet of Things, or IoT.

What is IoT? It could be anything connected to the internet — from a smart meter and appliances to smart smoke detectors and wearables.

These things are capable of collecting information about the physical world around us for processing.

The multimedia-based branch of IoT, called Internet of Multimedia Things, includes content like images, audio and video. This newer branch of technology opened the door to Iqbal’s latest interest: sensing the smart home more accurately via multimedia.

Humans as sensors

What does the future of the smart home look like from Iqbal’s perspective?  Here’s a clue: Humans serve as sensors in the ecosystem of connected things.

“Using ordinary microphones and cameras, we can gather information about users’ states,” Iqbal said. “This content is key to improving the smart home experience.”

Imagine turning up the heat in a room by saying it was cold or using sign language to turn lights on or off. Picture a system that prompts you to play a specific song or adjust a room’s atmosphere to match your mood.

“This includes recognizing voice commands and gestures, detecting emotions and differentiating between humans and pets or other objects, among other forms of sensing.”

These uses could support users’ convenience and mental well-being. Others could contribute to their physical safety and security.

“If an intruder entered the smart home, the more intuitive system could detect fear in a homeowner’s voice,” Iqbal said. “Then it could automatically dial a number and stream the conversation occurring in the home.”

The system could especially benefit a more vulnerable population of loved ones: the elderly. They may experience memory impairment, which makes remembering device commands for Amazon Alexa and Google Home difficult.

Elderly individuals also often experience falls, which the smart home system could detect.

“Existing fall detection systems rely on physical sensors. Some people take them off before showering or going to bed, rendering them useless,” Iqbal said. “Being able to analyze video data of smart devices to detect falls could be life-saving.”

Meeting in the middleware

Smart homes contain sensors and devices that allow homeowners to automate tasks remotely using a computer or phone.

These devices and sensors use set commands to control a limited range of functions – like turning things on or off. Their traditional IoT system can only process simple information with little data.

“Multimedia data is continuous and bulky. It demands more computer memory, bandwidth and processing power for real-time decision making,” Iqbal said. “Adding in a multimedia component to the IoT burdens the system further.”

Iqbal published research in the Elsevier Internet of Things journal. The article presents a novel framework for overcoming this challenge. Iqbal proposes that the high-processing software, called middleware, step in to help.

“He is one of the bright stars within our college and the greater university.” – Dr. Ridwan Sakidja

Middleware lies between the IoT operating system and the applications running on it. From there, it can intercept data from devices as they send it to the operating system.

Middleware converts multimedia content’s bulky data into a simpler form. Then, it sends the data back to the operating system to finish processing there.

Besides saving processing resources like battery power, middleware also saves users money.

“One smart device always leads to more. It would get very expensive if each had to process multimedia content itself,” Iqbal said. “Streaming the content to middleware is a much cheaper option — one that eliminates the need for expensive connecting devices.”

Building the foundation

Most of Iqbal’s research has centered on use of Internet of Multimedia Things in smart homes since joining Missouri State’s faculty in 2015. In his academic career, he has published more than 35 peer-reviewed articles in leading journals and conferences.

Iqbal is the founding director of the Multimedia Systems and Communications Laboratory at Missouri State. He collaborates with both undergraduate and graduate students on research there.

“I see it as an opportunity to instill in each student an understanding of and capacity for problem-solving, independent judgement and intellectual honesty.”

Iqbal’s research on multimedia content began as a graduate student, so he recognizes the importance of training his own students early on.

“Some might view it as an entryway into a research career,” Iqbal said. “I see it as an opportunity to instill in each student an understanding of and capacity for problem-solving, independent judgement and intellectual honesty.”

The work they produce will likely serve to make future technology more useful and user-friendly.

“Iqbal is always very open to exploring new ideas through collaboration,” Dr. Ridwan Sakidja, professor of physics and research collaborator of Iqbal, said. “He cares a great deal about his students’ success.”