Robots and artificial intelligence have been more than helpful at visualizing spaces and planning architectural designs. It is through these developments that people can enjoy new digital features like 3D walkthroughs.
Just recently, researchers at Facebook, University of Texas at Austin, and Carnegie Mellon University want to take it up a notch and have started exploring an AI tool that can reconstruct floor plans using short video clips.
According to the researchers, audio provides signals that complement image mapping because sound is driven by the geometry of objects. Audio bounces off surfaces and reveal the shape of the room. Even activities such as frying exposes the layouts based on where these are typically performed (i.e., the sound of frying shows the direction of the kitchen in the layout).
Called the AV-Map, the researchers’ approach converts videos into 2D floor plans. After gathering the audio and visual elements, the machine learning algorithm fuses the information using a decoder component. The floor plans created by AV-MP go beyond what the camera can capture, including discrete areas.
“A short video walk through a house can reconstruct the visible portions of the floorplan but is blind to many areas. We introduce audio-visual floor plan reconstruction, where sounds in the environment help infer both the geometric properties of the hidden areas as well as the semantic labels of the unobserved rooms (e.g., sounds of a person cooking behind a wall to the camera’s left suggest the kitchen),” shares the researchers through a paper detailing AV-Map.
In the future, they hope to refine the algorithm to infer multi-level floor plans, as well as add a robotic agent which can actively control the camera. “To our knowledge, ours is the first attempt to infer floor plans from audio-visual data.”
Add Comment