Stop Thief! Body language analysis software spots criminals in the crowd

The data analysis laboratory at the National Research Nuclear University MEPhI (Moscow Engineering Physics Institute) has developed software capable of detecting non-standard and potentially dangerous human behavior. The program has a variety of potential applications, such as identifying criminals in crowds.

“It's often the case that the perpetrators of a terrorist attack are captured on CCTV cameras several days prior to their attack, and subsequent analysis of the footage helps identify them,” said Vadim Danshin of MEPhI's Institute of Cyber-Intelligence Systems, who heads the project. “What gives them away is their behavior: instead of going about their business at the speed of the surrounding crowd, they are spying out CCTV cameras, analysing the terrain, and studying the behavioral patterns of police officers.”

How does it work?

Until now, such technology was only used in infrared cameras. The Russian developers are making it available for off-the-shelf webcams and smartphones. The camera may be installed in an arbitrary manner. The installation angle is entered into the program, which then computes target trajectories automatically.

All the algorithm needs is to be “shown” a moving person against a static background, and it then automatically analyzes the video and identifies the coordinates of the person's head, legs, elbow and knee joints, and gathers statistics on how they move. The coordinates are refreshed at a rate of 30 times per second. The software then builds a 3D model that moves synchronously with the person on the video footage.

This video shows the process.

“The purpose of this is to gather statistical data on how joints operate in the human body,” Danshin explained. “We will use this information to explain to a machine vision system how a waive of the hand in a casual conversation differs from an attempt to hit someone or steal personal belongings from someone in a crowd.”

People-monitoring algorithm

The algorithm is capable of identifying a person in a crowd by their gait, build, and the clothes they are wearing.

“If a person stands far away from the camera, the existing machine vision systems cannot recognize their face reliably,” Danshin said. “Our approach supports identification by additional parameters, such as clothes or a watch strap.”

The software has other potential uses: for example, it can measure time spent by a shopper in front of a particular display window to assess their interest in certain items for subsequent targeted advertising.

From sporting fans to hospital patients

The developers say their algorithm can also be used to create a new generation of interactive simulators for special forces, surgeons or firefighters. The software would gauge a test subject's reaction time in stressful situations to help improve efficiency.

The development team is currently testing the algorithm on video recordings of crowd behavior at soccer games in Germany. The goal is to prevent dangerous behavior by monitoring the body language of each individual fan captured on camera. In a separate project, which is being conducted in Canada, the algorithm is taught to recognize different types of behavior in hospital patients.

“Ordinary cameras simply record video, and you need a human to watch and analyze it,” Danshin said. “Machine vision systems can automate this process by singling out individuals in the crowd, classifying their behavior and then processing any exceptions.”

U.S. computer scientist Ben Usman believes similar algorithms could be used in vehicle control systems: “They are capable of predicting the behavior of pedestrians. The ambient conditions may present a problem though because the level of lighting and the quality of the camera used will result in imagery distortions. Creating a system that would be resilient to such distortions is a huge challenge.”

Read more: Kaspersky Lab teams up with futurologists to glimpse into Earth's future

If using any of Russia Beyond's content, partly or in full, always provide an active hyperlink to the original material.