Robotics

Researchers are developing robotic “fingertips” that could give surgeons back their sense of touch during minimally invasive and robotic operations

Modern surgery has gone from long incisions to tiny cuts guided by robots and AI. In the process, however, surgeons have lost something vital: the chance to feel inside the body directly. Without palpation, it becomes harder to detect tissue abnormalities during an operation.

A group of surgeons and engineers across Europe is now trying to bring back this vital aspect of surgery.

Working within an EU-funded research collaboration called PALPABLE, they are developing a soft robotic “fingertip” that can sense how firm or soft tissue is during minimally invasive and robotic surgery. The research runs until the end of 2026, with a first prototype expected to be tested by surgeons around March 2026.

By combining optical sensing, soft robotics and AI, the team is designing a probe that mimics the way a fingertip presses and feels during surgery. It would gently probe organs and create a visual map of tissue stiffness, displayed on a screen to guide surgeons as they operate.

Losing the surgeon’s touch

For many surgeons, the loss of direct touch has been one of the quiet trade-offs of modern surgery.

“We started 30 years ago with open surgery and using our fingers,” said Professor Alberto Arezzo from the University of Turin, Italy. He specialises in minimally invasive and robotic surgery and mostly treats patients with colorectal cancer.

“Then we moved into the era of keyhole surgery, which reduced tactile feedback because we began to use long instruments,” he said.

From the 1990s, keyhole surgery became increasingly common, allowing surgeons to operate through small incisions with the help of a camera. Patients benefited from less trauma, shorter hospital stays and faster recovery.

But this came at the expense of physical touch. That matters because tumours often feel different from healthy tissue – stiffer, less pliable or irregular – important differences that experienced hands can detect.

Finding tumour margins

When operating on cancer, surgeons walk a fine line: remove too much tissue and function may suffer; remove too little and cancer may remain, and then spread again, requiring more surgery. 

“We don’t want to do that. We want it done in one shot,” said Dr Gadi Marom at Hadassah Medical Centre in Jerusalem, one of the clinicians involved in the research, who specialises in minimally invasive and robotic surgery on patients with stomach and oesophagus diseases. 

This is where sensing technology could help. By translating physical contact into visual information, such as a colour-coded map showing softer and firmer areas, surgeons could regain a functional equivalent of touch. 

“With a new instrument, we want to be able to determine the margins around a tumour,” said Marom.

Using light to feel

To do that, engineers on the team are turning to light.

The probe they are developing contains fibre-optic cables embedded in a soft, flexible tip. When pressed against tissue, the tip deforms and the light travelling through the fibres changes.

“A silicone dome presses against soft tissue, allowing us to map both the direction and the magnitude of the applied force,” explained Dr Georgios Violakis at Hellenic Mediterranean University in Heraklion, Crete. 

Those tiny shifts in light intensity and wavelength are then translated into information about tissue stiffness.

In the lab, the team has already built and calibrated early versions of the soft membrane and light-based sensors, with partners contributing across the system.

Queen Mary University of London (UK) is helping design and refine the membranes, the Fraunhofer Institute (Germany) is developing the functional films, while Bendabl (Greece), Tech Hive Labs (Greece) and the University of Essex (UK) are advancing the software needed to visualise stiffness and tactile maps.

The prototype will be validated in lab tests before it is used on patients.

The fibre‑optic cables are each about the width of a human hair. Similar sensing technology has long been used to detect small movements in large structures such as aircraft, skyscrapers and nuclear reactors. Here it is being applied on a much smaller scale to detect subtle differences in human tissue. 

“For touching organs inside an anaesthetised patient, the device needs to be both highly accurate and high resolution,” said Professor Panagiotis Polygerinos, a soft robotics researcher at Hellenic Mediterranean University. 

“Something like this might have been possible sooner, but the technology would have been far more expensive and less precise, making it impractical for clinical use.” 

Bringing touch to robots

As surgery grows increasingly robotic, the loss of tactile feedback is becoming more pressing – and restoring a sense of touch even more vital.

“When I operate with a robot I have the advantage of 3D vision,” said Marom. “And I don’t have to stand for the entire surgery.” That matters in long procedures, such as removing a patient’s oesophagus, which can take up to eight hours.

Robotic surgery also raises new possibilities. Marom hopes it may eventually allow surgeons, in carefully selected cases, to remove small tumours from the oesophagus without removing the entire organ.

But there is a downside.

“In robotic surgery, tactile feedback is largely absent,” said Arezzo. “That’s why this work is so important.”

Both surgeons believe robotics will continue to expand in operating theatres, but only if surgeons are given better sensory information.

“Sooner or later, I believe the vast majority of surgeries will be robotic,” said Arezzo.

For Marom, working closely with engineers has been essential. “I am exposed to soft robotics and many new technologies,” he said. “I see how new instruments can be developed.”

“The bottom line is that we will be able to give better care to our patients,” he added.

Research in this article was funded by the EU’s Horizon Programme. The views of the interviewees don’t necessarily reflect those of the European Commission. If you liked this article, please consider sharing it on social media.

Text. Anthony King

This article was originally published in Horizon the EU Research and Innovation Magazine

This article was originally published in Horizon the EU Research and InnovationThis article was originally published in Horizon the EU Research and Innovation

The new multimodal system is one product of a three-year collaboration between Caltech's Center for Autonomous Systems and Technologies (CAST) and the Technology Innovation Institute (TII) in Abu Dhabi, United Arab Emirates. 

The robotic system demonstrates the kind of innovative and forward-thinking projects that are possible with the combined global expertise of the collaborators in autonomous systems, artificial intelligence, robotics, and propulsion systems.

"Right now, robots can fly, robots can drive, and robots can walk. Those are all great in certain scenarios," says Aaron Ames, the director and Booth-Kresa Leadership Chair of CAST and the Bren Professor of Mechanical and Civil Engineering, Control and Dynamical Systems, and Aerospace at Caltech. "But how do we take those different locomotion modalities and put them together into a single package, so we can excel from the benefits of all these while mitigating the downfalls that each of them have?"

Testing the capability of the X1 system, the team recently conducted a demonstration on Caltech's campus. The demo was based on the following premise: Imagine that there is an emergency somewhere on campus, creating the need to quickly get autonomous agents to the scene. For the test, the team modified an off-the-shelf Unitree G1 humanoid such that it could carry M4, Caltech's multimodal robot that can both fly and drive, as if it were a backpack.

The demo started with the humanoid in Gates–Thomas Laboratory. It walked through Sherman Fairchild Library and went outside to an elevated spot where it could safely deploy M4. The humanoid then bent forward at the waist, allowing M4 to launch in its drone mode. M4 then landed and transformed into driving mode to efficiently continue on wheels toward its destination. Before reaching that destination, however, M4 encountered the Turtle Pond, so it switched back to drone mode, quickly flew over the obstacle, and made its way to the site of the "emergency" near Caltech Hall. The humanoid and a second M4 eventually met up with the first responder.

"The challenge is how to bring different robots to work together so, basically, they become one system providing different functionalities. With this collaboration, we found the perfect match to solve this," says Mory Gharib (PhD '83), the Hans W. Liepmann Professor of Aeronautics and Medical Engineering at Caltech and CAST's founding director.

Gharib's group, which originally built the M4 robot, focuses on building flying and driving robots as well as advanced control systems. The Ames lab, for its part, brings expertise in locomotion and developing algorithms for the safe use of humanoid robots. Meanwhile, TII brings a wealth of knowledge about autonomy and sensing with robotic systems in urban environments. A Northeastern University team led by engineer Alireza Ramezani assists in the area of morphing robot design.

"The overall collaboration atmosphere was great. We had different researchers with different skill sets looking at really challenging robotics problems spanning from perception and sensor data fusion to locomotion modeling and controls, to hardware design," says Ramezani, an associate professor at Northeastern.

When TII engineers visited Caltech in July 2025, the partners built a new version of M4 that takes advantage of Saluki, a secure flight controller and computer technology developed by TII for onboard computing. In a future phase of work, the collaboration aims to give the entire system sensors, model-based algorithms, and machine learning-driven autonomy to navigate and adapt to its surroundings in real time.

"We install different kinds of sensors—lidar, cameras, range finders—and we combine all these data to understand where the robot is, and the robot understands where it is in order to go from one point to another," says Claudio Tortorici, director of TII. "So, we bring the capability of the robots to move around with autonomy."

Ames explains that even more was on display in the demo than meets the eye. For example, he says, the humanoid robot did more than simply walking around campus. Currently, the majority of humanoid robots are given data originally captured from human movements to achieve a particular movement, such as walking or kicking, and scaling that action to the robot. If all goes well, the robot can imitate that action repeatedly. But, Ames argues, "If we want to really deploy robots in complicated scenarios in the real world, we need to be able to generate these actions without necessarily having human references."

His group builds mathematical models that describe the physics of that application to a robot more broadly. When these are fused with machine learning techniques, the models imbue robots with more general abilities to navigate any situation they might encounter. "The robot learns to walk as the physics dictate," Ames says. "So X1 can walk; it can walk on different terrain types; it can walk up and down stairs, and importantly, it can walk with things like M4 on its back."

An overarching goal of the collaboration is to make such autonomous systems safer and more reliable. "I believe we are at a stage where people are starting to accept these robots," Tortorici says. " In order to have robots all around us, we need these robots to be reliable."

That is ongoing work for the team. "We're thinking about safety-critical control, making sure we can trust our systems, making sure they're secure," Ames says. "We have multiple projects that extend beyond this one that study all these different facets of autonomy, and these problems are really big. By having these different projects and facets of our collaboration, we are able to take on these much bigger problems and really move autonomy forward in a substantial and concerted way."

Text: Kimm Fesenmaier

https://www.caltech.edu/about/news/caltech-and-technology-innovation-institute-demo-multirobot-response-team

Image: Engineers from the CAST/TII collaboration with X1, a humanoid robot that can carry and launch M4, Caltech's morphing robot.Credit: Academic Media Technologies/Caltech

EU-funded researchers have developed robust mini robots with advanced sensors to help search and rescue teams find survivors in the aftermath of earthquakes and other disasters

In the critical 72 hours after an earthquake or explosion, a race against the clock begins to find survivors. After that window, the chances of survival drop sharply.

When a powerful earthquake hit central Italy on 24 August 2016, killing 299 people, over 5 000 emergency workers were mobilised in search and rescue efforts that saved dozens from the rubble in the immediate aftermath.

But the pressure to move fast can create risks for first responders, who often face unstable environments with little information about the dangers ahead. But this type of rescue work could soon become safer and more efficient thanks to a joint effort by EU and Japanese researchers.

Supporting first responders

Rescue organisations, research institutes and companies from both Europe and Japan worked together from 2019 to 2023 to develop a new generation of tools blending robotics, drone technology and chemical sensing to transform how emergency teams operate in disaster zones.

Their work was part of a four-year EU-funded international research initiative called CURSOR, which included partners from six EU countries, Norway and the UK. It also included Tohoku University, whose involvement was funded by the Japan Science and Technology Agency.

The researchers hope that the sophisticated rescue kit they have developed will help rescue workers locate trapped survivors faster, while also improving their own safety.

“In the field of search and rescue, we don’t have many technologies that support first responders, and the technologies that we do have, have a lot of limitations,” said Tiina Ristmäe, a research coordinator at the German Federal Agency for Technical Relief and vice president of the International Forum to Advance First Responder Innovation.

Meet the rescue bots

At the heart of the researcher’s work is a small robot called Soft Miniaturised Underground Robotic Finder (SMURF). The robot is designed to navigate through collapsed buildings and rubble piles to locate people who may be trapped underneath.

The idea is to allow rescue teams to do more of their work remotely, localising and finding humans from the most hazardous areas in the early stages of a rescue operation. The SMURF can be remotely controlled by operators who stay at a safe distance from the rubble.

“It is a prototype technology that did not exist before,” said Ristmäe. “We don’t send people, we send machines – robots – to do the often very dangerous job.”

The SMURF is compact and lightweight, with a two-wheel design that allows it to manoeuvre over debris and climb small obstacles.

“It moves and drops deep into the debris to find victims, with multiple robots covering the whole rubble pile,” said Professor Satoshi Tadokoro, a robotics expert at Tohoku University and one of the project’s lead scientists.

The development team tested many designs before settling on the final SMURF prototype.

“We investigated multiple options – multiple wheels or tracks, flying robots, jumping robots – but we concluded that this two-wheeled design is the most effective,” said Tadokoro.

Sniffing for survivors

The SMURF’s small “head” is packed with technology: video and thermal cameras, microphones and speakers for two-way communication, and a powerful chemical sensor known as the SNIFFER.

This sensor is capable of detecting substances that humans naturally emit, such as C02 and ammonia, and can even distinguish between living and deceased individuals.

Put to the test in real-world conditions, the SNIFFER has proved able to provide reliable information even when surrounded by competing stimuli, like smoke or rain.

According to the first responders who worked with the researchers, the information provided by the SNIFFER is highly valuable: it helps them to prioritise getting help to those who are still alive, said Ristmäe.

Drone delivery

To further improve the reach of the SMURF, the researchers also integrated drone support into the system. Customised drones are used to deliver the robots directly to the areas where they’re needed most – places that may be hard or dangerous to access on foot.

“You can transport several robots at the same time and drop them in different locations,” said Ristmäe.

Alongside these delivery drones, the CURSOR team developed a fleet of aerial tools designed to survey and assess disaster zones. One of the drones, dubbed the “mothership,” acts as a flying communications hub, linking all the devices on the ground with the rescue team’s command centre.

Other drones carry ground-penetrating radar to detect victims buried beneath debris. Additional drones capture overlapping high-definition footage that can be stitched together into detailed 3D maps of the affected area, helping teams to visualise the layout and plan their operations more strategically.

Along with speeding up search operations, these steps should slash the time emergency workers spend in dangerous locations like collapsed buildings.

Testing in the field

The combined system has already undergone real-world testing, including large-scale field trials in Japan and across Europe.

One of the most comprehensive tests took place in November 2022 in Afidnes, Greece, where the full range of CURSOR technologies was used in a simulated disaster scenario.

Though not yet commercially available, the prototype rescue kit has sparked global interest.

“We’ve received hundreds of requests from people wanting to buy it,” said Ristmäe. “We have to explain it’s not deployable yet, but the demand is there.”

The CURSOR team hopes to secure more funding to further enhance the technology and eventually bring it to market, potentially transforming the future of disaster response.

Research in this article was funded by the EU’s Horizon Programme. The views of the interviewees don’t necessarily reflect those of the European Commission.

Author: Michael Allen

This article was originally published in Horizon the EU Research and Innovation Magazine.

The SMARTEDGE project is using edge AI to transform autonomous robot collaboration and communication in factories and more

Since its launch in January 2023, the EU-funded SMARTEDGE project has been creating tools and methods that facilitate the development of edge intelligence solutions for real-world scenarios. Taking advantage of edge computing and the Internet of things to tackle challenges and optimise processes, the project is testing these methods and tools in smart factories, health monitoring, road intersection safety, manufacturing innovation and driving assistance.

One of these tools is smart swarm networks currently being developed in a lab at SMARTEDGE project coordinator National Inter-University Consortium for Telecommunications in Italy. At the lab, which emulates a factory floor at Dell Technologies, lead researcher Filippo Cugini and his colleagues are testing their software and hardware on three networked robots. The aim is to form and control a robot swarm using edge intelligence.

“We’re reproducing an operational area of the factory floor so we can understand exactly what happens to these autonomous robots,” explains Cugini in a news item published on ‘EE Times Europe’. “For example, we can walk through the lab, and the robots’ cameras will recognize there’s a person there as an obstacle and immediately stop.” The researcher goes on to explain that such factory settings require strong security and dynamic networking capabilities. “We also need to make sure that a robot claiming to be a certain robot is exactly that robot,” Cugini adds. “There are many issues.”

This work is the networking component of SMARTEDGE’s research into the development of autonomous mobile robot swarms that can communicate and collaborate on the edge. Other areas of research include a low-code engineering toolchain for edge intelligence developed by project partners Siemens, Dell Technologies and Technische Universität Berlin, and advanced computer vision developed by project partner Bosch. Another partner, Nvidia, has contributed to the project its BlueField-2 smart network interface card, a hardware accelerator.

Better communication, low latency

Cugini and his team have also developed a software layer that manages communications between robots, performing security validation and preventing messages from being intercepted. Latency limitations were taken into account during development: “After a couple years of working on this, we can say that this thin software layer doesn’t really add a lot of extra latency [to swarm networking], and the swarm can adapt to network changes in much less than 1 second,” Cugini notes, adding: “The extra latency that we added to enable this additional security and dynamicity—and, in total, the networking capability—strongly depends on the underlying hardware. If we use a simple Raspberry Pi that costs a few tens of euros, performance is on par with a home Wi-Fi network, but if we have a powerful Nvidia data processing unit, we’ve got a super-fast 100 gigabits per second or more.”

Experiments have only been conducted with up to 20 autonomous robots in a swarm, but – as reported in the news item – in simulations there can be 1 000. “We’ll never have 1,000 devices joining and leaving simultaneously on a factory floor, but we’re also considering swarms of cars in the project’s automotive use case,” Cugini explains. For this case, the SMARTEDGE (Semantic Low-code Programming Tools for Edge Intelligence) team ultimately aims to include around 1 000 autonomous vehicles in the combined practical and simulation studies.

Image: KW Foundation

In addition to training future players, the technology could expand the capabilities of other humanoid robots, such as for search and rescue

MIT engineers are getting in on the robotic ping pong game with a powerful, lightweight design that returns shots with high-speed precision.

The new table tennis bot comprises a multijointed robotic arm that is fixed to one end of a ping pong table and wields a standard ping pong paddle. Aided by several high-speed cameras and a high-bandwidth predictive control system, the robot quickly estimates the speed and trajectory of an incoming ball and executes one of several swing types — loop, drive, or chop — to precisely hit the ball to a desired location on the table with various types of spin.

In tests, the engineers threw 150 balls at the robot, one after the other, from across the ping pong table. The bot successfully returned the balls with a hit rate of about 88 percent across all three swing types. The robot’s strike speed approaches the top return speeds of human players and is faster than that of other robotic table tennis designs.

Now, the team is looking to increase the robot’s playing radius so that it can return a wider variety of shots. Then, they envision the setup could be a viable competitor in the growing field of smart robotic training systems.

Beyond the game, the team says the table tennis tech could be adapted to improve the speed and responsiveness of humanoid robots, particularly for search-and-rescue scenarios, and situations in which a robot would need to quickly react or anticipate.

“The problems that we’re solving, specifically related to intercepting objects really quickly and precisely, could potentially be useful in scenarios where a robot has to carry out dynamic maneuvers and plan where its end effector will meet an object, in real-time,” says MIT graduate student David Nguyen.

Nguyen is a co-author of the new study, along with MIT graduate student Kendrick Cancio and Sangbae Kim, associate professor of mechanical engineering and head of the MIT Biomimetics Robotics Lab. The researchers will present the results of those experiments in a paper at the IEEE International Conference on Robotics and Automation (ICRA) this month.

Precise Play

Building robots to play ping pong is a challenge that researchers have taken up since the 1980s. The problem requires a unique combination of technologies, including high-speed machine vision, fast and nimble motors and actuators, precise manipulator control, and accurate, real-time prediction, as well as higher-level planning of game strategy.

“If you think of the spectrum of control problems in robotics, we have on one end manipulation, which is usually slow and very precise, such as picking up an object and making sure you’re grasping it well. On the other end, you have locomotion, which is about being dynamic and adapting to perturbations in your system,” Nguyen explains. “Ping pong sits in between those. You’re still doing manipulation, in that you have to be precise in hitting the ball, but you have to hit it within 300 milliseconds. So, it balances similar problems of dynamic locomotion and precise manipulation.”

Ping pong robots have come a long way since the 1980s, most recently with designs by Omron and Google DeepMind that employ artificial intelligence techniques to “learn” from previous ping pong data, to improve a robot’s performance against an increasing variety of strokes and shots. These designs have been shown to be fast and precise enough to rally with intermediate human players.

“These are really specialized robots designed to play ping pong,” Cancio says. “With our robot, we are exploring how the techniques used in playing ping pong could translate to a more generalized system, like a humanoid or anthropomorphic robot that can do many different, useful things.”

Game Control

For their new design, the researchers modified a lightweight, high-power robotic arm that Kim’s lab developed as part of the MIT Humanoid — a bipedal, two-armed robot about the size of a small child. The group is using the robot to test various dynamic maneuvers, including navigating uneven terrain, jumping, running, and doing backflips, aiming to deploy such robots for search-and-rescue operations.

Each of the humanoid’s arms has four joints, or degrees of freedom, each controlled by an electrical motor. Cancio, Nguyen, and Kim built a similar robotic arm, adapted for ping pong with an extra degree of freedom in the wrist to control the paddle.

The team fixed the arm to one end of a standard ping pong table and set up high-speed motion capture cameras to track incoming balls. They developed control algorithms that predict, using math and physics, the paddle orientation and speed needed to return each type of swing — loop, drive, or chop.

Using three computers to process camera images and convert predictions into motor commands, the robot responded in real time. In tests, it returned 150 consecutive balls with nearly equal success across all swings: 88.4% for loop, 89.2% for chop, and 87.5% for drive. Later tests showed the robot could hit balls at 20 meters per second — faster than existing systems.

The average strike speed was 11 meters per second, approaching human advanced players (21–25 m/s). Further tuning pushed the robot’s speed to 19 meters per second (42 mph).

“Some of the goal of this project is to say we can reach the same level of athleticism that people have,” Nguyen says. “And in terms of strike speed, we’re getting really, really close.”

They also implemented aiming capabilities. The robot can now hit a ball to a preset location using trajectory prediction. However, since it's fixed to the table, it can only cover a crescent-shaped area. The team plans to mount it on a gantry or mobile base for greater range.

“A big thing about table tennis is predicting the spin and trajectory of the ball, given how your opponent hit it, which is information that an automatic ball launcher won’t give you,” Cancio says. “A robot like this could mimic real-game conditions to help humans train.”

This research is supported in part by the Robotics and AI Institute.

Source: Jennifer Chu | MIT News

A new approach could enable intuitive robotic helpers for household, workplace, and warehouse settings

For a robot, the real world is a lot to take in. Making sense of every data point in a scene can take a huge amount of computational effort and time. Using that information to then decide how to best help a human is an even thornier exercise.

Now, MIT roboticists have a way to cut through the data noise, to help robots focus on the features in a scene that are most relevant for assisting humans.

Their approach, which they aptly dub “Relevance,” enables a robot to use cues in a scene, such as audio and visual information, to determine a human’s objective and then quickly identify the objects that are most likely to be relevant in fulfilling that objective. The robot then carries out a set of maneuvers to safely offer the relevant objects or actions to the human.

The researchers demonstrated the approach with an experiment that simulated a conference breakfast buffet. They set up a table with various fruits, drinks, snacks, and tableware, along with a robotic arm outfitted with a microphone and camera. Applying the new Relevance approach, they showed that the robot was able to correctly identify a human’s objective and appropriately assist them in different scenarios.

In one case, the robot took in visual cues of a human reaching for a can of prepared coffee, and quickly handed the person milk and a stir stick. In another scenario, the robot picked up on a conversation between two people talking about coffee, and offered them a can of coffee and creamer.

Overall, the robot was able to predict a human’s objective with 90 percent accuracy and to identify relevant objects with 96 percent accuracy. The method also improved a robot’s safety, reducing the number of collisions by more than 60 percent, compared to carrying out the same tasks without applying the new method.

“This approach of enabling relevance could make it much easier for a robot to interact with humans,” says Kamal Youcef-Toumi, professor of mechanical engineering at MIT. “A robot wouldn’t have to ask a human so many questions about what they need. It would just actively take information from the scene to figure out how to help.”

Youcef-Toumi’s group is exploring how robots programmed with Relevance can help in smart manufacturing and warehouse settings, where they envision robots working alongside and intuitively assisting humans.

Youcef-Toumi, along with graduate students Xiaotong Zhang and Dingcheng Huang, will present their new method at the IEEE International Conference on Robotics and Automation (ICRA) in May. The work builds on another paper presented at ICRA the previous year.

Finding focus

The team’s approach is inspired by our own ability to gauge what’s relevant in daily life. Humans can filter out distractions and focus on what’s important, thanks to a region of the brain known as the Reticular Activating System (RAS). The RAS is a bundle of neurons in the brainstem that acts subconsciously to prune away unnecessary stimuli, so that a person can consciously perceive the relevant stimuli. The RAS helps to prevent sensory overload, keeping us, for example, from fixating on every single item on a kitchen counter, and instead helping us to focus on pouring a cup of coffee.

“The amazing thing is, these groups of neurons filter everything that is not important, and then it has the brain focus on what is relevant at the time,” Youcef-Toumi explains. “That’s basically what our proposition is.”

He and his team developed a robotic system that broadly mimics the RAS’s ability to selectively process and filter information. The approach consists of four main phases. The first is a watch-and-learn “perception” stage, during which a robot takes in audio and visual cues, for instance from a microphone and camera, that are continuously fed into an AI “toolkit.” This toolkit can include a large language model (LLM) that processes audio conversations to identify keywords and phrases, and various algorithms that detect and classify objects, humans, physical actions, and task objectives. The AI toolkit is designed to run continuously in the background, similarly to the subconscious filtering that the brain’s RAS performs.

The second stage is a “trigger check” phase, which is a periodic check that the system performs to assess if anything important is happening, such as whether a human is present or not. If a human has stepped into the environment, the system’s third phase will kick in. This phase is the heart of the team’s system, which acts to determine the features in the environment that are most likely relevant to assist the human.

To establish relevance, the researchers developed an algorithm that takes in real-time predictions made by the AI toolkit. For instance, the toolkit’s LLM may pick up the keyword “coffee,” and an action-classifying algorithm may label a person reaching for a cup as having the objective of “making coffee.” The team’s Relevance method would factor in this information to first determine the “class” of objects that have the highest probability of being relevant to the objective of “making coffee.” This might automatically filter out classes such as “fruits” and “snacks,” in favor of “cups” and “creamers.” The algorithm would then further filter within the relevant classes to determine the most relevant “elements.” For instance, based on visual cues of the environment, the system may label a cup closest to a person as more relevant — and helpful — than a cup that is farther away.

In the fourth and final phase, the robot would then take the identified relevant objects and plan a path to physically access and offer the objects to the human.

Helper mode

The researchers tested the new system in experiments that simulate a conference breakfast buffet. They chose this scenario based on the publicly available Breakfast Actions Dataset, which comprises videos and images of typical activities that people perform during breakfast time, such as preparing coffee, cooking pancakes, making cereal, and frying eggs. Actions in each video and image are labeled, along with the overall objective (frying eggs, versus making coffee).

Using this dataset, the team tested various algorithms in their AI toolkit, such that, when receiving actions of a person in a new scene, the algorithms could accurately label and classify the human tasks and objectives, and the associated relevant objects.

In their experiments, they set up a robotic arm and gripper and instructed the system to assist humans as they approached a table filled with various drinks, snacks, and tableware. They found that when no humans were present, the robot’s AI toolkit operated continuously in the background, labeling and classifying objects on the table.

When, during a trigger check, the robot detected a human, it snapped to attention, turning on its Relevance phase and quickly identifying objects in the scene that were most likely to be relevant, based on the human’s objective, which was determined by the AI toolkit.

“Relevance can guide the robot to generate seamless, intelligent, safe, and efficient assistance in a highly dynamic environment,” says co-author Zhang.

Going forward, the team hopes to apply the system to scenarios that resemble workplace and warehouse environments, as well as to other tasks and objectives typically performed in household settings.

“I would want to test this system in my home to see, for instance, if I’m reading the paper, maybe it can bring me coffee. If I’m doing laundry, it can bring me a laundry pod. If I’m doing repair, it can bring me a screwdriver,” Zhang says. “Our vision is to enable human-robot interactions that can be much more natural and fluent.”

Text: Jennifer Chu | MIT News

This research was made possible by the support and partnership of King Abdulaziz City for Science and Technology (KACST) through the Center for Complex Engineering Systems at MIT and KACST.