Challenge: We want to enable robots to learn everyday manipulation tasks from the Web and thus make abstract knowledge actionable. This challenge is important for robots that can perform a wide breadth of manipulation tasks and task variations, that can be performed by various robots using various tools for a variety of purposes. For a robot to successfully perform such a breadth of maniulation actions, it needs to understand given tasks. We start tackling this challenge by focusing on selected task domains, starting with cutting.

The idea is that our methodology can teach robots to cut any fruit with any to for any purpose, as visualised on top. For this, the robot needs to query the knowledge graph to parameterise its general action plan.

“The EASE household challenge is subject of the IEEE spectrum article ‘It’s (Still) Really Hard for Robots to Autonomously Do Household Chores’” by Evan Ackerman (IEEE Spectrum, 17.06.2021)

Example Videos

Architecture

Action Execution To achieve our goal of enabling a robotic agent to handle unkown task variations by parameterising general action plans using web knowledge, we employ the following architecture: In general, the robot needs to have access to a general action designator of cutting that can be parameterised. When the robot is given a task request, it can either query the graph database with the knowledge graph directly via its SPARQL REST API or use a knowledge framework with additional functionalities such as the KnowRob knowledge processing system1 and pose Prolog queries, which then are translated to SPARQL queries.
Research, Architecture 

Calls

Call for Submission at AKR^3 2024 The First International Workshop on Actionable Knowledge Representation and Reasoning for Robots (AKR³) and the Tutorial on Contextualizing and Executing Robot Manipulation Plans Using Web Knowledge are both co-located with the Extended Semantic Web Conference 2024 (ESWC24). Find more information about the call for papers here. Tutorial at AKR^3 2024 The AKR^3 workshop also includes a Tutorial session where you can get to know our approach in a walk-through.
Active Calls, Open, Paper, Workshop 

Knowledge Acquisition

Gathering and Linking Web Knowledge To support robotic agents in executing variations of Cutting on different fruits and vegetables, we collect two types of knowledge in our knowledge graph: action and object knowledge. Both kinds of knowledge need to be linked to enable task execution as explained here. Action Knowledge The action knowledge covers all properties of a specific manipulation action that are necessary for successfully completing the action and is thus also influenced by the participating objects.
Research, Knowledge Acquisition, Web Knowledge, Knowledge Graph, Action Knowledge, Object Knowledge, Knowledge Linking, WikiHow Analysis Tool 

Querying Like a Robot

We start tackling this challenge by focusing on selected task domains, starting with cutting. The idea is that our methodology can teach robots to cut any fruit with any to for any purpose, as visualised below. For this, the robot needs to query the knowledge graph to parameterise its general action plan.

We want to enable robots to learn everyday manipulation tasks from the Web and thus make abstract knowledge actionable.

Please choose a cutting verb and a food object from the dropdown lists below. Based on a food cutting knowledge graph, a sequence of motions will be returned that a robot can use for action parameterisation.



Show me the plan for the following
Query, Robot, Live Demo 

Robot Agent

Simulation We want to qualitativly show that a robot using the created ontology can infer the tools and motions needed to perform an unknown task like “Quartering an apple”. To simplify the experiment, we assume some existing knowledge and capabilities: While not knowing specific terms like “Quartering”, the robot already knows how to cut an object The robot is able to successfully perceive given objects for the experiment: a knife and an apple The robot is also able to grasp and hold a knife and an apple We use a simulation environment inside the Unreal Engine together with tools like URoboSim1 to gain complete control over the environment and precisely manipulate and monitor all factors that will affect the robot’s behaviour.
Research, Robot, Simulation