Neurosymbolic Robot Programming: Dissertation Abstract in KI - Künstliche Intelligenz

My article “Neurosymbolic Robot Programming” ↗ has been published in the Dissertation and Habilitation Abstracts section of KI - Künstliche Intelligenz. It is a three-page, open-access summary of my PhD dissertation.

Robots that assist people need to learn models of the world and optimise their own behaviour, but people also need to understand and control them to ensure safe execution. Deep learning serves the first requirement and symbolic programs serve the second. The dissertation connects the two by treating robot programs as predictive models of robot tasks.

The central representation is the Neurosymbolic Robot Program, which pairs a human-readable source program with a shadow program, a differentiable surrogate that mirrors the source program’s structure and predicts its execution outcomes. Because the shadow program is built automatically from the source program, existing robot programs become optimisable. Shadow Program Inversion propagates gradients of a task-level objective through the shadow program and writes the optimised parameters back to the source program, which stays auditable. Its extension SPI-DP adds a differentiable motion planner, and improved task success by up to 186% and reduced cycle times by up to 75% on a poka-yoke quality assurance task.

For creating the source programs in the first place, the MetaWizard family of systems reasons over knowledge bases that combine symbolic and subsymbolic information. It synthesises programs from VR demonstrations in retail assistance tasks and from natural-language dialogue in industrial surface treatment, and a third variant uses pretrained LLMs for task reasoning.

The article closes with follow-up work on conditioning shadow programs on camera images with multimodal transformers, and on combining LLM reasoning with structured knowledge in LODGE.