Neural Networks for Robotics Engineering

Can a single neural network policy generalize over poses, objects, obstacles, backgrounds, scene arrangements, in-hand objects, and start/goal states? Introducing Neural MP: A generalist policy for solving motion planning tasks in the real world 🤖 Quickly and dynamically moving around and in-between obstacles (motion planning) is a crucial skill for robots to manipulate the world around us. Traditional methods (sampling, optimization or search) can be slow and/or require strong assumptions to deploy in the real world. Instead of solving each new motion planning problem from scratch, we distill knowledge across millions of problems into a generalist neural network policy.  Our Approach: 1) large-scale procedural scene generation 2) multi-modal sequence modeling 3) test-time optimization for safe deployment Data Generation involves: 1) Sampling programmatic assets (shelves, microwaves, cubbys, etc.) 2) Adding in realistic objects from Objaverse 3) Generating data at scale using a motion planner expert (AIT*) - 1M demos! We distill all of this data into a single, generalist policy Neural policies can hallucinate just like ChatGPT - this might not be safe to deploy! Our solution: Using the robot SDF, optimize for paths that have the least intersection of the robot with the scene. This technique improves deployment time success rate by 30-50%! Across 64 real-world motion planning problems, Neural MP drastically outperforms prior work, beating out SOTA sampling-based planners by 23%, trajectory optimizers by 17% and learning-based planners by 79%, achieving an overall success rate of 95.83% Neural MP extends directly to unstructured, in-the-wild scenes! From defrosting meat in the freezer and doing the dishes to tidying the cabinet and drying the plates, Neural MP does it all! Neural MP generalizes gracefully to OOD scenarios as well. The sword in the first video is double the size of any in-hand object in the training set! Meanwhile the model has never seen anything like the bookcase during training time, but it's still able to safely and accurately place books inside it. Since, we train a closed-loop policy, Neural MP can perform dynamic obstacle avoidance as well! First, Jim tries to attack the robot with a sword, but it has excellent dodging skills. Then, he adds obstacles dynamically while the robot moves and it’s still able to safely reach its goal. This work is the culmination of a year-long effort at Carnegie Mellon University with co-lead Jiahui(Jim) Yang as well as Russell Mendonca, Youssef Khaky, Russ Salakhutdinov, and Deepak Pathak The model and hardware deployment code is open-sourced and on Huggingface!  Run Neural MP on your robot today, check out the following: Web: https://lnkd.in/emGhSV8k Paper: https://lnkd.in/eGUmaXKh Code: https://lnkd.in/e6QehB7R News: https://lnkd.in/enFWRvft

VLA models are systems that combine three capabilities into one framework: seeing the world through cameras, understanding natural language instructions like "pick up the red apple," and generating the actual motor commands to make a robot do it. Before these unified models existed, robots had separate modules for vision, language, and movement that were stitched together with manual engineering, which made them brittle and unable to handle new situations. This review paper covers over 80 VLA models published in the past three years, organizing them into a taxonomy based on their architectures—some use a single end-to-end network, others separate high-level planning from low-level control, some use diffusion models for smoother action sequences. The paper walks through how these models are trained using both internet data and robot demonstration datasets, then maps out where they're being applied. The later sections lay out the concrete technical problems that remain unsolved. Read online with an AI tutor: https://lnkd.in/eZdzYfdu PDF: https://lnkd.in/ezzncewE

Drawing insights from biological signal processing, neuromorphic computing promises a substantially lower power solution to improve energy efficiency of visual odometry (VO) in robotics. Published in Nature Machine Intelligence, this novel approach develops a VO algorithm built from neuromorphic building blocks called resonator networks. Demonstrated on Intel’s Loihi neuromorphic chip, the network generates and stores a working memory of the visual environment, while at the same time estimating the changing location and orientation of the camera. The system outperforms deep learning approaches on standard VO benchmarks in both precision and efficiency – relying on less than 100,000 neurons without any training. This work is a key step in using neuromorphic computing hardware for fast and power-efficient VO and the related task of simultaneous localization and mapping (SLAM), enabling robots to navigate reliably.   A companion paper explores how the neuromorphic resonator network can be applied to visual scene understanding. By formulating the generative model based on vector symbolic architectures (VSA), a scene can be described as a sum of vector products, which can then be efficiently factorized by a resonator network to infer objects and their poses. The work demonstrates a new path for solving problems of perception and many other complex inference problems using energy efficient neuromorphic algorithms and Intel hardware. Congratulations to researchers from the Institute of Neuroinformatics, University of Zurich and ETH Zurich, Accenture Labs, Redwood Center for Theoretical Neuroscience at UC Berkeley, and Intel Labs.   Learn more about neuromorphic VO: https://lnkd.in/gJCVVMCz   Learn how the VSA framework was developed for neuromorphic visual scene understanding based on a generative model (companion paper): https://lnkd.in/gjAENfpp   #iamintel #Neuromorphic #Robotics

 🧠 From Fall Detection to Reinforcement Learning: Connecting the Dots Over the past few weeks, I’ve been building in public a fall detection service that uses multiple neural networks working together. One model analyzes the visual features of a fall (a convolutional neural network, or CNN), another looks at timing and movement patterns (a long short term memory network, or LSTM), and a final layer makes the decision about whether or not to send an alert. While designing that decision layer, I stumbled across something that completely changed how I think about AI systems: reinforcement learning — the same technology used to train autonomous vehicles, drones, and robots to make realtime decisions in dynamic environments. And that discovery led to this weekend project. To really understand reinforcement learning, I decided to learn it hands on — by building a small 2D game simulation (of course with an AI coder) . 🎮 In this mini project, I’m creating a search and rescue drone navigating a "forest like" maze. The drone has sensors (like range finders or LIDAR), detects obstacles, and learns through trial and error how to reach a person in need — without colliding with anything. It's super simple, but my style of content creation is to show the beginning of my learning journey and then let the power of contrast do it's thing over time as I continue to post. So it's simple on the surface, but powerful for learning:  How to design an environment that teaches an AI agent to operate safely  How to reward good behavior and penalize mistakes  How to train policies that drive autonomous decisionmaking This small weekend simulation is a bridge between what I’ve built so far — fall detection — and where I’m heading next: autonomous drone systems that can sense, decide, and act on their own. 🎥 (Video attached: early version of my RL drone navigating the maze.) If you’d like to follow along as I go deeper into reinforcement learning and apply it to real world drone projects, I’ve started a newsletter and will be starting a Substack where I’ll be sharing the full journey 📬 Subscribe to get notified when new breakdowns go live. #ReinforcementLearning #AI #AutonomousSystems #DroneAI #DeepLearning #BuildingInPublic #WeekendProject #pygame #rl Umer Khan M. #firstPrinciples #elliottArnold

Liquid neural networks are a new type of adaptable AI system, first introduced in research around 2018, and later popularized in 2020 by Ramin Hasani’s work. These networks are "liquid" because they remain flexible after training, continuously adapting based on new inputs. A major advantage is their small size and efficiency, often requiring fewer neurons than traditional networks. For example, a car can be driven with just 19 nodes compared to the 100,000 nodes typical in conventional networks. Each node is governed by differential equations, allowing precise control with fewer resources. Liquid neural networks are well-suited for tasks like robotics, where real-time decision-making and adaptability are crucial. They run efficiently on low-power devices, such as Raspberry Pi, and don’t require cloud computing, making them ideal for safety-critical systems like autonomous driving. Their smaller size also makes them more interpretable and easier to debug than traditional, large neural networks, which often act like "black boxes" where the decision-making process is unclear. These networks have some limitations, such as their reliance on time-series data (e.g., videos) rather than static images. However, they offer a promising solution for robotic control systems by improving task-focused reasoning without the distractions of unnecessary environmental data. Generative AI can enhance these systems, enabling robots to operate in a more fluid, human-like manner. In summary, liquid neural networks provide a compact, efficient, and adaptable approach to AI, especially useful in robotics and safety-critical systems, with potential for more transparent decision-making processes compared to traditional, opaque AI models. Liquid AI