🌐 REALTECH News, November 2024

+ the Palantir scaling playbook, Waymo goes big and the battle for spatial AI

In today’s edition:

• How Palantir ignored Silicon Valley dogma to build and scale software in critical industries

• The fight for a winning ‘world model’, AI which can understand the physical world

• The startups solving the energy crisis for Big Tech and LLM model providers

• Waymo deploys end-to-end learning and raises $6BN

📣 PSAs

• 👂🏼Podcast: We just released an interview with Andrei Danescu from Dexory. Be sure to listen to it; click here.

• 🧑🏼‍🔬 Research: Alongside Dealroom, we have compiled 40+ pages of in-depth analysis on trends in REALTECH. We analysed 15k companies, 11k investors over 10 years - read it here

• ✈️ November is a busy month travel-wise; I will be in Barcelona (Nov 5th - 7th), Lisbon for WebSummit (Nov 11th - 14th) and Paris (Nov 27th - 28th)

If you’d like to listen to the audio version of this post, click here or subscribe via Spotify

Top Stories

New AI ‘world models’ and the many approaches to making AI understand the physical world

Fei-Fei Li, the godmother of AI

Fei Fei Li, the AI researcher known as the “godmother of AI”, just announced a $230m seed round for World Labs, a spatial intelligence startup. The stated goal of the company is to:

“[] aim to lift AI models from the 2D plane of pixels to full 3D worlds - both virtual and real - endowing them with spatial intelligence as rich as our own”

Spatial AI promises to bring intelligence to the physical world, automating physical processes with intelligent robots. The ultimate goal is for a “world model” startup to serve zero-shot intelligence to a robot via an API for a price, effectively “ChatGPT for the 3D world”.

These past few months have seen an inflection in spatial intelligence research and company announcements. Several new companies have raised eye-watering seed capital to tackle this problem, notably Physical Intelligence ($70m) and SkildAI ($300m). Sequoia and OpenAI are invested in both.

LLMs are currently constrained to “next token prediction” of textual input, which is great for blue-collar intelligence but less for understanding physics. Breakthroughs such as Deepmind RT-1, a vision language model (VLM), was heralded as a “ChatGPT moment” for robotics (covered here previously), but results have been mixed. Recent findings by researchers at UT Austin, in their paper “Does Spatial Cognition Emerge in Frontier Models?” confirm this. The findings show that current frontier models, including both LLMs and VLMs (eg ChatGPT-4o), perform poorly on a wide range of tasks, especially compared to human baseline performance. While VLMs connect language and internet-scale data with visual data, this approach fails to replicate real-world physics. Truly understanding physical dynamics requires AI architectures that don’t just ‘see’ but can interpret and predict based on foundational principles like force, momentum, and material interactions—a challenge that new models might address.

Today, there's an increase in approaches to this problem, including but not limited to multi-modal approaches (VLMs, VLAs), sensor-driven and physics-based models. This last month has seen a few new research releases:

• Physical Intelligence just previewed their π0 model, a VLA. π0 integrates vision, text, and action, learning from a vast, diverse dataset of robotic interactions and demonstrating low-level motor control across multiple robots. It leverages Internet-scale semantic knowledge from vision-language models, adapting this to perform continuous, high-frequency dexterous tasks like folding laundry, bussing tables, and assembling boxes

• 🍎 ArchetypeAI just released “Newton”, a model designed to learn foundational physics principles directly from nearly 600 million cross-modal sensor measurements. It encodes sensor data (such as temperature, electrical, and motion) into a universal representation, enabling it to perform zero-shot inference on new physical systems. Newton demonstrates strong predictive capabilities, accurately reconstructing past trajectories and forecasting future ones across various complex systems, from mechanical oscillators to real-world power grid dynamics, making it versatile in scenarios that require physical understanding

• Universal Physics Transformers (UPT), is a neural operator that embeds physics into its model by converting physical information from complex simulations (like fluid dynamics or particle movement) into a condensed internal representation called a latent space. This latent space captures essential physical dynamics, enabling UPT to simulate and predict changes in a system without needing to go back to detailed grids or particle setups each time

Who wins? The market is splitting between model providers (World Labs, Physical Intelligence, ArchetypeAI) and full-stack companies. The latter builds not only the AI models and hardware but also focuses on a specific use case - e.g., Electric Sheep, their world model powers their landscaping robots. There is a long history of automation struggling once out of the lab and deployed in real-world settings. If I were to bet on winners in this space, it would be the players speaking with end customers about solving real problems, not the AI labs.

---

Implementing technology in critical industries: lessons from Palantir

Embedding engineers with customers (source: Palantir)

I read a fantastic piece by Nabeel Qureshi on his time at Palantir, which I’d implore you to read in full. It touched at length on a topic I’m passionate about: implementing technology and software within critical industries. Specifically, it discussed how Palantir succeeded by using Forward-Deployed Engineers on-site with customers in a way that was heretical to common product-building wisdom in Silicon Valley.

The journey to integrating software and automation into critical industries is one of complexity, filled with data management hurdles and the need for deep organisational understanding. Most industrial companies remain in an era where data is locked in isolated machines, hand-recorded in logbooks, lab notebooks, QA reports, or stored in scanned, non-searchable PDFs.

Here’s an excerpt on how Palantir’s strategy to overcome this inertia and build their Foundry product:

FDEs were typically expected to ‘go onsite’ to the customer’s offices and work from there 3-4 days per week, which meant a ton of travel. This is, and was, highly unusual for a Silicon Valley company. 

There’s a lot to unpack about this model, but the key idea is that you gain intricate knowledge of business processes in difficult industries (manufacturing, healthcare, intel, aerospace, etc.) and then use that knowledge to design software that actually solves the problem. The PD engineers then ‘productize’ what the FDEs build, and – more generally – build software that provides leverage for the FDEs to do their work better and faster.

This is how much of the Foundry product took initial shape: FDEs went to customer sites, had to do a bunch of cruft work manually, and PD engineers built tools that automated the cruft work.

This led me to a rabbit hole and another great piece by ex-Palantirer Sarah Constantin, on ‘The Great Data Integration Schlep’, which describes the Palantir playbook in detail and can be summarised as this:

1. Target companies with urgent needs: Focus on companies facing critical issues where transformation is essential. Highlight how integration stabilises operations and cuts costs, showing ROI.

2. Secure top and bottom worker support: Gain C-Suite sponsorship for organizational alignment and fast-tracked access. Build frontline worker support by simplifying their work with helpful tools.

3. Streamline data collection: Inventory and digitize all data sources; centralize access in a secure, searchable format.

4. Address security & political barriers: Design integrations with robust security protocols from day one. Use executive and frontline allies to overcome departmental resistance.

5. Data standardization: Set clear formatting standards and automate repetitive cleaning tasks. Involve domain experts to contextualize and validate data.

6. Use Forward-Deployed Engineers (FDE): Embed FDEs onsite to gain firsthand insights and solve immediate pain points. Quickly implement fixes to demonstrate the value of new tools.

7. Scale automation gradually: Pilot small automation projects for high-impact, low-risk improvements. Adjust based on feedback, expanding incrementally to minimize disruption.

8. Prepare for long-term AI integration: Develop a data infrastructure roadmap with scalable storage solutions. Implement ongoing data quality management to ensure AI readiness

---

Intelligent infrastructure is destiny: Nuclear’s AI resurgence and emerging energy optimisations at every level of the AI stack

Three Mile Island, from nuclear meltdown to Microsoft deal

By now, you’ve probably read about the clean energy bottleneck in AI data centres. We’ve previously covered Big Tech’s $200bn AI datacenter spend, which is largely incompatible with their net-zero ambitions. While this is taking the headlines, several new companies are building energy optimisations at varying levels of the compute stack.

If you assume that AI demand stays constant, there are two ways to solve the energy bottleneck: 1) increasing the amount of available firm power through scaling renewables and nuclear, and 2) optimising the AI stack to become more energy efficient:

1. Big Tech is committing to nuclear energy programs through investments in small modular reactors (SMRs). Google partnered with Kairos Power for up to 500MW of power. Two days later, Amazon announced a partnership with X-Energy for up to 960MW. Microsoft reached a deal with Constellation Energy to restart nuclear power on Three Mile Island, home of one of only 3 nuclear meltdowns.

2. We’re seeing an increase in startups tackling optimisations at many levels of the AI stack - from chips, cooling, efficient hardware utilisation, energy-efficient model training, and scalable deployment tools across cloud and edge environments;

Source: author’s research

We’re seeing a lot of dollar movement in this space. Cerebras’ IPO is delayed due to a CFIUS investigation, and the EU is attempting to block Nvidia’s $700m acquisition of RunAI. New startups are popping up. Phaidra and Etched just raised large rounds of $12m and $120m, respectively. We can no longer rely on Moore’s Law for consistent improvements for power efficiency gains. Whilst Big Tech is focused on getting MW of clean energy online, startups are increasingly focusing on driving real efficiencies across the hardware stack.

---

The US DoDs Replicator 2 program - Counter drones to combat all those other drones

We’ve previously covered the US’s Defense Innovation Unit’s (DIU) Replicator program led by the US Deputy Secretary of Defense, Kathleen Hicks. The original Replicator program's mission is to:

“field attritable autonomous systems at scale of multiple thousands, in multiple domains, within the next 18-to-24 months”

In normal parlance, this means building thousands of autonomous drones fast. As we’ve seen in Ukraine, battlefield usage of drones has proliferated and is evolving week by week. The Replicator 2 program, released as an internal memo last month, is focused on counter-UAS:

“Replicator 2 will tackle the warfighter priority of countering the threat posed by small uncrewed aerial systems (C-sUAS)”

The US Army has said that a layered approach of multiple weapons and sensors is necessary for taking down drones, such as electronic warfare (EW) and kinetic and directed energy (e.g., lasers). An ever-increasing number of startups and primes are building new capabilities in this space.

🤓 Other stories you need to know

🏭 Siemens acquires Altair for $10bn. The acquisition will strengthen Siemens’ Teamcenter (PLM) industrial software offering with simulation and digital twins. It also now makes Siemens a viable competitor in the chip EDA space (Siemens)

🚨Tesla released a preview of its robotaxi: a night of Musk vaporware, scant details and overblown promises. Marques Brownlee has a breakdown video of the robotaxi

🚕 Waymo released its end-to-end learning research, the End-to-End Multimodal Model for Autonomous Driving (EMMA) is powered by Gemini and marks a continued shift to end-to-end architecture for autonomous vehicle companies (Waymo)

🍟 OpenAI is designing an inference chip with Broadcom and TSMC, which signals that it may be shelving its laughable plans to build its own foundry (Reuters)

🛩️The US Federal Aviation Authority released new regulatory guidelines for eVTOLs, marking the biggest change in airspace regulations in a century (FAA)

🔵 Meta released OMAT24 - a dataset with over 100m density functional theory (DFT) calculations for materials discovery (Hugging Face)

🛻 VW’s new EV pickup brand Scout Motors, will use software architecture provided by Rivian. This follows from VW and Rivian’s multi-$BN software JV and likely marks the nail in the coffin for VW’s software unit, Cariad. (TechCrunch)

💰Notable Funding Rounds

• Waymo($6BN) is an autonomous ride-hailing company spun out of Alphabet

• Form Energy ($405M Series E) develops 100-hour iron-air batteries for grid storage, led by TPG Rise Climate

• Purpose Green ($140M) funds building retrofit solutions with a focus on energy efficiency, backed by Fifth Wall

• Path Robotics ($100M Venture) uses AI for industrial robotic welding, the round was led by Matter Venture Partners and Drive Capital

• Auger ($100M Seed) is an AI venture focused on supply chain automation, led by ex Flexport CEO Dave Clark

• Terralayr ($77M Series B) supports geothermal technology for sustainable infrastructure, led by Creandum

• Carbon Robotics ($70M Series C) develops AI-powered robots for laser weed control in agriculture, led by investor Anthos Capital

• Beyond Aero (€44M Series A) is advancing hydrogen-propelled electric aviation, with investment from multiple sources.

• Vay (€34M) is rolling out teledriven car-sharing services, supported by the European Investment Bank (EIB)

• Zip ($56M Series D) provides AI-enhanced procurement software to optimize B2B purchasing, backed by Bond Capital.

• OroraTech (€25M Series B) focuses on predictive AI applications for environmental monitoring, funded by multiple investors

• Speckle ($12.5M Series A) aims to develop an Architecture, Engineering, and Construction (AEC) data hub led by Addition

• Dunia Innovations (£11.5M Seed) is creating a “self-driving lab” to expedite new material discovery, backed by Redalpine

• Forge Nano ($10M Strategic Investment) is innovating EV battery technology with a strategic investment from GM Ventures.

• Freeform ($14M Seed) leverages a novel 3D metal printing method developed by ex-SpaceX engineers, with early-stage funding.

• Keel (€6M Seed) is developing operational software for a streamlined workforce, funded by local globe

• Emidat (€4M Seed) is a construction emission management platform, with the round led by General Catalyst

• Nomos (€1.9M Seed) is creating dynamic electricity plans for European users, led by SpeedInvest