For an industrial robot built for the rigors of factories and power plants, tidying up a living room may seem like a light day at the office for Spot. Yet, a recent video of the robot picking up shoes and soda cans in a residential home represents the promise of AI models in robotics. In this case, Google’s visual-language model (VLM) Gemini Robotics-ER 1.5 was empowering Spot with embodied reasoning.

This particular demo grew out of a 2025 hackathon at Boston Dynamics that built on prior projects using Large Language Models (LLMs) and Visual Foundation Models (VFMs) to enable Spot to contextualize its environment and engage in more complex autonomous actions than a typical Autowalk mission. Rather than write formal software logic or a “state machine” program that defines each step of a given task, we interacted with Gemini Robotics using conversational language. In turn, it communicated with Spot on our behalf.

A Robust SDK and Natural Language Prompts Save Time

Using Spot’s SDK, we developed a layer that facilitated interaction between Gemini Robotics and Spot’s application programming interface (API). The API normally gives developers access to the robot’s capabilities to create custom applications or behaviors. For example, researchers at Meta have used Spot to test how an AI system could locate and retrieve objects it had never seen before.

Our ability to engage Gemini Robotics using natural language prompts was a huge timesaver, compared to traditional programming. We told Gemini Robotics it had access to a mobile robot equipped with cameras and a robotic arm. It also had a finite set of tools it could use to control the robot. A tool is a lightweight script that performs some internal logic and translates inputs from Gemini Robotics to actual API calls. We limited the actions to navigating between locations, capturing images, identifying objects, grasping them, and placing them somewhere else. 

The extent of our SDK means there are great examples one could leverage to add more access to the API with minimal development.

Giving Gemini Robotics a Baseline

To start we needed to explain to Gemini Robotics what we wanted it to do. We did experience a learning curve when writing these baseline prompts. Simple instructions like “put down an object” or “take a picture” weren’t detailed enough to produce expected behavior. We had to add context in our descriptions as we refined each tool. 

A good example is the detailed prompt for the “TakePicture” tool:

This command will cause the robot to take a picture with the specified camera. There is some nuance to choosing the correct camera. Once arriving at a location using GoTo, you should always start by taking a picture with the gripper camera, because it's the most informative.
If the robot has arrived at location and is already holding an object, you can do one of two things:
1. Immediately call PutDown
2. Search the area with either of the front cameras. The front cameras are low to the ground, so if you're trying to put things on an elevated surface, they won't give you useful information.

In this example, we gave Gemini Robotics no detailed description of the robot’s chassis or arm. Instead, we simply explained that Spot’s front cameras would be too low to photograph objects on elevated surfaces. We were able to iterate rapidly, as small changes in wording produced noticeably better results. Once it had this set of basic tools through the API, Gemini Robotics could sequence Spot’s actions and follow the handwritten instructions on a whiteboard on the day of the demonstration.

How Gemini Robotics and Spot Collaborate

Until the robot powers on, Gemini Robotics has no context for what specific tasks we might ask it to perform in a given demo. We only provided simple written instructions, such as, “Make sure all of the shoes at the front door are on the shoe rack.” Gemini Robotics evaluated images from Spot’s cameras and identified objects in the scene that matched the instructions. These objects became the reference points for Spot’s navigational and manipulation systems.

In many respects, Gemini Robotics was identical to an operator manually driving Spot using its tablet controller. For example, to pick up an object with Spot, an operator positions the robot near the object and then uses a grasp wizard to identify the target object. The operator provides high-level direction and Spot figures out the exact details. In this demonstration, Gemini Robotics functioned as both the operator and the tablet sending commands to the robot. This freed us up to act more like a team lead, providing a high-level to-do list and trusting Spot and Gemini Robotics do the rest.

Call and Response

When Gemini Robotics engages a given tool, the tool responds with results and context, such as, “I picked up the object,” or “I can’t pick up something while my hand is full.” Gemini Robotics then makes adjustments on the fly based on this feedback from Spot. For example, to pick up shoes, Gemini Robotics requests an image, identifies the shoes in that image, and calls the “pickup” command. By creating fundamental tools that semantically flow in conversation,  Gemini Robotics can manage the sequence of tasks required to clean up the room. Spot’s existing software stack manages the locomotion, navigation, and manipulation of the robot itself.

It’s important to note Gemini Robotics has strict boundaries in this scenario. It can’t invent new capabilities or control Spot beyond what is available through the API. This keeps Spot’s behavior predictable, while still allowing Gemini Robotics to adapt to different situations.

A Force Multiplier for Developers

For developers already working with Spot, this research has tremendous potential. Through Spot’s SDK, they have access to a robust toolkit of capabilities. Companies use these tools today to build applications for inspection, research, and industrial data analysis, among others.

An AI model like Gemini Robotics offers a way to expand those applications more rapidly. Rather than write extensive task logic on top of Spot’s APIs, developers can experiment with having AI systems interpret natural language instructions and dynamically choose to engage the robot. As a result, models like Gemini Robotics can act as force multipliers, amplifying the reliable toolkit and robust performance that is already delivering value for Boston Dynamics customers.

Our Next-Token Prediction for Spot and Gemini Robotics

Although this is still an experimental step and not a hardened application, it illustrates a compelling direction for robotics and physical AI. Robots like Spot are already extremely capable of navigating complex and changeable environments, collecting data and sensor readings, and manipulating objects. Rather than reinventing the wheel, AI foundation models offer a new way to expand these capabilities in new settings and to new applications.

Physical AI is a rapidly evolving field and our team is leading the way in the lab and in real applications of AI empowered robots. While we are early in our formal partnership with Google Deepmind, we’re excited for what the future holds with Atlas and we’ve already rolled out practical enhancements for Spot and Orbit, with AIVI-Learning powered by Google Gemini Robotics ER 1.6. This next evolution of our AI Visual Inspection tool unlocks a new level of visual intelligence, as users benefit from shared expertise bringing a deeper level of contextual intelligence to Spot and Orbit. Model improvements automatically happen behind the scenes, adding more capabilities to the same software and hardware.

Today, this demo points to a future where users can rely more on natural language to guide Spot’s actions, rather than complex code. The engineer’s role shifts toward setting goals and objectives. The multi-modal robot foundation model interprets the instructions to form complex and adaptive plans and Spot executes the action.

This article was contributed by Issac Ross and Nikhil Devraj, engineers on the Spot team.

BOSTON (AP) — Running the Boston Marathon is tough enough without having to jostle your way from Hopkinton to Copley Square.

So race organizers this year turned to an expert in crowd science to help them manage the field of more than 32,000 as it travels the 26.2 miles (42.195 kilometers) through eight Massachusetts cities and towns — some of it on narrow streets laid out during Colonial times.

“There are certain things that we can’t change — that we don’t want to change — because they make the Boston Marathon,” said Marcel Altenburg, a senior lecturer of crowd science at Manchester Metropolitan University in Britain. “Like, I’m a scientist, but I can’t be too science-y about the race. It should stay what it is because that’s what I love. That’s what the runners love.”

The world’s oldest and most prestigious annual marathon, the Boston race was inspired by the endurance test that made its debut at the inaugural modern Olympics in 1896 — itself a tribute to the route covered by the messenger Pheidippides, who ran to Athens with news of the Greek victory over the Persians in Marathon.

After sharing the news — “Rejoice, we conquer!” — Pheidippides dropped dead.

Organizers of the Boston race would prefer a more pleasant experience for their runners, even as the field has ballooned from 15 in 1897 to as many as 38,000 to meet demand for the 100th edition in 1996. It has settled at around 30,000 since 2015.

As the race grew, it tested the limits of the narrow New England roads and the host cities and towns, which are eager to reopen their streets for regular commutes and commerce as quickly as possible.

“It would be kind of great someday to be able to grow the race a little bit more,” race director Dave McGillivray said. “The problem with this race is that it’s about two things: time and space. We don’t have either. … So, we’re trying to be innovative.”

That’s where Altenburg comes in.

A former German army captain who runs ultra marathons himself, Altenburg has worked with all of the major races, other large sporting events, and airports and exhibitions that tend to attract large crowds on ways to keep things safe and flowing smoothly.

For the Boston Marathon, which draws hundreds of thousands of spectators in addition to the runners, his models allow him to run simulations that help him see how the race might play out under different conditions.

“We have simulated the Boston Marathon more than 100 times to run it once for real. That is the one that counts,” Altenburg said in a telephone interview. “They gave me, pretty much, all creative freedom to simulate more waves, simulate more runners and — within the existing time window — they allowed me to change pretty much anything for the betterment of the running experience.

“And then we checked every aid station, every mile, the finish, every important point, (asking): Is the result better for the runner? Is that something that we should explore further?”

The most noticeable difference on Monday will be that the runners are starting in six waves — groups organized by qualifying time — instead of three. The waves, which were first used in Boston in 2011, help spread things out so that runners don’t have to walk after the start, when Main Street in Hopkinton squeezes to just 39 feet wide.

Other, less obvious changes involve the unloading of the buses at the start, the placement of the water and aid stations, and the finish line chutes, where runners get their medals, perhaps a mylar blanket or a banana, and any medical treatment they might need.

“For an event that’s as old as ours, 130 years, it allowed us to be a startup all over again,” said Lauren Proshan, the chief of race operations and production for the Boston Athletic Association.

“The change isn’t meant to be earth-shattering. It’s to be a smooth experience from start to finish,” she said. “It’s one of those things that you work really, really hard behind the scenes and hope that no one notices — a behind-the-curtain change that makes you feel as if you’re just floating and having a great day.”

Shorter porta potty lines would also be nice.

“What I loved about working with the BAA was how aware they are of what the Boston Marathon is. And they won’t change anything lightly,” Altenburg said. “So it was very detailed work from literally the moment the race last year ended to now. That we check every single option. That we really make sure that if we change something about this historic race, then we know what we’re doing.”

The BAA will look at the feedback over the next three years before deciding about expansion or other changes.

“Fingers crossed, hope for the best, but we’ll get feedback from the participants,” McGillivray said. “And they’ll let us know whether or not it worked or not.”

But keeping the course open longer isn’t an option. And the route isn’t going to change. So there’s only so much that crowd science can help with at one of the toughest tests in sports.

“I can talk. I’m a scientist. I just press a button and it’s going to be,” Altenburg said. “But the runners still have to do it.”

___

AP sports: https://apnews.com/hub/sports

“I’ve done a lot of different things in my life, but there’s no question in my mind that the youth development aspect of what I’ve done with kids and tennis in Boston is the most important work I’ve ever done,” said Crane, who has dedicated the last 30 years of his life to youth tennis.

Once upon a time, Crane served as a sports journalist for the New York Post, the defender general of Vermont, and the executive director of the Massachusetts State Ethics Commission.

He has been the head boys’ tennis coach at Latin Academy since 2009, and last season led the Dragons to their first Division 3 semifinal appearance in program history.

This season, the Dragons are trying to repeat that success, and are doing so with players from five Boston high schools (Latin Academy, O’Bryant, Josiah Quincy Upper, East Boston, and New Mission).

Sophomore Mayfre Moreta, a New Mission student, has never crossed paths in the school hallways with his doubles partner, Gio Waterman, who attends Latin Academy, but the pair still managed to rally from a set down to clinch the deciding No. 2 doubles point in last year’s D3 quarterfinals.

“I think [that win] speaks to our identity as a program,” said Waterman. “It’s so nice to play with all these new guys from other city schools. We share that bond of representing the city of Boston.”

Along with the unique co-op structure, Crane runs a no-cut program that carries roughly 35 kids ranging from seventh to 12th grade every year who vary from beginners to experienced tournament players.

“We don’t cut because we want to teach kids from all over the city how to play the game,” said Crane. “We want to give them a sport that they’ll play for the rest of their lives.”

Mateus Washington, a Latin Academy senior, is in his sixth, and final, year with the program. Although Washington has dueled the state’s top players at No. 1 singles this season, he is just as proud that he gets to lead his teammates every day.

“It’s really cool to see how the seventh-graders of this generation look so much like I did in seventh grade,” said Washington, who has posted a 3-3 record this year. “It’s super eye-opening and enriching to be a part of their development.”

Crane recognizes that the team’s makeup is unique and oftentimes difficult to manage.

“Logistically, it’s difficult. The kids are coming from all over the city, and they can’t all show up at the same time because their schools get out at different times,” said Crane.

But above all, Crane is thankful he can give his kids — many of whom come from low-income situations — the chance to play tennis, as well as offer them summer jobs at Sportsmen’s, Franklin Park Tennis Association, and other tennis facilities around the city.

“What motivates me the most is getting to know these kids, building relationships with them, and figuring out how I can be of help to them. I want to help them grow, help them succeed on and off the court, and help them get ready for the rest of their lives.”

▪ Emily Cilley has yet to lose a match as the head coach of the Swampscott girls.

In Cilley’s first year with the program, the Big Blue (4-0) have put last season’s second-round loss to Dover-Sherborn in the rearview mirror.

Key to their success have been sophomore stars Nikki Carr and Ginger Gregoire. Carr has been dominant at first singles, posting a 4-0 record without dropping a set, and Gregoire has been a great option at second singles, logging a 3-1 record and securing the deciding 3-6, 6-1, 6-1 victory in the season opener against Bishop Fenwick.

“They are both very disciplined players who understand the balance between being cautious and being patient,” said Cilley. “Their technical skills are on point, and they aren’t intimidated by the person across from them.”

The Big Blue’s strong start has catapulted them to the top of the Northeastern Conference. They’ll look to continue their unbeaten streak against St. Mary’s next Saturday.

▪ The girls of Central Catholic are off to their best start in program history.

The Raiders boast a 6-0 record after taking down Lowell 5-0 on Saturday morning. The win was their fifth sweep of the season, with the only non-sweep coming in a 4-1 victory over Notre Dame (Tyngsborough).

Morgan Bateman has looked unstoppable at second singles, as she is yet to drop a set, and Ella Asmar has been just as impressive at third singles, posting an undefeated record.

Although Haley Wolters was responsible for the only loss by a Raiders player this season, she has logged impressive victories at first singles, such as a 6-2, 6-3 win against Chelmsford and a 6-1, 6-1 triumph over Lowell.

The Raiders have a chance to extend their winning streak to nine with matches against North Andover, Lowell, and Haverhill on the horizon, before they clash with undefeated Andover on April 30.

Webb Constable can be reached at webb.constable@globe.com. Follow him on X @webbconstable.