Niantic’s new game Peridot is part pet sim, part research project

Niantic, the maker of Pokémon Go, recently invited me to its London offices to see its new mobile pet sim Peridot. I saw a presentation, tried the game — which is released on iOS and Android on May 9 — and got to hang out with some of the developers, including the computer scientist heading the small research team that’s building the machine learning tech behind Peridot’s potentially groundbreaking new augmented reality technology.

But no demo could better illustrate the potential of Peridot than a half-hour I spent with the game a few days later, when I was given a test version to try at home. I have two young kids (ages 4 and 6), who were immediately charmed to see the cute creature of Niantic’s invention, called a Peridot, hatch on my phone screen and run around my house. We petted it, fed it, threw a ball for it that appeared to bounce off the walls in my living room, and watched it run around and behind the furniture, only slightly erratically.

Image: Niantic

Checking the baby Dot’s desires, I saw that it wanted to eat a dandelion, which needs to be foraged from grassy areas. Our backyard is paved, but there’s grass across the street, it’s a pleasant spring evening, and there’s a little time to kill before bath time — let’s go! We set out and the kids squealed to see the Peridot running ahead of us down the driveway. They were impressed that the game could tell the difference between grass, foliage, and paving stones, so the creature could forage different items from each.

I noticed that my Peridot wanted to forage from a Habitat, which is what this game calls the map-based local points of interest shared by all Niantic’s AR games (in Pokémon Go, they’re called PokéStops). I saw this was only a short stroll away and I hadn’t visited it before (I’m not a Pokémon Go player) — why not? We headed over. It turns out that the point of interest was a Victorian stink pipe (I live in London, can you tell?), which looks like a lamppost without a lamp and is in fact a sort of giant straw that was designed to release noxious gasses from the sewers below, well above the heads of the good Queen’s subjects. I’d never heard of these things or noticed this one before. We returned home having had a jolly time, got some unplanned exercise, and learned a little about local history (and leveled up our pet). Niantic couldn’t have scripted it better.

Peridot is a pretty typical pet sim, in the style of something like Nintendogs, crossed with Niantic’s vast mapping data resource and a new generation of AR tech. You interact with your pet to keep it happy and earn it growth points, leveling it up from baby to teen to adult. At the adult stage, the creature will want to be “released” at a habitat where it can breed with other Peridots (the animals are genderless) and sire a new baby for you to look after. (Producer Ziah Fogel says the plan was for players to say goodbye to their adult Peridots permanently at this point, but testers got too upset, so you get to hang on to them.) Niantic has built enough variables into the creatures’ DNA that each one is genetically unique, and every pairing will create a new, equally unique baby.

Image: Niantic

You can imagine the location-based features that Niantic has built into this genre of game; take your pet for walks, forage from different environments, engage in breeding with other local players as a kind of social endgame. You can also imagine the gentle but insistent trickle of objectives, progress, currencies, and rewards built into a free-to-play live service game such as this, as well as the customization features (yes, your pet can wear hats).

But Niantic has another mission with Peridot, which is to push the boundaries of AR and change how people think about it. The idea is that your pet should, on some level, be able to understand the real world that it (or rather, your phone camera) is looking at, and respond to it. This is the work being undertaken by Niantic’s R&D team, which is based over six floors of a narrow office building crammed into London’s Covent Garden area, and is led by Niantic’s chief research scientist Gabe Brostow.

The AI research work starts with accurate mapping of the 3D space and pathfinding around real-world objects, as well as obstacle occlusion that means your pet will disappear realistically behind chairs and tree trunks. (Brostow seems frustrated that the design team requested a shadow showing the Peridot’s position behind the object be put back into the game, lest players get too stressed out about losing their charges.) It continues with “semantic segmentation,” as Brostow puts it, of your phone’s video feed of the environment, using machine learning to train the code to tell the difference between grass and water, to recognize a TV screen so your Peridot can chide you about screen time, and to identify classes of object such as pets, people, and plates of food. The neural networks required for this weren’t possible on mobile phones even five years ago, Niantic says. Peridot certainly exercises your phone’s CPU and GPU, if its effect on battery life is anything to go by.

Image: Niantic

Peridot started life as a tech demo, and although the pet sim genre is a natural fit for Niantic’s efforts, you can tell. In some ways it still feels more like the skunkworks project of a tech company hungry for progress than a labor of love from a game developer. Brostow’s team is sharing its findings with the academic community, while the code will be available to third parties to license via Niantic’s Lightship suite of development tools. You can imagine the benefits the tech might have for robotics or other assistive technologies; Brostow likes to picture a future where he can hold his phone up to see the location of pipework behind walls and under floors when he’s doing some home improvement. There are probably more sinister uses for this combination of machine learning of environments with Niantic’s industrial-scale geopositional data farming, too, but don’t think about that — look at how cute these little critters are!

In practice, the tech is far from perfect: It can’t spot food that reliably, can see water outdoors better than in, and has trouble with windows. But there’s an undeniable magic to watching your Peridot scamper around your home, jump up onto tables, and notice your cat. It feels a little bit alive, and it will only feel more alive as the game is released and players begin to train Niantic’s neural nets en masse. As Fogel says, a game about growing little creatures and introducing them to the world is a good thematic match for technology that will itself improve its understanding of the world and the realism of its behaviors over time.

I think it’s impressive. But it’s not my reaction that matters, or that tells you whether Peridot will be a success. My kids were delighted by it within seconds, and my 4-year-old was wandering around the house calling for “pennydot” within minutes. Plus, we all know about Victorian stink pipes now.