Lego Town - Part 2: Point an iPad at the bricks, and a city appears

Me adding a few bricks, switching between the Bricks and City views, then zooming into a home (59 s).

Fire Chief Marco's mission from start to finish, on the real bricks (42 s).

Click a video to watch it full screen.

Straight to the hospital

My 8-year-old pointed the iPad at his Lego bricks. On the screen, he didn’t see bricks. He saw buildings, standing where his bricks were.

He went straight for the hospital. He zoomed in and found a doctor getting ready for surgery. He was fascinated. Then he turned to the next building, and the next, one by one, to see what was happening inside each of them.

In part 1, the city lived on a laptop screen. My first idea for what’s next was an iPad as a “magic window” over the board. That window now works, in augmented reality!

My kiddo touches the iPad screen, where a letter from the Nguyen family is open over the 3D city; the real bricks are on the board behind the iPad — My kiddo reading the Nguyen family's letter: there's no bakery, market or restaurant near their street. Another problem to fix in his city. The real bricks are behind the iPad.

In case you missed part 1

Lego Town is a side project I built with my kiddo over about two weeks, for under $500. Four green Duplo baseplates on the living-room floor make the board. Two ordinary webcams watch it, and a laptop turns the bricks into a living 3D city. The 3D structure behind it is a digital twin of the actual lego bricks.

Each brick color is a type of place: red for homes, dark blue for offices, black for factories, orange for parks, yellow for restaurants. Printed labels make special buildings, like the hospital, the fire station or city hall. Stack bricks higher and the building grows.

Citizens write letters to their mayor, my kiddo, about real planning problems, like factory smoke or no food shops nearby. He fixes them by moving bricks with his hands. It’s experiential learning, the cycle David Kolb described: build, notice, think, try again.

Claude and Replit helped me build it fast, with minimal time on coding and deployment. Nearly all my time went into the product and testing with my kiddo. I followed the build-measure-learn loop from Eric Ries’s The Lean Startup, but I measured what we learned about the experience itself, not revenue or engagement. Read more in part 1 here.

Augmented Reality: A city standing on the bricks

Lego Town now uses augmented reality, on an iPad Pro (M4).

My kiddo points the iPad at his bricks, and on the screen they turn into a city. Homes, shops, roads, people and cars stand where the bricks are, with the living room all around them. This experience is so much more immersive than the laptop was.

My kiddo kneels at the green board, building; the board sits in a cardboard frame lined with printed black-and-white cards — The board now sits in a frame of printed cards, which tell the iPad where the board is.

It also shrinks the gaze shift I wrote about in part 1. Before, he looked down at the bricks, up at the laptop and down again. Now, through the iPad, the bricks and the city are in the same place.

Then and now, side by side: on the left, the first version's city on a laptop screen; on the right, the city standing on the board in the living room, seen through the iPad — Part 1 put the city on a laptop screen. Now it stands on the bricks.

The iPad sits on a stand next to the board, so he doesn’t have to hold it. His hands stay free for the bricks, and he zooms in with his fingers. Three buttons switch the view: City shows the town, Bricks shows the bricks as the webcams read them, and Both shows the two together. A fourth, Check, uses the iPad’s LiDAR capability to flag bricks the webcams got wrong.

The iPad on its stand in front of the board, in Both mode: see-through colored bricks sit on the real board, with worried faces over some homes; my kiddo builds in the background — With Check on, see-through bricks show what the webcams read, on top of the real bricks. The worried faces mark homes far from food.

The iPad on its stand in City mode: the bricks have become a town with roads, buildings and trees, while my kiddo holds a red brick in the background — In City mode, the same bricks are a town. Behind the iPad, my kiddo is about to add a red brick: one more home.

Like a headset, without the headset

Looking through the iPad feels very close to wearing a headset like Meta Quest or Apple Vision Pro. The city stands in the room, on top of his bricks. But nothing goes on his head or covers his eyes. He can look away at any moment, and I can look at the same screen and help.

Headset makers set age limits. Meta doesn’t allow children under 10 to use its Quest headsets, and in the US, 10- to 12-year-olds need a parent-managed account (ConnectSafely). Apple says Vision Pro should not be used by children under 13 (Apple).

My kiddo is 8. For a child his age, I think an iPad on a stand is a much safer way to get that experience. And it allows us to both look at it together, a shared experience.

Something’s happening in every building!

The graphics are a huge step up too, thanks to Claude. In a few hours, Claude generated an entire world: buildings, people, pets and vehicles, with astonishing detail. I only provided a few guiding prompts on what I’d like to see and Claude spun-off many tasks to generate all the 3D graphics.

You can look inside every building. Zoom in, tap a building, and its walls turn see-through, with a card that says what’s happening inside:

Firefighters play ping-pong at the fire station.
At the hospital, a doctor gets ready for brain surgery.
In the science lab, a rocket is being tested.
Even the homes have families inside (for example, upstairs, the Rossi family is playing with a train set).

The iPad's X-ray view of the hospital at night: its walls are see-through, and a card lists what's happening inside — Looking inside the hospital at 7:48 pm: the patients are asleep, the night shift looks after them, and a brain is keeping cold in the frozen box. (My kiddo is fascinated by brain surgery...)

One letter, one mission

The missions work the same way as before, but now they play out on the bricks themselves. The second video at the top shows one of them from start to finish.

It starts with a letter from the town’s fire chief:

Dear Mayor, our fire truck can't get to some homes fast enough, because the fire station is too far from them. We have only one fire station, so could you move it nearer the middle of town?

Fire Chief Marco

The city shows the problem. An orange ring marks how far the truck can get in time, and worried faces float over the six homes outside it. Move the fire station’s label nearer the middle of town, and the city builds a new station there. The ring now covers every home: mission complete, and Chief Marco rings the bell, ding ding!

Fire Chief Marco's mission in four screens: 1, the letter arrives; 2, six homes are too far from the fire station; 3, the fire station label moves to the middle of town; 4, mission complete — The mission in four screens, from the same play session as the video.

Each mission ends with a real-world fact. This one: “Firefighters try to reach a fire within a few minutes. City planners think about where to build fire stations by looking at how far the trucks can drive in that time.”

Next: a budget

So far, bricks are free. Soon they won’t be. Next, I’m adding more advanced missions and a city budget: say $10 million in funding every year of the game, enough for 10 new bricks. A fire station, a park and a row of homes then compete for the same money, while the letters keep coming. Kids will have to choose which citizens’ problems to solve first.

That’s the real job of a mayor: solving problems under constraints. I hope kids learn that here, with their hands, and pick up some finance and economics along the way, through civics and city planning. A teacher could set the budget, write the letters live, and let the class argue about where the money should go.

Under the hood (for ML and AR readers)

The new part is a native iPad app, built in Xcode, running on an 11-inch iPad Pro (M4). The machine learning didn’t move. The laptop and its two webcams still read the bricks, as in part 1. The iPad gets the board from the laptop and draws the city on top of the real bricks.

Architecture: on the laptop, two webcams, find the plate and read the bricks; the board goes to the native iPad app, which pins the city on the bricks and runs the mayor's missions; the young mayor moves the bricks — From webcams to the iPad and back to the child: still one loop.

Pinning the city to the bricks. A frame of printed black-and-white cards now surrounds the board. The iPad’s camera finds the cards and works out where the board is, so the city stays on the bricks as the iPad moves. It spots the building labels too. A status line says what it sees: “Board pinned ✓ · sees 5 cards, 3 labels”.

A second opinion from LiDAR. The iPad Pro has a LiDAR scanner, which measures depth. Check mode uses it to catch the webcams’ mistakes. Each see-through brick shows what the webcams read, and it turns red where the iPad sees no brick at all. Two independent sensors now look at the same board.

Feature	Part 1	Part 2
Where the city appears	On the laptop screen	On the bricks, through the iPad
What reads the bricks	Two webcams and a laptop	The same
How the city lines up with the board	Not needed	Printed cards around the board (ArUco markers)
Checking the reading	The brick view, by eye	Check mode, with the iPad’s LiDAR
Inside the buildings	Not shown	X-ray views of every building
What it costs	Under $500	The same, plus an iPad Pro (M4) and a stand

What changed between the two versions.

Want to build it together?

I don’t know of anything quite like this. It’s still a side project, and still early: one family, one living room. I’d love to test it properly with educators and more kids aged 6 to 10.

If this sparks something, please get in touch:

if you have ideas, or want to work on it together
if you’d like to try it with your kids or in your classroom (it takes Duplo baseplates, two webcams and a laptop; the AR view adds an iPad Pro on a stand)

LEGO® and DUPLO® are trademarks of the LEGO Group of companies, which does not sponsor, authorize or endorse this project.