Built in 2 weeks: A living Lego town where my 8-year-old is mayor

Building the town from an empty board, in 53 seconds. The front webcam is bottom left, the brick view (what the cameras read) is bottom right, and the city is on top.

The first brick

The first time it worked, there was no city yet. Just an empty green baseplate on the floor, two webcams, a laptop and a 3D grid on the screen.

I asked my 8-year-old to put a brick on the plate. He put one down and waited. Suddenly, the same brick appeared on the screen, in 3D, in the same spot. His eyes went wide. It took him a moment to work out what had happened.

Then he put down another brick, and another, and another. He kept going for an hour, just to see what the 3D model would do.

Two weeks later, that plate is a living town, and my kiddo is its mayor.

My 8-year-old, seen from behind, placing bricks on four green baseplates on the living-room floor — Building the town, one brick at a time.

Bricks on the floor, a city on the screen

Four green Duplo baseplates make a 64×64-stud board, about a meter across. Two ordinary webcams watch it from about a meter up, one in front and one on the right, each held up by a chair. A laptop turns what they see into a 3D city, live.

Illustration of the setup: a baseplate about one meter across on the floor, two webcams on chairs aimed at it, and a laptop showing the city — The whole setup. The board is about a meter across.

Each brick color is a type of place: red for homes, dark blue for offices, black for factories, orange for parks, yellow for restaurants and bakeries. Stack bricks higher and the building grows. Roads appear on their own. People, cars and a helicopter start moving around.

My kiddo never touches a mouse or a keyboard to build. He builds with his hands, and the city follows. It’s a tactile experience.

Why I built it

For four years at Korbit, my former startup, I built an AI Teacher. It delivered personalized learning paths and generated content for more than 20,000 students learning machine learning, data science and computer science skills. (We pivoted later to AI code reviews, and ultimately got acquired by the cybersecurity company Boost Security.)

That work left me convinced that people learn best by doing, and kids most of all. Now as I’m experimenting with all of this, I start to think that augmented reality could move learning from “reading a textbook” to “playing in the real world”.

Educators call this experiential learning. David Kolb described it as a cycle: experience something, reflect on it, form an idea, then try it out. In the Lego town, one turn of that cycle takes minutes.

Kolb's experiential learning cycle drawn as four bricks: Build, Notice, Think, Try, with Grandma Rose's letter as the example — Kolb's four stages, as one mission plays out.

It’s a different experience from reading a textbook, listening to a lecture or playing a computer game. He builds something real with his hands, then watches it come alive on the screen.

Problem solving and the mayor’s mailbox

Free play is great, but I also wanted an element of problem solving to encourage him to think deeper about cities and the problems that come with building and running a city.

So, the town’s citizens write to their mayor. When something isn’t working, a letter lands in my kiddo’s mailbox, and a “Read it to me” button reads it aloud. Here’s an example:

Dear Mayor, the factory next to my house puffs smoke all day long. My washing turns grey, and I cough when I water my roses. Could the factories be farther away from our homes?

Love, Grandma Rose

My kiddo takes one letter at a time and fixes it with his hands, while the city shows the problem: smoke, coverage rings, sad faces over unhappy homes. Once the fix holds for three seconds: SUCCESS! 1,000 points and a sticker.

The city during Grandma Rose's mission: smoke over the factory, a ring showing how far it reaches, masked faces over the nearby homes, and a mission card reading 3 of 7 homes have clean air — Grandma Rose's mission in progress: 3 of 7 homes now have clean air.

Each of the eight missions is a real planning idea at kid size: buffers between industry and homes, parks within walking distance, where to put a hospital or fire station, enough jobs for the grown-ups, and food deserts. They pull against each other, too: factories make smoke, but they also bring jobs. And a fix only counts if the town keeps most of its citizens, so “bulldozing the problem away” is not allowed.

These are not made-up word problems. I believe real problems give kids a reason to need the math: counting jobs against grown-ups, comparing distances, finding the middle of town. I’d love to see a teacher run the town on a classroom projector, write new citizen letters live and work with their students to solve their problems.

Built with my kiddo, for my kiddo

Shortly after the city view arrived, we had our first meltdown. Lunch was ready. I called him once, twice, many times. He would not leave his city. The mayor was too busy for lunch.

In the end I had to switch the computer off. He came to the table almost in tears, telling me how upset he was and how annoying a daddy I always was. Every parent knows that moment.

Later, I asked what he wanted to change next: any building, any person, any car. He wanted the cars to fly. So I let him type the request to Claude himself. He wrote “magnetic livitation”, I helped him fix the spelling, and a few minutes later every car in town floated on glowing magnets.

A child's arm in a striped shirt points at the 3D city on a laptop screen — My kiddo showing me his city.

Built in two weeks, for under $500

I started two weeks ago, working a few hours a day. Everything I bought cost under $500 in total: bricks, baseplates and an AI subscription. The laptop, webcams and chairs were already in the house.

Claude and Replit helped me build it extremely fast, with minimal time spent on coding and deployment. I spent nearly all my time on the product itself: the concept and design, collecting data, and testing with my kiddo. Before AI coding tools, this would have taken months (or perhaps a year) with at least one ML engineer, one back-end developer and one developer expert in front-end and 3D rendering in the browser.

I followed the loop Eric Ries describes in The Lean Startup: build, measure, learn, repeat, fast. The outcome I measured was not revenue or engagement, but what we learned about the user experience. My kiddo played while the app recorded both cameras, we looked at what broke, and a fix shipped the same day. From a few hours of labeling photos on October 8 to the mayor’s missions on October 16, the app went through two dozen versions in barely nine days.

Under the hood (for ML readers)

Technically, the town is a digital twin of the living-room floor.

The ingredients are the same as in physical AI and industrial digital twins: cheap cameras, small models and careful state tracking.

Architecture: two webcams, find the plate and read the bricks in Python on the laptop; build the city and the mayor's missions in the browser; the young mayor moves the bricks — From webcams to missions and back to the child: one loop.

Reading the board. A small neural network (an MLP in NumPy) labels each pixel with a brick color and whether it’s a top or a side, measured relative to the green plate. The decoder draws candidate boards into both camera views and keeps the one that best explains both pictures. The second view breaks ties: from a low angle, a stack two cells back and one brick lower looks almost identical. A 10 ms check per frame waits for hands to leave, then re-reads only what changed.

Step, on the first n=39 photo build sequence	Occupied cells correct	Boards exactly correct
Render-and-compare with hand-set color rules, one photo at a time	71.5%	10 of 39
+ every stack is 2×2, and the previous board is the starting point	87.6%	24 of 39
+ learned pixel model, trained on the other two camera views	97.4%	35 of 39
+ a second view of each board (iPhone photos from the side)	100%	39 of 39

An occupied cell has a brick in the truth or in the reading; it's right when color and height both match. All 39 photos come from one build.

The game has a training mode to collect its own training data. In “Teach” mode, a square lights up, you place the brick and step back, and the app keeps four labeled picture pairs once both cameras are still. It takes patience, 5 to 10 seconds of stillness per example, so I recorded those rounds myself: n=308 labeled image pairs so far, in daylight and evening light.

Teach recording, held out of training	Light	Stacks correct	Changes correct
#1	daylight	73/73	13/13
#2	evening, strong window reflection	119/134	17/17
#3	evening	135/135	18/18
#4	evening	233/234	24/24

Each step is read starting from the correct board before it, so a stack counts at every step it stands.

What’s next, and where I’d love your ideas

I don’t know of anything quite like this, but there are at least four exciting directions:

True augmented reality. Today my kiddo looks down at the bricks, up at the screen, and down again. That gaze shift is a significant cognitive load for a young child. My next idea is an iPad as a “magic window”: hold it over the board and see the city standing on the bricks. I want it to feel like building a city, not placing bricks on a plate.
Multiplayer. Two or three kids negotiate one city: one runs housing (red bricks), one business, food and industry (blue, yellow and black), one public services like the hospitals, fire stations, schools etc. Together they build the city and answer their citizens’ letters.
Classrooms and homeschooling. Learning about cities, neighborhoods, planning and good governance by running one, with a teacher setting the problems.
Expanding to other subjects. Planting crops instead of buildings could teach agriculture. And perhaps could the same type of loop teach math or physics?

Want to build it together?

This is a side project, and it’s early: one family, one living room, two weeks. The early signs are exciting, and I’d love to test it properly with educators and more kids aged 6 to 10.

If this sparks something, please get in touch:

if you have ideas, or want to work on it together
if you’d like to try it with your kids or in your classroom (it takes Duplo baseplates, two webcams and a laptop)

LEGO® and DUPLO® are trademarks of the LEGO Group of companies, which does not sponsor, authorize or endorse this project.