A bit of physics simulation, and then I added the hand tracking sensor part.

The gadget used was just a regular HD projector to project the graphics on a regular wall surface, and a depth sensor for the hand tracking part. The sensor's position is rather flexible and the setup calibration part is just a couple of key presses. More details in TECH.INFO.

The movie below is a direct screen dump from the computer screen (size converted from 1080p to 720p) to see the graphics more clearly. 

All videos here have no sound or music background, which I think is better this way.

The texture of the bricks is from a photo I took in Anchang Town (安昌镇), an ancient water village in China.