significant challenges. Multi-touch UI toolkits do not currently capture all the richness in human hand gestures [21]. In particular, they do not attempt to map touch regions in the current time step to touch regions in the previous  time step. This required us to devise a new, fast approach to performing this mapping. Also, simulating the physical behavior of sand in real time is still a major challenge. We achieved real-time performance by optimizing an existing technique [28] and using graphics hardware acceleration.

In this paper, we make the following contributions:

  • We introduce a new digital artistic medium that leverages the expressiveness of hand gestures on a multi-touch platform to provide a visual experience that goes beyond physical sand
  • We analyze the sand animation process, highlighting common pouring and manipulation techniques and developing a taxonomy of hand
  • We present SandCanvas’s intuitive UI design and its enhanced digital
  • We describe our implementation, which employs new techniques for performing real time sand simulation in response to gestural
  • We evaluate SandCanvas with 1 professional artist, 4 amateur artists and 2 novice users to gain insight into the importance and unique affordances of this


SandCanvas is inspired by new media interactive installations and recent advances in interactive surfaces. We divide related work into three sections. First, we briefly summarize recent work in technologically enhanced static and performance art creation systems in general and for multi-touch surfaces. We then describe related systems and algorithms for sand motion simulation. Finally, we discuss systems closely related to SandCanvas.

Technologically Enhanced Art Creation Systems Performance art has a rich history that spans hundreds of years [17]. Myron Krueger's Videoplace, developed between 1969 and 1975, was an early interactive artwork that incorporated computer vision [15]. Since Videoplace, numerous audio-visual performance systems have been driven by human gestures [18, 27]. Examining numerous audio-visual performance systems, Levin derived a set of design goals for new performance art systems [18]. According to him, successful systems should be predictable, instantly knowable and indefinitely masterable. We pursued similar goals when designing SandCanvas.

In recent years, researchers have produced 2D animation by demonstration systems that could be used for performance art. K-Sketch [9] is a general purpose and informal sketch based 2D animation tool that allows novices to create animation quickly and easily, but all interaction must be done through a single point. In Video Puppetry [4], artists record simultaneous manipulations of multiple physical

puppets to create animation. Researchers have also explored the use of multiple touch points to record real-time deformation of characters [13].

Researchers have also produced notable multi-touch painting systems. Project Gustav attempts to create a realistic painting experience [22], while I/O brush [25] allows artists to paint with patterns and movements “picked up” from everyday materials. Fluid Paint [30] and IntuPaint

[31] use the entire region of contact between brush and surface to model brush strokes. SandCanvas also captures the entire region of contact between the surface and the artist’s hand to model interactions with sand.

SandCanvas bears a resemblance to each of the art creation systems presented here. It is a medium for performance art where the final performance is a kind of 2D animation. Instead of animating a fixed set of characters, however, the artist creates characters in sand using rich gestures that cannot be represented adequately with a set of discreet points. SandCanvas also has unique creative tools like recorded gestures that aren’t found in any of these systems.

Physical Sand Simulation

Since the sand particles used in sand animation are very fine, the number of sand particles is potentially huge. Hence, physically accurate interaction with sand is particularly challenging. Li and Morshell devised one simulation approach, but it assumes that sand is moved by convex objects only [19], which prevents touch regions of arbitrary shape from interacting with sand. Bell and colleagues devised a sand simulation method that handles arbitrary shapes [5], but it models each grain as a discrete element and will not produce real-time simulations on the scale needed for sand animation.

Summer and colleagues developed a faster technique that still falls short of real-time performance [28]. Onoue and colleagues sped it up by assuming that only rigid objects would interact with sand [23]. Our method is also based on Summer’s, but we do not assume objects are rigid, because an artist’s hand can change as it moves across the canvas.

Sand Art for Storytelling

We have found few sand art systems worth noting. Hancock and collegues’ sandtray therapy system allows storytelling on a sand background, but users manipulate figurines instead of sand [11]. Ura and colleagues developed a tool for painting with simulated sand, but it reduces input to discrete points [29]. iSand1 is an iPhone application for sand art that shares this limitation, and its sand granules are much larger than those used in traditional sand animation.

In contrast, SandCanvas captures rich human hand gestures in multiple areas instead of multiple points. It also preserves the expressive and playful nature of sand animation and adds new capabilities that go beyond traditional sand animation, such as recorded gestures and video mixing.