Dynamic Near-regular Texture Tracking and
Manipulation
Overview
A near-regular texture
(NRT) is a geometric and photometric deformation from its regular origin -- a
congruent wallpaper pattern formed by 2D translations of a single tile. A
dynamic NRT is an NRT under motion. Correspondingly, the basic unit of a
dynamic NRT is a well-defined texton, as a geometrically and photometrically
deformed tile, moving through a 3D spatiotemporal space. Although NRTs are
pervasive in man-made and natural environments, effective computational
algorithms for NRTs are few. Through a
systematic and quantitative comparison study of multiple texture synthesis
algorithms, we are able to show that faithful NRT synthesis has challenged most
of the state of the art texture synthesis algorithms. Our recent work on static
NRTs analysis and manipulation (SIGGRAPH 2004) is the first algorithmic
treatment aimed specifically to preserve the regularity and randomness in real
world near regular textures.
The theme of this
project is to address computational issues in modeling, tracking and
manipulating dynamic NRTs. One basic observation on dynamic NRT is its topology
invariance property: the lattice structure of a dynamic NRT remains invariant
despite its drastic geometry or appearance variations. We propose a
lattice-based Markov-Random-Field (MRF) model for dynamic NRT in a 3D
spatiotemporal space. Our dynamic NRT model consists of a global lattice
structure that characterizes the topological constraint among multiple textons
and an image observation model that handles local geometry and appearance
variations. Our model behaves like a network of statistically varied springs.
Based on our dynamic NRT model, we develop a tracking algorithm that can
effectively handle the special challenges of dynamic NRT tracking, including:
ambiguous correspondences, occlusions, illumination variations, and appearance
variations. Furthermore, we implement a dynamic NRT manipulation system that
can replace and superimpose augmented images on a dynamic NRT from an unknown
environment.
Publications
·
Wen-Chieh Lin and Yanxi Liu, “Tracking
Dynamic Near-regular Textures under Occlusions and Rapid Movements,” 9th
European Conference on Computer Vision.
·
Wen-Chieh
Lin, “A Lattice-based MRF Model for Dynamic
Near-regular Texture Tracking and Manipulation,” Technical Report
CMU-RI-TR-05-58, Ph.D. Thesis, Robotics Institute, Carnegie Mellon University,
Dec, 2005.
·
Wen-Chieh Lin and
Yanxi Liu, “A Lattice-based MRF Model for Dynamic
Near-regular Texture Tracking,” IEEE Transactions on Pattern Analysis and
Machine Intelligence, Vol. 29, No. 5, 2007, pp. 777-792.
Links to results
To play the following
videos, a quicktime player and a media player are required.
The avi format is DivX
5.2 (http://www.divx.com/divx or http://www.divx-digest.com/software/divxcodec5.html)
If you use the
Microsoft IE browser, please temporarily allow pop-ups and active controls.
Please click on images
to play videos.
Texton Detection Results
Tracking Dynamic NRTs without Occlusion
Slowly waving cloth
Underwater texture tracking and replacement
(A texture placed under a water tank was seen through disturbed water)
Playing at normal speed, 30 fps |
Playing 10 times slower, 3 fps |
Playing at normal speed, 30 fps |
Playing 4 times slower, 7.5fps |
Tracking Dynamic NRTs with Occlusion
Crowd motion
(Special thanks to Yu
and Wu for running their algorithm on this video) |
Fabric texture
(Special thanks to Guskov for
providing the input video) |
top-left: visibility map of aligned textons |
results with
visibility map |
Validation and Comparison
Different neighborhood systems
Multiple texton templates vs. PCA texton template
The tracking result using multiple texton templates
is better (see frames 16, 68, 88)
Different initial texton positions
Video Editing Applications
Texture replacement of a
fabric texture
tracked lattices and lighting deformation field |
texture replacement
results |
Texture replacement of
an underwater texture
Video superimposing:
“