CoViz_2021: Experiments in Computer Vision

Created by Tyler Klein Longmire

A live motion-capture digital media installation.

Produced with the support of Quickdraw Animation Society.

Initially commissioned for Alberta Culture Days.

Photography by Chelsea Yang-Smith.

APPEARANCES:
  • High Line Brewing, Calgary AB, September 2021
  • Particle+Wave Festival, Calgary AB, March 2022
  • AMAAS Conference, Vulcan AB, July 2022
  • Found Festival, Edmonton AB, July 2023

Downloads

This project is available for bookings!
 Reach out for more information!

BRIEF:

CoViz_2021 is a durational installation where the audience is the performer. Up to four audience members at a time dance in front of a motion-capture camera, and their actions are mapped onto cartoonish digital avatars in real-time.

PROJECT DESCRIPTION:

CoViz_2021 is an experiment in computer vision and real-time rendering of live motion-capture data. Using scavenged hardware, a beefy computer, custom CGI, and the free Unity game engine, I wanted to create a fun and accessible party favour, where the users’ movements get mapped onto outlandish characters in a 3D, cartoon world. Up to four people can dance together in front of the Kinect camera, and their movements are translated onto their cartoon avatars in real-time. They move when you move, dance how you dance. For a brief moment, you’re a digital puppeteer. It’s glitchy, it’s imprecise, but that’s all part of the fun.

Computer vision is fast becoming essential to our workflows and our daily lives. Once the domain of high-end VFX houses, it is now possible to hack together your own basic motion-capture system using consumer devices. Smart, AI-powered cameras are increasingly prevalent in our homes and workplaces, feeding identifying data into corporate databases. What if we used that tech for something silly instead? Does every photon of digital media need to be commodified and sold? This installation is an attempt to explore these questions, to vibe in the metaverse for a night, to make a silly thing for my friends.

TYLER'S NOTES:

I've been working with motion-capture technology for a while now, mostly using Kinect sensors through an escalating complexity of softwares. I initially made this project as a test of a new workflow for a theatrical livestream show I was working on: 366 Days, with Major Matt Mason Theatre Collective. Previous to this, I had used the Kinect in Processing, then in TouchDesigner, but it was always really fussy. So after some experimentation, I got it working using Unity instead. And boy, was it an improvement. So much more flexible than anything I had tried so far. TouchDesigner works, but was really tempermental, and it doesn't have as much control of the entire "scene" the way a Unity scene does.

So I made this weird little installation to prototype this motion-capture workflow into a Unity scene. Working with Quickdraw, we pitched it to Alberta Culture Days and got a bit of cash to support me developing it to a presentable piece. At the time, summer 2021, the height of the pandemic, I was getting really depressed and struggling to take care of myself. Whether I eat breakfast or not is a reliable marker of where my mental health is at. I was thinking a lot about mocap while choking down my granola and yogurt one morning, and decided to make a few character rigs for this project based on my breakfast foods: a bro banana, a bowl of cereal, a cracked egg. An anxious green enby gremlin would be their focus point. Together, they make a healthy breakfast, ready to go explore the world. I wanted to go run around in the forest, and so I put them into a digital one.

There is a secondary scene of the green fellow sitting at a table in a kitchen, phone on the counter, basically having a panic attack while their manic breakfast foods dance around them. It was more reflective of how I was feeling at the time, but that scene wasn't as popular with the crowds!

This installation was designed to be extremely casual. I wanted to see if people would "get it" if they just walked up to it, and maybe got a small bit of explanation from the "operator". And it works! No special suits needed, just wiggle yourself in front of the camera. It is extremely popular, and fun, and people interacting with it have an awesome time. They try to find how the rigs break, and giggle when the banana wanders off-screen by accident, or limbs go where they have no right going. All in all, it's a fun little festival party favour, a nice little installation to liven up the mood and get people moving.