MARS & Data-Augmented Streaming

MARS (Metadata Augmented Real-time Streaming) is a system for enhancing game streaming through real-time, metadata-driven overlays rendered over the streamed video signal. Rather than a one-size-fits-all overlay embedded by the streamer, MARS lets viewers receive personalized, context-aware augmentations of the stream — dynamic highlights that track in-game entities, for instance. The project is a collaboration with Jessica Hammer and doctoral student Noor Hammad, for whom it forms a substantial component of her thesis work.

The architecture introduces two key innovations. First, it supports real-time synchronization between game state and stream overlays, enabling responsive and interactive experiences. Second, it allows for differentiated access to information across viewers, opening up possibilities for asymmetric interaction: unlike traditional overlay tools or audience participation games, which offer shared outputs, MARS enables individualized outputs while maintaining a shared input space.

After MARS was presented as a systems contribution at UIST, the project expanded into a design workshop study exploring how streamers, viewers, designers, and developers envision the system’s possibilities, and into a line of inquiry — with other members of the HCII — into how game-aware streaming might support blind and low-vision viewers, a group largely excluded from a mostly-visual medium. Ongoing directions include extending the system to additional game engines and exploring real-world deployment and evaluation at scale.