Dan Jenkins
CEO at Nimble Ape
The LLM in the Gallery: No Director Required
In broadcast, the gallery is where the director sits, calling cuts, switching cameras, managing the composition of the live output. It takes skill, experience, and a lot of fast decision-making.
Smelter from Software Mansion, treats video composition as code. Every source, layout, transition and overlay is an API call. And if your production stack is just code, there's nothing stopping you handing the keyboard to an LLM and walking out of the gallery.
In this session, we'll do exactly that. We'll set up a live stream with multiple sources available – video feeds, graphics, lower thirds – give an LLM a system prompt telling it it's the director, and let it run the show in real time. The LLM isn't flying blind either: it receives a live speech transcription of what's happening on stage, and has access to image frames from both camera and screen share feeds. It can see and hear the show, and direct accordingly.
You'll watch it reason about what to put on screen, when to cut, and how to compose the output, with no human intervention. It will almost certainly do something unexpected. That's the point.
This isn't a pitch for replacing directors. It's a practical exploration of what falls out when you combine code-first video composition with a multimodal LLM that can reason about live content and issue API calls in response. We'll cover what it tells us about the future of AI-assisted live production.
We don't know exactly how it's going to go. Come along for the ride.
Read bio
Dan Jenkins is the founder of Nimble Ape Ltd and Everycast Labs, a real-time communications consultant, and an IETF editor of the WHEP specification. He organises CommCon, the real-time communications conference, and has spent the last decade building things with WebRTC, SIP, and voice AI that probably shouldn't work but mostly do. He enjoys presenting talks that involve live demos with uncertain outcomes.