I still have the commit where I gave up. 3:14 a.m., the interactive documentary about Bologna’s porticoes had 47 branching paths, and the JavaScript file that managed narrative state was 2,800 lines of nested if/else statements. Adding a new scene meant tracing callbacks through five files, and the video player would sometimes jump to the wrong chapter because a boolean flag got toggled twice in the same frame. The project shipped, but I knew I’d built a house of cards. That failure taught me something I should have learned from the start: interactive narrative structure is an engineering problem, not a creative afterthought.

When we talk about web multimedia, we usually talk about codecs, buffering strategies, and the Web Audio API. But the moment you introduce branching—a viewer choosing which interview to watch next, an audio-driven choose-your-own-adventure, an accessible documentary walkthrough—you’re no longer just delivering media. You’re building a state machine that must stay coherent across page loads, browser tab discards, and assistive technology navigation. This article is about how to plot that machine, debug it, and keep it maintainable, using standards that already exist in the browser.

Why Branching Logic Becomes Spaghetti

The root problem is that HTML5 media elements are linear by design. A <video> or <audio> element has a currentTime, a duration, and a play() method. It doesn’t have a concept of “the user chose path B, so the next segment should be chapter 4b instead of 4a.” We have to layer that logic on top, and the naive approach—scattering event listeners that set global variables—works for three branches and collapses at thirty.

In my Bologna project, each video segment had an ended event handler that checked a userChoice variable and loaded the next source. But some segments could be reached from multiple paths, so the handler also had to know the previous state. Soon I had a matrix of origin-destination pairs maintained by hand. When a subject-matter expert asked me to reorder two chapters, I had to update seven different conditionals. That’s not a creative workflow; that’s a reliability incident waiting to happen. The Google SRE book has a chapter on simplicity that applies directly here: “Software systems are inherently dynamic and unstable. A system that is too complex to be understood by the people who operate it is a system that will fail in unexpected ways.”¹ My narrative logic had become exactly that kind of system.

Modeling the Story as a Directed Graph

The first step toward sanity is to stop thinking of the narrative as a timeline and start thinking of it as a directed graph. Each node is a media segment—a video chapter, an audio clip, a text-and-image interlude. Each edge is a transition triggered by a user choice, a timer expiring, or an accessibility preference. This graph can be expressed in a single JSON file that becomes the single source of truth for the entire experience.

Here’s a minimal schema I’ve settled on after several projects:

{
  "nodes": {
    "intro": {
      "src": "media/intro.mp4",
      "type": "video",
      "chapters": [
        { "title": "Opening", "startTime": 0, "endTime": 45 },
        { "title": "The Question", "startTime": 45, "endTime": 90 }
      ],
      "choices": [
        { "label": "Follow the architect", "target": "architect_interview" },
        { "label": "Walk the portico", "target": "portico_walk" }
      ]
    },
    "architect_interview": {
      "src": "media/architect.webm",
      "type": "video",
      "chapters": [
        { "title": "Early work", "startTime": 0, "endTime": 120 }
      ],
      "choices": [
        { "label": "Back to the walk", "target": "portico_walk" }
      ]
    },
    "portico_walk": {
      "src": "media/walk.mp4",
      "type": "video",
      "chapters": [
        { "title": "Pavaglione", "startTime": 0, "endTime": 60 }
      ],
      "choices": []
    }
  },
  "startNode": "intro"
}

This structure gives you several things immediately. You can validate the graph before any media loads: every target must exist in nodes, there must be a startNode, and there should be no unreachable nodes. You can write a small script that traverses the graph and reports orphaned segments. You can serialize the current node name to sessionStorage and restore it when the user returns, even if the browser discarded the tab. And you can generate a visual debugger overlay that shows the entire graph with the active node highlighted—something I’ll describe later.

Synchronizing State with Media Fragments and WebVTT

A graph model solves the branching logic, but it doesn’t solve the synchronization problem. When the user is inside a node, you still need to know where they are within that media segment, and you need to expose that structure to assistive technology. This is where Media Fragments and WebVTT chapters come in.

Media Fragments are a W3C specification that lets you address a portion of a media resource using a URL fragment: media/intro.mp4#t=45,90. Browser support is uneven—Firefox handles them well, Chromium-based browsers are spotty—but you can polyfill the behavior by parsing the fragment yourself and setting currentTime on the media element. The real value is that Media Fragments give you a URL-addressable way to link to a specific chapter. If your graph node’s src is a full file, you can construct chapter-specific URLs that users can bookmark or share. That turns an ephemeral interactive experience into something citable.

WebVTT chapters, on the other hand, give you a way to embed navigation points directly into the media timeline. A WebVTT file with chapter cues looks like this:

WEBVTT

00:00:00.000 --> 00:00:45.000
Opening

00:00:45.000 --> 00:01:30.000
The Question

When you add this track to a video element with <track kind="chapters" src="chapters.vtt">, the browser exposes the chapter list through the textTracks API. Screen readers can announce chapter titles when the user navigates by chapter. More importantly, you can listen for cuechange events and use them to update your graph state. If the user scrubs past a chapter boundary, your state machine can react without polling currentTime.

The combination is powerful: the JSON graph defines the macro-structure (which segment comes next), while WebVTT chapters define the micro-structure (what’s happening inside this segment). Media Fragments let you link to any point in either structure. Together, they form a standards-based spine for interactive narrative that doesn’t depend on any framework.

Building a Narrative Debugger

Once you have a graph model, you can build tooling that would be impossible with ad-hoc conditionals. The most valuable tool I’ve built is a narrative debugger: a side panel that renders the entire story graph as a force-directed layout, highlights the current node, and shows the path the user took to get there.

I implement this with an offscreen <canvas> and the OffscreenCanvas API to keep layout calculations off the main thread. Each node is a circle labeled with the node ID; each edge is a line labeled with the choice text. The current node pulses. Clicked nodes show a breadcrumb trail. The debugger reads the same JSON file that drives the experience, so it’s always in sync.

During development, this debugger catches logic errors that would otherwise surface as “the video just stopped.” I can see at a glance whether a choice leads to the wrong node, whether a node has no outgoing edges, or whether the user’s path through the graph matches the narrative design. In production, I hide the debugger behind a query parameter (?debug=true) so stakeholders and testers can use it without touching the code.

This approach echoes a principle from the NIST Cybersecurity Framework: “Identify, Protect, Detect, Respond, Recover.”² While that framework targets organizational risk, the mental model applies to interactive media. You identify your narrative assets (the graph nodes), protect the integrity of the state machine (single source of truth JSON), detect anomalies (the debugger), respond to bugs (fix the graph, not scattered conditionals), and recover from failures (serialize state so the user doesn’t lose progress). It’s a structured way to think about a problem that’s usually treated as purely creative.

Handling the Hard Cases

A graph model solves the structural problem, but interactive documentaries throw edge cases that require specific engineering attention. The three that have bitten me hardest are state persistence across sessions, accessible choice presentation, and media loading transitions.

Users close laptops. Browsers discard background tabs. If your documentary takes 45 minutes to explore, you cannot assume a single continuous session. I store the current node ID and the user’s path history in sessionStorage (for tab-scoped persistence) and offer an explicit “save progress” button that writes to localStorage. On load, the player checks for saved state and offers to resume. The key detail: I store the path as an array of node IDs, not just the current node. That way, if the graph structure changes between visits (you reordered chapters), the player can detect that the saved path is no longer valid and fall back to the nearest valid ancestor node, rather than crashing.

When a segment ends and the user must make a choice, that choice interface must work for keyboard, screen reader, and switch device users. I render choices as a <fieldset> with <legend> “What would you like to do next?” and radio buttons for each option. The change event triggers the graph transition. This is standard HTML, but it’s easy to forget when you’re focused on video playback. The graph model helps here too: the choices array in each node can include ariaLabel and description fields that the UI renders appropriately.

When the user makes a choice, you need to load the next media segment. A naive approach pauses the current video, sets src on the same element, and calls play(). That causes a visible stutter and loses audio context. Instead, I use two media elements and crossfade between them. While the current segment plays, the next segment preloads in a hidden element. When the transition triggers, I fade the audio gain of the outgoing element down and the incoming element up using the Web Audio API’s GainNode, and crossfade the video using CSS opacity. The graph model tells me which node to preload: I can look ahead at all possible targets from the current node and start preloading them speculatively, respecting the user’s data-saver preference.

Why This Approach Scales

The JSON graph approach scales because it separates concerns that are usually tangled. The narrative designer (who may not be a developer) can edit a JSON file—or better, use a tool that generates it—without touching playback logic. The developer can optimize media loading, accessibility, and state persistence independently. The debugger works on any graph-shaped narrative, not just the one it was built for.

I’ve used this pattern for an audio-driven city tour with 12 branching points, a video-based ethics training module with conditional pathways based on learner responses, and a documentary where the viewer’s choices determined which archival footage they saw. In each case, the graph model kept the codebase manageable and made it possible to hand the project off to another developer without a week of oral tradition.

When I need to draft the initial narrative structure—the nodes, the choices, the chapter boundaries—I often start with a plain-text outline. I’ll sometimes use a tool like the Unsloppy AI Writing App to generate a structured draft from a narrative brief, which I then refine into the JSON schema. The key is that the output is a starting point for human judgment, not a final artifact. The graph model doesn’t care how the JSON was authored; it only cares that it’s valid.

Debugging in Production

Even with a debugger, things go wrong in production. A CDN caches an old version of a WebVTT file. A browser’s autoplay policy blocks the first transition. A user’s assistive technology doesn’t announce the chapter change. You need telemetry that maps user-reported problems back to the graph state.

I instrument the graph transitions with a lightweight event log: each time the player moves from one node to another, it pushes an object with { from, to, timestamp, choiceLabel } to an array. If the user hits an error, the player sends that log along with the error report. This is infinitely more useful than a stack trace, because it tells me exactly what path the user took through the narrative. I can replay that path in my debugger and see what went wrong.

This is the same principle as a postmortem culture in site reliability engineering: you can’t fix what you can’t reconstruct. The Google SRE book’s chapter on postmortems emphasizes blamelessness and learning from failure. When a user’s interactive documentary session breaks, the goal isn’t to assign fault to a particular if statement; it’s to understand the sequence of states that led to the break and fix the graph or the transition logic so it can’t happen again.

What I’d Do Differently Next Time

If I were starting the Bologna project today, I’d build the graph model first, before writing a single line of media playback code. I’d validate the graph with a test suite that checks for orphan nodes, missing targets, and cycles that could trap the user. I’d render the graph as an SVG on the project wiki so the whole team—including the subject-matter experts—could argue about structure without reading code. And I’d treat the narrative state as a first-class citizen, with the same care I give to codec selection and buffer management.

Interactive web narratives are still a frontier. The browser gives us the primitives: <video>, <audio>, WebVTT, Media Fragments, the Web Audio API. What it doesn’t give us is a narrative engine. That’s our job to build, and it’s a job that rewards engineering discipline as much as creative ambition. The open web deserves stories that branch and breathe, and those stories deserve a structure that doesn’t collapse under their own weight.