Last September I shipped a transcript-first video player for a 42-minute conference recording. It broke inside ten minutes of user testing. The recording carried live EBU-TT-D captions from a stenographer at the event, and I had structured the page so the transcript panel was the primary navigation surface — click a line, the video seeks. When a sighted developer tried scrubbing through the transcript to find a code snippet, the aria-live region I had wired to cue enter/exit events flooded VoiceOver so aggressively that the screen-reader user sitting next to her couldn’t hear anything. Both users were navigating the same content through the same component. I had built something that served neither.

The fix took three weeks. What follows is the experiment log from that build: why I chose DataCue over standard text cues for structured metadata, how I throttled aria-live announcements without losing semantic chapter information, and why cue timing drift at 1.5x playback rate forced me to treat the transcript component as a small distributed system with its own reliability budget. The Google SRE book frames this well — when a system’s timing guarantees degrade under load, you need alerting and compensation, not wishful thinking about browser precision (Google SRE – Site reliability engineering book). Cue timing drift is a reliability defect, not a cosmetic UX issue.

Environment

The conference recording was a 1080p H.264 MP4, 42 minutes 17 seconds, recorded at Open Source Lisbon 2024. The stenographer delivered EBU-TT-D captions in real time via a WebSocket feed, which I converted to WebVTT for browser playback. The test environment spanned three devices:

• Desktop: Firefox 130.0 on Ubuntu 24.04 LTS, Intel i5-1240P, 16 GB RAM, 1440p display
• Laptop: Chrome 129.0.6533.88 on macOS 14.6.1, Apple M2 Pro, 16 GB RAM, 1512×982 viewport
• Mobile: Safari 18.0 on iOS 18.0, iPhone 13 mini, 375×812 viewport, VoiceOver enabled

The WebVTT file contained 847 text cues (the stenographer’s captions) and 12 chapter cues I authored manually after the event. The player used Media Source Extensions for adaptive playback via hls.js 1.5.13, with the transcript component built as a web component using Shadow DOM to isolate styling. I tested with NVDA 2024.3 on Firefox and VoiceOver on both macOS and iOS.

Problem Statement: Why Transcript-First, Not Transcript-Alongside

Most web video players treat the transcript as an optional sidebar — a nice-to-have accessibility feature that mirrors the audio. I wanted to invert that hierarchy. The conference recording was dense with code snippets, library names, and configuration flags. Developers watching it wanted to search, scan, and jump to specific moments. Screen-reader users wanted the same thing, but the standard pattern — a visually scrolling transcript that highlights the active cue — gave them nothing. They got a live region that either announced every caption (unusable) or nothing at all (equally unusable).

The design goal was a component where the transcript text is the primary navigation surface for all users. Sighted users click or tap a line to seek. Screen-reader users navigate by semantic chapter, with the active transcript line announced in a controlled, throttled manner. The video element is present but secondary — it responds to transcript navigation rather than driving it.

Method: DataCue for Structured Metadata

The first decision was how to carry chapter metadata. Standard WebVTT text cues can carry arbitrary string content, but I needed structured data: a chapter title, a long-form description (3–5 sentences summarizing the segment), a list of referenced tools, and timestamp anchors for key moments. Parsing all of that from a text cue’s content string would have meant inventing a microformat and writing a parser for it. Fragile and unpleasant.

DataCue, part of the HTML spec’s timed text track API, lets you attach an arbitrary JavaScript value to a cue. Browser support is incomplete: Firefox 130 supports it, Chrome 129 supports it, Safari 18 supports it for text track cues but with some quirks around structured cloning of certain object shapes. The safe path is to store a JSON-serializable plain object and avoid any class instances or cyclic references.

Here is the chapter cue structure I used:

{
  "type": "chapter",
  "title": "Encoding Pipeline Architecture",
  "description": "The speaker walks through a two-pass FFmpeg pipeline for SVT-AV1, explaining why CRF 32 was chosen over target bitrate for this content type. Includes a comparison of VAAPI and Vulkan hardware paths on Intel Arc A380.",
  "references": ["FFmpeg 7.0", "SVT-AV1 2.1.0", "VAAPI", "Vulkan"],
  "keyMoments": [
    {"time": 742.5, "label": "CRF vs target bitrate discussion"},
    {"time": 815.0, "label": "VAAPI benchmark numbers"}
  ]
}

I loaded these as a separate text track (kind=”metadata”) alongside the caption track (kind=”captions”). The transcript component listened to cuechange events on both tracks: caption cues updated the active transcript line, chapter cues updated the chapter heading and the aria-live announcement. Keeping them on separate tracks meant I could independently control the announcement frequency for each.

The aria-live Flooding Problem

Here is where the first user test failed catastrophically. I had wired a single aria-live=”polite” region to every cuechange event. At normal playback speed, the stenographer’s captions produced a new cue roughly every 4–7 seconds — manageable. But when a sighted user clicked a transcript line to seek, the active cue changed instantly, and if they clicked through several lines rapidly (as developers do when scanning for a specific snippet), the live region queued up 8–10 announcements in a second. VoiceOver on iOS 18 tried to read them all, overlapping and cutting off each announcement mid-word.

The NIST Cybersecurity Framework’s approach to risk management applies here more directly than I expected: accessibility-tree flooding is a systemic risk requiring framework-level mitigation, not individual patching of each edge case (Cybersecurity Framework | NIST). I needed a structured strategy, not a series of ad-hoc fixes.

That same discipline applies to editorial structure: before publishing, editors need a way to test scattered notes become an argument readers can follow, which is where an AI writing software that fits the project can function as a planning aid rather than a substitute for domain evidence.

The solution was a two-tier announcement system. Chapter cues announced through an aria-live=”assertive” region, but only on enter — never on exit. Caption cues did not announce through aria-live at all. Instead, the active transcript line received aria-current=”true” and was focusable. Screen-reader users could navigate the transcript with standard heading and element navigation, and chapter changes were announced proactively. This reduced announcements from roughly 847 per 42 minutes (every caption) to 12 per 42 minutes (every chapter), while preserving the ability to read individual captions on demand.

The throttle for rapid seeking was a 300ms debounce on the aria-live region’s text content update. If a user clicked through five transcript lines in 300ms, only the last one was announced. This matched the existing focus behavior — the last-clicked line received focus, and the announcement aligned with it.

Cue Timing Drift at 1.5x Playback Rate

The second failure was subtler and took longer to diagnose. At 1.5x playback rate, the transcript highlight and the active caption in the video began to drift apart. After about 8 minutes of playback at 1.5x, the transcript highlight was a full cue ahead of the audio. At 2x, the drift appeared within 3 minutes.

The root cause: WebVTT cue timing is specified in seconds relative to the media timeline, but the cuechange event fires based on the media element’s currentTime — which at increased playback rates advances faster than wall-clock time. The transcript component’s highlight logic was independently tracking elapsed time using performance.now() for the scroll-into-view animation, creating a growing skew between the two clocks.

The fix was to stop using performance.now() for any timing related to cue synchronization. Everything — highlight updates, scroll-into-view calculations, aria-live announcements — had to be driven by the video element’s timeupdate event and currentTime property. The tradeoff is that timeupdate fires at roughly 4–66ms intervals (the HTML spec says 15–250ms, but modern browsers fire much more frequently), which is coarse enough that scroll animations look slightly less smooth at normal speed. I compensated by interpolating the scroll position between timeupdate events using requestAnimationFrame, but only for visual polish — never for cue logic.

Here is the measured drift before and after the fix:

// Before fix (performance.now-based highlight):
//   1.0x rate: 0ms drift after 10 min
//   1.5x rate: ~1200ms drift after 10 min
//   2.0x rate: ~2400ms drift after 10 min

// After fix (currentTime-driven highlight):
//   1.0x rate: 0ms drift after 10 min
//   1.5x rate: 0ms drift after 10 min
//   2.0x rate: ~40ms drift after 10 min (timeupdate granularity)

The residual 40ms drift at 2x comes from the interval between timeupdate events. At 2x playback, 20ms of real time equals 40ms of media time. If a timeupdate event fires 20ms late, the highlight is 40ms behind. This is within the threshold where users don’t perceive desynchronization, so I accepted it rather than adding a more complex interpolation layer.

Generating Chapter Descriptions: Manual vs. AI-Assisted Drafting

The 12 chapter cues each needed a 3–5 sentence description summarizing the segment. Writing these manually from the transcript took about 90 minutes — I read through the raw WebVTT, identified natural breakpoints, and drafted descriptions. The descriptions needed to serve two purposes: give a screen-reader user enough context to decide whether to listen to that chapter, and give a sighted developer enough information to know whether the segment contained the code snippet they were looking for.

For comparison, I also drafted chapter descriptions using AI writing software fed with the raw transcript text for each segment, asking for a structured summary matching my DataCue schema. The goal was not to replace manual annotation but to see whether AI-assisted drafting could reduce the 90-minute burden while producing output that met the same accessibility requirements — specifically, descriptions detailed enough to serve as standalone navigation labels for screen-reader users.

The comparison was instructive. The AI-assisted drafts were grammatically clean and structurally consistent, but they consistently under-specified technical details. Where my manual description for the encoding chapter read “CRF 32 was chosen over target bitrate for this content type,” the AI-assisted draft read “a constant quality setting was chosen for encoding.” For a sighted developer scanning for the CRF value, the manual description is immediately actionable. The AI-assisted one requires listening to the segment. I ended up using the AI-assisted drafts as a first pass, then manually revising each one to restore the specific technical references. This brought the total time down to about 50 minutes — a useful improvement, but not the 10-minute shortcut I had hoped for.

The workflow lesson: AI-assisted drafting for accessibility metadata needs a verification pass against the same requirements you’d apply to manual annotation. The drafts are a productivity tool, not a replacement for domain expertise. If your chapter descriptions need to carry specific values, names, and configuration flags — and for a developer-facing conference recording, they absolutely do — then the AI draft is a starting structure, not a finished deliverable.

Tradeoffs

The transcript-first architecture introduced costs I did not anticipate:

• Initial load weight: The full WebVTT file (847 caption cues + 12 chapter cues) is fetched and parsed on page load. For this 42-minute recording, the file was 74 KB gzipped. For a 3-hour recording, that would be roughly 320 KB — still manageable, but worth lazy-loading if you have multiple videos on a page.
• Scroll performance: Rendering 847 transcript lines in the DOM simultaneously caused jank on the iPhone 13 mini during fast scrolling. I switched to virtualized rendering (only 30 lines in the DOM at a time) which fixed the jank but broke Ctrl+F in-browser search. The tradeoff: I built a custom search input that filters transcript lines and renders matching results outside the virtualized list.
• Keyboard navigation complexity: The transcript component needed its own keyboard model. Arrow keys move between transcript lines, Enter seeks the video, and Tab moves between the transcript panel and the video controls. This had to be documented in an aria-label and a visible help tooltip, because it diverges from standard video player keyboard conventions.
• Screen-reader testing burden: The two-tier announcement system required testing across three screen readers (NVDA, VoiceOver macOS, VoiceOver iOS) and two browsers. NVDA 2024.3 on Firefox handled aria-current=”true” correctly. VoiceOver on iOS 18 sometimes skipped the chapter announcement if the user was navigating the transcript with swipe gestures at the moment a chapter boundary was crossed. I added a manual “Announce chapter” button as a fallback.

What I’d Do Differently

Three things, in order of how much pain they would have saved:

First, I would have started with the aria-live architecture instead of bolting it on after the first user test. The instinct to wire every cuechange event to a live region is strong because it feels like the accessible thing to do. It is not. It is the thing that makes your component unusable for the people you’re trying to serve. Design the announcement budget before writing any code: how many announcements per minute is acceptable, what triggers them, and what happens when the user is actively navigating.

Second, I would have used currentTime-driven timing from the start. The performance.now() approach felt cleaner because it decoupled the transcript component from the video element’s event loop. But that decoupling is exactly what creates drift. The video element is the clock. Everything else reads from it.

Third, I would have built the virtualized transcript list before the first user test, not after. 847 DOM elements is too many for a mobile screen reader to handle gracefully, and the jank was visible to sighted users too. The custom search input that replaced Ctrl+F is a decent compromise, but it’s a compromise I made under time pressure. A designed-from-scratch search experience would be better.

Maintenance Note

Watch for Safari’s DataCue behavior on version upgrades. Safari 18 handles structured cloning of plain objects correctly, but earlier Safari versions (17.x) silently failed when the DataCue value contained nested arrays. If you’re targeting a Safari range that includes 17.x, serialize your metadata to a JSON string and parse it in the cuechange handler instead of relying on DataCue’s value property. The performance cost is negligible for chapter cues (you’ll have dozens, not thousands), and the compatibility gain is significant.

Also watch for the WebVTT file growing beyond what the browser will parse synchronously. Chrome 129 parses a 500 KB WebVTT file in under 10ms, but Firefox 130 on the same hardware took 45ms for the same file. If your transcript file approaches 1 MB (roughly a 6-hour recording with dense captions), consider splitting it into chapter-segmented files and loading them on demand as the user navigates.