#
AI video generation company HeyGen recently open-sourced a framework called HyperFrames, with the core idea of bringing the front-end development workflow to video production. Developers can now use familiar technologies like HTML, CSS, and the GreenSock Animation Platform (GSAP) to precisely control and generate video content by writing code, opening up new possibilities for creating dynamic charts, product demos, and batch-produced video content.
Code as Video: A New Production Paradigm
Traditional video production workflows, which heavily rely on GUI software like After Effects or Premiere, have several inherent pain points:
- Efficiency Bottlenecks: Manually adjusting keyframes is a labor-intensive task, where even minor changes can be time-consuming.
- Difficult Version Control: Project files are typically in binary formats, which cannot be effectively managed by version control systems like Git. Team collaboration often degenerates into passing files back and forth.
- Poor Scalability: The process for batch-generating videos based on templates is complex and rigid.
HyperFrames proposes a ‘Code as Video’ solution, defining every element, style, and animation of a video as plain text code. This approach allows video projects to be seamlessly integrated into modern development workflows, leveraging the significant advantages of version control, code reuse, and automated scripting. For developers who need to produce large volumes of data visualizations, product introductions, or dynamic subtitles, HyperFrames can significantly boost efficiency.
Core Architecture: A Layer-by-Layer Look at the Rendering Pipeline
HyperFrames uses a layered architecture to ensure a clear separation of concerns, making the process from user command to final pixel output efficient and controllable. Its four-layer structure is as follows:
- CLI (
@hyperframes/render): The user entry point, providing command-line tools like hyperframes render to initiate the entire rendering task.
- Producer (
@hyperframes/producer): Acts as the ‘orchestrator’ of the rendering pipeline, responsible for parsing input, managing task distribution, coordinating parallel processing, and final audio/video compositing.
- Engine (
@hyperframes/engine): The core frame-capturing engine, responsible for launching a headless browser instance and performing frame-by-frame rendering.
- Core (
@hyperframes/core): The underlying runtime, providing the basic APIs, type definitions, and the FrameAdapter for interfacing with animation libraries that run within the browser page.
The entire process is initiated by the CLI. The Producer dispatches one or more Engine instances, which use logic injected by the Core layer to drive page animations and capture images frame by frame, ultimately generating a video file.
Key Technologies: Achieving Determinism and High Performance
To ensure that code can consistently produce pixel-perfect videos, HyperFrames employs a series of key technologies.
The “Seek-and-Capture” Loop Mechanism
HyperFrames abandons the traditional real-time playback and recording model in favor of a more controllable ‘Seek-and-Capture’ loop. This mechanism breaks down the video rendering process into a series of independent static frame snapshot generations. Its pseudo-code is as follows:
javascript
for (let frame = 0; frame <= totalFrames; frame++) {
// Using integer arithmetic to avoid floating-point errors
const time = Math.floor(frame) / fps;
// Instruct the animation library to jump to the exact time point
await adapter.seekFrame(frame);
// Capture the pixels of the current browser view
}
This method calculates time using precise frame numbers, completely eliminating dependency on the system clock and forming the cornerstone of deterministic rendering.
Efficient Frame Capture via CDP
For frame capture, the Engine launches a minimal headless browser, chrome-headless-shell, and uses the Chrome DevTools Protocol (CDP) to call the experimental HeadlessExperimental.beginFrame API. This API allows an external program to explicitly command the browser’s compositor to render a single frame and directly return the pixel buffer. Its advantages include:
- Atomicity: Each frame’s rendering is an atomic operation, avoiding timing issues that could lead to capturing a ‘partially rendered’ frame.
- High Performance: Pixel data is obtained directly from the GPU compositor, bypassing the overhead of traditional screenshot IPC (Inter-Process Communication).
Protocol-Driven Decoupled Design
HyperFrames achieves flexibility and extensibility through two core protocols:
FrameAdapter Protocol: Defines a standard interface that allows any animation library supporting on-demand seeking to be integrated. The adapter implementation for GSAP is quite intuitive: it directly controls the global timeline via gsap.globalTimeline.seek(time). This protocol requires the seekFrame method to be idempotent and support random access, ensuring stateless rendering.
window.__hf Protocol: Serves as the communication bridge between the Node.js rendering engine and the browser page. After the page loads, the Core runtime attaches an object compliant with this protocol to window.__hf. The engine drives the in-page animation by calling page.evaluate(() => window.__hf.seek(t)), thus enabling cross-process control.
Audio-Visual Separation and Parallel Rendering
A browser’s rendering capabilities are limited to the visual layer. To address this, HyperFrames completely separates audio processing. The Producer parses <audio> and <video> tags and their data-* attributes from the HTML, using FFmpeg to independently extract, trim, and mix all audio tracks. After all video frames are rendered and encoded into a silent video, FFmpeg then merges (muxes) the main audio track with the video stream to create the final MP4 file.
To increase rendering speed, the Producer can launch multiple parallel Engine sessions (workers) based on the number of CPU cores. Each worker is an independent Chrome process responsible for rendering a chunk of frames. Finally, all chunks are merged in sequence, making full use of multi-core processor capabilities.
Practical Application: From Animation Definition to Subtitle Synchronization
In a HyperFrames project, developers declare the timeline using data attributes in HTML:
html
<div
class=“clip”
data-start=“0”
data-duration=“5”
data-track-index=“1”
<h1>Hello World</h1>
</div>
Animations, on the other hand, are defined using GSAP’s JavaScript code and registered to the window.__timelines object for the framework to call. For example, to implement a simple entrance animation:
javascript
// Create and register the timeline
var tl = gsap.timeline({ paused: true });
window.timelines = window.timelines || {};
window.__timelines[“main”] = tl;
// At 0.2s, the title slides in from 100px below
tl.from(“.title”, {
y: 100,
opacity: 0,
duration: 1.0,
ease: “power3.out”
}, 0.2);
By linking animation time points with external data (like subtitle timestamps), developers can easily achieve precise narration-subtitle synchronization, which is particularly useful for generating social media feed videos and educational content. The emergence of HyperFrames opens new, programmatic and engineering-driven doors for video content creation, ensuring that video production is no longer the exclusive domain of designers.
