video-use Released: A Text-Driven AI Tool for Automated Video Editing
Recently, a team called browser-use has open-sourced its video editing project, video-use. The tool aims to highly automate the video editing process through interaction with a large language model (LLM). Users can simply place their source material into a designated folder and generate a final edited video file through natural language conversation.
Core Features and Automated Workflow
video-use automates a series of tedious post-production tasks. Its main features include:
- Intelligent Content Editing: Automatically identifies and cuts filler words (like “um,” “uh”), false starts, and long silences between takes.
- Automated Color Grading: Provides one-click color grading for video clips with built-in presets like “Warm Cinematic” and “Neutral Punch.” It also supports custom color grading via
ffmpeg command chains.
- Audio Processing: Automatically applies a 30ms audio fade-in/out at each cut point to eliminate potential popping sounds or abrupt audio changes.
- Subtitle Generation and Burn-in: Capable of automatically generating and burning subtitles into the video, with support for highly customizable subtitle styles.
- Dynamic Overlay Layers: Supports generating animated overlays using libraries like
Manim, Remotion, or PIL, and can process multiple animation effects in parallel.
- Session Memory: Persists conversation history in a
project.md file, allowing users to pause and resume their work from where they left off.
Core Mechanism: “The LLM Reads, Not Watches, the Video”
The most innovative aspect of video-use is its method of processing video. Traditional approaches, which feed video frame-by-frame into an LLM, would generate massive amounts of data and incur high computational costs. video-use takes a different path by converting video content into structured text, which LLMs are better at handling. Its core is a two-layer structure:
Layer One: Comprehensive Audio Transcription
The project first uses the ElevenLabs Scribe service to transcribe the audio from all video footage. This step generates not only text with word-level timestamps but also includes speaker diarization and information about non-speech audio events like laughter or applause. All this information is packed into an approximately 12KB takes_packed.md file, which serves as the primary basis for the LLM’s editing decisions. This method transforms unstructured video data into structured text that is easy for the LLM to understand and manipulate.
Layer Two: On-Demand Visual References
When a critical decision is needed, such as determining whether a pause is for dramatic effect or a mistake, the system generates a composite image called timeline_view on demand. This image includes filmstrip thumbnails, audio waveforms, and word-by-word annotations for a specific time segment, providing the LLM with necessary visual context. This “on-demand” strategy significantly reduces data processing volume, striking a balance between efficiency and accuracy.
Closed-Loop Workflow and Quality Control
The tool’s workflow is designed as a closed-loop system to ensure reliable output quality. The complete process is as follows:
Transcribe → Pack → LLM Inference → Generate EDL (Edit Decision List) → Render → Self-Assess
Here, “Self-Assess” is a critical quality control step. After each render, the system automatically runs a timeline_view check at every cut point to evaluate common editing errors like jump cuts, audio pops, or obscured subtitles. The video is presented to the user only after passing all self-assessment tests. If issues are found, the system attempts to automatically fix them and re-render, looping up to three times to ensure a professional final product.
Design Principles and Tech Stack
The development of video-use follows several clear design principles, including “audio-first, visuals-follow,” “confirm strategy before execution,” and “self-assess and persist after execution.” On the technical side, the project relies on Claude Code as its core interactive environment and requires ffmpeg for video processing. The audio transcription feature requires an ElevenLabs API key. The project’s code has been released on GitHub at https://github.com/browser-use/video-use.
