Video Understand
Extract frames and transcribe video locally with ffmpeg and Whisper
What You Can Do
You can intelligently extract representative frames from videos using scene detection, keyframe analysis, or regular intervals, while simultaneously transcribing audio to text using OpenAI's Whisper model. This skill processes videos entirely offline, giving you frame-by-frame visual context and complete transcripts without relying on external APIs or cloud services.
Features
automatically identifies shot changes and extracts frames at transitions
isolates the most important frames based on visual content analysis
captures frames at regular time intervals for consistent sampling
converts video audio to text using offline speech recognition
control maximum frames extracted to balance detail vs. file size
choose from tiny to large models based on accuracy needs
extract visual content without transcription for faster processing
structured metadata with timestamps, frame paths, and transcription data
Example Output
Scene Detection with Transcription
{
"video": "interview.mp4",
"duration_seconds": 180,
"frames_extracted": 12,
"frames": [
{"timestamp": 0.0, "mode": "scene", "path": "frame_0000.jpg"},
{"timestamp": 15.3, "mode": "scene", "path": "frame_0001.jpg"}
],
"transcription": {
"full_text": "Welcome to the interview. Today we're discussing video production techniques...",
"segments": [
{"start": 0.0, "end": 5.2, "text": "Welcome to the interview."},
{"start": 5.2, "end": 12.1, "text": "Today we're discussing video production techniques..."}
]
}
}
Keyframe-Only Output
✓ Extracted 8 keyframes from video.mp4
✓ Identified critical visual moments
✓ Saved frames: frame_0000.jpg, frame_0001.jpg, ...
What's Included
- SKILL.md: complete setup and CLI reference
- understand_video.py: main script with scene detection, keyframe, and interval extraction modes
- Whisper integration: optional local transcription with configurable model sizes
- CLI flags & options: control frame limits, models, output formats, and processing modes
- JSON schema templates: structured output format for frame metadata and transcription segments
Who It's For
- Video editors — quickly understand video structure and content before detailed editing
- Content creators — extract key moments and auto-generate captions locally
- Researchers — analyze video datasets offline without API costs or privacy concerns
- Accessibility specialists — transcribe video audio to create subtitles and transcripts
- Post-production teams — preview shot sequences and identify editing points visually
Best For
- Transcribing video interviews and podcasts to searchable text
- Extracting thumbnail candidates and key frames for previews
- Analyzing shot composition and scene transitions
- Creating subtitle files from video audio tracks
- Cataloging video library content without external dependencies
You might also like
You'll convert narrative scripts into comprehensive shot-by-shot breakdowns that specify camera movement, framing, timing estimates, emotional beats, and editorial cues. The skill generates detailed visual specifications that serve directors, cinematographers, production teams, and editors—eliminating guesswork between creative intent and execution while reducing coverage waste and accelerating the editorial process.
by dliangthinks
This skill helps you architect complete color strategies before grading sessions begin. You'll generate detailed shot-by-shot grading maps that synchronize visual mood to dynamic audio, develop LUT strategies for specific music genres with documented creative reasoning, establish technical parameter ranges per scene or musical section, and create mood board documentation that aligns stakeholder expectations early. By treating color as a primary creative element rather than post-production cleanup, you reduce revision cycles and deliver grades that amplify both narrative and musical intent.
by dliangthinks
This skill generates comprehensive audio design documentation that bridges creative vision and technical execution. You'll produce detailed mixing specifications with priority hierarchies, platform-specific audio budgets, middleware configurations for Wwise/FMOD, and implementation workflows tailored to your target engines (Unity, Unreal, custom). It accelerates pre-production planning, clarifies audio requirements for programmers, and reduces iteration cycles during integration.
by dliangthinks
You can transform raw video footage into structured, professional content by leveraging AI to identify key moments, organize clips, transcribe audio, and augment with voiceovers, overlays, and subtitles. This skill guides you through a proven pipeline that emphasizes compression and smart structuring rather than AI generation—turning hours of footage into minutes of compelling content optimized for any platform.
by dliangthinks
You can generate production-ready audio documentation that bridges creative sound design and technical implementation. This skill produces asset inventories with precise technical specifications, engine-specific integration instructions for Unreal, Unity, Godot and proprietary engines, platform-aware optimization strategies for mobile/console/PC, and audio middleware configuration guides that development teams can directly implement without iteration cycles.
by dliangthinks
You can build full video production workflows that generate promotional videos, product demos, social media content, and animated explainers from scratch. The skill provides a complete project scaffold with pre-built component libraries, text-to-speech integration, theme systems, and hot-reload development—allowing you to iterate quickly and output broadcast-quality motion graphics without manual keyframing.
by dliangthinks
Build detailed, shoot-ready plans that anticipate technical challenges and capture decisive moments under pressure. You'll generate venue-specific shot lists tailored to unique logistics, develop contingency workflows for unpredictable conditions (weather, lighting, timing delays), and coordinate multi-team coverage with clear positioning and equipment redundancy strategies. This skill transforms vague event briefs into actionable pre-production blueprints that keep you organized and confident on-set.
by dliangthinks
Turn any text idea into a finished video in minutes. Describe your concept, and Claude guides HeyGen's Video Agent to automatically write scripts, select avatars and voices, generate visuals, add professional voiceovers, synchronize pacing, and embed captions. Perfect for creating product demos, explainer videos, marketing content, or social media clips without hiring a production team.
by dliangthinks