KeiSeiKit-1.0/skills/video-gen/SKILL.md

name	description	arguments
video-gen	Use when generating video from frame sequences — FFmpeg extraction, WebP/AVIF conversion, sprite sheets, scroll-synced playback, video encoding/transcoding. Covers the full pipeline from video source to web-ready frame sequence or optimized video.	name description required source Source: video file path, image sequence directory, or AI-generated frames true name description required target Target: frame-sequence, sprite-sheet, optimized-video, gif (default: frame-sequence) false

Video-Gen Skill — Frame Sequence Pipeline

Pipeline Overview

Source Video (MP4/MOV/ProRes)
  │
  ├─→ [Frame Extraction] FFmpeg → PNG sequence
  │     │
  │     ├─→ [Optimize] cwebp → WebP sequence (primary)
  │     ├─→ [Optimize] avifenc → AVIF sequence (smaller, slower encode)
  │     └─→ [Sprite Sheet] ImageMagick montage → single image
  │
  ├─→ [Video Scrub] FFmpeg re-encode → optimized MP4 for scroll scrub
  │
  └─→ [Web Playback] Canvas + ScrollTrigger / video.currentTime

---

1. Frame Extraction [E1]

Basic Extraction

# Extract all frames at source FPS
ffmpeg -i source.mp4 -qscale:v 2 frames/frame_%04d.png

# Extract at specific FPS (30fps → 150 frames for 5s video)
ffmpeg -i source.mp4 -vf "fps=30" frames/frame_%04d.png

# Extract with resolution scaling
ffmpeg -i source.mp4 -vf "fps=30,scale=1920:1080" frames/frame_%04d.png

# Extract specific time range (2s to 7s)
ffmpeg -i source.mp4 -ss 2 -t 5 -vf "fps=30" frames/frame_%04d.png

Frame Count Guidelines

Duration	Desktop (30fps)	Mobile (15fps)	Notes
3 seconds	90 frames	45 frames	Short reveal
5 seconds	150 frames	75 frames	Product showcase
10 seconds	300 frames	150 frames	Full story section
15 seconds	450 frames	225 frames	Max recommended

Rule: More frames = smoother but heavier. 120-180 is the sweet spot for most scroll animations.

---

2. Format Conversion [E1]

PNG to WebP (Recommended)

# Single file
cwebp -q 80 frame_0001.png -o frame_0001.webp

# Batch convert all PNGs
for f in frames/*.png; do
  cwebp -q 80 "$f" -o "${f%.png}.webp"
done

# Parallel batch (faster)
find frames/ -name "*.png" | xargs -P 8 -I {} sh -c '
  cwebp -q 80 "{}" -o "$(echo {} | sed s/.png/.webp/)"
'

PNG to AVIF (Smaller, Slower Encode)

# Requires avifenc (brew install libavif)
avifenc --min 20 --max 30 -s 6 frame_0001.png frame_0001.avif

# Batch
for f in frames/*.png; do
  avifenc --min 20 --max 30 -s 6 "$f" "${f%.png}.avif"
done

Format Comparison

Format	Quality at q80	Size vs PNG	Encode Speed	Browser Support
WebP	Excellent	-85-90%	Fast	All modern [E1]
AVIF	Excellent	-90-95%	Slow (10x)	Chrome, Firefox, Safari 16+ [E1]
JPEG	Good	-80-85%	Fastest	Universal [E1]
PNG	Lossless	Baseline	Fast	Universal [E1]

Decision: WebP for production (best balance). AVIF if encode time is not an issue and you need minimum size.

---

3. Size Budgets [E2]

Per-Frame Targets

Resolution	WebP q80	AVIF q30	Target per frame
960x540 (mobile)	12-18 KB	8-12 KB	<15 KB
1280x720 (tablet)	18-28 KB	12-20 KB	<25 KB
1920x1080 (desktop)	25-45 KB	18-30 KB	<35 KB

Total Budget

Frames	Desktop total	Mobile total	Acceptable?
60	~1.5 MB	~0.7 MB	Great
120	~3.0 MB	~1.4 MB	Good
180	~4.5 MB	~2.1 MB	Acceptable
300	~7.5 MB	~3.5 MB	Heavy, needs lazy load

Hard limit: <5MB total for initial load. Lazy load anything beyond.

---

4. Responsive Frame Sets [E2]

Directory Structure

/public/frames/
  /desktop/    # 1920x1080, 150 frames
  /tablet/     # 1280x720, 120 frames
  /mobile/     # 960x540, 75 frames

FFmpeg Multi-Resolution Script

#!/bin/bash
SOURCE="source.mp4"

# Desktop: 1920x1080, 30fps
mkdir -p frames/desktop
ffmpeg -i "$SOURCE" -vf "fps=30,scale=1920:1080" frames/desktop/frame_%04d.png

# Tablet: 1280x720, 24fps
mkdir -p frames/tablet
ffmpeg -i "$SOURCE" -vf "fps=24,scale=1280:720" frames/tablet/frame_%04d.png

# Mobile: 960x540, 15fps
mkdir -p frames/mobile
ffmpeg -i "$SOURCE" -vf "fps=15,scale=960:540" frames/mobile/frame_%04d.png

# Convert all to WebP
for dir in frames/desktop frames/tablet frames/mobile; do
  for f in "$dir"/*.png; do
    cwebp -q 80 "$f" -o "${f%.png}.webp"
    rm "$f"  # remove PNG after conversion
  done
done

Responsive Loading (JS)

function getBreakpoint() {
  const w = window.innerWidth;
  if (w >= 1280) return { dir: "desktop", count: 150 };
  if (w >= 768)  return { dir: "tablet",  count: 120 };
  return { dir: "mobile", count: 75 };
}

const { dir, count } = getBreakpoint();
const basePath = `/frames/${dir}/frame_`;

---

5. Sprite Sheet (Alternative) [E2]

For fewer frames (<60), a single sprite sheet can be faster than individual files:

# Create sprite sheet with ImageMagick
montage frames/frame_*.webp -tile 10x6 -geometry 480x270+0+0 spritesheet.webp
# 60 frames, 10 columns x 6 rows, each 480x270

CSS Sprite Animation

.sprite-player {
  width: 480px;
  height: 270px;
  background: url("spritesheet.webp");
  background-size: 4800px 1620px; /* 10 cols x 6 rows */
}

JS Sprite + Scroll

const sprite = document.querySelector(".sprite-player");
const cols = 10, rows = 6, total = 60;
const frameW = 480, frameH = 270;

gsap.to({ frame: 0 }, {
  frame: total - 1,
  snap: "frame",
  ease: "none",
  scrollTrigger: {
    trigger: "#sprite-section",
    start: "top top",
    end: "+=2000",
    pin: true,
    scrub: 0.5,
  },
  onUpdate: function() {
    const i = Math.round(this.targets()[0].frame);
    const col = i % cols;
    const row = Math.floor(i / cols);
    sprite.style.backgroundPosition = `-${col * frameW}px -${row * frameH}px`;
  }
});

---

6. Video Scrub (Alternative to Frame Sequence) [E2]

Apple's modern approach: single compressed video + scroll-driven playback.

Optimize Video for Scrub

# Encode for web scrub: low bitrate, many keyframes
ffmpeg -i source.mp4 \
  -c:v libx264 \
  -preset slow \
  -crf 23 \
  -g 1 \          # keyframe every frame (critical for scrub)
  -an \           # no audio
  -movflags +faststart \
  -vf "scale=1920:1080" \
  output-scrub.mp4

Key: -g 1 makes every frame a keyframe, enabling instant seeking. File will be larger than normal video but smaller than frame sequence.

Playback

const video = document.getElementById("scrub-video");

// Ensure video is loaded
video.preload = "auto";

gsap.to(video, {
  currentTime: video.duration || 5, // fallback if metadata not loaded
  ease: "none",
  scrollTrigger: {
    trigger: "#video-section",
    start: "top top",
    end: "+=4000",
    pin: true,
    scrub: true,
  }
});

// Alternative: manual scroll control
video.addEventListener("loadedmetadata", () => {
  const section = document.getElementById("video-section");
  window.addEventListener("scroll", () => {
    const rect = section.getBoundingClientRect();
    const progress = Math.max(0, Math.min(1,
      -rect.top / (rect.height - window.innerHeight)
    ));
    video.currentTime = progress * video.duration;
  });
});

Frame Sequence vs Video Scrub

Factor	Frame Sequence	Video Scrub
Smoothness	Best (instant)	Good (may drop on mobile)
File size	2-5 MB (150 frames)	1-3 MB (one file)
HTTP requests	150 requests	1 request
Memory usage	High (all frames in RAM)	Low (decoded on demand)
Mobile perf	Good	Variable
Complexity	More code	Simpler

Decision: Frame sequence for hero sections where smoothness is critical. Video scrub for secondary content or when bandwidth is limited.

---

7. Preloading Strategy [E1]

Priority Loading

async function loadFrames(basePath, count) {
  const frames = [];

  // Phase 1: Load first 10 frames immediately (show something fast)
  const priority = Array.from({ length: 10 }, (_, i) =>
    loadImage(`${basePath}${String(i + 1).padStart(4, "0")}.webp`)
  );
  const first10 = await Promise.all(priority);
  frames.push(...first10);

  // Phase 2: Load remaining frames in background
  for (let i = 10; i < count; i++) {
    const img = new Image();
    img.src = `${basePath}${String(i + 1).padStart(4, "0")}.webp`;
    frames.push(img);
  }

  return frames;
}

function loadImage(src) {
  return new Promise((resolve, reject) => {
    const img = new Image();
    img.onload = () => resolve(img);
    img.onerror = reject;
    img.src = src;
  });
}

IntersectionObserver Trigger

// Only start loading when section is near viewport
const observer = new IntersectionObserver(
  ([entry]) => {
    if (entry.isIntersecting) {
      loadFrames("/frames/desktop/frame_", 150);
      observer.disconnect();
    }
  },
  { rootMargin: "200px" } // start 200px before visible
);
observer.observe(document.getElementById("sequence-section"));

---

Workflow

1. Source video — get MP4/MOV at highest quality available
2. Plan frame count — duration * fps, aim for 120-180 sweet spot
3. Extract frames — FFmpeg with target resolution and fps
4. Convert to WebP — cwebp q80, check total size budget
5. Create responsive sets — desktop/tablet/mobile with different counts
6. Implement playback — Canvas + GSAP ScrollTrigger (or video scrub)
7. Add preloading — priority first 10, lazy rest, IntersectionObserver
8. Test on mobile — check memory usage, reduce frames if needed
9. Add fallback — static image for prefers-reduced-motion