Skip to content

video-interaction-mapper

by Figma · official repository

This skill should be used when the user asks to analyze a UI screen recording and map interaction states into Figma. Trigger for requests such as "put video frames in Figma", "extract states from my recording", "map interactions from video to Figma", "analyze this screen recording", "create a storyboard from my video", "deconstruct this interaction in Figma", "annotate the UI states in my recording", or "pull the key moments from this video into Figma". Also trigger when the user references a video file (.mp4, .mov, .webm, .avi) together with Figma, design review, interaction analysis, prototypes, or UI states. The skill extracts key visual moments from a video, infers interaction triggers, and builds an annotated Figma Design storyboard using native Figma annotations and uploaded screensh… [10 more characters not shown]

OfficialRuns commands5 scripts

Labels from its files: runs commands. 7 files, read Oct 21, 2026. Nothing was run.

npx skills add https://github.com/figma/mcp-server-guide --skill video-interaction-mapper

What it lets an agent do

Each label, and the line in the skill's files that set it.

  • Runs commands

    Allows a shell or ships scripts the agent runs.

    • It ships 5 scripts the agent can run.

Findings

Matches of fixed checks, with the text they matched.

No check matched anything in this skill's files.

Not a safety rating, and no finding is not a safety guarantee. These are fixed checks of the files as read on Oct 21, 2026. Nothing was run. How skills are checked.

SKILL.md

As the publisher wrote it. Shown as text, never rendered.

Publisher text
---
name: video-interaction-mapper
description: >
  This skill should be used when the user asks to analyze a UI screen recording and
  map interaction states into Figma. Trigger for requests such as "put video frames
  in Figma", "extract states from my recording", "map interactions from video to
  Figma", "analyze this screen recording", "create a storyboard from my video",
  "deconstruct this interaction in Figma", "annotate the UI states in my recording",
  or "pull the key moments from this video into Figma". Also trigger when the user
  references a video file (.mp4, .mov, .webm, .avi) together with Figma, design
  review, interaction analysis, prototypes, or UI states. The skill extracts key
  visual moments from a video, infers interaction triggers, and builds an annotated
  Figma Design storyboard using native Figma annotations and uploaded screenshot
  assets.
---

# Video Interaction Mapper

Turn a UI recording into a static, annotated Figma storyboard. Extract the important
before/after states, infer what triggered each change, then place clean screenshots,
blue interaction markers, and concise annotations into a Figma Design file.

## Required Inputs and Tools

Start with:

- A local video file path (`.mp4`, `.mov`, `.webm`, or `.avi`).
- `ffmpeg` and `ffprobe` available on the machine.
- Python with Pillow installed.
- Figma MCP tools available: `create_new_file`, `use_figma`, `upload_assets`, and
  `get_screenshot` or `get_design_context`.

Run the bundled files in `scripts/` as executable workflow helpers. They are part
of the skill's implementation, not reference material. Read or modify them only
when debugging, adapting to an unusual environment, or changing the skill itself.

Use a Figma Design file (`figma.com/design/...`) as the target. The generated
Plugin API code creates pages and image-fill frames, which are Design-file
operations. If the user provides a FigJam or Slides URL, ask for a Design file or
create a new one.

Before any `use_figma` call, load the Figma API guidance skill if available
(`figma-use`). Before creating a new Figma file, load the file-creation guidance
skill if available (`figma-create-new-file`). Pass `skillNames:
"figma-use,video-interaction-mapper"` on `use_figma` calls when the client supports
that parameter.

## Workflow

### 1. Validate and Prepare

Confirm the video path exists, then check `ffmpeg -version`. If Pillow is missing,
install it in the active Python environment:

```bash
pip install Pillow --break-system-packages -q
```

Do local analysis before creating or modifying the Figma file. Create a Figma file
only after `key_moments.json`, `upload_manifest.json`, and generated Figma scripts
are ready, unless the user explicitly asks to create the file first. This avoids
leaving partial pages behind when local frame analysis changes.

### 2. Scout the Timeline Quickly

Start with a low-resolution scout pass and contact sheet:

```bash
python <SKILL_DIR>/scripts/extract_key_frames.py \
  --input "<video_path>" \
  --output /tmp/vim_frames_<timestamp>/ \
  --mode scout
```

The scout pass extracts at 1 fps by default, resizes frames to 640 px wide, scores
scene changes, writes `<output_dir>/frames_manifest.json`, and creates
`<output_dir>/contact_sheet.jpg`. Inspect the contact sheet first to identify likely
interaction moments.

Use the slower full extraction only when the scout pass is insufficient:

```bash
python <SKILL_DIR>/scripts/extract_key_frames.py \
  --input "<video_path>" \
  --output /tmp/vim_frames_<timestamp>/full_frames/ \
  --mode full \
  --max-width 1600
```

For very long videos, ask whether to focus on a specific time range before
processing the whole recording. For too many key frames, re-run extraction with a
higher `--scene-threshold`, such as `0.4`.

### 3. Analyze Key Moments

Read `frames_manifest.json` and inspect `contact_sheet.jpg` or relevant extracted
frames. For each meaningful transition, identify:

- Visual change: modal, drawer, dropdown, tooltip, snackbar, overlay, navigation,
  loaded content, selected state, validation state, input focus, or scroll position.
- Trigger: click, tap, keyboard input, hover, scroll, swipe, form submit, auto/timer,
  or unknown trigger when the cause is ambiguous.
- Interaction target and result coordinates when visible and useful.

Use normalized coordinates from `0` to `1` relative to the screenshot. Percent-like
values such as `45` or `94` are accepted by the generated Figma script and converted
to `0.45` or `0.94`. Coordinates are machine metadata for marker placement only.
Do not include coordinate values in visible annotation text, moment labels, or user
summaries.

Only add marker coordinates when the point lands on a clearly visible UI element or
state. If a target is inferred, between visible controls, or based mainly on cursor
position, mark it with `inferred: true`, `uncertain: true`, `visible: false`, or
`confidence: "medium"`/`"low"`. The generated script suppresses markers for those
points by default so the canvas does not show a confident pulse over empty UI. If
the target or result cannot be located usefully, set `interaction` to `null`.

Use `confidence: "high"` or omit `confidence` only when the marker should be drawn.
Use `show_marker: true` only when an uncertain point is still worth marking; the
generated marker uses an alternate style for these forced uncertain markers.

Aim for 5 to 15 key moments. Prioritize high scene scores and genuinely different
states over tiny visual shifts.

Write the analysis to `<output_dir>/key_moments.json` as an array. When using the
scout flow, provide timestamps first; resolve high-quality frame files in the next
step.

```json
[
  {
    "moment_index": 1,
    "timestamp_s": 3.4,
    "before_timestamp_s": 3.2,
    "after_timestamp_s": 3.4,
    "trigger": "Click",
    "trigger_emoji": "",
    "short_label": "Dropdown menu opens",
    "annotation": "The account avatar is activated. A menu appears below it with profile, billing, and sign-out actions.",
    "interaction": {
      "target_frame": "before",
      "target": {
        "x": 0.91,
        "y": 0.08,
        "label": "Account avatar",
        "confidence": "high"
      },
      "result_frame": "after",
      "result": {
        "x": 0.86,
        "y": 0.2,
        "label": "Opened account menu"
      },
      "annotation": "Activating the avatar opens the account menu."
    }
  }
]
```

Keep annotations human-readable: `Target: Account avatar`, not a label with a raw
coordinate suffix. Coordinates stay hidden in JSON and appear only as visual marker
placement.

If a marker is drawn, the native Figma annotation is attached to that marker node
rather than the screenshot frame. If a marker is suppressed because the point is
uncertain or not visible, the annotation remains attached to the screenshot frame.

### 4. Resolve Selected Frames

If `key_moments.json` uses timestamps, extract only the selected before/after
frames at higher quality:

```bash
python <SKILL_DIR>/scripts/resolve_moment_frames.py \
  --video "<video_path>" \
  --moments-file /tmp/vim_frames_<timestamp>/key_moments.json \
  --output /tmp/vim_frames_<timestamp>/key_moments_resolved.json \
  --frames-dir /tmp/vim_frames_<timestamp>/selected_frames/ \
  --max-width 1600
```

Use `key_moments_resolved.json` in later steps. If a full extraction already
produced precise `before_frame_path` and `after_frame_path` values, this step can
be skipped.

### 5. Prepare Uploadable Screenshot Assets

Compress and resize the before/after screenshots for upload:

```bash
python <SKILL_DIR>/scripts/prepare_upload_frames.py \
  --manifest /tmp/vim_frames_<timestamp>/frames_manifest.json \
  --moments-file /tmp/vim_frames_<timestamp>/key_moments_resolved.json \
  --output /tmp/vim_frames_<timestamp>/upload_manifest.json
```

The script writes optimized JPEGs under `<output_dir>/upload_assets/` and creates
`upload_manifest.json`. Defaults are 1440 px wide for landscape recordings and 900
px wide for portrait recordings at JPEG quality 75. It records image dimensions,
actual JPEG quality, file size, and whether each asset fits the upload budget.

### 6. Generate Figma Call Files

Generate the self-contained Plugin API scripts:

```bash
python <SKILL_DIR>/scripts/generate_figma_calls.py \
  --prepared /tmp/vim_frames_<timestamp>/upload_manifest.json \
  --output-dir /tmp/vim_frames_<timestamp>/figma_calls/
```

This writes:

- `figma_storyboard.js`: one combined storyboard creation call.
- `figma_apply_fills_template.js`: a fill-pass template for uploaded image hashes.
- `figma_manifest.json`: execution order and upload instructions.
- `run_manifest.json`: resumable status, selected moments, expected uploads, node
  IDs, image hashes, fill status, and verification status.

The generated JavaScript does not embed image bytes and does not call
`figma.createImage`. The storyboard script creates all screenshot containers,
labels, native annotations, and marker overlays in one call, then returns
`uploadTargets` containing Figma node IDs and local JPEG paths. In a newly created
blank file, the script reuses and renames the default page instead of creating a
second page.

### 7. Execute in Figma and Upload Assets

Create a new Figma Design file or open the target Design file now. Run the generated
files in `figma_manifest.json`:

1. Read `figma_storyboard.js`.
2. Call `use_figma` against the target Design file with the full JavaScript content.
3. Save the result JSON if practical, then update the run manifest:

```bash
python <SKILL_DIR>/scripts/update_run_manifest.py \
  --run-manifest /tmp/vim_frames_<timestamp>/figma_calls/run_manifest.json \
  --storyboard-result /tmp/vim_frames_<timestamp>/storyboard_result.json
```

For each returned `uploadTargets` item:

- Call `upload_assets` with the target `fileKey`, `count: 1`, `nodeId`, and
  `scaleMode: "FILL"`.
- POST the local file at `target.path` to the returned upload URL.
- Record `{ "nodeId": "...", "imageHash": "...", "scaleMode": "FILL" }` in an
  uploads JSON file.

Generate and run the fill pass:

```bash
python <SKILL_DIR>/scripts/update_run_manifest.py \
  --run-manifest /tmp/vim_frames_<timestamp>/figma_calls/run_manifest.json \
  --uploads-file /tmp/vim_frames_<timestamp>/uploaded_images.json \
  --write-fill-script /tmp/vim_frames_<timestamp>/figma_calls/figma_apply_fills.js
```

Run `figma_apply_fills.js` once with `use_figma`. This explicit image-hash fill pass
is preferred because it makes uploaded screenshots render reliably in the generated
containers.

### 8. Verify

Run `get_screenshot`, `download_assets`, or `get_design_context` to verify that:

- Screenshots are visible and nonblank.
- Blue marker rings sit on the intended target/result locations.
- Native annotations use human labels and do not expose coordinate values.
- Ambiguous or inferred targets are suppressed unless explicitly forced with
  `show_marker: true`.

Update `run_manifest.json` with verification artifacts when available:

```bash
python <SKILL_DIR>/scripts/update_run_manifest.py \
  --run-manifest /tmp/vim_frames_<timestamp>/figma_calls/run_manifest.json \
  --verification-screenshot /tmp/vim_frames_<timestamp>/figma_storyboard.png \
  --verification-notes "Screenshots and markers verified"
```

## Output Expectations

The storyboard layout is left-to-right:

```text
[Title block]

[Before] [After]   [Before] [After]   [Before] [After]
moment 1           moment 2           moment 3

"Before state"     "01 Dropdown menu opens"
                   Annotation text...
```

Blue ring/dot overlays show the target and result locations when coordinates are
available and confident. Native Figma annotations attach to the marker nodes when
markers exist; otherwise they attach to the relevant before/after screenshot nodes.
Annotation labels should name the UI element or state change, never raw coordinate
percentages.

Report back with:

- The number of key moments mapped.
- The Figma file URL.
- Any moments whose trigger was ambiguous.
- Any uploads or frames that failed.
- The verification status and any inferred/uncertain markers.

## Edge Cases

- Long video: ask for a time range or warn that processing can take longer.
- Too many moments: raise `--scene-threshold` or manually keep the most meaningful
  5 to 15 transitions.
- Mobile recording: keep the portrait default width of 900 px so text remains
  readable without oversized assets.
- Missing trigger evidence: use `Unknown trigger` instead of guessing.
- Asset upload failure: retry the individual `upload_assets` target; the generated
  screenshot container remains in the file and can be reused.
- `upload_assets` succeeds but images do not render: run the generated fill pass
  with the returned image hashes.
- `upload_assets` is unavailable: warn that the fallback path is slower because it
  must embed image data directly in `use_figma` calls.

Scripts (5)

Read as text. anlyon.dev never runs them.

scripts/extract_key_frames.pyShow
#!/usr/bin/env python3
"""
Extract key frames from a video file using ffmpeg.

This script:
1. Extracts frames at a given fps
2. Runs ffmpeg scene-change detection to score each frame
3. Flags frames with scene scores above the threshold as key frames
4. Optionally writes a compact contact sheet for quick visual review
5. Outputs JPEG frames + a frames_manifest.json

Usage:
    python extract_key_frames.py --input video.mp4 --output /tmp/frames/ --mode scout
"""

import argparse
import json
import os
import re
import subprocess
import sys
from pathlib import Path


def get_video_info(video_path: str) -> dict:
    """Get basic video metadata using ffprobe."""
    cmd = [
        "ffprobe", "-v", "quiet",
        "-print_format", "json",
        "-show_streams", "-show_format",
        video_path
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0:
        raise RuntimeError(f"ffprobe failed: {result.stderr}")

    data = json.loads(result.stdout)

    # Find video stream
    video_stream = next(
        (s for s in data.get("streams", []) if s.get("codec_type") == "video"),
        None
    )
    if not video_stream:
        raise RuntimeError("No video stream found in file")

    duration = float(data.get("format", {}).get("duration", 0))
    width = int(video_stream.get("width", 0))
    height = int(video_stream.get("height", 0))

    # Parse framerate
    fps_str = video_stream.get("r_frame_rate", "30/1")
    fps_parts = fps_str.split("/")
    fps = float(fps_parts[0]) / float(fps_parts[1]) if len(fps_parts) == 2 else float(fps_str)

    return {
        "duration_s": duration,
        "width": width,
        "height": height,
        "native_fps": fps,
        "orientation": "portrait" if height > width else "landscape"
    }


def extract_all_frames(video_path: str, output_dir: str, fps: int = 5, max_frames: int = 3000) -> list[str]:
    """Extract all frames at the given fps as JPEG files."""
    os.makedirs(output_dir, exist_ok=True)

    output_pattern = os.path.join(output_dir, "frame_%06d.jpg")

    cmd = [
        "ffmpeg", "-i", video_path,
        "-vf", f"fps={fps}",
        "-q:v", "3",          # JPEG quality (1-31, lower = better)
        "-vframes", str(max_frames),
        output_pattern,
        "-y", "-v", "quiet"
    ]

    print(f"Extracting frames at {fps}fps...", flush=True)
    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0:
        raise RuntimeError(f"ffmpeg frame extraction failed: {result.stderr}")

    # Collect output frames in order
    frames = sorted(Path(output_dir).glob("frame_*.jpg"))
    print(f"Extracted {len(frames)} frames", flush=True)
    return [str(f) for f in frames]


def detect_scene_changes(video_path: str, fps: int = 5) -> dict[float, float]:
    """
    Use ffmpeg's scene filter to get scene change scores at each timestamp.
    Returns a dict mapping timestamp (seconds) -> scene score (0.0-1.0).
    """
    # ffmpeg scene detection — outputs to stderr with showinfo
    cmd = [
        "ffmpeg", "-i", video_path,
        "-vf", f"fps={fps},select='gte(scene,0)',metadata=print:file=-",
        "-an", "-f", "null", "-",
        "-v", "quiet"
    ]

    print("Running scene change detection...", flush=True)
    result = subprocess.run(cmd, capture_output=True, text=True)
    # Scene metadata goes to stdout via metadata=print:file=-
    output = result.stdout

    scene_scores = {}
    current_ts = None

    for line in output.splitlines():
        # Look for pts_time and lavfi.scene_score
        ts_match = re.search(r'pts_time:([0-9.]+)', line)
        if ts_match:
            current_ts = float(ts_match.group(1))
        score_match = re.search(r'lavfi\.scene_score=([0-9.]+)', line)
        if score_match and current_ts is not None:
            scene_scores[current_ts] = float(score_match.group(1))
            current_ts = None

    print(f"Got scene scores for {len(scene_scores)} timestamps", flush=True)
    return scene_scores


def match_frames_to_scores(
    frame_paths: list[str],
    scene_scores: dict[float, float],
    fps: int,
    threshold: float
) -> list[dict]:
    """
    Build the manifest by matching each extracted frame to its scene score.
    Frame index N corresponds to timestamp N/fps seconds.
    """
    frames = []
    for i, path in enumerate(frame_paths):
        timestamp = i / fps

        # Find the closest scene score timestamp within a small window
        window = 1.0 / fps  # within one frame's width
        closest_score = 0.0
        for ts, score in scene_scores.items():
            if abs(ts - timestamp) <= window:
                if score > closest_score:
                    closest_score = score

        is_key = closest_score >= threshold

        frames.append({
            "index": i,
            "path": path,
            "timestamp_s": round(timestamp, 3),
            "scene_score": round(closest_score, 4),
            "is_key_frame": is_key
        })

    return frames


def resize_frames(frame_paths: list[str], max_width: int = 1600):
    """
    Resize extracted frames to a max width while preserving aspect ratio.
    Modifies files in-place using PIL.
    """
    if max_width <= 0:
        print("Skipping frame resize", flush=True)
        return

    try:
        from PIL import Image
    except ImportError:
        print("Pillow not available, skipping resize", flush=True)
        return

    print(f"Resizing frames to max {max_width}px wide...", flush=True)
    for path in frame_paths:
        with Image.open(path) as img:
            if img.width > max_width:
                ratio = max_width / img.width
                new_h = int(img.height * ratio)
                img = img.resize((max_width, new_h), Image.LANCZOS)
                img.save(path, "JPEG", quality=85, optimize=True)


def sample_frames_for_contact_sheet(frames: list[dict], max_frames: int) -> list[dict]:
    """Return up to max_frames frames, evenly sampled from the manifest."""
    if max_frames <= 0 or len(frames) <= max_frames:
        return frames
    if max_frames == 1:
        return [frames[0]]

    sampled = []
    step = (len(frames) - 1) / (max_frames - 1)
    seen = set()
    for sample_index in range(max_frames):
        frame_index = round(sample_index * step)
        if frame_index not in seen:
            sampled.append(frames[frame_index])
            seen.add(frame_index)
    return sampled


def write_contact_sheet(
    frames: list[dict],
    output_path: str,
    cols: int = 5,
    thumb_width: int = 260,
    max_frames: int = 80,
) -> str | None:
    """Create a timeline contact sheet for fast visual scanning."""
    if not frames:
        return None

    try:
        from PIL import Image, ImageDraw
    except ImportError:
        print("Pillow not available, skipping contact sheet", flush=True)
        return None

    selected = sample_frames_for_contact_sheet(frames, max_frames)
    cols = max(1, cols)
    rows = (len(selected) + cols - 1) // cols
    label_h = 34
    gutter = 10

    first = Image.open(selected[0]["path"])
    thumb_height = max(1, int(first.height * (thumb_width / first.width)))
    first.close()

    sheet_w = cols * thumb_width + (cols + 1) * gutter
    sheet_h = rows * (thumb_height + label_h) + (rows + 1) * gutter
    sheet = Image.new("RGB", (sheet_w, sheet_h), (246, 246, 248))
    draw = ImageDraw.Draw(sheet)

    for cell_index, frame in enumerate(selected):
        col = cell_index % cols
        row = cell_index // cols
        x = gutter + col * (thumb_width + gutter)
        y = gutter + row * (thumb_height + label_h + gutter)

        with Image.open(frame["path"]) as img:
            if img.mode != "RGB":
                img = img.convert("RGB")
            thumb = img.resize((thumb_width, thumb_height), Image.LANCZOS)
            sheet.paste(thumb, (x, y))

        label = f"#{frame['index']}  t={frame['timestamp_s']:.1f}s  score={frame['scene_score']:.2f}"
        if frame.get("is_key_frame"):
            label = "* " + label
        draw.text((x, y + thumb_height + 6), label, fill=(32, 32, 36))

    Path(output_path).parent.mkdir(parents=True, exist_ok=True)
    sheet.save(output_path, "JPEG", quality=86, optimize=True)
    return output_path


def main():
    parser = argparse.ArgumentParser(description="Extract key frames from a video file")
    parser.add_argument("--input", required=True, help="Path to input video file")
    parser.add_argument("--output", required=True, help="Output directory for frames")
    parser.add_argument(
        "--mode",
        choices=["scout", "full"],
        default="full",
        help="scout extracts a low-fps review timeline; full extracts denser frames (default: full)",
    )
    parser.add_argument("--fps", type=int, default=None, help="Frames per second to extract")
    parser.add_argument("--scene-threshold", type=float, default=0.25,
                        help="Scene change threshold 0.0-1.0 (default: 0.25). "
                             "Higher = fewer key frames detected.")
    parser.add_argument("--max-width", type=int, default=None,
                        help="Max frame width in pixels. Use 0 to keep native resolution.")
    parser.add_argument("--max-frames", type=int, default=3000,
                        help="Maximum extracted frames (default: 3000)")
    parser.add_argument("--contact-sheet", dest="contact_sheet", action="store_true",
                        help="Write contact_sheet.jpg next to the manifest")
    parser.add_argument("--no-contact-sheet", dest="contact_sheet", action="store_false",
                        help="Skip contact sheet generation")
    parser.set_defaults(contact_sheet=None)
    parser.add_argument("--contact-sheet-cols", type=int, default=5,
                        help="Columns in the contact sheet (default: 5)")
    parser.add_argument("--contact-sheet-max-frames", type=int, default=80,
                        help="Maximum thumbnails in the contact sheet (default: 80)")
    args = parser.parse_args()

    fps = args.fps if args.fps is not None else (1 if args.mode == "scout" else 5)
    max_width = args.max_width if args.max_width is not None else (640 if args.mode == "scout" else 1600)
    make_contact_sheet = args.contact_sheet if args.contact_sheet is not None else args.mode == "scout"

    if not os.path.exists(args.input):
        print(f"Error: video file not found: {args.input}", file=sys.stderr)
        sys.exit(1)

    # Check ffmpeg
    check = subprocess.run(["ffmpeg", "-version"], capture_output=True)
    if check.returncode != 0:
        print("Error: ffmpeg not found. Install with: brew install ffmpeg (macOS) or apt install ffmpeg (Linux)",
              file=sys.stderr)
        sys.exit(1)

    os.makedirs(args.output, exist_ok=True)

    # Get video metadata
    print(f"Analyzing video: {args.input}", flush=True)
    video_info = get_video_info(args.input)
    print(f"  Duration: {video_info['duration_s']:.1f}s | "
          f"Resolution: {video_info['width']}x{video_info['height']} | "
          f"Orientation: {video_info['orientation']}", flush=True)

    # Extract all frames
    frame_paths = extract_all_frames(args.input, args.output, fps, args.max_frames)

    if not frame_paths:
        print("Error: no frames were extracted", file=sys.stderr)
        sys.exit(1)

    # Resize frames, preserving enough resolution for Figma inspection.
    if max_width > 0:
        resize_frames(frame_paths, max_width)
    else:
        print("Keeping extracted frames at native resolution", flush=True)

    # Detect scene changes
    scene_scores = detect_scene_changes(args.input, fps)

    # Build manifest
    frames = match_frames_to_scores(frame_paths, scene_scores, fps, args.scene_threshold)

    key_count = sum(1 for f in frames if f["is_key_frame"])
    print(f"Found {key_count} key frames out of {len(frames)} total "
          f"(threshold: {args.scene_threshold})", flush=True)

    contact_sheet_path = None
    if make_contact_sheet:
        contact_sheet_path = os.path.join(args.output, "contact_sheet.jpg")
        contact_sheet_path = write_contact_sheet(
            frames,
            contact_sheet_path,
            cols=args.contact_sheet_cols,
            max_frames=args.contact_sheet_max_frames,
        )
        if contact_sheet_path:
            print(f"Contact sheet written to: {contact_sheet_path}", flush=True)

    manifest = {
        "video_path": os.path.abspath(args.input),
        "output_dir": os.path.abspath(args.output),
        "mode": args.mode,
        "fps_extracted": fps,
        "scene_threshold": args.scene_threshold,
        "max_width": max_width,
        "video_info": video_info,
        "contact_sheet_path": os.path.abspath(contact_sheet_path) if contact_sheet_path else None,
        "total_frames": len(frames),
        "key_frame_count": key_count,
        "frames": frames
    }

    manifest_path = os.path.join(args.output, "frames_manifest.json")
    with open(manifest_path, "w") as f:
        json.dump(manifest, f, indent=2)

    print(f"\nManifest written to: {manifest_path}")
    print(f"Key frames: {key_count} / {len(frames)}")

    # Print a quick summary of key frame timestamps
    key_frames = [f for f in frames if f["is_key_frame"]]
    if key_frames:
        print("\nKey frame timestamps:")
        for kf in key_frames[:20]:  # show first 20
            print(f"  t={kf['timestamp_s']:.2f}s  score={kf['scene_score']:.3f}  {kf['path']}")
        if len(key_frames) > 20:
            print(f"  ... and {len(key_frames) - 20} more")


if __name__ == "__main__":
    main()
scripts/generate_figma_calls.pyShow
#!/usr/bin/env python3
"""
Generate Figma Plugin API JavaScript for a complete interaction storyboard.

The generated storyboard script creates all screenshot containers, labels,
native annotations, and visual interaction markers in one use_figma call. It
does not embed image bytes. It returns uploadTargets so the agent can upload
optimized screenshots with upload_assets, then run the generated fill script.

Usage:
    python generate_figma_calls.py \
      --prepared /tmp/vim_frames/upload_manifest.json \
      --output-dir /tmp/vim_frames/figma_calls/
"""

import argparse
import datetime
import json
import os
import re
import sys
from pathlib import Path


STORYBOARD_TEMPLATE = r'''// AUTO-GENERATED by generate_figma_calls.py - do not edit manually
// Creates the full interaction storyboard. Upload images separately with upload_assets.
const PAGE_NAME = __PAGE_NAME_JSON__;
const STORYBOARD = __STORYBOARD_JSON__;

let page = figma.root.children.find(p => p.type === "PAGE" && p.name === PAGE_NAME);
if (!page) {
  const reusableBlankPage = figma.root.children.length === 1 ? figma.root.children[0] : null;
  page = reusableBlankPage || figma.createPage();
  page.name = PAGE_NAME;
}
await figma.setCurrentPageAsync(page);

// Make re-runs deterministic for this generated page.
for (const child of [...page.children]) {
  child.remove();
}

await figma.loadFontAsync({ family: "Inter", style: "Regular" });
await figma.loadFontAsync({ family: "Inter", style: "Semi Bold" });
await figma.loadFontAsync({ family: "Inter", style: "Bold" });

const createdNodeIds = [];
const mutatedNodeIds = [page.id];
const uploadTargets = [];

const frameW = STORYBOARD.frameWidth;
const frameH = STORYBOARD.frameHeight;
const GAP = 80;
const MOMENT_GAP = 160;
const TITLE_HEIGHT = 240;
const LABEL_GAP = 16;
const NOTE_GAP = 8;
const ns = "videoInteractionMapper";

const colors = {
  titleBg: { r: 0.067, g: 0.067, b: 0.067 },
  white: { r: 1, g: 1, b: 1 },
  muted: { r: 0.58, g: 0.58, b: 0.62 },
  mutedText: { r: 0.45, g: 0.45, b: 0.45 },
  accent: { r: 0.38, g: 0.38, b: 0.93 },
  frameBg: { r: 0.92, g: 0.92, b: 0.95 },
  frameStroke: { r: 0.78, g: 0.78, b: 0.84 },
  marker: { r: 0.098, g: 0.435, b: 1 },
  inferredMarker: { r: 1.0, g: 0.62, b: 0.05 }
};

const makeText = (chars, size, style, color, x, y, maxWidth, parent = page) => {
  const node = figma.createText();
  node.fontName = { family: "Inter", style };
  node.characters = chars || "";
  node.fontSize = size;
  node.fills = [{ type: "SOLID", color }];
  node.x = x;
  node.y = y;
  if (maxWidth) {
    node.resize(maxWidth, node.height);
    node.textAutoResize = "HEIGHT";
  }
  parent.appendChild(node);
  createdNodeIds.push(node.id);
  return node;
};

const makeUploadFrame = (name, x, y, width, height) => {
  const frame = figma.createFrame();
  frame.name = name;
  frame.resize(width, height);
  frame.x = x;
  frame.y = y;
  frame.fills = [{ type: "SOLID", color: colors.frameBg }];
  frame.strokes = [{ type: "SOLID", color: colors.frameStroke }];
  frame.strokeWeight = 1;
  frame.cornerRadius = 8;
  frame.clipsContent = true;
  page.appendChild(frame);
  createdNodeIds.push(frame.id);
  return frame;
};

const toUnit = (value) => {
  const number = Number(value);
  if (!Number.isFinite(number)) return null;
  return Math.max(0, Math.min(1, number > 1 ? number / 100 : number));
};

const normalizePoint = (point) => {
  if (!point) return null;
  if (Array.isArray(point)) return { x: point[0], y: point[1], label: "" };
  return point;
};

const isUncertainPoint = (point) => {
  const normalized = normalizePoint(point);
  if (!normalized) return false;
  if (normalized.uncertain === true || normalized.inferred === true) return true;
  if (typeof normalized.confidence === "string") {
    return ["medium", "low", "inferred", "uncertain"].includes(normalized.confidence.toLowerCase());
  }
  if (typeof normalized.confidence === "number") {
    return normalized.confidence < 0.85;
  }
  return false;
};

const pointHasCoordinates = (point) => {
  const normalized = normalizePoint(point);
  if (!normalized) return false;
  return toUnit(normalized.x) !== null && toUnit(normalized.y) !== null;
};

const hasExplicitMarkerOverride = (point, value) => {
  const normalized = normalizePoint(point);
  if (!normalized) return false;
  return normalized.marker === value ||
    normalized.draw_marker === value ||
    normalized.drawMarker === value ||
    normalized.show_marker === value ||
    normalized.showMarker === value;
};

const shouldDrawMarker = (point) => {
  const normalized = normalizePoint(point);
  if (!normalized || !pointHasCoordinates(normalized)) return false;
  if (normalized.visible === false || hasExplicitMarkerOverride(normalized, false)) return false;
  if (hasExplicitMarkerOverride(normalized, true)) return true;
  return !isUncertainPoint(normalized);
};

const shouldDrawResultMarker = (point) => {
  return hasExplicitMarkerOverride(point, true);
};

const labelForPoint = (point, fallback) => {
  const normalized = normalizePoint(point);
  const label = normalized && normalized.label ? normalized.label : fallback;
  return isUncertainPoint(normalized) ? `${label} (inferred)` : label;
};

const addInteractionMarker = (frame, point, markerName) => {
  const normalized = normalizePoint(point);
  if (!shouldDrawMarker(normalized)) return null;
  const x = toUnit(normalized.x);
  const y = toUnit(normalized.y);
  if (x === null || y === null) return null;

  const uncertain = isUncertainPoint(normalized);
  const markerColor = uncertain ? colors.inferredMarker : colors.marker;
  const size = uncertain ? 56 : 48;

  const ring = figma.createEllipse();
  ring.name = uncertain ? `${markerName}_inferred` : markerName;
  ring.resize(size, size);
  ring.x = (x * frameW) - (size / 2);
  ring.y = (y * frameH) - (size / 2);
  ring.fills = [{ type: "SOLID", color: markerColor, opacity: uncertain ? 0.1 : 0.14 }];
  ring.strokes = [{ type: "SOLID", color: markerColor }];
  ring.strokeWeight = uncertain ? 3 : 4;
  frame.appendChild(ring);
  createdNodeIds.push(ring.id);

  const dot = figma.createEllipse();
  dot.name = `${ring.name}_dot`;
  dot.resize(10, 10);
  dot.x = (x * frameW) - 5;
  dot.y = (y * frameH) - 5;
  dot.fills = [{ type: "SOLID", color: markerColor }];
  frame.appendChild(dot);
  createdNodeIds.push(dot.id);

  return ring;
};

const getInteractionCategory = async () => {
  if (!figma.annotations) return null;
  const categories = await figma.annotations.getAnnotationCategoriesAsync();
  const existing = categories.find(category => category.label === "Interaction");
  return existing || await figma.annotations.addAnnotationCategoryAsync({
    label: "Interaction",
    color: "blue"
  });
};

const appendNativeAnnotation = (node, labelMarkdown, categoryId) => {
  if (!categoryId) return;
  node.annotations = [
    ...(node.annotations || []),
    { labelMarkdown, categoryId }
  ];
  mutatedNodeIds.push(node.id);
};

const titleBg = figma.createFrame();
titleBg.name = "_title_block";
titleBg.resize(860, 200);
titleBg.x = 0;
titleBg.y = 0;
titleBg.fills = [{ type: "SOLID", color: colors.titleBg }];
titleBg.cornerRadius = 12;
page.appendChild(titleBg);
createdNodeIds.push(titleBg.id);

makeText("Interaction Map", 28, "Bold", colors.white, 32, 32, 780, titleBg);
makeText(STORYBOARD.videoName, 14, "Regular", colors.muted, 32, 80, 780, titleBg);
makeText(STORYBOARD.meta, 12, "Regular", colors.muted, 32, 104, 780, titleBg);
makeText(`${STORYBOARD.moments.length} interaction states extracted`, 14, "Semi Bold", colors.accent, 32, 140, 780, titleBg);

const category = await getInteractionCategory();
let nextX = 0;
const frameY = TITLE_HEIGHT;

for (const moment of STORYBOARD.moments) {
  if (nextX > 0) nextX += MOMENT_GAP - GAP;

  const beforeX = nextX;
  const beforeFrame = makeUploadFrame(moment.beforeLabel, beforeX, frameY, frameW, frameH);
  if (moment.beforePath) {
    uploadTargets.push({
      assetKey: moment.beforeAssetKey,
      momentIndex: moment.momentIndex,
      role: "before",
      nodeId: beforeFrame.id,
      path: moment.beforePath,
      scaleMode: "FILL"
    });
  }
  makeText("Before state", 11, "Regular", colors.mutedText, beforeX, frameY + frameH + LABEL_GAP, frameW);

  nextX = beforeX + frameW + GAP;
  const afterX = nextX;
  const afterFrame = makeUploadFrame(moment.afterLabel, afterX, frameY, frameW, frameH);
  if (moment.afterPath) {
    uploadTargets.push({
      assetKey: moment.afterAssetKey,
      momentIndex: moment.momentIndex,
      role: "after",
      nodeId: afterFrame.id,
      path: moment.afterPath,
      scaleMode: "FILL"
    });
  }

  const interaction = moment.interaction;
  const pickFrame = (name, fallback) => {
    if (name === "after") return afterFrame;
    if (name === "before") return beforeFrame;
    return fallback;
  };

  if (interaction && interaction.target) {
    const targetFrame = pickFrame(interaction.target_frame || interaction.targetFrame, beforeFrame);
    const resultFrame = pickFrame(interaction.result_frame || interaction.resultFrame, afterFrame);
    const targetLabel = labelForPoint(interaction.target, "User action");
    const resultLabel = labelForPoint(interaction.result, "Resulting UI change");
    const summary = interaction.annotation || `${targetLabel} -> ${resultLabel}`;

    const targetMarker = addInteractionMarker(targetFrame, interaction.target, "_interaction_target_marker");
    const targetAnnotationNode = targetMarker || targetFrame;
    appendNativeAnnotation(
      targetAnnotationNode,
      `**Interaction target**\n${summary}\n\nTarget: ${targetLabel}`,
      category && category.id
    );

    if (interaction.result) {
      const resultMarker = shouldDrawResultMarker(interaction.result)
        ? addInteractionMarker(resultFrame, interaction.result, "_interaction_result_marker")
        : null;
      const resultAnnotationNode = resultMarker || resultFrame;
      if (resultAnnotationNode.id !== targetAnnotationNode.id) {
        appendNativeAnnotation(
          resultAnnotationNode,
          `**Interaction result**\n${summary}\n\nResult: ${resultLabel}`,
          category && category.id
        );
      }
    }
  }

  const triggerLabel = makeText(
    moment.afterLabel,
    13,
    "Semi Bold",
    colors.accent,
    afterX,
    frameY + frameH + LABEL_GAP,
    frameW
  );

  makeText(
    moment.annotation,
    11,
    "Regular",
    colors.mutedText,
    afterX,
    frameY + frameH + LABEL_GAP + triggerLabel.height + NOTE_GAP,
    frameW
  );

  nextX = afterX + frameW + GAP;
}

page.setSharedPluginData(ns, "runManifestPath", STORYBOARD.runManifestPath || "");
page.setSharedPluginData(ns, "pageName", PAGE_NAME);
page.setSharedPluginData(ns, "frameWidth", String(frameW));
page.setSharedPluginData(ns, "frameHeight", String(frameH));

figma.viewport.scrollAndZoomIntoView(page.children);

return {
  pageId: page.id,
  pageName: PAGE_NAME,
  createdNodeIds,
  mutatedNodeIds,
  uploadTargets,
  nextX
};
'''


APPLY_FILLS_TEMPLATE = r'''// AUTO-GENERATED by generate_figma_calls.py - do not edit manually
// Paste upload results into uploadedImages, then run this with use_figma.
// Expected item shape: { "nodeId": "3:8", "imageHash": "abc123" }
const uploadedImages = __UPLOAD_RESULTS_JSON__;

if (!uploadedImages.length) {
  throw new Error("No uploaded image hashes supplied. Replace uploadedImages with upload_assets results first.");
}

const mutatedNodeIds = [];
for (const item of uploadedImages) {
  const node = await figma.getNodeByIdAsync(item.nodeId);
  if (!node) {
    throw new Error(`Could not find image target node ${item.nodeId}`);
  }
  node.fills = [{
    type: "IMAGE",
    imageHash: item.imageHash,
    scaleMode: item.scaleMode || "FILL"
  }];
  mutatedNodeIds.push(node.id);
}

return {
  mutatedNodeIds,
  appliedImageCount: mutatedNodeIds.length
};
'''


def safe_filename(value: str) -> str:
    cleaned = re.sub(r"[^a-zA-Z0-9_-]", "_", value.lower())
    cleaned = re.sub(r"_+", "_", cleaned).strip("_")
    return cleaned[:30] or "moment"


def frame_size(prepared: dict) -> tuple[int, int]:
    width = prepared.get("frame_width")
    height = prepared.get("frame_height")
    if width and height:
        return int(width), int(height)

    video_info = prepared.get("video_info", {})
    source_width = int(video_info.get("width", 1440) or 1440)
    source_height = int(video_info.get("height", 810) or 810)
    max_width = 900 if source_height > source_width else 1440
    if source_width > max_width:
        ratio = max_width / source_width
        return max_width, int(source_height * ratio)
    return source_width, source_height


def asset_key(moment_index: int, role: str, label: str) -> str:
    return f"moment-{moment_index:03d}-{role}-{safe_filename(label)}"


def point_has_coordinates(point: object) -> bool:
    if not point:
        return False
    if isinstance(point, list) and len(point) >= 2:
        return True
    if isinstance(point, dict):
        return point.get("x") is not None and point.get("y") is not None
    return False


def point_label(point: object, fallback: str) -> str:
    if isinstance(point, dict) and point.get("label"):
        return str(point["label"])
    return fallback


def point_value(point: object, key: str) -> object:
    if isinstance(point, dict):
        return point.get(key)
    return None


def point_marker_override(point: object) -> bool | None:
    if not isinstance(point, dict):
        return None
    for key in ("marker", "draw_marker", "drawMarker", "show_marker", "showMarker"):
        if key in point:
            return bool(point[key])
    return None


def point_is_uncertain(point: object) -> bool:
    if not isinstance(point, dict):
        return False
    if point.get("uncertain") is True or point.get("inferred") is True:
        return True
    confidence = point.get("confidence")
    if isinstance(confidence, str):
        return confidence.lower() in {"medium", "low", "inferred", "uncertain"}
    if isinstance(confidence, (int, float)):
        return confidence < 0.85
    return False


def point_marker_decision(point: object, *, default_draw: bool = True) -> str:
    if not point_has_coordinates(point):
        return "skipped_no_coordinates"
    if isinstance(point, dict) and point.get("visible") is False:
        return "skipped_not_visible"
    override = point_marker_override(point)
    if override is False:
        return "skipped_by_marker_override"
    if override is True:
        return "drawn_by_marker_override"
    if point_is_uncertain(point):
        return "skipped_uncertain"
    return "drawn" if default_draw else "skipped_default_result_annotation"


def build_storyboard(prepared: dict, page_name: str, run_manifest_path: str) -> tuple[dict, list[dict]]:
    frame_w, frame_h = frame_size(prepared)
    video_path = prepared.get("video_path", "unknown")
    video_name = os.path.basename(video_path)
    total_moments = prepared.get("total_moments", len(prepared.get("moments", [])))
    generated_at = datetime.datetime.now().isoformat(timespec="seconds")
    meta = f"{total_moments} key moments - {generated_at[:10]}"

    moments_out = []
    expected_uploads = []

    for index, moment in enumerate(prepared.get("moments", []), start=1):
        moment_index = int(moment.get("moment_index", index))
        short_label = moment.get("short_label", f"Moment {moment_index}")
        trigger_marker = moment.get("trigger_emoji", "")
        before_path = moment.get("before_upload_path") or moment.get("before_frame_path")
        after_path = moment.get("after_upload_path") or moment.get("after_frame_path")
        before_key = asset_key(moment_index, "before", short_label)
        after_key = asset_key(moment_index, "after", short_label)
        after_label = f"{moment_index:02d} {trigger_marker} {short_label}".replace("  ", " ").strip()

        moments_out.append(
            {
                "momentIndex": moment_index,
                "timestampS": moment.get("timestamp_s"),
                "trigger": moment.get("trigger"),
                "shortLabel": short_label,
                "annotation": moment.get("annotation", ""),
                "beforeLabel": f"Before - {short_label}",
                "afterLabel": after_label,
                "beforePath": before_path,
                "afterPath": after_path,
                "beforeAssetKey": before_key,
                "afterAssetKey": after_key,
                "interaction": moment.get("interaction"),
            }
        )

        if before_path:
            expected_uploads.append(
                {
                    "assetKey": before_key,
                    "momentIndex": moment_index,
                    "role": "before",
                    "path": before_path,
                    "scaleMode": "FILL",
                    "nodeId": None,
                    "imageHash": None,
                    "status": "pending",
                }
            )
        if after_path:
            expected_uploads.append(
                {
                    "assetKey": after_key,
                    "momentIndex": moment_index,
                    "role": "after",
                    "path": after_path,
                    "scaleMode": "FILL",
                    "nodeId": None,
                    "imageHash": None,
                    "status": "pending",
                }
            )

    storyboard = {
        "videoName": video_name,
        "videoPath": video_path,
        "generatedAt": generated_at,
        "meta": meta,
        "frameWidth": frame_w,
        "frameHeight": frame_h,
        "runManifestPath": run_manifest_path,
        "moments": moments_out,
    }

    return storyboard, expected_uploads


def build_run_manifest(
    prepared: dict,
    storyboard: dict,
    expected_uploads: list[dict],
    page_name: str,
    storyboard_script: str,
    fill_script_template: str,
) -> dict:
    selected_moments = []
    for moment in prepared.get("moments", []):
        interaction = moment.get("interaction") or {}
        target = interaction.get("target") if isinstance(interaction, dict) else None
        result = interaction.get("result") if isinstance(interaction, dict) else None
        selected_moments.append(
            {
                "moment_index": moment.get("moment_index"),
                "timestamp_s": moment.get("timestamp_s"),
                "short_label": moment.get("short_label"),
                "before_upload_path": moment.get("before_upload_path"),
                "after_upload_path": moment.get("after_upload_path"),
                "interaction": {
                    "target_frame": interaction.get("target_frame") or interaction.get("targetFrame"),
                    "target_label": point_label(target, "User action") if target else None,
                    "target_has_coordinates": point_has_coordinates(target),
                    "target_confidence": point_value(target, "confidence"),
                    "target_inferred": bool(point_value(target, "inferred")),
                    "target_uncertain": bool(point_value(target, "uncertain")),
                    "target_visible": point_value(target, "visible"),
                    "target_marker_decision": point_marker_decision(target),
                    "result_frame": interaction.get("result_frame") or interaction.get("resultFrame"),
                    "result_label": point_label(result, "Resulting UI change") if result else None,
                    "result_has_coordinates": point_has_coordinates(result),
                    "result_confidence": point_value(result, "confidence"),
                    "result_inferred": bool(point_value(result, "inferred")),
                    "result_uncertain": bool(point_value(result, "uncertain")),
                    "result_visible": point_value(result, "visible"),
                    "result_marker_decision": point_marker_decision(re… [4564 more characters not shown]
scripts/prepare_upload_frames.pyShow
#!/usr/bin/env python3
"""
Prepare key frame images for Figma upload_assets.

Takes a frames manifest and a list of analyzed key moments, then writes resized
JPEG assets for each before/after state plus an upload_manifest.json consumed by
generate_figma_calls.py.

Usage:
    python prepare_upload_frames.py \
      --manifest /tmp/vim_frames/frames_manifest.json \
      --moments-file /tmp/vim_frames/key_moments.json \
      --output /tmp/vim_frames/upload_manifest.json
"""

import argparse
import json
import os
from pathlib import Path


def prepare_image(
    input_path: str,
    output_path: Path,
    max_width: int,
    quality: int,
    max_bytes: int,
) -> dict:
    """Resize and compress an image for upload_assets, returning final metadata."""
    try:
        from PIL import Image
    except ImportError as exc:
        raise RuntimeError("Pillow is required: pip install Pillow --break-system-packages") from exc

    with Image.open(input_path) as img:
        if img.mode != "RGB":
            img = img.convert("RGB")

        if max_width > 0 and img.width > max_width:
            ratio = max_width / img.width
            new_height = int(img.height * ratio)
            img = img.resize((max_width, new_height), Image.LANCZOS)

        output_path.parent.mkdir(parents=True, exist_ok=True)

        current_quality = min(max(quality, 1), 95)
        while True:
            img.save(output_path, format="JPEG", quality=current_quality, optimize=True)
            size_bytes = output_path.stat().st_size
            if size_bytes <= max_bytes:
                break

            if current_quality > 45:
                current_quality = max(45, current_quality - 10)
                continue

            next_width = max(480, int(img.width * 0.85))
            if next_width >= img.width:
                break
            ratio = next_width / img.width
            img = img.resize((next_width, int(img.height * ratio)), Image.LANCZOS)

        return {
            "width": img.width,
            "height": img.height,
            "quality": current_quality,
            "size_bytes": output_path.stat().st_size,
            "within_budget": output_path.stat().st_size <= max_bytes,
        }


def safe_stem(label: str) -> str:
    cleaned = "".join(ch.lower() if ch.isalnum() else "_" for ch in label)
    cleaned = "_".join(part for part in cleaned.split("_") if part)
    return cleaned[:48] or "moment"


def main() -> None:
    parser = argparse.ArgumentParser(description="Prepare frame images for Figma upload_assets")
    parser.add_argument("--manifest", required=True, help="Path to frames_manifest.json")
    parser.add_argument("--moments-file", required=True, help="Path to key_moments.json")
    parser.add_argument("--output", required=True, help="Output path for upload_manifest.json")
    parser.add_argument(
        "--assets-dir",
        help="Directory for optimized JPEGs. Defaults to <output-dir>/upload_assets",
    )
    parser.add_argument(
        "--max-width",
        type=int,
        default=0,
        help="Max frame width. Auto-detected from video orientation if not set.",
    )
    parser.add_argument(
        "--quality",
        type=int,
        default=75,
        help="JPEG quality 1-95 (default: 75)",
    )
    parser.add_argument(
        "--max-file-mb",
        type=float,
        default=9.5,
        help="Maximum uploaded image size in MB (default: 9.5)",
    )
    args = parser.parse_args()

    with open(args.manifest) as f:
        manifest = json.load(f)

    with open(args.moments_file) as f:
        moments = json.load(f)

    output_path = Path(args.output)
    assets_dir = Path(args.assets_dir) if args.assets_dir else output_path.parent / "upload_assets"

    video_info = manifest.get("video_info", {})
    orientation = video_info.get("orientation", "landscape")
    if args.max_width > 0:
        max_width = args.max_width
    elif orientation == "portrait":
        max_width = 900
    else:
        max_width = 1440

    prepared_moments = []
    frame_width = None
    frame_height = None
    max_bytes = int(args.max_file_mb * 1024 * 1024)

    print(
        f"Preparing {len(moments)} moments for upload_assets "
        f"({max_width}px wide, JPEG q{args.quality})...",
        flush=True,
    )

    for index, moment in enumerate(moments, start=1):
        prepared = dict(moment)
        moment_index = prepared.get("moment_index", index)
        label = safe_stem(prepared.get("short_label", f"moment_{moment_index}"))

        for role in ("before", "after"):
            source_key = f"{role}_frame_path"
            upload_key = f"{role}_upload_path"
            source_path = prepared.get(source_key)

            if source_path and os.path.exists(source_path):
                output_file = assets_dir / f"moment_{moment_index:03d}_{role}_{label}.jpg"
                info = prepare_image(source_path, output_file, max_width, args.quality, max_bytes)
                prepared[upload_key] = str(output_file)
                prepared[f"{role}_upload_width"] = info["width"]
                prepared[f"{role}_upload_height"] = info["height"]
                prepared[f"{role}_upload_quality"] = info["quality"]
                prepared[f"{role}_upload_size_bytes"] = info["size_bytes"]
                prepared[f"{role}_upload_within_budget"] = info["within_budget"]
                frame_width = frame_width or info["width"]
                frame_height = frame_height or info["height"]
                status = "ok" if info["within_budget"] else "over budget"
                print(
                    f"  [{index}/{len(moments)}] {role}: {output_file.name} "
                    f"({info['width']}x{info['height']}, q{info['quality']}, "
                    f"{info['size_bytes'] / 1024:.0f} KB, {status})",
                    flush=True,
                )
            else:
                prepared[upload_key] = None
                print(f"  [{index}/{len(moments)}] {role}: missing ({source_path})", flush=True)

        prepared_moments.append(prepared)

    if frame_width is None or frame_height is None:
        width = int(video_info.get("width", max_width) or max_width)
        height = int(video_info.get("height", max_width) or max_width)
        if width > max_width:
            ratio = max_width / width
            width = max_width
            height = int(height * ratio)
        frame_width = width
        frame_height = height

    output = {
        "video_path": manifest.get("video_path"),
        "video_info": video_info,
        "frame_width": frame_width,
        "frame_height": frame_height,
        "image_budget": {
            "max_width": max_width,
            "requested_quality": args.quality,
            "max_file_mb": args.max_file_mb,
            "max_file_bytes": max_bytes,
        },
        "total_moments": len(prepared_moments),
        "assets_dir": str(assets_dir),
        "moments": prepared_moments,
    }

    output_path.parent.mkdir(parents=True, exist_ok=True)
    with open(output_path, "w") as f:
        json.dump(output, f, indent=2)

    print(f"\nPrepared {len(prepared_moments)} moments")
    print(f"Assets: {assets_dir}")
    print(f"Output: {output_path}")


if __name__ == "__main__":
    main()
scripts/resolve_moment_frames.pyShow
#!/usr/bin/env python3
"""
Resolve analyzed key moments into concrete before/after screenshot files.

This lets the workflow start with a low-resolution scout/contact sheet and only
extract the high-quality frames that will actually be uploaded to Figma.

Usage:
    python resolve_moment_frames.py \
      --video input.mp4 \
      --moments-file key_moments.json \
      --output key_moments_resolved.json \
      --frames-dir selected_frames/
"""

import argparse
import json
import os
import subprocess
import sys
from pathlib import Path


def safe_stem(label: str) -> str:
    cleaned = "".join(ch.lower() if ch.isalnum() else "_" for ch in label)
    cleaned = "_".join(part for part in cleaned.split("_") if part)
    return cleaned[:48] or "moment"


def numeric_value(value: object) -> float | None:
    try:
        number = float(value)
    except (TypeError, ValueError):
        return None
    return number if number >= 0 else 0.0


def timestamp_for(moment: dict, role: str, before_offset: float, after_offset: float) -> float | None:
    for key in (
        f"{role}_timestamp_s",
        f"{role}_time_s",
        f"{role}_timestamp",
    ):
        timestamp = numeric_value(moment.get(key))
        if timestamp is not None:
            return timestamp

    base = numeric_value(moment.get("timestamp_s"))
    if base is None:
        return None
    if role == "before":
        return max(0.0, base - before_offset)
    return max(0.0, base + after_offset)


def optimize_jpeg(path: Path, max_width: int, quality: int) -> tuple[int | None, int | None]:
    try:
        from PIL import Image
    except ImportError:
        if max_width > 0:
            print("Pillow not available, selected frames will not be resized", flush=True)
        return None, None

    with Image.open(path) as img:
        if img.mode != "RGB":
            img = img.convert("RGB")
        if max_width > 0 and img.width > max_width:
            ratio = max_width / img.width
            img = img.resize((max_width, int(img.height * ratio)), Image.LANCZOS)
        img.save(path, "JPEG", quality=quality, optimize=True)
        return img.width, img.height


def extract_frame(video_path: str, timestamp_s: float, output_path: Path, max_width: int, quality: int) -> dict:
    output_path.parent.mkdir(parents=True, exist_ok=True)
    cmd = [
        "ffmpeg",
        "-ss",
        f"{timestamp_s:.3f}",
        "-i",
        video_path,
        "-frames:v",
        "1",
        "-q:v",
        "2",
        "-y",
        "-v",
        "quiet",
        str(output_path),
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0 or not output_path.exists():
        raise RuntimeError(f"ffmpeg failed at {timestamp_s:.3f}s: {result.stderr}")

    width, height = optimize_jpeg(output_path, max_width, quality)
    return {
        "path": str(output_path),
        "timestamp_s": round(timestamp_s, 3),
        "width": width,
        "height": height,
        "size_bytes": output_path.stat().st_size,
    }


def main() -> None:
    parser = argparse.ArgumentParser(description="Extract selected before/after frames for key moments")
    parser.add_argument("--video", required=True, help="Source video path")
    parser.add_argument("--moments-file", required=True, help="Input key_moments.json")
    parser.add_argument("--output", required=True, help="Output resolved key_moments JSON")
    parser.add_argument("--frames-dir", required=True, help="Directory for selected frame JPEGs")
    parser.add_argument("--max-width", type=int, default=1600, help="Max selected frame width (default: 1600)")
    parser.add_argument("--quality", type=int, default=86, help="JPEG quality for selected frames (default: 86)")
    parser.add_argument("--before-offset", type=float, default=0.2,
                        help="Fallback seconds before timestamp_s for before frames")
    parser.add_argument("--after-offset", type=float, default=0.0,
                        help="Fallback seconds after timestamp_s for after frames")
    parser.add_argument("--force", action="store_true", help="Re-extract even when frame paths already exist")
    args = parser.parse_args()

    if not os.path.exists(args.video):
        print(f"Error: video file not found: {args.video}", file=sys.stderr)
        sys.exit(1)

    with open(args.moments_file) as f:
        moments = json.load(f)

    if not isinstance(moments, list):
        print("Error: moments file must contain a JSON array", file=sys.stderr)
        sys.exit(1)

    frames_dir = Path(args.frames_dir)
    resolved = []
    warnings = []

    for index, moment in enumerate(moments, start=1):
        next_moment = dict(moment)
        moment_index = int(next_moment.get("moment_index", index))
        label = safe_stem(next_moment.get("short_label", f"moment_{moment_index}"))

        for role in ("before", "after"):
            path_key = f"{role}_frame_path"
            existing_path = next_moment.get(path_key)
            if existing_path and os.path.exists(existing_path) and not args.force:
                continue

            timestamp_s = timestamp_for(next_moment, role, args.before_offset, args.after_offset)
            if timestamp_s is None:
                warnings.append(f"moment {moment_index} {role}: no frame path or timestamp")
                continue

            output_path = frames_dir / f"moment_{moment_index:03d}_{role}_{label}.jpg"
            info = extract_frame(args.video, timestamp_s, output_path, args.max_width, args.quality)
            next_moment[path_key] = info["path"]
            next_moment[f"{role}_timestamp_s"] = info["timestamp_s"]
            next_moment[f"{role}_frame_size_bytes"] = info["size_bytes"]
            if info["width"] and info["height"]:
                next_moment[f"{role}_frame_width"] = info["width"]
                next_moment[f"{role}_frame_height"] = info["height"]
            print(f"  moment {moment_index:02d} {role}: {output_path.name}", flush=True)

        resolved.append(next_moment)

    output_path = Path(args.output)
    output_path.parent.mkdir(parents=True, exist_ok=True)
    output_path.write_text(json.dumps(resolved, indent=2))

    print(f"\nResolved {len(resolved)} moments")
    print(f"Output: {output_path}")
    if warnings:
        print("\nWarnings:")
        for warning in warnings:
            print(f"  - {warning}")


if __name__ == "__main__":
    main()
scripts/update_run_manifest.pyShow
#!/usr/bin/env python3
"""
Update a video-interaction-mapper run_manifest.json as Figma work completes.

This helper keeps the workflow resumable:
- record the storyboard use_figma result, including node IDs and upload targets
- record upload_assets image hashes
- generate a ready-to-run fill script from the fill template
- record fill and verification status

Usage:
    python update_run_manifest.py \
      --run-manifest figma_calls/run_manifest.json \
      --storyboard-result storyboard_result.json

    python update_run_manifest.py \
      --run-manifest figma_calls/run_manifest.json \
      --uploads-file uploaded_images.json \
      --write-fill-script figma_calls/figma_apply_fills.js
"""

import argparse
import json
import re
from pathlib import Path


def load_json(path: str) -> object:
    data = json.loads(Path(path).read_text())
    if isinstance(data, list) and len(data) == 1 and isinstance(data[0], dict) and "text" in data[0]:
        return json.loads(data[0]["text"])
    if isinstance(data, dict) and "text" in data and isinstance(data["text"], str):
        return json.loads(data["text"])
    return data


def as_list(value: object, keys: tuple[str, ...]) -> list:
    if isinstance(value, list):
        return value
    if isinstance(value, dict):
        for key in keys:
            candidate = value.get(key)
            if isinstance(candidate, list):
                return candidate
    return []


def update_upload_targets(run_manifest: dict, upload_targets: list[dict]) -> None:
    run_manifest["figma"]["upload_targets"] = upload_targets
    expected_by_key = {
        item.get("assetKey"): item
        for item in run_manifest.get("expected_uploads", [])
        if item.get("assetKey")
    }
    for target in upload_targets:
        match = expected_by_key.get(target.get("assetKey"))
        if match:
            match["nodeId"] = target.get("nodeId")
            match["status"] = "ready_to_upload"
    if upload_targets:
        run_manifest["stages"]["storyboard"] = "complete"


def normalize_uploaded_images(value: object) -> list[dict]:
    uploads = as_list(value, ("uploadedImages", "uploads", "results", "items"))
    normalized = []
    for item in uploads:
        if not isinstance(item, dict):
            continue
        node_id = item.get("nodeId") or item.get("node_id")
        image_hash = item.get("imageHash") or item.get("image_hash") or item.get("hash")
        if node_id and image_hash:
            normalized.append(
                {
                    "assetKey": item.get("assetKey"),
                    "momentIndex": item.get("momentIndex"),
                    "role": item.get("role"),
                    "nodeId": node_id,
                    "imageHash": image_hash,
                    "scaleMode": item.get("scaleMode", "FILL"),
                }
            )
    return normalized


def update_uploaded_images(run_manifest: dict, uploaded_images: list[dict]) -> None:
    run_manifest["figma"]["uploaded_images"] = uploaded_images
    expected = run_manifest.get("expected_uploads", [])
    expected_by_node = {item.get("nodeId"): item for item in expected if item.get("nodeId")}
    expected_by_key = {item.get("assetKey"): item for item in expected if item.get("assetKey")}
    for uploaded in uploaded_images:
        match = expected_by_node.get(uploaded.get("nodeId"))
        if not match and uploaded.get("assetKey"):
            match = expected_by_key.get(uploaded.get("assetKey"))
        if match:
            match["imageHash"] = uploaded.get("imageHash")
            match["status"] = "uploaded"
    if expected and all(item.get("status") == "uploaded" for item in expected):
        run_manifest["stages"]["uploads"] = "complete"


def write_fill_script(run_manifest: dict, output_path: str) -> None:
    template_path = run_manifest["figma"].get("fill_script_template")
    if not template_path:
        raise RuntimeError("run manifest has no fill_script_template path")
    uploaded_images = run_manifest["figma"].get("uploaded_images", [])
    if not uploaded_images:
        raise RuntimeError("run manifest has no uploaded_images to apply")

    template = Path(template_path).read_text()
    replacement = "const uploadedImages = " + json.dumps(uploaded_images, indent=2) + ";"
    script = re.sub(r"const uploadedImages = \[\];", replacement, template, count=1)
    Path(output_path).write_text(script)
    run_manifest["figma"]["fill_script"] = output_path


def main() -> None:
    parser = argparse.ArgumentParser(description="Update a video interaction mapper run manifest")
    parser.add_argument("--run-manifest", required=True, help="Path to run_manifest.json")
    parser.add_argument("--storyboard-result", help="JSON file containing the storyboard use_figma result")
    parser.add_argument("--uploads-file", help="JSON file containing uploaded nodeId/imageHash pairs")
    parser.add_argument("--fill-result", help="JSON file containing the fill-pass use_figma result")
    parser.add_argument("--write-fill-script", help="Write a ready-to-run fill script to this path")
    parser.add_argument("--verification-screenshot", help="Path to a downloaded verification screenshot")
    parser.add_argument("--verification-notes", default="", help="Verification notes")
    parser.add_argument("--output", help="Optional output manifest path. Defaults to in-place update.")
    args = parser.parse_args()

    run_manifest_path = Path(args.run_manifest)
    run_manifest = json.loads(run_manifest_path.read_text())

    if args.storyboard_result:
        result = load_json(args.storyboard_result)
        if isinstance(result, dict):
            run_manifest["figma"]["page_id"] = result.get("pageId") or run_manifest["figma"].get("page_id")
            run_manifest["figma"]["created_node_ids"] = result.get("createdNodeIds", [])
            update_upload_targets(run_manifest, result.get("uploadTargets", []))

    if args.uploads_file:
        uploads = normalize_uploaded_images(load_json(args.uploads_file))
        update_uploaded_images(run_manifest, uploads)

    if args.write_fill_script:
        write_fill_script(run_manifest, args.write_fill_script)

    if args.fill_result:
        fill_result = load_json(args.fill_result)
        if isinstance(fill_result, dict):
            run_manifest["figma"]["fill_pass"] = {
                "status": "complete",
                "mutated_node_ids": fill_result.get("mutatedNodeIds", []),
                "applied_image_count": fill_result.get("appliedImageCount"),
            }
            run_manifest["stages"]["fills"] = "complete"

    if args.verification_screenshot:
        run_manifest["figma"]["verification"] = {
            "status": "complete",
            "screenshot_path": args.verification_screenshot,
            "notes": args.verification_notes,
        }
        run_manifest["stages"]["verification"] = "complete"

    output_path = Path(args.output) if args.output else run_manifest_path
    output_path.write_text(json.dumps(run_manifest, indent=2))
    print(f"Updated run manifest: {output_path}")


if __name__ == "__main__":
    main()