video-interaction-mapper
by Figma · official repository
This skill should be used when the user asks to analyze a UI screen recording and map interaction states into Figma. Trigger for requests such as "put video frames in Figma", "extract states from my recording", "map interactions from video to Figma", "analyze this screen recording", "create a storyboard from my video", "deconstruct this interaction in Figma", "annotate the UI states in my recording", or "pull the key moments from this video into Figma". Also trigger when the user references a video file (.mp4, .mov, .webm, .avi) together with Figma, design review, interaction analysis, prototypes, or UI states. The skill extracts key visual moments from a video, infers interaction triggers, and builds an annotated Figma Design storyboard using native Figma annotations and uploaded screensh… [10 more characters not shown]
Labels from its files: runs commands. 7 files, read Oct 21, 2026. Nothing was run.
npx skills add https://github.com/figma/mcp-server-guide --skill video-interaction-mapperWhat it lets an agent do
Each label, and the line in the skill's files that set it.
- Runs commands
Allows a shell or ships scripts the agent runs.
- It ships 5 scripts the agent can run.
Findings
Matches of fixed checks, with the text they matched.
No check matched anything in this skill's files.
Not a safety rating, and no finding is not a safety guarantee. These are fixed checks of the files as read on Oct 21, 2026. Nothing was run. How skills are checked.
SKILL.md
As the publisher wrote it. Shown as text, never rendered.
---
name: video-interaction-mapper
description: >
This skill should be used when the user asks to analyze a UI screen recording and
map interaction states into Figma. Trigger for requests such as "put video frames
in Figma", "extract states from my recording", "map interactions from video to
Figma", "analyze this screen recording", "create a storyboard from my video",
"deconstruct this interaction in Figma", "annotate the UI states in my recording",
or "pull the key moments from this video into Figma". Also trigger when the user
references a video file (.mp4, .mov, .webm, .avi) together with Figma, design
review, interaction analysis, prototypes, or UI states. The skill extracts key
visual moments from a video, infers interaction triggers, and builds an annotated
Figma Design storyboard using native Figma annotations and uploaded screenshot
assets.
---
# Video Interaction Mapper
Turn a UI recording into a static, annotated Figma storyboard. Extract the important
before/after states, infer what triggered each change, then place clean screenshots,
blue interaction markers, and concise annotations into a Figma Design file.
## Required Inputs and Tools
Start with:
- A local video file path (`.mp4`, `.mov`, `.webm`, or `.avi`).
- `ffmpeg` and `ffprobe` available on the machine.
- Python with Pillow installed.
- Figma MCP tools available: `create_new_file`, `use_figma`, `upload_assets`, and
`get_screenshot` or `get_design_context`.
Run the bundled files in `scripts/` as executable workflow helpers. They are part
of the skill's implementation, not reference material. Read or modify them only
when debugging, adapting to an unusual environment, or changing the skill itself.
Use a Figma Design file (`figma.com/design/...`) as the target. The generated
Plugin API code creates pages and image-fill frames, which are Design-file
operations. If the user provides a FigJam or Slides URL, ask for a Design file or
create a new one.
Before any `use_figma` call, load the Figma API guidance skill if available
(`figma-use`). Before creating a new Figma file, load the file-creation guidance
skill if available (`figma-create-new-file`). Pass `skillNames:
"figma-use,video-interaction-mapper"` on `use_figma` calls when the client supports
that parameter.
## Workflow
### 1. Validate and Prepare
Confirm the video path exists, then check `ffmpeg -version`. If Pillow is missing,
install it in the active Python environment:
```bash
pip install Pillow --break-system-packages -q
```
Do local analysis before creating or modifying the Figma file. Create a Figma file
only after `key_moments.json`, `upload_manifest.json`, and generated Figma scripts
are ready, unless the user explicitly asks to create the file first. This avoids
leaving partial pages behind when local frame analysis changes.
### 2. Scout the Timeline Quickly
Start with a low-resolution scout pass and contact sheet:
```bash
python <SKILL_DIR>/scripts/extract_key_frames.py \
--input "<video_path>" \
--output /tmp/vim_frames_<timestamp>/ \
--mode scout
```
The scout pass extracts at 1 fps by default, resizes frames to 640 px wide, scores
scene changes, writes `<output_dir>/frames_manifest.json`, and creates
`<output_dir>/contact_sheet.jpg`. Inspect the contact sheet first to identify likely
interaction moments.
Use the slower full extraction only when the scout pass is insufficient:
```bash
python <SKILL_DIR>/scripts/extract_key_frames.py \
--input "<video_path>" \
--output /tmp/vim_frames_<timestamp>/full_frames/ \
--mode full \
--max-width 1600
```
For very long videos, ask whether to focus on a specific time range before
processing the whole recording. For too many key frames, re-run extraction with a
higher `--scene-threshold`, such as `0.4`.
### 3. Analyze Key Moments
Read `frames_manifest.json` and inspect `contact_sheet.jpg` or relevant extracted
frames. For each meaningful transition, identify:
- Visual change: modal, drawer, dropdown, tooltip, snackbar, overlay, navigation,
loaded content, selected state, validation state, input focus, or scroll position.
- Trigger: click, tap, keyboard input, hover, scroll, swipe, form submit, auto/timer,
or unknown trigger when the cause is ambiguous.
- Interaction target and result coordinates when visible and useful.
Use normalized coordinates from `0` to `1` relative to the screenshot. Percent-like
values such as `45` or `94` are accepted by the generated Figma script and converted
to `0.45` or `0.94`. Coordinates are machine metadata for marker placement only.
Do not include coordinate values in visible annotation text, moment labels, or user
summaries.
Only add marker coordinates when the point lands on a clearly visible UI element or
state. If a target is inferred, between visible controls, or based mainly on cursor
position, mark it with `inferred: true`, `uncertain: true`, `visible: false`, or
`confidence: "medium"`/`"low"`. The generated script suppresses markers for those
points by default so the canvas does not show a confident pulse over empty UI. If
the target or result cannot be located usefully, set `interaction` to `null`.
Use `confidence: "high"` or omit `confidence` only when the marker should be drawn.
Use `show_marker: true` only when an uncertain point is still worth marking; the
generated marker uses an alternate style for these forced uncertain markers.
Aim for 5 to 15 key moments. Prioritize high scene scores and genuinely different
states over tiny visual shifts.
Write the analysis to `<output_dir>/key_moments.json` as an array. When using the
scout flow, provide timestamps first; resolve high-quality frame files in the next
step.
```json
[
{
"moment_index": 1,
"timestamp_s": 3.4,
"before_timestamp_s": 3.2,
"after_timestamp_s": 3.4,
"trigger": "Click",
"trigger_emoji": "",
"short_label": "Dropdown menu opens",
"annotation": "The account avatar is activated. A menu appears below it with profile, billing, and sign-out actions.",
"interaction": {
"target_frame": "before",
"target": {
"x": 0.91,
"y": 0.08,
"label": "Account avatar",
"confidence": "high"
},
"result_frame": "after",
"result": {
"x": 0.86,
"y": 0.2,
"label": "Opened account menu"
},
"annotation": "Activating the avatar opens the account menu."
}
}
]
```
Keep annotations human-readable: `Target: Account avatar`, not a label with a raw
coordinate suffix. Coordinates stay hidden in JSON and appear only as visual marker
placement.
If a marker is drawn, the native Figma annotation is attached to that marker node
rather than the screenshot frame. If a marker is suppressed because the point is
uncertain or not visible, the annotation remains attached to the screenshot frame.
### 4. Resolve Selected Frames
If `key_moments.json` uses timestamps, extract only the selected before/after
frames at higher quality:
```bash
python <SKILL_DIR>/scripts/resolve_moment_frames.py \
--video "<video_path>" \
--moments-file /tmp/vim_frames_<timestamp>/key_moments.json \
--output /tmp/vim_frames_<timestamp>/key_moments_resolved.json \
--frames-dir /tmp/vim_frames_<timestamp>/selected_frames/ \
--max-width 1600
```
Use `key_moments_resolved.json` in later steps. If a full extraction already
produced precise `before_frame_path` and `after_frame_path` values, this step can
be skipped.
### 5. Prepare Uploadable Screenshot Assets
Compress and resize the before/after screenshots for upload:
```bash
python <SKILL_DIR>/scripts/prepare_upload_frames.py \
--manifest /tmp/vim_frames_<timestamp>/frames_manifest.json \
--moments-file /tmp/vim_frames_<timestamp>/key_moments_resolved.json \
--output /tmp/vim_frames_<timestamp>/upload_manifest.json
```
The script writes optimized JPEGs under `<output_dir>/upload_assets/` and creates
`upload_manifest.json`. Defaults are 1440 px wide for landscape recordings and 900
px wide for portrait recordings at JPEG quality 75. It records image dimensions,
actual JPEG quality, file size, and whether each asset fits the upload budget.
### 6. Generate Figma Call Files
Generate the self-contained Plugin API scripts:
```bash
python <SKILL_DIR>/scripts/generate_figma_calls.py \
--prepared /tmp/vim_frames_<timestamp>/upload_manifest.json \
--output-dir /tmp/vim_frames_<timestamp>/figma_calls/
```
This writes:
- `figma_storyboard.js`: one combined storyboard creation call.
- `figma_apply_fills_template.js`: a fill-pass template for uploaded image hashes.
- `figma_manifest.json`: execution order and upload instructions.
- `run_manifest.json`: resumable status, selected moments, expected uploads, node
IDs, image hashes, fill status, and verification status.
The generated JavaScript does not embed image bytes and does not call
`figma.createImage`. The storyboard script creates all screenshot containers,
labels, native annotations, and marker overlays in one call, then returns
`uploadTargets` containing Figma node IDs and local JPEG paths. In a newly created
blank file, the script reuses and renames the default page instead of creating a
second page.
### 7. Execute in Figma and Upload Assets
Create a new Figma Design file or open the target Design file now. Run the generated
files in `figma_manifest.json`:
1. Read `figma_storyboard.js`.
2. Call `use_figma` against the target Design file with the full JavaScript content.
3. Save the result JSON if practical, then update the run manifest:
```bash
python <SKILL_DIR>/scripts/update_run_manifest.py \
--run-manifest /tmp/vim_frames_<timestamp>/figma_calls/run_manifest.json \
--storyboard-result /tmp/vim_frames_<timestamp>/storyboard_result.json
```
For each returned `uploadTargets` item:
- Call `upload_assets` with the target `fileKey`, `count: 1`, `nodeId`, and
`scaleMode: "FILL"`.
- POST the local file at `target.path` to the returned upload URL.
- Record `{ "nodeId": "...", "imageHash": "...", "scaleMode": "FILL" }` in an
uploads JSON file.
Generate and run the fill pass:
```bash
python <SKILL_DIR>/scripts/update_run_manifest.py \
--run-manifest /tmp/vim_frames_<timestamp>/figma_calls/run_manifest.json \
--uploads-file /tmp/vim_frames_<timestamp>/uploaded_images.json \
--write-fill-script /tmp/vim_frames_<timestamp>/figma_calls/figma_apply_fills.js
```
Run `figma_apply_fills.js` once with `use_figma`. This explicit image-hash fill pass
is preferred because it makes uploaded screenshots render reliably in the generated
containers.
### 8. Verify
Run `get_screenshot`, `download_assets`, or `get_design_context` to verify that:
- Screenshots are visible and nonblank.
- Blue marker rings sit on the intended target/result locations.
- Native annotations use human labels and do not expose coordinate values.
- Ambiguous or inferred targets are suppressed unless explicitly forced with
`show_marker: true`.
Update `run_manifest.json` with verification artifacts when available:
```bash
python <SKILL_DIR>/scripts/update_run_manifest.py \
--run-manifest /tmp/vim_frames_<timestamp>/figma_calls/run_manifest.json \
--verification-screenshot /tmp/vim_frames_<timestamp>/figma_storyboard.png \
--verification-notes "Screenshots and markers verified"
```
## Output Expectations
The storyboard layout is left-to-right:
```text
[Title block]
[Before] [After] [Before] [After] [Before] [After]
moment 1 moment 2 moment 3
"Before state" "01 Dropdown menu opens"
Annotation text...
```
Blue ring/dot overlays show the target and result locations when coordinates are
available and confident. Native Figma annotations attach to the marker nodes when
markers exist; otherwise they attach to the relevant before/after screenshot nodes.
Annotation labels should name the UI element or state change, never raw coordinate
percentages.
Report back with:
- The number of key moments mapped.
- The Figma file URL.
- Any moments whose trigger was ambiguous.
- Any uploads or frames that failed.
- The verification status and any inferred/uncertain markers.
## Edge Cases
- Long video: ask for a time range or warn that processing can take longer.
- Too many moments: raise `--scene-threshold` or manually keep the most meaningful
5 to 15 transitions.
- Mobile recording: keep the portrait default width of 900 px so text remains
readable without oversized assets.
- Missing trigger evidence: use `Unknown trigger` instead of guessing.
- Asset upload failure: retry the individual `upload_assets` target; the generated
screenshot container remains in the file and can be reused.
- `upload_assets` succeeds but images do not render: run the generated fill pass
with the returned image hashes.
- `upload_assets` is unavailable: warn that the fallback path is slower because it
must embed image data directly in `use_figma` calls.
Scripts (5)
Read as text. anlyon.dev never runs them.
scripts/extract_key_frames.pyShowHide
#!/usr/bin/env python3
"""
Extract key frames from a video file using ffmpeg.
This script:
1. Extracts frames at a given fps
2. Runs ffmpeg scene-change detection to score each frame
3. Flags frames with scene scores above the threshold as key frames
4. Optionally writes a compact contact sheet for quick visual review
5. Outputs JPEG frames + a frames_manifest.json
Usage:
python extract_key_frames.py --input video.mp4 --output /tmp/frames/ --mode scout
"""
import argparse
import json
import os
import re
import subprocess
import sys
from pathlib import Path
def get_video_info(video_path: str) -> dict:
"""Get basic video metadata using ffprobe."""
cmd = [
"ffprobe", "-v", "quiet",
"-print_format", "json",
"-show_streams", "-show_format",
video_path
]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
raise RuntimeError(f"ffprobe failed: {result.stderr}")
data = json.loads(result.stdout)
# Find video stream
video_stream = next(
(s for s in data.get("streams", []) if s.get("codec_type") == "video"),
None
)
if not video_stream:
raise RuntimeError("No video stream found in file")
duration = float(data.get("format", {}).get("duration", 0))
width = int(video_stream.get("width", 0))
height = int(video_stream.get("height", 0))
# Parse framerate
fps_str = video_stream.get("r_frame_rate", "30/1")
fps_parts = fps_str.split("/")
fps = float(fps_parts[0]) / float(fps_parts[1]) if len(fps_parts) == 2 else float(fps_str)
return {
"duration_s": duration,
"width": width,
"height": height,
"native_fps": fps,
"orientation": "portrait" if height > width else "landscape"
}
def extract_all_frames(video_path: str, output_dir: str, fps: int = 5, max_frames: int = 3000) -> list[str]:
"""Extract all frames at the given fps as JPEG files."""
os.makedirs(output_dir, exist_ok=True)
output_pattern = os.path.join(output_dir, "frame_%06d.jpg")
cmd = [
"ffmpeg", "-i", video_path,
"-vf", f"fps={fps}",
"-q:v", "3", # JPEG quality (1-31, lower = better)
"-vframes", str(max_frames),
output_pattern,
"-y", "-v", "quiet"
]
print(f"Extracting frames at {fps}fps...", flush=True)
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
raise RuntimeError(f"ffmpeg frame extraction failed: {result.stderr}")
# Collect output frames in order
frames = sorted(Path(output_dir).glob("frame_*.jpg"))
print(f"Extracted {len(frames)} frames", flush=True)
return [str(f) for f in frames]
def detect_scene_changes(video_path: str, fps: int = 5) -> dict[float, float]:
"""
Use ffmpeg's scene filter to get scene change scores at each timestamp.
Returns a dict mapping timestamp (seconds) -> scene score (0.0-1.0).
"""
# ffmpeg scene detection — outputs to stderr with showinfo
cmd = [
"ffmpeg", "-i", video_path,
"-vf", f"fps={fps},select='gte(scene,0)',metadata=print:file=-",
"-an", "-f", "null", "-",
"-v", "quiet"
]
print("Running scene change detection...", flush=True)
result = subprocess.run(cmd, capture_output=True, text=True)
# Scene metadata goes to stdout via metadata=print:file=-
output = result.stdout
scene_scores = {}
current_ts = None
for line in output.splitlines():
# Look for pts_time and lavfi.scene_score
ts_match = re.search(r'pts_time:([0-9.]+)', line)
if ts_match:
current_ts = float(ts_match.group(1))
score_match = re.search(r'lavfi\.scene_score=([0-9.]+)', line)
if score_match and current_ts is not None:
scene_scores[current_ts] = float(score_match.group(1))
current_ts = None
print(f"Got scene scores for {len(scene_scores)} timestamps", flush=True)
return scene_scores
def match_frames_to_scores(
frame_paths: list[str],
scene_scores: dict[float, float],
fps: int,
threshold: float
) -> list[dict]:
"""
Build the manifest by matching each extracted frame to its scene score.
Frame index N corresponds to timestamp N/fps seconds.
"""
frames = []
for i, path in enumerate(frame_paths):
timestamp = i / fps
# Find the closest scene score timestamp within a small window
window = 1.0 / fps # within one frame's width
closest_score = 0.0
for ts, score in scene_scores.items():
if abs(ts - timestamp) <= window:
if score > closest_score:
closest_score = score
is_key = closest_score >= threshold
frames.append({
"index": i,
"path": path,
"timestamp_s": round(timestamp, 3),
"scene_score": round(closest_score, 4),
"is_key_frame": is_key
})
return frames
def resize_frames(frame_paths: list[str], max_width: int = 1600):
"""
Resize extracted frames to a max width while preserving aspect ratio.
Modifies files in-place using PIL.
"""
if max_width <= 0:
print("Skipping frame resize", flush=True)
return
try:
from PIL import Image
except ImportError:
print("Pillow not available, skipping resize", flush=True)
return
print(f"Resizing frames to max {max_width}px wide...", flush=True)
for path in frame_paths:
with Image.open(path) as img:
if img.width > max_width:
ratio = max_width / img.width
new_h = int(img.height * ratio)
img = img.resize((max_width, new_h), Image.LANCZOS)
img.save(path, "JPEG", quality=85, optimize=True)
def sample_frames_for_contact_sheet(frames: list[dict], max_frames: int) -> list[dict]:
"""Return up to max_frames frames, evenly sampled from the manifest."""
if max_frames <= 0 or len(frames) <= max_frames:
return frames
if max_frames == 1:
return [frames[0]]
sampled = []
step = (len(frames) - 1) / (max_frames - 1)
seen = set()
for sample_index in range(max_frames):
frame_index = round(sample_index * step)
if frame_index not in seen:
sampled.append(frames[frame_index])
seen.add(frame_index)
return sampled
def write_contact_sheet(
frames: list[dict],
output_path: str,
cols: int = 5,
thumb_width: int = 260,
max_frames: int = 80,
) -> str | None:
"""Create a timeline contact sheet for fast visual scanning."""
if not frames:
return None
try:
from PIL import Image, ImageDraw
except ImportError:
print("Pillow not available, skipping contact sheet", flush=True)
return None
selected = sample_frames_for_contact_sheet(frames, max_frames)
cols = max(1, cols)
rows = (len(selected) + cols - 1) // cols
label_h = 34
gutter = 10
first = Image.open(selected[0]["path"])
thumb_height = max(1, int(first.height * (thumb_width / first.width)))
first.close()
sheet_w = cols * thumb_width + (cols + 1) * gutter
sheet_h = rows * (thumb_height + label_h) + (rows + 1) * gutter
sheet = Image.new("RGB", (sheet_w, sheet_h), (246, 246, 248))
draw = ImageDraw.Draw(sheet)
for cell_index, frame in enumerate(selected):
col = cell_index % cols
row = cell_index // cols
x = gutter + col * (thumb_width + gutter)
y = gutter + row * (thumb_height + label_h + gutter)
with Image.open(frame["path"]) as img:
if img.mode != "RGB":
img = img.convert("RGB")
thumb = img.resize((thumb_width, thumb_height), Image.LANCZOS)
sheet.paste(thumb, (x, y))
label = f"#{frame['index']} t={frame['timestamp_s']:.1f}s score={frame['scene_score']:.2f}"
if frame.get("is_key_frame"):
label = "* " + label
draw.text((x, y + thumb_height + 6), label, fill=(32, 32, 36))
Path(output_path).parent.mkdir(parents=True, exist_ok=True)
sheet.save(output_path, "JPEG", quality=86, optimize=True)
return output_path
def main():
parser = argparse.ArgumentParser(description="Extract key frames from a video file")
parser.add_argument("--input", required=True, help="Path to input video file")
parser.add_argument("--output", required=True, help="Output directory for frames")
parser.add_argument(
"--mode",
choices=["scout", "full"],
default="full",
help="scout extracts a low-fps review timeline; full extracts denser frames (default: full)",
)
parser.add_argument("--fps", type=int, default=None, help="Frames per second to extract")
parser.add_argument("--scene-threshold", type=float, default=0.25,
help="Scene change threshold 0.0-1.0 (default: 0.25). "
"Higher = fewer key frames detected.")
parser.add_argument("--max-width", type=int, default=None,
help="Max frame width in pixels. Use 0 to keep native resolution.")
parser.add_argument("--max-frames", type=int, default=3000,
help="Maximum extracted frames (default: 3000)")
parser.add_argument("--contact-sheet", dest="contact_sheet", action="store_true",
help="Write contact_sheet.jpg next to the manifest")
parser.add_argument("--no-contact-sheet", dest="contact_sheet", action="store_false",
help="Skip contact sheet generation")
parser.set_defaults(contact_sheet=None)
parser.add_argument("--contact-sheet-cols", type=int, default=5,
help="Columns in the contact sheet (default: 5)")
parser.add_argument("--contact-sheet-max-frames", type=int, default=80,
help="Maximum thumbnails in the contact sheet (default: 80)")
args = parser.parse_args()
fps = args.fps if args.fps is not None else (1 if args.mode == "scout" else 5)
max_width = args.max_width if args.max_width is not None else (640 if args.mode == "scout" else 1600)
make_contact_sheet = args.contact_sheet if args.contact_sheet is not None else args.mode == "scout"
if not os.path.exists(args.input):
print(f"Error: video file not found: {args.input}", file=sys.stderr)
sys.exit(1)
# Check ffmpeg
check = subprocess.run(["ffmpeg", "-version"], capture_output=True)
if check.returncode != 0:
print("Error: ffmpeg not found. Install with: brew install ffmpeg (macOS) or apt install ffmpeg (Linux)",
file=sys.stderr)
sys.exit(1)
os.makedirs(args.output, exist_ok=True)
# Get video metadata
print(f"Analyzing video: {args.input}", flush=True)
video_info = get_video_info(args.input)
print(f" Duration: {video_info['duration_s']:.1f}s | "
f"Resolution: {video_info['width']}x{video_info['height']} | "
f"Orientation: {video_info['orientation']}", flush=True)
# Extract all frames
frame_paths = extract_all_frames(args.input, args.output, fps, args.max_frames)
if not frame_paths:
print("Error: no frames were extracted", file=sys.stderr)
sys.exit(1)
# Resize frames, preserving enough resolution for Figma inspection.
if max_width > 0:
resize_frames(frame_paths, max_width)
else:
print("Keeping extracted frames at native resolution", flush=True)
# Detect scene changes
scene_scores = detect_scene_changes(args.input, fps)
# Build manifest
frames = match_frames_to_scores(frame_paths, scene_scores, fps, args.scene_threshold)
key_count = sum(1 for f in frames if f["is_key_frame"])
print(f"Found {key_count} key frames out of {len(frames)} total "
f"(threshold: {args.scene_threshold})", flush=True)
contact_sheet_path = None
if make_contact_sheet:
contact_sheet_path = os.path.join(args.output, "contact_sheet.jpg")
contact_sheet_path = write_contact_sheet(
frames,
contact_sheet_path,
cols=args.contact_sheet_cols,
max_frames=args.contact_sheet_max_frames,
)
if contact_sheet_path:
print(f"Contact sheet written to: {contact_sheet_path}", flush=True)
manifest = {
"video_path": os.path.abspath(args.input),
"output_dir": os.path.abspath(args.output),
"mode": args.mode,
"fps_extracted": fps,
"scene_threshold": args.scene_threshold,
"max_width": max_width,
"video_info": video_info,
"contact_sheet_path": os.path.abspath(contact_sheet_path) if contact_sheet_path else None,
"total_frames": len(frames),
"key_frame_count": key_count,
"frames": frames
}
manifest_path = os.path.join(args.output, "frames_manifest.json")
with open(manifest_path, "w") as f:
json.dump(manifest, f, indent=2)
print(f"\nManifest written to: {manifest_path}")
print(f"Key frames: {key_count} / {len(frames)}")
# Print a quick summary of key frame timestamps
key_frames = [f for f in frames if f["is_key_frame"]]
if key_frames:
print("\nKey frame timestamps:")
for kf in key_frames[:20]: # show first 20
print(f" t={kf['timestamp_s']:.2f}s score={kf['scene_score']:.3f} {kf['path']}")
if len(key_frames) > 20:
print(f" ... and {len(key_frames) - 20} more")
if __name__ == "__main__":
main()
scripts/generate_figma_calls.pyShowHide
#!/usr/bin/env python3
"""
Generate Figma Plugin API JavaScript for a complete interaction storyboard.
The generated storyboard script creates all screenshot containers, labels,
native annotations, and visual interaction markers in one use_figma call. It
does not embed image bytes. It returns uploadTargets so the agent can upload
optimized screenshots with upload_assets, then run the generated fill script.
Usage:
python generate_figma_calls.py \
--prepared /tmp/vim_frames/upload_manifest.json \
--output-dir /tmp/vim_frames/figma_calls/
"""
import argparse
import datetime
import json
import os
import re
import sys
from pathlib import Path
STORYBOARD_TEMPLATE = r'''// AUTO-GENERATED by generate_figma_calls.py - do not edit manually
// Creates the full interaction storyboard. Upload images separately with upload_assets.
const PAGE_NAME = __PAGE_NAME_JSON__;
const STORYBOARD = __STORYBOARD_JSON__;
let page = figma.root.children.find(p => p.type === "PAGE" && p.name === PAGE_NAME);
if (!page) {
const reusableBlankPage = figma.root.children.length === 1 ? figma.root.children[0] : null;
page = reusableBlankPage || figma.createPage();
page.name = PAGE_NAME;
}
await figma.setCurrentPageAsync(page);
// Make re-runs deterministic for this generated page.
for (const child of [...page.children]) {
child.remove();
}
await figma.loadFontAsync({ family: "Inter", style: "Regular" });
await figma.loadFontAsync({ family: "Inter", style: "Semi Bold" });
await figma.loadFontAsync({ family: "Inter", style: "Bold" });
const createdNodeIds = [];
const mutatedNodeIds = [page.id];
const uploadTargets = [];
const frameW = STORYBOARD.frameWidth;
const frameH = STORYBOARD.frameHeight;
const GAP = 80;
const MOMENT_GAP = 160;
const TITLE_HEIGHT = 240;
const LABEL_GAP = 16;
const NOTE_GAP = 8;
const ns = "videoInteractionMapper";
const colors = {
titleBg: { r: 0.067, g: 0.067, b: 0.067 },
white: { r: 1, g: 1, b: 1 },
muted: { r: 0.58, g: 0.58, b: 0.62 },
mutedText: { r: 0.45, g: 0.45, b: 0.45 },
accent: { r: 0.38, g: 0.38, b: 0.93 },
frameBg: { r: 0.92, g: 0.92, b: 0.95 },
frameStroke: { r: 0.78, g: 0.78, b: 0.84 },
marker: { r: 0.098, g: 0.435, b: 1 },
inferredMarker: { r: 1.0, g: 0.62, b: 0.05 }
};
const makeText = (chars, size, style, color, x, y, maxWidth, parent = page) => {
const node = figma.createText();
node.fontName = { family: "Inter", style };
node.characters = chars || "";
node.fontSize = size;
node.fills = [{ type: "SOLID", color }];
node.x = x;
node.y = y;
if (maxWidth) {
node.resize(maxWidth, node.height);
node.textAutoResize = "HEIGHT";
}
parent.appendChild(node);
createdNodeIds.push(node.id);
return node;
};
const makeUploadFrame = (name, x, y, width, height) => {
const frame = figma.createFrame();
frame.name = name;
frame.resize(width, height);
frame.x = x;
frame.y = y;
frame.fills = [{ type: "SOLID", color: colors.frameBg }];
frame.strokes = [{ type: "SOLID", color: colors.frameStroke }];
frame.strokeWeight = 1;
frame.cornerRadius = 8;
frame.clipsContent = true;
page.appendChild(frame);
createdNodeIds.push(frame.id);
return frame;
};
const toUnit = (value) => {
const number = Number(value);
if (!Number.isFinite(number)) return null;
return Math.max(0, Math.min(1, number > 1 ? number / 100 : number));
};
const normalizePoint = (point) => {
if (!point) return null;
if (Array.isArray(point)) return { x: point[0], y: point[1], label: "" };
return point;
};
const isUncertainPoint = (point) => {
const normalized = normalizePoint(point);
if (!normalized) return false;
if (normalized.uncertain === true || normalized.inferred === true) return true;
if (typeof normalized.confidence === "string") {
return ["medium", "low", "inferred", "uncertain"].includes(normalized.confidence.toLowerCase());
}
if (typeof normalized.confidence === "number") {
return normalized.confidence < 0.85;
}
return false;
};
const pointHasCoordinates = (point) => {
const normalized = normalizePoint(point);
if (!normalized) return false;
return toUnit(normalized.x) !== null && toUnit(normalized.y) !== null;
};
const hasExplicitMarkerOverride = (point, value) => {
const normalized = normalizePoint(point);
if (!normalized) return false;
return normalized.marker === value ||
normalized.draw_marker === value ||
normalized.drawMarker === value ||
normalized.show_marker === value ||
normalized.showMarker === value;
};
const shouldDrawMarker = (point) => {
const normalized = normalizePoint(point);
if (!normalized || !pointHasCoordinates(normalized)) return false;
if (normalized.visible === false || hasExplicitMarkerOverride(normalized, false)) return false;
if (hasExplicitMarkerOverride(normalized, true)) return true;
return !isUncertainPoint(normalized);
};
const shouldDrawResultMarker = (point) => {
return hasExplicitMarkerOverride(point, true);
};
const labelForPoint = (point, fallback) => {
const normalized = normalizePoint(point);
const label = normalized && normalized.label ? normalized.label : fallback;
return isUncertainPoint(normalized) ? `${label} (inferred)` : label;
};
const addInteractionMarker = (frame, point, markerName) => {
const normalized = normalizePoint(point);
if (!shouldDrawMarker(normalized)) return null;
const x = toUnit(normalized.x);
const y = toUnit(normalized.y);
if (x === null || y === null) return null;
const uncertain = isUncertainPoint(normalized);
const markerColor = uncertain ? colors.inferredMarker : colors.marker;
const size = uncertain ? 56 : 48;
const ring = figma.createEllipse();
ring.name = uncertain ? `${markerName}_inferred` : markerName;
ring.resize(size, size);
ring.x = (x * frameW) - (size / 2);
ring.y = (y * frameH) - (size / 2);
ring.fills = [{ type: "SOLID", color: markerColor, opacity: uncertain ? 0.1 : 0.14 }];
ring.strokes = [{ type: "SOLID", color: markerColor }];
ring.strokeWeight = uncertain ? 3 : 4;
frame.appendChild(ring);
createdNodeIds.push(ring.id);
const dot = figma.createEllipse();
dot.name = `${ring.name}_dot`;
dot.resize(10, 10);
dot.x = (x * frameW) - 5;
dot.y = (y * frameH) - 5;
dot.fills = [{ type: "SOLID", color: markerColor }];
frame.appendChild(dot);
createdNodeIds.push(dot.id);
return ring;
};
const getInteractionCategory = async () => {
if (!figma.annotations) return null;
const categories = await figma.annotations.getAnnotationCategoriesAsync();
const existing = categories.find(category => category.label === "Interaction");
return existing || await figma.annotations.addAnnotationCategoryAsync({
label: "Interaction",
color: "blue"
});
};
const appendNativeAnnotation = (node, labelMarkdown, categoryId) => {
if (!categoryId) return;
node.annotations = [
...(node.annotations || []),
{ labelMarkdown, categoryId }
];
mutatedNodeIds.push(node.id);
};
const titleBg = figma.createFrame();
titleBg.name = "_title_block";
titleBg.resize(860, 200);
titleBg.x = 0;
titleBg.y = 0;
titleBg.fills = [{ type: "SOLID", color: colors.titleBg }];
titleBg.cornerRadius = 12;
page.appendChild(titleBg);
createdNodeIds.push(titleBg.id);
makeText("Interaction Map", 28, "Bold", colors.white, 32, 32, 780, titleBg);
makeText(STORYBOARD.videoName, 14, "Regular", colors.muted, 32, 80, 780, titleBg);
makeText(STORYBOARD.meta, 12, "Regular", colors.muted, 32, 104, 780, titleBg);
makeText(`${STORYBOARD.moments.length} interaction states extracted`, 14, "Semi Bold", colors.accent, 32, 140, 780, titleBg);
const category = await getInteractionCategory();
let nextX = 0;
const frameY = TITLE_HEIGHT;
for (const moment of STORYBOARD.moments) {
if (nextX > 0) nextX += MOMENT_GAP - GAP;
const beforeX = nextX;
const beforeFrame = makeUploadFrame(moment.beforeLabel, beforeX, frameY, frameW, frameH);
if (moment.beforePath) {
uploadTargets.push({
assetKey: moment.beforeAssetKey,
momentIndex: moment.momentIndex,
role: "before",
nodeId: beforeFrame.id,
path: moment.beforePath,
scaleMode: "FILL"
});
}
makeText("Before state", 11, "Regular", colors.mutedText, beforeX, frameY + frameH + LABEL_GAP, frameW);
nextX = beforeX + frameW + GAP;
const afterX = nextX;
const afterFrame = makeUploadFrame(moment.afterLabel, afterX, frameY, frameW, frameH);
if (moment.afterPath) {
uploadTargets.push({
assetKey: moment.afterAssetKey,
momentIndex: moment.momentIndex,
role: "after",
nodeId: afterFrame.id,
path: moment.afterPath,
scaleMode: "FILL"
});
}
const interaction = moment.interaction;
const pickFrame = (name, fallback) => {
if (name === "after") return afterFrame;
if (name === "before") return beforeFrame;
return fallback;
};
if (interaction && interaction.target) {
const targetFrame = pickFrame(interaction.target_frame || interaction.targetFrame, beforeFrame);
const resultFrame = pickFrame(interaction.result_frame || interaction.resultFrame, afterFrame);
const targetLabel = labelForPoint(interaction.target, "User action");
const resultLabel = labelForPoint(interaction.result, "Resulting UI change");
const summary = interaction.annotation || `${targetLabel} -> ${resultLabel}`;
const targetMarker = addInteractionMarker(targetFrame, interaction.target, "_interaction_target_marker");
const targetAnnotationNode = targetMarker || targetFrame;
appendNativeAnnotation(
targetAnnotationNode,
`**Interaction target**\n${summary}\n\nTarget: ${targetLabel}`,
category && category.id
);
if (interaction.result) {
const resultMarker = shouldDrawResultMarker(interaction.result)
? addInteractionMarker(resultFrame, interaction.result, "_interaction_result_marker")
: null;
const resultAnnotationNode = resultMarker || resultFrame;
if (resultAnnotationNode.id !== targetAnnotationNode.id) {
appendNativeAnnotation(
resultAnnotationNode,
`**Interaction result**\n${summary}\n\nResult: ${resultLabel}`,
category && category.id
);
}
}
}
const triggerLabel = makeText(
moment.afterLabel,
13,
"Semi Bold",
colors.accent,
afterX,
frameY + frameH + LABEL_GAP,
frameW
);
makeText(
moment.annotation,
11,
"Regular",
colors.mutedText,
afterX,
frameY + frameH + LABEL_GAP + triggerLabel.height + NOTE_GAP,
frameW
);
nextX = afterX + frameW + GAP;
}
page.setSharedPluginData(ns, "runManifestPath", STORYBOARD.runManifestPath || "");
page.setSharedPluginData(ns, "pageName", PAGE_NAME);
page.setSharedPluginData(ns, "frameWidth", String(frameW));
page.setSharedPluginData(ns, "frameHeight", String(frameH));
figma.viewport.scrollAndZoomIntoView(page.children);
return {
pageId: page.id,
pageName: PAGE_NAME,
createdNodeIds,
mutatedNodeIds,
uploadTargets,
nextX
};
'''
APPLY_FILLS_TEMPLATE = r'''// AUTO-GENERATED by generate_figma_calls.py - do not edit manually
// Paste upload results into uploadedImages, then run this with use_figma.
// Expected item shape: { "nodeId": "3:8", "imageHash": "abc123" }
const uploadedImages = __UPLOAD_RESULTS_JSON__;
if (!uploadedImages.length) {
throw new Error("No uploaded image hashes supplied. Replace uploadedImages with upload_assets results first.");
}
const mutatedNodeIds = [];
for (const item of uploadedImages) {
const node = await figma.getNodeByIdAsync(item.nodeId);
if (!node) {
throw new Error(`Could not find image target node ${item.nodeId}`);
}
node.fills = [{
type: "IMAGE",
imageHash: item.imageHash,
scaleMode: item.scaleMode || "FILL"
}];
mutatedNodeIds.push(node.id);
}
return {
mutatedNodeIds,
appliedImageCount: mutatedNodeIds.length
};
'''
def safe_filename(value: str) -> str:
cleaned = re.sub(r"[^a-zA-Z0-9_-]", "_", value.lower())
cleaned = re.sub(r"_+", "_", cleaned).strip("_")
return cleaned[:30] or "moment"
def frame_size(prepared: dict) -> tuple[int, int]:
width = prepared.get("frame_width")
height = prepared.get("frame_height")
if width and height:
return int(width), int(height)
video_info = prepared.get("video_info", {})
source_width = int(video_info.get("width", 1440) or 1440)
source_height = int(video_info.get("height", 810) or 810)
max_width = 900 if source_height > source_width else 1440
if source_width > max_width:
ratio = max_width / source_width
return max_width, int(source_height * ratio)
return source_width, source_height
def asset_key(moment_index: int, role: str, label: str) -> str:
return f"moment-{moment_index:03d}-{role}-{safe_filename(label)}"
def point_has_coordinates(point: object) -> bool:
if not point:
return False
if isinstance(point, list) and len(point) >= 2:
return True
if isinstance(point, dict):
return point.get("x") is not None and point.get("y") is not None
return False
def point_label(point: object, fallback: str) -> str:
if isinstance(point, dict) and point.get("label"):
return str(point["label"])
return fallback
def point_value(point: object, key: str) -> object:
if isinstance(point, dict):
return point.get(key)
return None
def point_marker_override(point: object) -> bool | None:
if not isinstance(point, dict):
return None
for key in ("marker", "draw_marker", "drawMarker", "show_marker", "showMarker"):
if key in point:
return bool(point[key])
return None
def point_is_uncertain(point: object) -> bool:
if not isinstance(point, dict):
return False
if point.get("uncertain") is True or point.get("inferred") is True:
return True
confidence = point.get("confidence")
if isinstance(confidence, str):
return confidence.lower() in {"medium", "low", "inferred", "uncertain"}
if isinstance(confidence, (int, float)):
return confidence < 0.85
return False
def point_marker_decision(point: object, *, default_draw: bool = True) -> str:
if not point_has_coordinates(point):
return "skipped_no_coordinates"
if isinstance(point, dict) and point.get("visible") is False:
return "skipped_not_visible"
override = point_marker_override(point)
if override is False:
return "skipped_by_marker_override"
if override is True:
return "drawn_by_marker_override"
if point_is_uncertain(point):
return "skipped_uncertain"
return "drawn" if default_draw else "skipped_default_result_annotation"
def build_storyboard(prepared: dict, page_name: str, run_manifest_path: str) -> tuple[dict, list[dict]]:
frame_w, frame_h = frame_size(prepared)
video_path = prepared.get("video_path", "unknown")
video_name = os.path.basename(video_path)
total_moments = prepared.get("total_moments", len(prepared.get("moments", [])))
generated_at = datetime.datetime.now().isoformat(timespec="seconds")
meta = f"{total_moments} key moments - {generated_at[:10]}"
moments_out = []
expected_uploads = []
for index, moment in enumerate(prepared.get("moments", []), start=1):
moment_index = int(moment.get("moment_index", index))
short_label = moment.get("short_label", f"Moment {moment_index}")
trigger_marker = moment.get("trigger_emoji", "")
before_path = moment.get("before_upload_path") or moment.get("before_frame_path")
after_path = moment.get("after_upload_path") or moment.get("after_frame_path")
before_key = asset_key(moment_index, "before", short_label)
after_key = asset_key(moment_index, "after", short_label)
after_label = f"{moment_index:02d} {trigger_marker} {short_label}".replace(" ", " ").strip()
moments_out.append(
{
"momentIndex": moment_index,
"timestampS": moment.get("timestamp_s"),
"trigger": moment.get("trigger"),
"shortLabel": short_label,
"annotation": moment.get("annotation", ""),
"beforeLabel": f"Before - {short_label}",
"afterLabel": after_label,
"beforePath": before_path,
"afterPath": after_path,
"beforeAssetKey": before_key,
"afterAssetKey": after_key,
"interaction": moment.get("interaction"),
}
)
if before_path:
expected_uploads.append(
{
"assetKey": before_key,
"momentIndex": moment_index,
"role": "before",
"path": before_path,
"scaleMode": "FILL",
"nodeId": None,
"imageHash": None,
"status": "pending",
}
)
if after_path:
expected_uploads.append(
{
"assetKey": after_key,
"momentIndex": moment_index,
"role": "after",
"path": after_path,
"scaleMode": "FILL",
"nodeId": None,
"imageHash": None,
"status": "pending",
}
)
storyboard = {
"videoName": video_name,
"videoPath": video_path,
"generatedAt": generated_at,
"meta": meta,
"frameWidth": frame_w,
"frameHeight": frame_h,
"runManifestPath": run_manifest_path,
"moments": moments_out,
}
return storyboard, expected_uploads
def build_run_manifest(
prepared: dict,
storyboard: dict,
expected_uploads: list[dict],
page_name: str,
storyboard_script: str,
fill_script_template: str,
) -> dict:
selected_moments = []
for moment in prepared.get("moments", []):
interaction = moment.get("interaction") or {}
target = interaction.get("target") if isinstance(interaction, dict) else None
result = interaction.get("result") if isinstance(interaction, dict) else None
selected_moments.append(
{
"moment_index": moment.get("moment_index"),
"timestamp_s": moment.get("timestamp_s"),
"short_label": moment.get("short_label"),
"before_upload_path": moment.get("before_upload_path"),
"after_upload_path": moment.get("after_upload_path"),
"interaction": {
"target_frame": interaction.get("target_frame") or interaction.get("targetFrame"),
"target_label": point_label(target, "User action") if target else None,
"target_has_coordinates": point_has_coordinates(target),
"target_confidence": point_value(target, "confidence"),
"target_inferred": bool(point_value(target, "inferred")),
"target_uncertain": bool(point_value(target, "uncertain")),
"target_visible": point_value(target, "visible"),
"target_marker_decision": point_marker_decision(target),
"result_frame": interaction.get("result_frame") or interaction.get("resultFrame"),
"result_label": point_label(result, "Resulting UI change") if result else None,
"result_has_coordinates": point_has_coordinates(result),
"result_confidence": point_value(result, "confidence"),
"result_inferred": bool(point_value(result, "inferred")),
"result_uncertain": bool(point_value(result, "uncertain")),
"result_visible": point_value(result, "visible"),
"result_marker_decision": point_marker_decision(re… [4564 more characters not shown]scripts/prepare_upload_frames.pyShowHide
#!/usr/bin/env python3
"""
Prepare key frame images for Figma upload_assets.
Takes a frames manifest and a list of analyzed key moments, then writes resized
JPEG assets for each before/after state plus an upload_manifest.json consumed by
generate_figma_calls.py.
Usage:
python prepare_upload_frames.py \
--manifest /tmp/vim_frames/frames_manifest.json \
--moments-file /tmp/vim_frames/key_moments.json \
--output /tmp/vim_frames/upload_manifest.json
"""
import argparse
import json
import os
from pathlib import Path
def prepare_image(
input_path: str,
output_path: Path,
max_width: int,
quality: int,
max_bytes: int,
) -> dict:
"""Resize and compress an image for upload_assets, returning final metadata."""
try:
from PIL import Image
except ImportError as exc:
raise RuntimeError("Pillow is required: pip install Pillow --break-system-packages") from exc
with Image.open(input_path) as img:
if img.mode != "RGB":
img = img.convert("RGB")
if max_width > 0 and img.width > max_width:
ratio = max_width / img.width
new_height = int(img.height * ratio)
img = img.resize((max_width, new_height), Image.LANCZOS)
output_path.parent.mkdir(parents=True, exist_ok=True)
current_quality = min(max(quality, 1), 95)
while True:
img.save(output_path, format="JPEG", quality=current_quality, optimize=True)
size_bytes = output_path.stat().st_size
if size_bytes <= max_bytes:
break
if current_quality > 45:
current_quality = max(45, current_quality - 10)
continue
next_width = max(480, int(img.width * 0.85))
if next_width >= img.width:
break
ratio = next_width / img.width
img = img.resize((next_width, int(img.height * ratio)), Image.LANCZOS)
return {
"width": img.width,
"height": img.height,
"quality": current_quality,
"size_bytes": output_path.stat().st_size,
"within_budget": output_path.stat().st_size <= max_bytes,
}
def safe_stem(label: str) -> str:
cleaned = "".join(ch.lower() if ch.isalnum() else "_" for ch in label)
cleaned = "_".join(part for part in cleaned.split("_") if part)
return cleaned[:48] or "moment"
def main() -> None:
parser = argparse.ArgumentParser(description="Prepare frame images for Figma upload_assets")
parser.add_argument("--manifest", required=True, help="Path to frames_manifest.json")
parser.add_argument("--moments-file", required=True, help="Path to key_moments.json")
parser.add_argument("--output", required=True, help="Output path for upload_manifest.json")
parser.add_argument(
"--assets-dir",
help="Directory for optimized JPEGs. Defaults to <output-dir>/upload_assets",
)
parser.add_argument(
"--max-width",
type=int,
default=0,
help="Max frame width. Auto-detected from video orientation if not set.",
)
parser.add_argument(
"--quality",
type=int,
default=75,
help="JPEG quality 1-95 (default: 75)",
)
parser.add_argument(
"--max-file-mb",
type=float,
default=9.5,
help="Maximum uploaded image size in MB (default: 9.5)",
)
args = parser.parse_args()
with open(args.manifest) as f:
manifest = json.load(f)
with open(args.moments_file) as f:
moments = json.load(f)
output_path = Path(args.output)
assets_dir = Path(args.assets_dir) if args.assets_dir else output_path.parent / "upload_assets"
video_info = manifest.get("video_info", {})
orientation = video_info.get("orientation", "landscape")
if args.max_width > 0:
max_width = args.max_width
elif orientation == "portrait":
max_width = 900
else:
max_width = 1440
prepared_moments = []
frame_width = None
frame_height = None
max_bytes = int(args.max_file_mb * 1024 * 1024)
print(
f"Preparing {len(moments)} moments for upload_assets "
f"({max_width}px wide, JPEG q{args.quality})...",
flush=True,
)
for index, moment in enumerate(moments, start=1):
prepared = dict(moment)
moment_index = prepared.get("moment_index", index)
label = safe_stem(prepared.get("short_label", f"moment_{moment_index}"))
for role in ("before", "after"):
source_key = f"{role}_frame_path"
upload_key = f"{role}_upload_path"
source_path = prepared.get(source_key)
if source_path and os.path.exists(source_path):
output_file = assets_dir / f"moment_{moment_index:03d}_{role}_{label}.jpg"
info = prepare_image(source_path, output_file, max_width, args.quality, max_bytes)
prepared[upload_key] = str(output_file)
prepared[f"{role}_upload_width"] = info["width"]
prepared[f"{role}_upload_height"] = info["height"]
prepared[f"{role}_upload_quality"] = info["quality"]
prepared[f"{role}_upload_size_bytes"] = info["size_bytes"]
prepared[f"{role}_upload_within_budget"] = info["within_budget"]
frame_width = frame_width or info["width"]
frame_height = frame_height or info["height"]
status = "ok" if info["within_budget"] else "over budget"
print(
f" [{index}/{len(moments)}] {role}: {output_file.name} "
f"({info['width']}x{info['height']}, q{info['quality']}, "
f"{info['size_bytes'] / 1024:.0f} KB, {status})",
flush=True,
)
else:
prepared[upload_key] = None
print(f" [{index}/{len(moments)}] {role}: missing ({source_path})", flush=True)
prepared_moments.append(prepared)
if frame_width is None or frame_height is None:
width = int(video_info.get("width", max_width) or max_width)
height = int(video_info.get("height", max_width) or max_width)
if width > max_width:
ratio = max_width / width
width = max_width
height = int(height * ratio)
frame_width = width
frame_height = height
output = {
"video_path": manifest.get("video_path"),
"video_info": video_info,
"frame_width": frame_width,
"frame_height": frame_height,
"image_budget": {
"max_width": max_width,
"requested_quality": args.quality,
"max_file_mb": args.max_file_mb,
"max_file_bytes": max_bytes,
},
"total_moments": len(prepared_moments),
"assets_dir": str(assets_dir),
"moments": prepared_moments,
}
output_path.parent.mkdir(parents=True, exist_ok=True)
with open(output_path, "w") as f:
json.dump(output, f, indent=2)
print(f"\nPrepared {len(prepared_moments)} moments")
print(f"Assets: {assets_dir}")
print(f"Output: {output_path}")
if __name__ == "__main__":
main()
scripts/resolve_moment_frames.pyShowHide
#!/usr/bin/env python3
"""
Resolve analyzed key moments into concrete before/after screenshot files.
This lets the workflow start with a low-resolution scout/contact sheet and only
extract the high-quality frames that will actually be uploaded to Figma.
Usage:
python resolve_moment_frames.py \
--video input.mp4 \
--moments-file key_moments.json \
--output key_moments_resolved.json \
--frames-dir selected_frames/
"""
import argparse
import json
import os
import subprocess
import sys
from pathlib import Path
def safe_stem(label: str) -> str:
cleaned = "".join(ch.lower() if ch.isalnum() else "_" for ch in label)
cleaned = "_".join(part for part in cleaned.split("_") if part)
return cleaned[:48] or "moment"
def numeric_value(value: object) -> float | None:
try:
number = float(value)
except (TypeError, ValueError):
return None
return number if number >= 0 else 0.0
def timestamp_for(moment: dict, role: str, before_offset: float, after_offset: float) -> float | None:
for key in (
f"{role}_timestamp_s",
f"{role}_time_s",
f"{role}_timestamp",
):
timestamp = numeric_value(moment.get(key))
if timestamp is not None:
return timestamp
base = numeric_value(moment.get("timestamp_s"))
if base is None:
return None
if role == "before":
return max(0.0, base - before_offset)
return max(0.0, base + after_offset)
def optimize_jpeg(path: Path, max_width: int, quality: int) -> tuple[int | None, int | None]:
try:
from PIL import Image
except ImportError:
if max_width > 0:
print("Pillow not available, selected frames will not be resized", flush=True)
return None, None
with Image.open(path) as img:
if img.mode != "RGB":
img = img.convert("RGB")
if max_width > 0 and img.width > max_width:
ratio = max_width / img.width
img = img.resize((max_width, int(img.height * ratio)), Image.LANCZOS)
img.save(path, "JPEG", quality=quality, optimize=True)
return img.width, img.height
def extract_frame(video_path: str, timestamp_s: float, output_path: Path, max_width: int, quality: int) -> dict:
output_path.parent.mkdir(parents=True, exist_ok=True)
cmd = [
"ffmpeg",
"-ss",
f"{timestamp_s:.3f}",
"-i",
video_path,
"-frames:v",
"1",
"-q:v",
"2",
"-y",
"-v",
"quiet",
str(output_path),
]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0 or not output_path.exists():
raise RuntimeError(f"ffmpeg failed at {timestamp_s:.3f}s: {result.stderr}")
width, height = optimize_jpeg(output_path, max_width, quality)
return {
"path": str(output_path),
"timestamp_s": round(timestamp_s, 3),
"width": width,
"height": height,
"size_bytes": output_path.stat().st_size,
}
def main() -> None:
parser = argparse.ArgumentParser(description="Extract selected before/after frames for key moments")
parser.add_argument("--video", required=True, help="Source video path")
parser.add_argument("--moments-file", required=True, help="Input key_moments.json")
parser.add_argument("--output", required=True, help="Output resolved key_moments JSON")
parser.add_argument("--frames-dir", required=True, help="Directory for selected frame JPEGs")
parser.add_argument("--max-width", type=int, default=1600, help="Max selected frame width (default: 1600)")
parser.add_argument("--quality", type=int, default=86, help="JPEG quality for selected frames (default: 86)")
parser.add_argument("--before-offset", type=float, default=0.2,
help="Fallback seconds before timestamp_s for before frames")
parser.add_argument("--after-offset", type=float, default=0.0,
help="Fallback seconds after timestamp_s for after frames")
parser.add_argument("--force", action="store_true", help="Re-extract even when frame paths already exist")
args = parser.parse_args()
if not os.path.exists(args.video):
print(f"Error: video file not found: {args.video}", file=sys.stderr)
sys.exit(1)
with open(args.moments_file) as f:
moments = json.load(f)
if not isinstance(moments, list):
print("Error: moments file must contain a JSON array", file=sys.stderr)
sys.exit(1)
frames_dir = Path(args.frames_dir)
resolved = []
warnings = []
for index, moment in enumerate(moments, start=1):
next_moment = dict(moment)
moment_index = int(next_moment.get("moment_index", index))
label = safe_stem(next_moment.get("short_label", f"moment_{moment_index}"))
for role in ("before", "after"):
path_key = f"{role}_frame_path"
existing_path = next_moment.get(path_key)
if existing_path and os.path.exists(existing_path) and not args.force:
continue
timestamp_s = timestamp_for(next_moment, role, args.before_offset, args.after_offset)
if timestamp_s is None:
warnings.append(f"moment {moment_index} {role}: no frame path or timestamp")
continue
output_path = frames_dir / f"moment_{moment_index:03d}_{role}_{label}.jpg"
info = extract_frame(args.video, timestamp_s, output_path, args.max_width, args.quality)
next_moment[path_key] = info["path"]
next_moment[f"{role}_timestamp_s"] = info["timestamp_s"]
next_moment[f"{role}_frame_size_bytes"] = info["size_bytes"]
if info["width"] and info["height"]:
next_moment[f"{role}_frame_width"] = info["width"]
next_moment[f"{role}_frame_height"] = info["height"]
print(f" moment {moment_index:02d} {role}: {output_path.name}", flush=True)
resolved.append(next_moment)
output_path = Path(args.output)
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(json.dumps(resolved, indent=2))
print(f"\nResolved {len(resolved)} moments")
print(f"Output: {output_path}")
if warnings:
print("\nWarnings:")
for warning in warnings:
print(f" - {warning}")
if __name__ == "__main__":
main()
scripts/update_run_manifest.pyShowHide
#!/usr/bin/env python3
"""
Update a video-interaction-mapper run_manifest.json as Figma work completes.
This helper keeps the workflow resumable:
- record the storyboard use_figma result, including node IDs and upload targets
- record upload_assets image hashes
- generate a ready-to-run fill script from the fill template
- record fill and verification status
Usage:
python update_run_manifest.py \
--run-manifest figma_calls/run_manifest.json \
--storyboard-result storyboard_result.json
python update_run_manifest.py \
--run-manifest figma_calls/run_manifest.json \
--uploads-file uploaded_images.json \
--write-fill-script figma_calls/figma_apply_fills.js
"""
import argparse
import json
import re
from pathlib import Path
def load_json(path: str) -> object:
data = json.loads(Path(path).read_text())
if isinstance(data, list) and len(data) == 1 and isinstance(data[0], dict) and "text" in data[0]:
return json.loads(data[0]["text"])
if isinstance(data, dict) and "text" in data and isinstance(data["text"], str):
return json.loads(data["text"])
return data
def as_list(value: object, keys: tuple[str, ...]) -> list:
if isinstance(value, list):
return value
if isinstance(value, dict):
for key in keys:
candidate = value.get(key)
if isinstance(candidate, list):
return candidate
return []
def update_upload_targets(run_manifest: dict, upload_targets: list[dict]) -> None:
run_manifest["figma"]["upload_targets"] = upload_targets
expected_by_key = {
item.get("assetKey"): item
for item in run_manifest.get("expected_uploads", [])
if item.get("assetKey")
}
for target in upload_targets:
match = expected_by_key.get(target.get("assetKey"))
if match:
match["nodeId"] = target.get("nodeId")
match["status"] = "ready_to_upload"
if upload_targets:
run_manifest["stages"]["storyboard"] = "complete"
def normalize_uploaded_images(value: object) -> list[dict]:
uploads = as_list(value, ("uploadedImages", "uploads", "results", "items"))
normalized = []
for item in uploads:
if not isinstance(item, dict):
continue
node_id = item.get("nodeId") or item.get("node_id")
image_hash = item.get("imageHash") or item.get("image_hash") or item.get("hash")
if node_id and image_hash:
normalized.append(
{
"assetKey": item.get("assetKey"),
"momentIndex": item.get("momentIndex"),
"role": item.get("role"),
"nodeId": node_id,
"imageHash": image_hash,
"scaleMode": item.get("scaleMode", "FILL"),
}
)
return normalized
def update_uploaded_images(run_manifest: dict, uploaded_images: list[dict]) -> None:
run_manifest["figma"]["uploaded_images"] = uploaded_images
expected = run_manifest.get("expected_uploads", [])
expected_by_node = {item.get("nodeId"): item for item in expected if item.get("nodeId")}
expected_by_key = {item.get("assetKey"): item for item in expected if item.get("assetKey")}
for uploaded in uploaded_images:
match = expected_by_node.get(uploaded.get("nodeId"))
if not match and uploaded.get("assetKey"):
match = expected_by_key.get(uploaded.get("assetKey"))
if match:
match["imageHash"] = uploaded.get("imageHash")
match["status"] = "uploaded"
if expected and all(item.get("status") == "uploaded" for item in expected):
run_manifest["stages"]["uploads"] = "complete"
def write_fill_script(run_manifest: dict, output_path: str) -> None:
template_path = run_manifest["figma"].get("fill_script_template")
if not template_path:
raise RuntimeError("run manifest has no fill_script_template path")
uploaded_images = run_manifest["figma"].get("uploaded_images", [])
if not uploaded_images:
raise RuntimeError("run manifest has no uploaded_images to apply")
template = Path(template_path).read_text()
replacement = "const uploadedImages = " + json.dumps(uploaded_images, indent=2) + ";"
script = re.sub(r"const uploadedImages = \[\];", replacement, template, count=1)
Path(output_path).write_text(script)
run_manifest["figma"]["fill_script"] = output_path
def main() -> None:
parser = argparse.ArgumentParser(description="Update a video interaction mapper run manifest")
parser.add_argument("--run-manifest", required=True, help="Path to run_manifest.json")
parser.add_argument("--storyboard-result", help="JSON file containing the storyboard use_figma result")
parser.add_argument("--uploads-file", help="JSON file containing uploaded nodeId/imageHash pairs")
parser.add_argument("--fill-result", help="JSON file containing the fill-pass use_figma result")
parser.add_argument("--write-fill-script", help="Write a ready-to-run fill script to this path")
parser.add_argument("--verification-screenshot", help="Path to a downloaded verification screenshot")
parser.add_argument("--verification-notes", default="", help="Verification notes")
parser.add_argument("--output", help="Optional output manifest path. Defaults to in-place update.")
args = parser.parse_args()
run_manifest_path = Path(args.run_manifest)
run_manifest = json.loads(run_manifest_path.read_text())
if args.storyboard_result:
result = load_json(args.storyboard_result)
if isinstance(result, dict):
run_manifest["figma"]["page_id"] = result.get("pageId") or run_manifest["figma"].get("page_id")
run_manifest["figma"]["created_node_ids"] = result.get("createdNodeIds", [])
update_upload_targets(run_manifest, result.get("uploadTargets", []))
if args.uploads_file:
uploads = normalize_uploaded_images(load_json(args.uploads_file))
update_uploaded_images(run_manifest, uploads)
if args.write_fill_script:
write_fill_script(run_manifest, args.write_fill_script)
if args.fill_result:
fill_result = load_json(args.fill_result)
if isinstance(fill_result, dict):
run_manifest["figma"]["fill_pass"] = {
"status": "complete",
"mutated_node_ids": fill_result.get("mutatedNodeIds", []),
"applied_image_count": fill_result.get("appliedImageCount"),
}
run_manifest["stages"]["fills"] = "complete"
if args.verification_screenshot:
run_manifest["figma"]["verification"] = {
"status": "complete",
"screenshot_path": args.verification_screenshot,
"notes": args.verification_notes,
}
run_manifest["stages"]["verification"] = "complete"
output_path = Path(args.output) if args.output else run_manifest_path
output_path.write_text(json.dumps(run_manifest, indent=2))
print(f"Updated run manifest: {output_path}")
if __name__ == "__main__":
main()
