A Twitch stream is a long file with a few minutes you would actually post. The boring pipeline underneath every twitch clip maker is the same: get the VOD, guess where the moments are, then export a vertical file.

I work on FragCut. The product details I mention later are the current published plan facts (first match analyzed free, pay to render clips, Creator and Pro pricing, 14-day refund, payments through Paddle). Everything measured below is from a local run on this machine: Python 3.13.5 and FFmpeg 7.1.5. I am not going to invent detection accuracy numbers, clip counts, or processing times for the hosted product.

This post builds the three steps so you can see what they cost in code, and where a hosted tool is simply doing the same jobs for you.

1. Download your own VOD, not someone else's weekend

Twitch keeps past broadcasts as video on demand for a limited time; the details live in Twitch's own Video on Demand help article. Highlights and uploads follow different rules. If the VOD has expired, no clipper can invent it.

For a file you have the right to download — typically your own broadcast — yt-dlp is the straightforward CLI. The project's usage pattern is:

yt-dlp -F "https://www.twitch.tv/videos/VIDEO_ID"
yt-dlp -o "vod.%(ext)s" "https://www.twitch.tv/videos/VIDEO_ID"

-F lists the formats the site is offering right now. I do that first because VOD availability and muxes change. Then I download with an output template so I do not inherit a 90-character title as a filename.

I am not going to run yt-dlp against a live Twitch URL in this article. That would mean copying someone else's stream or republishing a large copyrighted file I do not need. Use it on recordings you are allowed to keep. If you already have the file from your own encoder or console, skip this step and go to heuristics.

Two practical checks after the download:

• Duration matches what you remember streaming. A format that resolved to an unexpected audio-only or highlight-only file will waste every later step.
• You have either a single muxed file or a video-plus-audio pair you can join. Highlight code is easier when the timeline is one media file.

2. Two highlight heuristics, measured on purpose

Scrubbing a three-hour VOD by hand works until you do it every night. Automated "this might be a clip" suggestions usually come from one of two cheap signals:

1. Audio energy. A kill, a win, a yell, or a sudden crowd of game sound raises the loudness.
2. Chat rate. Viewers type more when something happens.

Neither signal knows what an ace is. Both can still point you at timestamps worth opening. The honest way to test them is not a marketing clip. It is synthetic data with known answers.

Audio peaks

I wrote an 8-second mono WAV (16 kHz, 16-bit) with three 440 Hz beeps at 1.5 s, 4.0 s, and 6.5 s, each 180 milliseconds long, on an otherwise silent track. Then I computed RMS energy in 100 ms windows and kept local maxima at least 0.6 s apart whose energy was at least two standard deviations above the mean.

def local_peaks(series, z=2.0, min_gap_s=0.6):
    values = [v for _, v in series]
    mean = sum(values) / len(values)
    var = sum((v - mean) ** 2 for v in values) / len(values)
    std = var ** 0.5
    threshold = mean + z * std
    candidates = [(t, v) for t, v in series if v >= threshold]
    kept = []
    for t, v in sorted(candidates, key=lambda x: -x[1]):
        if any(abs(t - kt) < min_gap_s for kt, _ in kept):
            continue
        kept.append((t, v))
    return sorted(kept)

On this run the file produced 80 windows. The detector returned peak times at 1.5 s, 4.0 s, and 6.5 s — exactly the beep onsets — each matched within 150 ms. Peak RMS was 11123.8 on all three tones, which is what you want from identical beeps.

That is the best case: silence, then a clean event. Real VODs are messier. Music beds, constant game loudness, and a streamer who talks through every round will raise the baseline so a shout is no longer two standard deviations above anything. A per-stream threshold, or a comparison against a rolling median instead of the global mean, is the usual next patch. I left the global mean in this article because I can show it recover known timestamps without pretending it is production-ready on a Valorant VOD.

If you use this on a real file, export a few seconds on either side of each peak. The interesting part of a clip is often the setup, not the peak frame itself.

Chat spikes

Chat is a different sensor. It lags the play (people type after they react) and it piles up for reasons that have nothing to do with a highlight: a raid, a bot command, a giveaway. It is still useful, especially when the audio is already loud.

I built a 30-second series with a baseline of 2 messages per second and three crowded seconds at t = 8, 17, and 24, each with 18 messages. Second 9 has a hangover of 7 messages, which is how real chat decays after a spike.

def chat_spikes(counts, z=2.0):
    mean = sum(counts) / len(counts)
    var = sum((c - mean) ** 2 for c in counts) / len(counts)
    std = var ** 0.5
    threshold = mean + z * std
    return [(i, c) for i, c in enumerate(counts) if c >= threshold]

On this run the mean was 3.77 messages per second. The function flagged (8, 18), (17, 18), and (24, 18). Second 9, with 7 messages, stayed under the line.

That is the behavior I want: clear spikes pass, the decay after a spike does not become a second clip. If your chat log is stored as timestamps rather than per-second buckets, histogram it first. Many downloaders and chat archives already give you a timestamp per message; collections.Counter(int(t) for t in times) is enough to feed this function.

How I would combine them

A simple merge: take audio peaks and chat spikes independently, then keep a window if either signal fires and promote a window if both fire within a second or two. I did not invent a precision number for that merge, because I did not run it on a real VOD in this sitting. The table is the claim; the synthetic recoveries are the evidence I do have.

The full script is in code/highlight_heuristics.py next to this draft. It writes the WAV, prints the matched beep times, and prints the chat spikes.

3. Export 9:16 without guessing the crop filter

Most VODs arrive as 16:9. Shorts, Reels, and TikTok want 9:16. The mechanical version of that conversion is: crop a vertical window out of the widescreen frame, then scale to the size you need.

I generated a 3-second 1920×1080 color test pattern at 30 fps with FFmpeg's testsrc2, then ran:

ffmpeg -i src_16x9.mp4 \
  -vf "crop=ih*9/16:ih,scale=1080:1920" \
  -c:v libx264 -pix_fmt yuv420p \
  out_9x16.mp4

crop=ih*9/16:ih keeps the full height and a 9:16-wide slice from the center. scale=1080:1920 turns that slice into a 1080×1920 file. FFmpeg is doing the work; the filter graph is the entire creative decision if you accept a center crop.

ffprobe on the two files from this run:

Center crop is honest and often wrong. FPS gameplay keeps the crosshair in the middle, so a center 9:16 window is a reasonable default. A talking-head cam in the top-left, or an action that happens on the far side of a wide map, will get shaved off. The next upgrade is a crop origin you choose per clip (crop=w:h:x:y) or a face/body tracker. I am not going to fake a tracker demo here. If the subject is not in the middle, do not pretend a default crop is "vertical-ready."

For a real VOD you would first cut the highlight window (-ss / -to or a stream copy into a temporary file), then crop. Doing crop-and-scale on the entire three-hour file just to post one moment is how you discover how large intermediate files can get.

When I would run this myself, and when I would not

I would run this pipeline when I already have the VOD locally, I care about seeing the heuristic fire on timestamps I can audit, and I want a specific crop. It is also a good way to learn why automated clippers disagree with your memory: your favorite moment may have been quiet, or chat may have exploded two seconds late.

I would not run it on a night when I just want candidates from a match I played and I do not want to babysit FFmpeg. That is the hosted-tool case.

FragCut's twitch clip maker is the product I work on for that case. Under the current plans, the first match is analyzed free; you pay when you render clips. Creator is $19.99 a month or $199 a year and includes 30 matches a month. Pro is $49.99 a month or $499 a year with no match cap. Refunds are available for 14 days, and payment goes through Paddle.

I am not going to describe hidden model internals or quote processing-time claims here. If you use the hosted path, you are trading the three local steps above for a web workflow. If you use the local path, you keep every file and every timestamp in view. Both are valid. The bad version is downloading a VOD you do not have rights to, or posting a 16:9 kill without noticing that the vertical crop removed the actual play.

The code, the synthetic WAV, and the two test videos from this run sit in the code/ folder beside the draft. Re-run highlight_heuristics.py if you change the beep times; the printout should keep matching them within 150 ms. Re-run the FFmpeg pair if you change the filter, and read ffprobe before you trust the filename.

Face-shape apps usually hide the boring part: four ratios, a handful of thresholds, and a long list of faces that sit between two labels.

I wanted the boring part in the open. This post:

1. Runs MediaPipe Face Landmarker (478 landmarks — the current Tasks API successor to classic Face Mesh) on a synthetic plaster-bust render, not a photo of a person.
2. Computes the same four ratio families product face-shape tools talk about: length ÷ width, forehead ÷ cheeks, jaw ÷ cheeks, jaw angle.
3. Morphs that mesh and records where a transparent 7-class rule set flips — especially the oval ↔ oblong boundary and the square ↔ triangle confusion.

Every number below was measured in this run. The classifier in the repo is a demo rule set for the article, not Face Shape Haircut AI's production model. Disclosure: I work on that product; the live face shape detector is what we ship. This post is about the measurement geometry, and where brittle rules lie.

Environment

Python 3.13
mediapipe 1.2.2
opencv-python-headless 5.x
numpy, matplotlib
Face Landmarker model: face_landmarker.task (float16, Google storage)

Classic mp.solutions.face_mesh is gone in this MediaPipe build. The Tasks API is what you get now.

Why the input image is synthetic on purpose

I need a detector input that:

• Is obviously not a photograph of a real person (so the article can show mesh diagrams without a face photo).
• Still has enough "face-like" structure that Face Landmarker returns a full 478-point mesh.

make_synthetic_bust.py draws a matte oval head with soft shading, dark eye sockets, a nose bridge shadow, and a mouth line. No skin texture, no identity.

# make_synthetic_bust.py (trimmed) — full file is in /code
import cv2, numpy as np
H, W = 800, 480
img = np.zeros((H, W, 3), np.uint8)
cx, cy = W // 2, H // 2 - 40
yy, xx = np.mgrid[0:H, 0:W]
head = ((xx - cx) / 125) ** 2 + ((yy - cy) / 210) ** 2 <= 1
# ... plaster fill, soft light, abstract eye sockets, mouth line ...
cv2.imwrite("synthetic_plaster_bust.png", cv2.cvtColor(img, cv2.COLOR_RGB2BGR))

MediaPipe still returned 478 landmarks on that image. That surprised me the first time. It also means every downstream ratio is grounded in a real detector output, not hand-drawn points.

Measuring the four ratios

I used the public Face Mesh topology indices:

def ratios_from_landmarks(lm, w, h):
    top, chin = xy(lm, 10, w, h), xy(lm, 152, w, h)
    cl, cr = xy(lm, 234, w, h), xy(lm, 454, w, h)
    fl, fr = xy(lm, 54, w, h), xy(lm, 284, w, h)
    jl, jr = xy(lm, 172, w, h), xy(lm, 397, w, h)
    length = np.linalg.norm(chin - top)
    cheek_w = np.linalg.norm(cr - cl)
    forehead_w = np.linalg.norm(fr - fl)
    jaw_w = np.linalg.norm(jr - jl)
    # jaw angle = mean of angles at jaw points (cheek→jaw→chin)
    ...
    return length/cheek_w, forehead_w/cheek_w, jaw_w/cheek_w, jaw_angle_deg

On the synthetic bust, Face Landmarker gave:

Under the demo rules (below), that lands on round. Visually the bust is a short-ish oval head — fair enough. The interesting part is what happens when we stretch it.

A transparent 7-class rule set (for probing, not shipping)

Production classifiers are usually messy ensembles. For this article I want something you can argue with:

• Oblong if length/width ≳ 1.45 and the top-to-bottom widths are similar.
• Oval if length/width sits roughly 1.30–1.45 with balanced widths.
• Round if length/width is short (< ~1.28) with full cheeks.
• Square if the jaw is wide and the angle is sharper.
• Heart if the forehead is wide and the jaw is narrow.
• Diamond if forehead and jaw are narrow relative to the cheeks.
• Triangle if the jaw is clearly wider than the forehead.

When two scores are close, the code records a leaning label. That matches how "between two shapes" shows up in real faces.

Again: this is not the production face shape detector. It's a ruler for finding brittle edges.

Experiment 1: stretch only the length

I took the detected 478 points and scaled every landmark's vertical distance from the mid-face by a factor s from 0.90 to 1.80, leaving horizontal structure alone. Forehead÷cheek and jaw÷cheek stayed at 0.809 / 0.818 the whole way. Only length ÷ width moved.

Measured transitions:

The oval ↔ oblong gate in these rules is 1.45. The flip happened at scale 1.18, where length ÷ width crossed from 1.4269 → 1.4515.

The misclassification that matters

Between scale 1.06 and 1.16, the demo rules did not say oval. They said diamond leaning oval/oblong.

Why? Because the synthetic bust's forehead and jaw were already narrow relative to the cheeks (0.81). When you only add length, you enter the oval/oblong length band while still looking "narrow top and bottom" to a diamond rule. The label flips through diamond as an artifact of a length-only morph.

That's the first boundary bug: a length threshold and a width-ratio rule can fight each other. A face that is simply "a bit longer than average" with slightly narrow forehead/jaw gets called diamond until length finally overwhelms everything else.

Experiment 2: directed morphs toward each class

I also pushed the same mesh toward each of the seven labels with combined length and width-band morphs. Measured results:

Two failures are useful:

1. toward_oval → diamond leaning oval. I only lengthened the mesh into the oval band. I didn't widen the forehead/jaw toward balance. The diamond rule still fired. "Make it oval" is not a single-axis edit.
2. toward_triangle → square leaning triangle. Wide jaw + narrow forehead should be triangle, but a sharp jaw angle also scores square. On this morph the square score won, with triangle as the lean. Square ↔ triangle is a real boundary; a single winner label is dishonest here.

Heart and diamond morphs that also moved length into the 1.37–1.40 band behaved. Oblong and round were easy once length was extreme enough. The fragile middle is the point.

Running it yourself

python -m venv .venv && source .venv/bin/activate
pip install mediapipe opencv-python-headless numpy matplotlib
# download face_landmarker.task into code/models/
python make_synthetic_bust.py
python face_ratios.py

face_ratios.py writes results.json, the mesh plot, and both charts. Full source is next to this post under code/.

What I would not conclude

• These thresholds are not universal anthropometry. They're a probe.
• A synthetic bust is easier than a real selfie: no glasses, no hair covering landmarks, no expression.
• MediaPipe's landmark indices are stable; your ratio definitions (which point counts as "forehead width"?) will move the numbers. Document the indices.
• Production systems that show a "leaning" second shape are doing the user a favor. A hard seven-way argmax is where support tickets come from.

Why this matters for a face shape detector

If you only ship a single label, boundary faces get a confident wrong answer. If you ship the four ratios next to the label — length ÷ width, forehead, jaw, jaw angle — the user can see why the model hesitated.

That's the product stance I care about: measure first, label second, and admit when two shapes are close. The live tool is the face shape detector; this post is the geometry underneath, run on a synthetic mesh so the numbers stay reproducible.

If you re-run the sweep on a real selfie (your own), I'd be curious where your oval ↔ oblong flip lands, and whether diamond shows up as a false bridge the way it did here.

So this post goes one step past "speech to text." It takes Reel video files you already have, transcribes them with word-level timestamps, and pulls out the hook, the call to action and the speaking pace. I tested it on three short ad-style clips with three Whisper model sizes. The smallest model made a mistake I wouldn't have caught by skimming, and the largest one broke my first version of the code.

Where the video files come from

I'm not covering downloading Reels here. Instagram doesn't offer a public download API, scraping it means logging in with an account, and that conflicts with its terms. The files this script expects are ones you're entitled to have: your own Reels, UGC videos creators deliver to your brand before posting, or clips from a client's account that they handed over.

For reproducible tests I made three 16-second vertical clips myself. I wrote three short ad scripts (a water bottle, sunscreen tips, running shoes), voiced them with the open-source Piper TTS engine, and wrapped the audio in a 1080×1920 video with ffmpeg:

echo "$SCRIPT" | piper -m en_US-lessac-medium.onnx -f bottle.wav
ffmpeg -f lavfi -i color=c=0x223344:s=1080x1920:r=30 -i bottle.wav \
  -shortest -c:v libx264 -tune stillimage -c:a aac -b:a 128k reel_bottle.mp4

Synthetic voices are cleaner than a creator talking over music, so treat the accuracy here as a best case. The point of the clips is that I know exactly what was said in each one, which makes it possible to grade the output.

Transcribing with word timestamps

I used faster-whisper, a CTranslate2 reimplementation of Whisper that runs well on CPU with int8 weights. The one setting that matters here is word_timestamps=True. Segment-level timestamps tell you roughly when a sentence was spoken; word-level ones let you say "the ask starts at 12.5 seconds."

import json, re, subprocess, sys, time
import numpy as np
from faster_whisper import WhisperModel

CTA = re.compile(r"\b(comment|save this|send this|link in (my )?bio|follow for|"
                 r"tap the link|dm me|share this)\b", re.I)
HOOK_SECONDS = 3.0

def load_audio(video):
    """ffmpeg -> 16 kHz mono float32 in memory; no temp files, no PyAV."""
    raw = subprocess.run(["ffmpeg", "-nostdin", "-v", "error", "-i", video, "-vn",
                          "-ac", "1", "-ar", "16000", "-f", "f32le", "-"],
                         capture_output=True, check=True).stdout
    return np.frombuffer(raw, np.float32)

def transcribe(model, video):
    segments, info = model.transcribe(load_audio(video), word_timestamps=True,
                                      vad_filter=True, beam_size=5)
    words = [w for s in segments for w in s.words]
    return words, info

load_audio exists because of an error. faster-whisper decodes files with PyAV, and the PyAV version in my environment rejected an argument faster-whisper passes (metadata_errors), so every call failed before transcribing anything. Piping raw float32 samples out of ffmpeg and passing the NumPy array directly avoids PyAV entirely. It also handles any container ffmpeg can read, which matters when people send you .mov files from iPhones.

Pulling out the structure

The regex is a short list of the calls to action Reels actually use. Very few say "buy now"; the common ones are engagement asks like comment a word, save this, send this to someone, or link in bio. You'll want to extend the list for your niche.

My first version split the transcript into sentences at periods and question marks, then flagged any sentence that matched the regex. It worked with the tiny model. With the larger ones, it reported the running-shoes CTA as starting at 6.3 seconds (base.en) and then at 0.0 seconds (small.en), because those models returned the whole clip as one long sentence held together with commas. Whisper's punctuation is a guess, and bigger models don't guess the same way.

The version that works ignores sentences. It builds the text word by word, records which word every character came from, and maps the regex match back to that word's timestamp:

def structure(words):
    text, owner = "", []
    for i, w in enumerate(words):
        text += w.word
        owner += [i] * len(w.word)
    hook = "".join(w.word for w in words if w.start < HOOK_SECONDS).strip()
    m = CTA.search(text)
    cta_at = words[owner[m.start()]].start if m else None
    cta = text[m.start():].strip() if m else ""
    dur = words[-1].end - words[0].start
    return {"hook_first_3s": hook, "cta": cta,
            "cta_starts_at": round(cta_at, 1) if m else None,
            "words": len(words), "speech_seconds": round(dur, 1),
            "words_per_minute": round(len(words) / dur * 60),
            "transcript": text.strip()}

Results

All runs on an 8-vCPU cloud machine, CPU only, int8. Processing time is per clip and doesn't include loading the model.

After the fix, all three model sizes find the same CTA in every clip, and their start times agree within half a second. The hooks came out as the full opening line in each case ("Okay, stop scrolling if your water bottle smells weird."), which is a useful check in itself: if your first sentence doesn't fit in three seconds at normal pace, it isn't a hook.

Pace was the number I found most useful. All three scripts sit around 190–230 words per minute, well above typical conversation. That's normal for short-form video, and it's also why Reels without captions lose people.

The mistake that mattered

tiny.en is the fastest, and on structure it was as good as the others. But in the running-shoes clip, the line "I'm not a doctor, so talk to someone if it's bad" came out as:

I met a doctor, so talk to someone if it's bad.

One wrong word flips a disclaimer into a claim. If you're transcribing a creator's video to check it against your brand's health or legal guidelines, that's the exact sentence you needed to get right. base.en and small.en both got it right. My rule after this: use tiny for bulk research where you only want structure, and something bigger whenever the transcript itself will be read for compliance or quoted back to a creator.

The other differences were cosmetic: "Ok" versus "Okay," "4 months" versus "four months," and "1." versus "One," for the numbered list. Normalize numbers before you compare transcripts across models, or you'll count those as errors.

When you don't have the file

All of this assumes the video is on your disk. For research on public Reels from other brands, you usually just have a link. That's the case where a hosted tool is easier: UGCfy AI runs a free instagram reel transcript generator that takes a public Reel URL. It reads the captions when the Reel has them and falls back to transcribing the audio when it doesn't. There's no signup, with a small daily cap on the free tier. You can then paste that text into the structure() function above and get the same hook, CTA and pace breakdown.

What I'd add next

• A second pass over the hook that flags the common formats (question, "stop scrolling," a numbered promise like "three things") so you can count which ones a competitor leans on.
• Silence detection between hook and body. A long pause right after the hook is usually a cut, and cuts are worth counting.
• Running the CTA regex on the on-screen caption text as well, because a lot of Reels put the ask on screen and never say it.

The transcript is the raw material. The timestamps are what make it something you can compare across fifty videos.

This is where AI clip generation tools are starting to make a real difference.

The editing bottleneck

Most gaming content isn't actually that hard to record. The hard part is turning raw footage into something worth watching. You need to identify the moments that matter — the clutch plays, the funny fails, the close calls — cut the dead time between them, pace the edits so the video feels energetic, and add captions and music.

For a two-hour session, this takes two to six hours depending on skill level. For full-time creators, that's just the cost of doing business. For part-time creators with jobs, families, or school, it's often the thing that kills the habit.

What AI clip generation does

Tools like FragCut are designed specifically for this problem. The AI analyzes your footage and automatically identifies the interesting moments — kills, deaths, streaks, comebacks, high-action sequences — then assembles them into a polished highlight reel.

It's not a raw clip dump. The AI handles pacing, trims dead time before and after each moment, and orders clips for impact. Some tools add captions and background music automatically. What a skilled editor would spend an hour producing can be done in minutes.

Who this is actually for

The technology has a clear sweet spot.

Part-time creators with full-time jobs are the obvious fit. "I'll post when I have time to edit" becomes "I can post tonight." Streamers who want to repurpose three-hour streams into 90-second TikToks get a workflow that was previously either expensive to outsource or impossible to sustain solo. New creators who aren't sure which moments are worth keeping get a useful starting point.

Full-time professionals who want complete creative control over every cut will still edit manually. AI tools aren't trying to replace them.

The consistency advantage

The biggest benefit isn't the time savings on any single video — it's what consistent posting does over time.

Algorithms reward regular uploads. Audiences expect them. But "post consistently" is advice that's easy to give and hard to follow when every video requires four hours of editing. When the time from "finished playing" to "posted clip" drops to twenty minutes, posting three times a week becomes feasible for someone who couldn't manage once a week before.

For creators serious about growing their presence, that's the real value.

Getting started

If you've been putting off posting because editing feels like too much work, the barrier to entry with AI tools is low. Most let you start with just an upload.

The time savings become obvious immediately.