Feeding YouTube links straight to Gemini: one API call, and four things the docs will not tell you

The requirement was simple: the user pastes a YouTube link, the app returns a set of highlight cards — what the video covers, its key points, and where in the video each point starts.

The intuitive route is a three-stage pipeline: scrape the transcript → feed it to an LLM → structure the output. But transcript scraping is a swamp of its own: not every video has one, auto-captions are unreliable, and scraping methods attract platform defenses.

It turns out none of that is needed. Gemini's generateContent has a capability few people notice: the fileData field accepts a YouTube URL directly.

The entire core is one request

const res = await fetch(vertexUrl(env, model, 'generateContent'), {
  method: 'POST',
  headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({
    system_instruction: { parts: [{ text: systemPrompt }] },
    contents: [{
      role: 'user',
      parts: [
        { fileData: { fileUri: videoUrl, mimeType: 'video/*' } },  // ← this line
        { text: userPrompt },
      ],
    }],
    generationConfig: {
      temperature: 0.3,
      maxOutputTokens: 4096,
      responseMimeType: 'application/json',
      mediaResolution: 'MEDIA_RESOLUTION_LOW',
    },
  }),
});

Put the YouTube URL in fileUri, set mimeType to video/*, and the model goes and watches the video — visuals and audio both, no dependence on captions. No download, no transcoding, no transcript scraping. The whole pipeline is one HTTP request.

Getting it running took about ten minutes. The next four things are what the road from running to user-facing actually looked like.

One: an entire class of videos returns 403, with a cryptic message

Testing went smoothly until I fed it a music video: Vertex returned 403, and the error text contained the phrase "not owned".

The measured pattern is clean: copyright-protected videos (Content ID — music videos being the bulk) are rejected with 403 across the board; talks, tutorials, podcasts, and tech presentations passed without exception.

This must be handled at the product layer, because "403 not owned" is gibberish to a user. I translate that one specific error into a human sentence:

if (res.status === 403 && errText.includes('not owned')) {
  throw new Error('VIDEO_COPYRIGHT');
}
// Upstream, this becomes a user-facing message:
// "This video is copyright-protected and cannot be analyzed
//  (music videos etc.). Talks and tutorials are unaffected."

Note the condition is "403 AND the message contains not owned" — 403 alone is not enough, because misconfigured permissions are also a 403. Conflate the two and you will report your own configuration mistakes as "this video is copyrighted".

Two: mediaResolution is the main cost switch

Video input token consumption scales directly with how closely the model looks. The mediaResolution field in generationConfig controls how many tokens represent each sampled frame, and at the default resolution a full-length talk consumes a substantial prompt.

My use case is listening — for talks and tutorials the information density is in the audio track, while the visuals are mostly slides and a speaker. With MEDIA_RESOLUTION_LOW, measured output quality showed no perceptible difference: the key points are the same key points, and the timestamps stay accurate.

The cost difference, however, is multiples. If your use case is watching — UI walkthroughs, visual analysis — that is a different conversation; for listening-type content, LOW is close to a free discount. The field is easy to miss precisely because leaving it unset also "works". Only the bill is wrong.

On measuring cost: the response's usageMetadata carries promptTokenCount and candidatesTokenCount — log their sum and you know what each video actually costs. We also count this feature against the user's daily AI quota: video is the most expensive input type there is, and shipping it un-metered is an open invitation to strawman your own bill.

Three: not one field of the returned JSON can be trusted directly

The request sets responseMimeType: application/json, and the model returns legal JSON most of the time. In production, "most of the time" means "will break".

Variants I have actually seen: a response that is not JSON at all (parse throws), a field that should be a string arriving as an array, nulls mixed into cards, points arriving as one long string instead of an array. Any of these white-screens a naive client.

So the parsing layer is defensive normalization — every field coerced to its type, every absence given a default:

let data;
try {
  data = JSON.parse(result.text);
} catch {
  data = { title: 'YouTube video', tldr: '', cards: [], tags: [] };
}

// Normalize each field: strings are strings, arrays are arrays, nulls dropped
const asStr = (v) => Array.isArray(v) ? v.filter(x => x != null).join('\n')
  : v == null || typeof v === 'string' ? v : String(v);

data.cards = Array.isArray(data.cards)
  ? data.cards.filter(c => c && typeof c === 'object').map(c => ({
      heading: asStr(c.heading) || '',
      points: Array.isArray(c.points) ? c.points.filter(p => p != null).map(String) : [],
      timestamp: asStr(c.timestamp) || '',
    }))
  : [];

The principle: LLM output is always external input. responseMimeType raises a probability, not a guarantee; the final schema defense has to live in your own code.

Four: timestamps do not appear on their own — demand them in the prompt

Cards need to link back to the moment in the video — that is the heart of the experience. The model understands the video's timeline, but unless you ask explicitly it will not include timestamps — or it will, in a different format every time ("three minutes twenty", "3:20", "around the 200-second mark").

The fix is plain but necessary: define the format inside the output schema.

"cards": [
  {"heading": "Point title", "points": ["point 1", "point 2"],
   "timestamp": "mm:ss when this point begins"}
]
3-7 cards depending on video length; every point concrete
and informative — no filler.

Writing the format example directly into the schema description (mm:ss) works far better than explaining it in separate instructions. The same prompt also bounds the card count — leave it unbounded and short videos get padded to seven thin cards while long ones get crushed into three.

Where the call lives in the architecture

The last decision has nothing to do with the API and matters just as much: where this call sits.

The answer is a server-side gateway, not the app. Three reasons: the API key never ships (a key inside an app binary is a public key); quota and permissions are enforced server-side against the JWT rather than trusting the client; and the model and prompt can change any day without an app release.

The app sees one clean interface: POST a YouTube URL, receive structured cards or a human-readable error. Every piece of dirty work in this article stays inside the gateway.

If you are building the same thing

One — verify your video category passes first. Test with real videos of your target type; music and film content will 403, and launch day is the wrong time to learn that.

Two — measure cost before shipping: log usageMetadata, choose mediaResolution by use case, set quotas from day one.

Three — write the parsing layer as if consuming an untrusted external API: JSON parse with a fallback, every field normalized.

Four — pin the output format (timestamps included) inside the prompt's schema; do not leave it to the model's mood.

Five — keep the call server-side. Your key, your quota, and your prompt iteration speed will all thank you.