Building an API for a product that has none: reliability engineering for browser automation

NotebookLM is one of my most-used AI tools, and it has no public API. If you want it inside an automated workflow — add sources, ask questions, collect answers — there is exactly one road: browser automation.

So I built an MCP server that drives the real NotebookLM page with Playwright, wraps "add source", "ask", and "extract answer" into callable tools, and shipped it to PyPI.

Version one worked after a weekend. Then I spent the next two releases fixing a single category of problem: it would return things that looked entirely plausible and were wrong.

Why browser automation errors are uniquely poisonous

Ordinary API integrations fail honestly: 4xx, 5xx, timeouts. You see them; you handle them.

Browser automation is different. You are scraping a dynamic page designed for humans, and its failure mode is "grabbed something — just not the thing you wanted":

All three look like 200 OK to the caller. And if the caller is another AI agent, it will happily keep reasoning on top of the wrong content — the error does not stop here; it propagates.

Failure one: an answer only counts when it is consecutively stable

AI answers stream; the DOM node keeps growing. When is it "done"?

Waiting a fixed number of seconds is the intuitive answer and the wrong one: short answers overwait, long answers underwait. The correct signal is not time — it is stability:

# Poll the answer node: content must be identical
# across 3 consecutive checks to count as complete
if text == last_text:
    stable_answer_checks += 1
else:
    stable_answer_checks = 0    # still changing — reset
    last_text = text

if has_changed_answer and stable_answer_checks >= 3:
    return text                 # new, and stable

Both conditions are required. "Stable" blocks partial answers. "New" blocks stale ones — record the reply already on screen before asking, and require the harvested content to differ from it. Only content that changed and then stopped changing is this question's complete answer.

Failure two: detecting UI pollution

The third class is the most insidious. NotebookLM's workspace is full of feature entry points — AI slides, flashcards, mind maps, Audio Overview — and when a selector starts matching too wide a node after a UI update, those button labels get stitched into an "answer".

It is not garbage bytes. It is a string of real words that fails to mean anything. A human spots it instantly; a program cannot.

My fix is a signature library: list the workspace's control-element strings and feature names as markers, and count hits in the extracted content:

def detect_answer_ui_pollution(text: str) -> list[str]:
    """Detect workspace UI captured as if it were answer text."""
    normalized = re.sub(r"\s+", " ", text).strip().lower()
    control_hits = [m for m in _ANSWER_UI_CONTROL_MARKERS if m in normalized]
    feature_hits = [m for m in _ANSWER_UI_FEATURE_MARKERS if m in normalized]

    # >=2 control hits, or >=4 feature hits: this is interface, not an answer
    if len(control_hits) >= 2 or len(feature_hits) >= 4:
        return [*control_hits, *feature_hits]
    return []

The thresholds are the design: a real answer occasionally mentions "mind map" once (the sources happen to discuss it), so a single hit cannot condemn it; but four feature names in one passage, or two pure control-element strings, is almost never natural-language prose.

What happens on detection? This was the most important decision in the project:

Return an explicit ANSWER_EXTRACTION_FAILED error instead of passing along whatever was scraped. Better the caller knows "nothing this time" than receives wrong content it will treat as true. For tools consumed by AI agents this principle cannot be overstated — agents have zero immunity against well-formatted wrong content.

Failure three: layered selector defense

With no API contract, the DOM is the contract — and the other side may change it at will. Selectors are not a "write them correctly" problem; they are a "how do they age" problem.

Every target element gets a fallback array, ordered most-specific to most-generic:

"sources_panel": [
    "section.source-panel",          # current real structure: precise
    '[data-testid="sources-panel"]', # test attribute: fairly stable
    '[aria-label*="Sources"]',       # a11y label: survives redesigns well
    '[aria-label*="來源"]',           # same layer, Chinese UI
    '[class*="sources"]',            # widest net: last resort
],

The ordering is the strategy. The top entries are precise but brittle — a redesign snaps them. The bottom ones live long but match wide — which is precisely one source of UI pollution. So layered selectors and pollution detection are a pair: wide selectors keep basic function alive across redesigns, and pollution detection catches what the wide selectors wrongly grab.

Two supporting rules: every optional element read gets a short timeout (a removed element must not stall the whole flow), and "page still loading" gets an explicit state — a loading notebook page naively reads as "an unnamed notebook with zero sources", which is again plausible-looking wrong data.

Verification: 133 tests, plus a real signed-in run

Testing such a project has one peculiar difficulty: you cannot mock NotebookLM. Unit tests cover the parsing logic — pollution detection, stability judgment, error classification — but "do the selectors still match the real page" has exactly one verification method: a real account, real notebooks, real long streaming answers.

So every release passes two layers before it ships: 133 unit tests, then a live pass against a signed-in NotebookLM — health check, notebook metadata, source counts, long-answer extraction, citation extraction. All green, or no release.

If you are wrapping a product with no API

Five principles, in order of importance:

One — distrust extracted content by default. Everything passes integrity checks (stable? new? not UI?) before leaving the tool.

Two — prefer explicit failure. An error code always beats plausible-looking wrong content, doubly so when the caller is an AI.

Three — wait on stability signals, never fixed durations.

Four — layer your selectors, and accept that wide selectors must be balanced by pollution detection.

Five — real-environment verification is part of the release process, not a post-incident remedy.

Browser automation is always a temporary bridge — the day an official API ships, all this code can be thrown away. But the discipline — distrust what you scraped, fail loudly — transfers to every cross-system integration you will ever build.