Designing the Catalog

For the last decade, we have optimized our design logic for human eyes. We build beautiful case studies, polished mockups, and pixel-perfect prototypes. We rely on the visual fidelity of these artifacts to communicate our intent to engineering teams. But in the agentic era, the primary consumer of your design work is entirely different, and it operates under a vastly different set of cognitive rules.

To help autonomous agents build software reliably, we have the opportunity to move beyond treating design as a static visual export, and elevate it to programmable infrastructure.

The Illusion of Computer Vision

When we hear the phrase “computer vision,” it is a comforting metaphor. It implies the machine looks at a Figma mockup and intuitively understands the visual hierarchy, the grouping of elements, and the brand aesthetic.

In reality, AI doesn’t have eyes. What a multimodal model actually “sees” is a massive grid of floating-point values representing colors, borders, margins, and mathematical probabilities. It has to perform incredibly complex computations just to guess that a cluster of blue pixels might be a primary Submit button. As design leader Ryan Rumsey recently pointed out, if you hand an AI a picture of a user interface, you are giving it a puzzle to solve. If you hand it explicit text and structured data, you are giving it your answer key.

Relying solely on images creates a translation gap. AI coding tools operating without explicit structured constraints can suffer from “design blindness,” generating code that functions technically but invents new padding scales or drifts off-brand. To unlock reliable generation, we get to evolve how we deliver our work.

Generative UI and the Return of SDUI

It is computationally expensive for an AI agent to invent a UI component from scratch. Every decision regarding border radius, color hex codes, and font weights burns tokens and introduces the risk of errors. However, referencing a component that already exists in a structured format is highly efficient.

This architectural shift mirrors the rise of Server-Driven UI (SDUI) from the mobile engineering era. In SDUI, the backend server dictates the structural intent using a payload of structured data, and the client application dynamically assembles the interface using pre-approved, hardcoded components. The server knows nothing about CSS or pixels; it only knows the declarative rules.

Today, generative AI operates as the ultimate dynamic server. Frameworks like Vercel’s json-render and Google’s A2UI operate on this exact premise. Instead of agents writing custom syntax for every interaction, the application maintains a strict catalog of trusted UI components.

We are already seeing this programmable design infrastructure in action. As demonstrated recently by Google Cloud’s Prashanth Subrahmanyam, pairing an agentic IDE like Google Antigravity with a design automation tool like Google Stitch fundamentally changes the workflow. By connecting to the Stitch MCP server, the Antigravity coding assistant can securely interact with your live design system. The agent doesn’t just guess at aesthetics; it actively queries your theme tokens, component libraries, and layout rules directly into its context to build the UI.

The AI processes natural language prompts, but it is strictly guardrailed to outputting declarative JSON payloads that call upon your specific catalog. You define the system constraints; the AI generates the layout within them.

{
  "type": "MetricCard",
  "props": {
    "title": "Monthly Recurring Revenue",
    "value": "$124,000",
    "trend": "positive",
    "action": "open_report"
  }
}

By abstracting the design system into a machine-readable schema, the agent never has to guess what a MetricCard should look like. It only needs to determine when to use it and what data to pass into it.

Curating the Machine-Readable Catalog

How do you effectively communicate this catalog and your broader design intent to an autonomous worker? The answer is curating a UX Context Manifesto.

In an agentic workflow, the visual mockup is relegated to a reference for human stakeholders; plain-text Markdown files and JSON schemas act as the source of truth for the machine. Instead of painstakingly annotating wireframes with paragraphs of interaction behaviors, Intent Architects maintain lightweight configuration files—such as DESIGN.md or ux_context.md—that live directly inside the project repository.

When an agent initializes, it ingests these files to form its baseline reality. A robust DESIGN.md explicitly codifies your taste and systemic rules:

# UI Catalog & Brand Context

**Component: PrimaryButton**
*   **Description:** Use for the primary call-to-action on any given view.
*   **Props:** `label` (string), `isLoading` (boolean), `icon` (string)
*   **Aesthetic Rules:** NEVER use generic system fonts or hallucinated hex codes. Always use `var(--color-primary-500)` for backgrounds.

By aligning your component naming conventions with the statistical probability of the model’s training data (e.g., using <article> or PrimaryButton rather than obscure internal names), you create desire paths for the agent. The AI flows through your logic effortlessly because your intent is explicit.

The UX Pull Request

Providing structured context is only half the equation; the other half is validation. When an agent generates a complex interface based on your catalog, reviewing thousands of lines of raw syntax isn’t a great use of human talent. This is where we can leverage the UX Pull Request.

As we transition from Operators to Editors, we can utilize tools that provide visual, three-way diffs. A proper UX Pull Request presents the original state of the application, the documented JSON intent, and a live render of the AI’s new implementation side-by-side. It automatically audits the Accessibility (A11y) tree and verifies that the agent adhered to the design tokens defined in your context files.

AI can fix the padding, but it cannot fix the logic. That remains your domain. By treating design as programmable infrastructure, we move from drawing screens to directing the system, scaling our judgment infinitely across the agentic workforce.