CAD-AI Evaluation & Prompt Engineering Standards

🎯 Benchmark Metrics (MVP)

Note: "Kernel success" is measured by running the generated JS under Replicad/OpenCascade headless; failures throw.

🔄 Replicad API → LLM Format

Processing Pipeline

1. Start from gen/api/replicad/replicad-clean-with-jsdoc.d.ts
2. Strip compiler-only noise with regex:
   Copy

   /^.*\b(gp_|TopoDS_|BRep|Adaptor3d_|Handle_|Bnd_).*$/gm

3. Remove generics, private|protected, overload duplicates (keep max-arity)
4. Save each top-level export with its JSDoc into JSON chunks:
   Copy

   {
     "id": "drawRoundedRectangle",
     "signature": "export declare function drawRoundedRectangle(width: number, height: number, r?: number | { rx?: number; ry?: number; }): Drawing;",
     "jsDoc": "Creates the Drawing of a rectangle with rounded corners…"
   }

5. Persist array to gen/api/replicad/replicad-chunks.json (≈ 900 objects)

🗃️ Vector Store (RAG)

Database Setup

• Database: PostgreSQL 15 + pgvector extension
  Copy

  CREATE EXTENSION IF NOT EXISTS vector;

Schema

CREATE TABLE replicad_chunks (
  id text PRIMARY KEY,
  signature text,
  jsdoc text,
  embedding vector(1536)
);

Usage

• CLI build script (scripts/build-replicad-chunks.ts) inserts with INSERT … ON CONFLICT
• Runtime retrieval:
  Copy

  const top = await db
    .select()
    .from('replicad_chunks')
    .orderBy(sql`embedding <-> $1`, [queryEmbedding])
    .limit(8);

🧠 Prompt-side Memory Strategy

• Vector memory: Store (summary, embedding) for every completed assistant/cad message
• Sliding window: Keep last 4 messages verbatim
• Image captions: Base64 never sent; use CLIP caption + embedding
• Tool call summaries: 1-sentence description replaces raw JSON

📦 Package Setup

pnpm install pg pgvector           # DB client + vector helpers

The server must enable the pgvector extension once per database.

These standards are the source-of-truth for future evaluation harnesses and prompt refactors.