AIditorEcosystemDocumentationDesign Log
GitHubGet Started

Sitemap Generation And Robots Validation

Design Log #175 — Sitemap Generation and robots.txt Validation

Written for AI agents. See Log Methodology Note below for details.

Background

Jay Stack projects serve static files from public/ — Vite serves them in dev, and the build pipeline copies them to the frontend root for production (DL#134a). Files like robots.txt can be placed in public/ and work in both environments.

However, sitemap.xml needs to list all concrete page URLs. The framework already enumerates these during production builds (route scanning + loadParams + cross-product + dedup), but this capability isn't exposed as a standalone feature.

Currently there is no validation that robots.txt or sitemap.xml exist, and no tooling to generate a sitemap.

Production dynamism

In production, pages aren't static — the slow render server (DL#134c) can rebuild instances when data changes. When a new product is added or removed, POST /_jay/rebuild creates or updates instances, and the route-manifest.json is atomically rewritten with the new instance list. build-metadata.json is also updated, triggering the main server to reload.

This means a build-time sitemap goes stale whenever instances change. The route-manifest.json is the live source of truth for what pages exist.

Problem

  1. No sitemap generation — Projects must manually create and maintain sitemap.xml, which is error-prone as routes change
  2. Sitemap staleness — Even if generated at build time, the sitemap becomes stale when instances are added/removed via rebuild
  3. No robots.txt validation — Missing robots.txt is an SEO gap that goes undetected
  4. No site.baseUrl validation — If configured, it should be validated; if missing, sitemap features should warn

Design

Part 1: Dynamic sitemap served by the main server

Instead of a static public/sitemap.xml, the main server serves /sitemap.xml dynamically from the route manifest. This way it's always in sync with the live set of pages.

How it works

The main server already loads route-manifest.json and reloads it when build-metadata.json changes. Serving the sitemap means:

  1. Register a handler for GET /sitemap.xml
  2. On manifest load/reload, generate sitemap.xml to a temp file in the build directory, then rename into the frontend directory
  3. On GET /sitemap.xml, serve the file (or let the static file handler serve it)

Streaming and caching

Sitemaps can be large (thousands of URLs). Instead of building an XML string in memory:

  • Write to disk on manifest load/reload — generate sitemap.xml as a file in the frontend directory (where static files are served from)
  • The existing static file handler serves it — no special response logic needed
  • On rebuild, regenerate the file (atomic write via temp + rename)

Base URL

Required config in jay-stack.config.yaml:

site:
  baseUrl: https://example.com

If not configured, /sitemap.xml is not generated.

noindex in manifest

Add a noindex: true flag to RouteEntry in the route manifest. Populated at build time by statically checking the page's jay-html for <meta name="robots" content="noindex">. Only static noindex is supported — dynamic noindex (from ViewState bindings) is not detected.

Routes with noindex: true are excluded from the sitemap.

Where to detect: During the build pipeline, after parsing each page's jay-html, check headMeta for a robots meta tag containing "noindex". Store the flag on the RouteEntry.

Filtering

  • Exclude routes with devOnly: true
  • Exclude routes with noindex: true

Dev server

The dev server returns a placeholder for /sitemap.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Sitemap is generated by the production server from the route manifest. -->
<!-- Run jay-stack build && jay-stack serve to see the full sitemap. -->
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" />

This avoids the overhead of running loadParams in dev just for a sitemap, while still making /sitemap.xml resolve (useful for testing robots.txt references).

Part 2: jay-stack sitemap CLI command

For projects that want a static sitemap (e.g., static hosting without a main server), a CLI command that writes public/sitemap.xml. Uses the same URL enumeration pipeline as the build:

  1. scanRoutes() + plugin routes (exclude devOnly)
  2. loadParams() for dynamic routes
  3. crossProductParams() + materializeRouteParams() + dedupeByUrl() (all from param-routing.ts)

Writes as a stream to public/sitemap.xml. Optionally runs as part of jay-stack build if site.baseUrl is configured.

Part 3: Validation

robots.txt existence

Project-level warning in jay-stack validate when public/robots.txt is missing.

Where: In validate.ts, after the per-file validation loop but before plugin validators.

Severity: warning

Message:

⚠ public/robots.txt not found — search engines may crawl pages you don't intend to expose.
  Create public/robots.txt with at minimum:
  User-agent: *
  Allow: /
  Sitemap: https://your-domain.com/sitemap.xml

Where NOT to add: Not in the SEO plugin — robots.txt is a project-level concern, not per-page.

site.baseUrl configuration

Warning in jay-stack validate when site.baseUrl is not configured:

⚠ site.baseUrl not configured — sitemap.xml will not be generated.
  Add to jay-stack.config.yaml: site: { baseUrl: "https://your-domain.com" }

Implementation Plan

Phase 1: Validation (standalone, no dependencies)

  1. Add site.baseUrl to config schema (config.ts)
  2. In validate.ts, add project-level checks:
    • Warn if public/robots.txt missing
    • Warn if site.baseUrl not configured

Phase 2: noindex in manifest

  1. During build, detect static <meta name="robots" content="noindex"> from parsed jay-html headMeta
  2. Add noindex?: boolean to RouteEntry type
  3. Populate in build-pipeline.ts when building route entries

Phase 3: Sitemap generation in main server

  1. On manifest load/reload, generate sitemap.xml from routes × instances using buildUrl()
  2. Stream-write to frontend/sitemap.xml (atomic via temp + rename)
  3. Skip routes with devOnly or noindex
  4. Skip if site.baseUrl not in config

Phase 4: Dev server placeholder

  1. Add GET /sitemap.xml handler that returns the placeholder XML

Phase 5: jay-stack sitemap CLI command (for static hosting)

  1. Extract URL enumeration from build pipeline into reusable enumerateUrls()
  2. Add sitemap command to CLI
  3. Optionally integrate into jay-stack build

Verification Criteria

  1. jay-stack validate warns when public/robots.txt is missing
  2. jay-stack validate warns when site.baseUrl is not configured
  3. Production main server serves /sitemap.xml with all current page URLs
  4. After a rebuild adds/removes instances, /sitemap.xml reflects the change
  5. Pages with static <meta name="robots" content="noindex"> are excluded
  6. devOnly routes are excluded
  7. Dev server returns placeholder XML for /sitemap.xml
  8. jay-stack sitemap generates a valid public/sitemap.xml for static hosting

Log Methodology Note

Note: These design logs are written primarily for AI agents as part of the Design Log methodology and made accessible here for human readers. The language and structure are optimized for machine consumption — expect precise, specification-style prose rather than narrative documentation.