DL#169 — Data Files Plugin

Written for AI agents. See Log Methodology Note below for details.

Background

Jay projects often need structured data that doesn't come from an external CMS — team members, FAQ items, feature comparisons, changelog entries, pricing tiers. Today these are either hardcoded in jay-html or require a custom plugin with a service, contract, and component.

The markdown plugin solved this for content-heavy pages (one file = one page). Data files solve it for structured tabular data: one file = many items, with list views, item views, and per-item pages.

Related Design Logs

• DL#155 — Markdown plugin (file-based pages, loadParams, slug routing)
• DL#60 — Plugin system refinement and dynamic contracts
• DL#80 — Materializing dynamic contracts for agentic generation
• DL#156 — Keyed headless component props

Problem

1. No lightweight data source — structured lists (team, FAQ, features) require custom services and components. Too much boilerplate for simple data.
2. CSV/YAML/JSON is natural — designers and content editors already work with spreadsheets and structured files. No database, no API, no credentials.
3. Cross-referencing — an item in one list may reference an item in another list or a markdown file. Today there's no convention for this.
4. Schema discoverability — agents need to know the data shape to write templates. Without a schema file, the agent must read the raw data.

Design

Supported formats

Auto-detected by file extension:

Extension	Format	Notes
.csv	CSV with header row	All values are strings; nested JSON in cells supported
.yaml / .yml	YAML array of objects	Types inferred (string, number, boolean); nested objects supported
.json	JSON array of objects	Types inferred; nested objects supported
.jsonl	JSON Lines (one object per line)	Types inferred

Schema-as-contract

Each data directory requires a schema file — a .jay-contract that describes the data shape. The schema uses the standard contract format with meta fields for data-files-specific annotations. If missing, the plugin emits a validation error pointing to the generate-schema action which auto-generates one from the data file:

# content/team/team.jay-contract
name: team
description: Team members directory
tags:
  - tag: slug
    type: data
    dataType: string
    meta:
      slug: 'true' # marks this field as the slug/identifier
  - tag: name
    type: data
    dataType: string
  - tag: role
    type: data
    dataType: string
  - tag: photo
    type: data
    dataType: string
  - tag: bio
    type: data
    dataType: html-string # rendered HTML content — use {bio} binding (not escaped)
  - tag: manager
    type: sub-contract
    link: ./team.jay-contract # self-referencing link — a manager is another team member

Self-referencing schemas (a contract linking to itself) are valid — they express relationships within the same collection. Circular data (A.manager=B, B.manager=A) is detected and rejected at render time.

Why contract format: it's already understood by the compiler, agents, and the agent-kit. The meta field on ContractTag (already in the type system — meta?: Record<string, string>) carries plugin-specific annotations without extending the contract spec. Cross-references use the standard link field.

meta fields:

meta key	Value	Meaning
slug	"true"	This field is the item identifier for routing and lookup

Schema requirement

The schema contract file is always required. If missing, the plugin issues a validation error:

Error: Data file "content/team/data.csv" has no schema contract.
Create content/team/team.jay-contract to define the data shape.
Run `jay-stack run data-files/generate-schema` to auto-generate from the data file.
See: agent-kit/designer/data-files-usage.md

The plugin provides a generate-schema action that reads the data file and generates a base contract file. The agent can then refine it (add descriptions, mark the slug field, define references).

Schema vs component contracts: the schema contract (e.g., team.jay-contract) defines the data shape but is NOT used directly in jay-html. Agents must use the materialized component contracts (data-pages, data-list, data-item) generated by jay-stack agent-kit. If an agent imports the schema contract directly, the validator emits:

Error: "team.jay-contract" is a data-files schema, not a component contract.
Use the materialized component contracts instead:
  - data-pages (per-item pages): plugin="@jay-framework/data-files" contract="team-data-pages"
  - data-list (list view): plugin="@jay-framework/data-files" contract="team-data-list"
  - data-item (single item): plugin="@jay-framework/data-files" contract="team-data-item"
See: agent-kit/designer/data-files-usage.md

End-to-end workflow

Typical usage in a project (e.g., jay-website):

1. Fetch data — a pre-build script fetches data from any source (API, spreadsheet, database) and writes CSV/YAML/JSON files to a content directory
2. Define schema — create a .jay-contract file describing the data shape (or run generate-schema to auto-generate one)
3. Generate agent-kit — run jay-stack agent-kit to materialize the schema into component contracts (data-pages, data-list, data-item) per data file. This is how the designer agent discovers available fields.
4. Use in templates — import the data-files plugin in jay-html with the content directory and file as props
5. Build — the plugin reads the files at render time, resolves cross-references, and produces ViewState

project/
├── content/
│   ├── team/
│   │   ├── data.csv          # the data
│   │   └── team.jay-contract # the schema
│   └── faq/
│       ├── data.yaml
│       └── faq.jay-contract
├── scripts/
│   └── fetch-data.sh         # pre-build: fetches data from external sources
├── src/pages/
│   ├── team/
│   │   ├── page.jay-html     # list view (data-list)
│   │   └── [slug]/
│   │       └── page.jay-html # per-item page (data-pages)
│   └── faq/
│       └── page.jay-html     # FAQ list (data-list)

Three components

1. data-pages — per-item pages (like markdown-pages)

Each row becomes a page. Used with [slug] dynamic routes.

<head>
  <script
    type="application/jay-headless"
    plugin="@jay-framework/data-files"
    contract="team-data-pages"
    key="member"
  >
    contentDir: content/team
    file: data.csv
  </script>
</head>
<body>
  <div>
    <h1>{member.name}</h1>
    <p>{member.role}</p>
    <img src="{member.photo}" alt="{member.name}" />
    <div>{member.bio}</div>
  </div>
</body>

Props: contentDir, file.

The slug field is read from the schema contract (the tag with meta.slug: "true").

withLoadParams: reads the data file, yields { slug: row[slugField] } for each row.

withSlowlyRender: finds the row matching props.slug, resolves cross-references, returns all fields as ViewState.

2. data-list — all items as a list (keyed headless)

<head>
  <script
    type="application/jay-headless"
    plugin="@jay-framework/data-files"
    contract="team-data-list"
    key="team"
  >
    contentDir: content/team
    file: data.csv
  </script>
</head>
<body>
  <ul>
    <li forEach="team.items" trackBy="slug">
      <a href="/team/{slug}">{name} — {role}</a>
    </li>
  </ul>
</body>

withSlowlyRender: reads the file, resolves cross-references for all rows, returns as items array.

3. data-item — single item by slug (instance headless)

<jay:team-data-item slug="jane" contentDir="content/team" file="data.csv">
  <div class="card">
    <h3>{name}</h3>
    <p>{role}</p>
  </div>
</jay:team-data-item>

Cross-references via contract links

Cross-references and nested objects both use sub-contracts. The distinction between them is data-driven:

Inline sub-contract — always inline data

A sub-contract with tags: directly in the schema expects the data file to have a nested object:

# Schema
- tag: nutrition
  type: sub-contract
  tags:
    - tag: calories
      type: data
      dataType: number
    - tag: protein
      type: data
      dataType: number

# Data — inline object
- slug: carbonara
  nutrition:
    calories: 450
    protein: 25

Linked sub-contract — reference or inline, determined by data value

A sub-contract with link: can be either a cross-reference (string slug) or inline data (nested object). The plugin checks the actual value type in the data file:

• String value → reference by slug — look up the item in the linked data file
• Object value → inline data — use directly, type-checked against the linked schema

# Schema
tags:
  - tag: author
    type: sub-contract
    link: ../team/team.jay-contract

# Data case 1: reference (string → resolve by slug from team data file)
- slug: carbonara
  author: jane

# Data case 2: inline (object → use directly)
- slug: carbonara
  author:
    name: Jane Doe
    role: Guest Chef

Both produce the same ViewState shape (defined by the linked contract). The resolution path differs based on the value type.

Full example

# content/recipes/recipes.jay-contract
name: recipes
tags:
  - tag: slug
    type: data
    dataType: string
    meta:
      slug: 'true'
  - tag: title
    type: data
    dataType: string
  - tag: author
    type: sub-contract
    link: ../team/team.jay-contract # linked → string=reference, object=inline
  - tag: nutrition
    type: sub-contract # inline → always expects nested object
    tags:
      - tag: calories
        type: data
        dataType: number
  - tag: relatedGuide
    type: sub-contract
    link: ../../docs/content.jay-contract # linked → string=reference to markdown

# content/recipes/data.yaml
- slug: carbonara
  title: Pasta Carbonara
  author: jane # string → resolved from team data file
  nutrition:
    calories: 450 # inline object → used directly
  relatedGuide: pasta-basics # string → resolved from markdown content

Resolved ViewState:

{
  "slug": "carbonara",
  "title": "Pasta Carbonara",
  "author": {
    "slug": "jane",
    "name": "Jane Doe",
    "role": "CTO"
  },
  "nutrition": {
    "calories": 450
  },
  "relatedGuide": {
    "title": "Pasta Basics",
    "content": "<p>...</p>"
  }
}

This is cleaner than a custom meta.ref mechanism — it reuses the standard contract linking that the compiler, agents, and agent-kit already understand.

Performance and large files

Data files can be large (thousands of items). Two concerns: lookup performance and circular references.

Indexed lookup: on first read of a data file, the plugin builds a Map<string, Row> keyed by slug. All subsequent lookups (cross-references, data-item by slug) are O(1). The index is cached per-file for the duration of the build.

Size limit: data files are for small/medium static datasets. The plugin validates file size and emits a warning above a threshold (e.g., 10,000 rows):

Warning: "content/products/data.csv" has 50,000 rows.
Data files are designed for small/medium datasets. For large catalogs,
use a CMS plugin with API-based pagination instead.
See: agent-kit/designer/data-files-usage.md

This prevents memory issues from loading large files into a Map and reinforces the architectural boundary between data files (static, full rebuild) and CMS plugins (dynamic, incremental updates).

Circular references are not allowed. During reference resolution, the plugin maintains a Set<string> of visited file:slug pairs. If a cycle is detected, emit a validation error:

Error: Circular reference detected: recipes/carbonara → team/jane → recipes/carbonara
Circular references are not supported in data files. Remove one of the references.

Build characteristics

Data files require a full route rebuild when data changes. Unlike CMS plugins (e.g., wix-stores) that can invalidate and rebuild a single item page, data-files reads the entire collection file at render time. Changing one row in data.csv triggers a rebuild of all pages that use that file.

This is a fundamental tradeoff:

Concern	Data Files	CMS Plugin
Data source	Local files (CSV/YAML/JSON)	External API
Update granularity	Full rebuild per collection	Single item invalidation
Credentials	None	API keys required
Setup complexity	Zero	Plugin setup + credentials
Best for	Small/medium static datasets	Large, frequently updated catalogs

This tradeoff must be documented in the agent-kit guide so agents choose the right plugin for the use case.

Type inference (for generate-schema action only)

The schema contract is always required and is the source of truth for types. Type inference is only used by the generate-schema action when auto-generating a base schema from a data file:

Source	Inferred types
CSV	All fields → string; nested JSON in a cell → sub-contract
YAML/JSON	string, number, boolean inferred; nested objects → sub-contract; arrays → repeated sub-contract
HTML content	Detected by < prefix → html-string dataType

After generation, the agent refines the schema — adding descriptions, marking the slug field, converting fields to html-string where appropriate, and defining cross-reference links.

Contract naming convention

Each schema generates three materialized component contracts. The names are derived from the schema's name field:

Schema name	Component	Materialized contract name
team	data-pages	team-data-pages
team	data-list	team-data-list
team	data-item	team-data-item

The designer uses these names in jay-html: contract="team-data-pages".

Multiple data files

Each data file needs its own schema contract. A directory can contain multiple data files with separate schemas:

content/people/
├── members.csv
├── members.jay-contract    # name: members
├── alumni.csv
└── alumni.jay-contract     # name: alumni

The schema and data file are associated by convention: the schema name field matches the data file name (without extension). The file prop in jay-html selects which data file to use.

Dev mode file watching

The data files plugin registers content files for dev server watching (same pattern as the markdown plugin). When a CSV/YAML/JSON file changes, affected pages re-render automatically.

Dynamic contracts

At jay-stack agent-kit time, the plugin:

1. Scans project for directories with data files + schema contracts
2. Generates three materialized contracts per schema ({name}-data-pages, {name}-data-list, {name}-data-item)
3. Attaches metadata: { contentDir, file } for the component to use

Questions

1. Should the plugin support nested YAML objects? Yes — nested objects become sub-contracts. Nested JSON in CSV cells is also supported.

2. Eager or lazy reference resolution? Eager — the designer gets {author.name} directly.

3. Filtering/sorting? Not at this stage.

4. Inline data component? No — file-based only.

5. Plugin name? @jay-framework/data-files.

6. Should circular references be detected? Yes — circular references are not allowed. The plugin detects cycles and emits a validation error.

7. Should the schema contract support params for the data-pages component? Only slug is supported as a param. Filtering, sorting, and categories require a CMS plugin, not flat data files.

Implementation Plan

Phase 1: Plugin scaffold

• Create packages/plugins/data-files/
• plugin.yaml with three contracts
• Package setup (vite config, tsconfig, package.json)

Phase 2: Data parsing

• lib/parse-data.ts — reads CSV, YAML, JSON, JSONL files
• Returns Array<Record<string, unknown>>
• Use papaparse for CSV (lightweight, handles quoted fields and nested JSON)
• Use js-yaml for YAML (already in the monorepo)

Phase 3: Schema loading

• lib/load-schema.ts — reads the .jay-contract schema file from the content directory
• Extracts slug field from meta.slug
• Identifies cross-references from linked sub-contracts (link: field)
• Validation: error if no schema file exists, error if slug field not marked

Phase 4: Components

• data-pages: withLoadParams + withSlowlyRender (follow markdown-pages pattern)
• data-list: withSlowlyRender returning items array
• data-item: withSlowlyRender finding one item by slug prop

Phase 5: Cross-references

• Reference resolution at render time
• Circular reference detection
• Markdown file reading (reuse @jay-framework/markdown parsing)

Phase 6: generate-schema action

• Plugin action: reads data file, generates base .jay-contract
• Auto-detects types, picks first candidate slug field
• Agent refines the generated contract

Phase 7: Agent-kit guide

• agent-kit/designer/data-files-usage.md — end-to-end workflow
• Document CSV/YAML/JSON format, schema contract, cross-references
• Examples: team page, FAQ, changelog, recipes with cross-references
• Document the pre-build script pattern for external data sources

Phase 8: Tests and verification

Fixture-based tests — all data is files, making tests self-contained:

test/fixtures/
├── team/                    # basic CSV
│   ├── data.csv
│   └── team.jay-contract
├── faq/                     # YAML with nested objects
│   ├── data.yaml
│   └── faq.jay-contract
├── recipes/                 # cross-references
│   ├── data.yaml
│   ├── recipes.jay-contract # links to ../team/team.jay-contract
│   └── ...
├── no-schema/               # missing schema → validation error
│   └── data.csv
├── circular/                # circular data → validation error
│   ├── data.yaml
│   └── circular.jay-contract
└── large/                   # >10K rows → size warning
    ├── data.csv
    └── large.jay-contract

• Data parsing: all four formats (CSV, YAML, JSON, JSONL), nested objects, type detection
• Schema loading: slug extraction, linked sub-contracts, missing schema error
• Components: data-pages loadParams + slowRender, data-list slowRender, data-item slug lookup
• Cross-references: string→resolve, object→inline, self-referencing schema, circular data detection
• Validation: missing schema, missing slug field, size limit warning, schema-vs-component-contract misuse
• yarn confirm

Trade-offs

Choice	Pro	Con
Schema always required	Discoverable, type-safe, agent-friendly	One extra file per data source
meta on contract tags	Uses existing contract spec, no extensions	Less visible than a dedicated config
File-based data	Zero infrastructure, designer-friendly	Not suitable for large/dynamic datasets
Eager reference resolution	Simple for designer	Larger ViewState, needs cycle detection
Four formats (CSV/YAML/JSON/JSONL)	Covers all common sources	More parsing code

Implementation Results

What was implemented

Plugin scaffold (packages/plugins/data-files/):

• package.json, plugin.yaml, vite.config.ts, tsconfig.json
• No external CSV parser — built-in CSV parsing (split-based, handles simple CSVs)
• Dependencies: js-yaml only (already in the monorepo)

Data parsing (lib/parse-data.ts):

• Four formats: CSV, YAML, JSON, JSONL
• Extension validation before file read
• buildSlugIndex for O(1) lookups

Schema loading (lib/load-schema.ts):

• Reads .jay-contract schema from content directory
• Extracts slug field from meta.slug: "true"
• Validation errors for missing schema and missing slug field
• Row count validation (10K threshold)

Three components (lib/components/):

• data-pages: withLoadParams yields slugs, withSlowlyRender finds row by slug
• data-list: withSlowlyRender returns all rows as items array
• data-item: withSlowlyRender finds one row by slug prop

Cross-reference resolution (lib/resolve-references.ts):

• Linked sub-contracts: string value → slug reference (resolved from linked data file), object value → inline data
• Inline sub-contracts: always inline nested data
• Circular reference detection via visited Set<string>
• Depth cap (10 levels)
• File cache (Map) for indexed lookups within a build

Dynamic contract generation (lib/contract-generator.ts):

• Three async generators: generateDataPagesContract, generateDataListContract, generateDataItemContract
• Scans content/ for directories with schema contracts
• Generates materialized contracts with actual field tags from schema
• Listed in plugin.yaml under dynamic_contracts

generate-schema command (lib/generate-schema.ts):

• Reads data file, infers types (string, number, boolean, html-string, sub-contract)
• Picks slug field from common candidates (slug, id, key, name)
• Handles nested objects → sub-contract, arrays → repeated sub-contract
• Outputs .jay-contract YAML ready for agent refinement
• Listed in plugin.yaml under commands

Agent-kit guide (agent-kit/designer/data-files-usage.md):

• End-to-end workflow, data formats, schema contract syntax
• Three components with examples
• Contract naming convention
• Cross-references, nested objects, limitations
• Pre-build script pattern for external data

Contracts (lib/contracts/):

• Base contracts for data-pages, data-list, data-item

What was deferred

• Markdown file reference resolution — refMarkdown links to markdown content not yet implemented. Would reuse @jay-framework/markdown parsing.
• Dev mode file watching — data file changes don't trigger automatic re-render yet
• Schema-vs-component-contract validation — detecting when an agent imports the schema contract directly instead of the materialized component contract

Tests

23 tests, all passing:

• Data parsing: CSV, YAML (flat + nested), JSON, JSONL, unsupported format
• Slug index: building, empty slug handling
• Schema loading: valid schema, missing schema, slug detection, linked sub-contracts
• Cross-references: passthrough fields, inline nested, linked reference resolution, unresolved reference, circular detection
• generate-schema: CSV inference, YAML type inference, nested objects, JSON, slug field picking

---

Log Methodology Note

Note: These design logs are written primarily for AI agents as part of the Design Log methodology and made accessible here for human readers. The language and structure are optimized for machine consumption — expect precise, specification-style prose rather than narrative documentation.