Brok's ForgeAI Engineering
Docs
GitHubHome
All documentation

Introduction

  • What is Broks Forge?
  • The AI Engineering Operating System
  • Getting Started
  • The Five Layers

Core Concepts

  • Core Concepts
  • Engineering Intelligence
  • Engineering Memory
  • Knowledge
  • Why Observability Is Not Enough
  • Deterministic Engineering Reasoning

Capabilities

  • Registry
  • AI Git
  • Forge Graph
  • Execution Graph & Failure Graph
  • Evolution
  • Brok — the Engineering Partner
  • Root Cause Explorer
  • Evaluations & Metrics

Working With Broks Forge

  • The Engineering Workflow
  • Examples
  • Best Practices

Comparisons

  • Comparisons Overview
  • Broks Forge vs LangFuse
  • Broks Forge vs LangSmith
  • Broks Forge vs Promptfoo
  • Broks Forge vs Helicone
  • Broks Forge vs Weights & Biases

Developer Documentation

  • Architecture Overview
  • Data Model
  • REST API
  • Module Structure
  • Extension Points
  • Developer Setup & Build
  • Engineering Principles

Reference

  • FAQ
  • Glossary

Engineering Handbook

  • Master Architecture
  • Engineering Handbook
  • Developer Guide
  • Project Rules
  • Coding Standards
  • API Guidelines
  • Security
  • Error Handling
  • Testing Strategy
  • Performance
  • Deployment
  • Contributing
  • Roadmap
Docs/Developer Documentation

Data Model

Broks Forge stores a small number of real tables and derives everything else on read. Understanding that split is understanding the platform.

Stored entities

All persisted, Flyway-migrated, and scoped by organization (and usually project).

Identity and tenancy

EntityNotes
usersAccounts, credentials, verification state
organizationsThe top-level tenant
organization_membersMembership with a role: OWNER, ADMIN, MEMBER, VIEWER
projectsA workspace inside an organization
api_keysProgrammatic access, hashed at rest

Artifacts

EntityVersionedNotes
agents✅ agent_versionsRegistered by HTTP endpoint; framework and language are metadata
prompts✅ prompt_versionsTemplate text; the notes field becomes Engineering Memory
datasets✅ dataset_versions, dataset_itemsGround truth: input + expected output per item
providers—Model provider config; credentials encrypted

A versioned artifact carries a pointer to its current active version — the promoted revision. That pointer moving is the act that derives a Decision.

Evaluation

EntityNotes
evaluation_jobsThe pinned configuration and lifecycle: PENDING → RUNNING → COMPLETED / FAILED / CANCELLED
evaluation_runsOne per dataset item: output, latency, tokens, cost, HTTP status, error. PENDING / RUNNING / SUCCEEDED / FAILED
evaluation_resultsPer-metric outcome for a run: passed, score, detail
evaluation_profilesReusable metric configurations
benchmarks, benchmark_entriesVariant comparison
regression_checksCandidate vs baseline
reportsGenerated report records

Derived objects

None of these have tables. They are computed from the rows above every time they are read.

ObjectDerived from
ObservationAn evaluation's outcome against an artifact
ClaimA promoted revision plus the evaluations covering it
DecisionThe act of promoting or deprecating, carrying the version's notes as rationale
EvidenceAn evaluation, framed as support for a claim or decision
KnowledgeA Decision and Evidence, together
Engineering MemoryThe rationale on Decisions, recalled verbatim
Forge GraphReal references between artifacts
EvolutionGraph traversal — dependencies, dependents, transitive impact
AI Git timelineVersion rows plus active/rollback state
PrecedentEarlier troubled evaluations sharing an artifact
Investigations, Brok answers, briefsReadings of all of the above

Composite ids

Derived objects need stable, linkable identity without a primary key. They use composite ids:

   decision:prompt-version:3f2a…      a decision about a prompt version
   evidence:evaluation:9c1b…          an evaluation framed as evidence
   knowledge:prompt:7e44…             knowledge about a prompt
   observation:evaluation:0ab2…       a measured outcome
   claim:agent:5d90…                  a claim about an agent

Artifact node ids follow the same shape — prompt:<uuid>, evaluation:<uuid>, run:<uuid> — and are used consistently by the graph, the Registry, Brok references, investigation timelines and deep links. One id vocabulary across the entire product.

This is also how grounding is verified: an automated check asserts that every reference in every Brok answer and every investigation starts with a known prefix, which makes a fabricated reference a test failure.

Why derive instead of store

Stored reasoningDerived reasoning
DriftInevitableImpossible
Migration when logic improvesBackfill requiredNone
Can be fabricatedYes — anything can be insertedNo — no insert path exists
CostCheap readsSome CPU per read
Deletion / correctionMust cascade by handAutomatic

The cost is real. Assembling an investigation reads the evaluation, its runs, the artifacts, their revisions, the knowledge catalog and the precedent set. At the scale this platform targets — an engineering estate, not a traffic firehose — that is the right trade, and it buys a guarantee that would otherwise be a promise.

Entity relationships

   organization
     ├── organization_members ── users
     └── project
          ├── agent ──────── agent_version
          │     └── provider
          ├── prompt ─────── prompt_version
          ├── dataset ────── dataset_version ── dataset_item
          └── evaluation_job
               ├── (pins agent, prompt_version, dataset_version, provider, model)
               └── evaluation_run ── evaluation_result

An evaluation job pins its inputs rather than referencing them loosely — which is what makes a result reproducible and what lets an investigation say exactly what was measured.

Time

Two timestamps matter for reasoning:

  • `createdAt` — when the record was made.
  • `completedAt` — when an evaluation finished.

The platform reads "when did this happen" as completion where recorded, otherwise creation. That single rule makes the engineering timeline orderable across objects that finish at different points in their lifecycle.

See also: Architecture Overview · Core Concepts · REST API

PreviousArchitecture OverviewNextREST API