The AI Engineering Operating System.
Your AI system is a sequence of engineering decisions — a prompt promoted, a dataset regenerated, a model swapped. Broks Forge records those decisions with the evidence and the reasoning behind them, then answers the questions that follow: why did this fail, what changed, has this happened before, and can we defend it?
- Build
- agents, prompts, datasets, providers
- Evaluate
- reproducible measurement
- Understand
- intelligence, knowledge, memory
- Evolve
- AI Git, promotion, rollback
An example answer from Brok. Question: has this happened before? Answer, near-certain: yes. Checkout Quality number one failed nineteen days ago against the same agent and dataset, with an identical recorded cause, which makes this a recurrence rather than a coincidence. This is derived, read from earlier evaluations sharing an artifact with this one. Both failures recorded “Connection refused”. Engineering memory records “Moved the endpoint behind the internal gateway”. Two downstream artifacts are left without evidence.
Every statement declares how it is known. Nothing is generated.
The problem
Teams lose the reasoning behind their AI systems.
The prompt is in version control, but why v8 replaced v7 is in a Slack thread. The evaluation results are on a dashboard, but which decision they justified is nowhere. The engineer who knew has left. Six weeks later something fails, and nobody can reconstruct which change mattered.
An ordinary engineering question
Tracing & eval tools
Broks Forge
What did this request do?
Which calls were slow or expensive?
Did this prompt change break anything?
Why was this prompt promoted?
What evidence supports the current configuration?
Has this failure happened before, and what did we do?
Which of our decisions have no evidence behind them?
What would break if I changed this dataset?
Why is the system the way it is?
The bottom half of that table is not a feature gap a better dashboard closes. Those questions are about artifacts, versions, decisions and evidence — objects that tracing does not model, because it was never trying to. Answering them needs a different data model.
The solution
Record the engineering act. Then reason over it.
You register artifacts and evaluate them — work you were doing anyway. From that, Broks Forge derives observations, claims, decisions, evidence and durable knowledge. Nobody writes any of it down.
Derived, not authored
Promoting a version is a decision. Running an evaluation is evidence. The record is a by-product of engineering, so it never rots.
Nothing is fabricated
The reasoning layer is deterministic over real rows — no language model. When the record cannot answer, it says so.
Every claim is labelled
Derived, inferred, suggested or unknown — with a three-step confidence ladder instead of an invented percentage.
Absence is not health
An artifact nobody measured is reported as unknown, never as passing. A distinct verdict state, on purpose.
Deterministic reasoning
There is no language model in the reasoning layer.
Not as a limitation — as the point. An answer you have to go and verify has not saved you the work of finding it. Broks Forge resolves your question to an engineering intent and composes the answer from real rows, so the same question always returns the same answer, and every sentence can be traced to the record that produced it.
A generated answer
“The evaluation likely failed due to a configuration issue with the agent's credentials. You may want to check your provider settings and re-run the test.”
- Plausible whether or not it is true.
- No way to tell which part came from your system.
- Ask twice, get two different answers.
- You still have to go and check.
A derived answer
“6 of 8 sampled runs returned HTTP 401 against Local OpenAI-compatible. Credentials on that provider were rotated 3 hours before this ran.”
- Derived — the run rows say so.
- Traceable — every clause opens the row behind it.
- Reproducible — the same question returns the same answer.
- Bounded — proximity is offered as proximity, not as proof.
Knowledge
Derived from work already done — never written, never generated.
Documentation rots because keeping it true is a second job nobody is promoted for. Broks Forge never asks. Promoting a version is a decision. Running an evaluation is evidence. Knowledge appears only where a decision and its evidence genuinely both exist — which is also why the platform can tell you which of your decisions have nothing behind them at all.
Derived
Read directly from the record.
“4 evaluations cover this revision.”
Inferred
A conclusion drawn from what the record holds, and labelled as one.
“These failures read as infrastructure, not quality.”
Suggested
A recommendation, carrying the evidence behind it.
“Do not promote v4 — nothing has measured it.”
Unknown
The record cannot answer. Said plainly, never filled in.
“No evaluation has run against this dataset.”
Absence is never reported as health. An artifact nobody has measured comes back as unknown — a distinct verdict, on purpose, because “no failures” and “no evidence” are not the same claim.
How the reasoning worksArchitecture
Five layers, each earning the one above it.
A layer may read everything beneath it and nothing above it, and no layer may duplicate a layer below it. That rule is why adding a reasoning surface never means adding a table.
Brok
An engineering partner, not a chatbot.
Ask why something failed, whether to promote it, or what the team should do next — and get an answer read from your own engineering record. There is no language model: Brok resolves your question to one of 25 engineering intents and composes the answer from real rows. Ask it something the record cannot support and it refuses, then offers what it can answer.
How Brok worksRoot Cause Explorer
When you ask why, you get an investigation.
Open a failure and everything the record holds about it is already assembled: a dated chronology of the engineering that led there, the cause at four depths, the evidence and AI Git chains, the engineering memory, and every earlier failure on the same ground. No hunting across five pages.
How the Explorer worksAI Git
Version control for engineering reasoning.
Not source control. AI Git answers engineering questions: what was promoted, why, what it replaced, what evidence covered it — and whether production is running the newest revision or an older one somebody rolled back to at 2am. The reason you record becomes Engineering Memory, recalled verbatim forever.
How AI Git worksEngineering Intelligence
The reasoning layer nobody has to write.
Observations, claims, decisions, evidence and knowledge — all derived from real engineering work. Promoting a version is a decision; running an evaluation is evidence. Knowledge exists only where both are genuinely present, which is why it can be trusted, and why the platform can also tell you which of your decisions have nothing behind them.
How Engineering Intelligence worksForge Graph
The architecture diagram that cannot go stale.
Every architecture document is out of date the week after it is written, because keeping it true is manual. The Forge Graph is not maintained — it is the relationships the system actually recorded while you worked. Ask what depends on a dataset before you regenerate it and the blast radius is a fact, not a recollection. Narrow the same map to one run and it becomes the Execution Graph; narrow it to one failure and it becomes the Failure Graph.
How the Forge Graph worksThe product
Five surfaces, one engineering record.
No surface owns its own data. Each is a different lens on the same record, which is why they can never disagree with one another.
What should I know this morning?
The record, briefed.
Brok opens on standing briefs read straight from the engineering record — what ran, what was promoted, what is failing, what knowledge was derived overnight. Ask it a question and the answer is composed from real rows, each statement declaring how it is known.
How it works
Captured from a running instance with seeded data. Nothing here is a mockup.
The engineering journey
A loop, not a set of pages.
Every surface in the product serves one step of this loop — and the loop closes. Learning becomes the precedent the next failure searches.
Problem
Quality dropped, cost rose, something failed overnight.
Execution
Run a reproducible evaluation with a pinned configuration.
Evidence
The result enters the record as an observation, then as evidence.
Knowledge
Where a decision and evidence both exist, a durable fact emerges.
Decision
Promote or don't. Brok refuses to bless an unmeasured revision.
Revision
A new version — and one honest sentence saying why.
Promotion
AI Git records what was promoted and what it superseded.
Deployment
Production runs it. A rollback is displayed as a rollback.
Learning
It becomes precedent. The next failure can find it.
The shortest useful loop: register, write a reason on every version, evaluate before you promote, investigate instead of re-running.
The full workflowComparisons
A different scope, not a better dashboard.
Tracing, evaluation and gateway tools are good at what they do, and several are more mature than this project. Most teams should run one alongside Broks Forge. The useful question is not which is better — it is what each one is for.
Generalised across the tools compared below; capabilities differ between them and change over time. Each comparison page is specific, and cites what the other tool does well.
Who it is for
Built for the questions asked after the demo works.
Getting an AI feature working is the first week. Explaining, defending and evolving it is every week after that.
AI / ML engineers
“Why did this fail, and has it failed before?”
Reproducible evaluations, a versioned history of every prompt and agent, and an assembled investigation instead of a log search.
Staff & lead engineers
“Can we defend this configuration?”
One engineering record to reason over — including precedent, contradictions, and decisions carrying no evidence.
Engineering managers & CTOs
“What can we actually prove?”
Evidence behind decisions. Which promotions are defensible, which are being carried on faith, and what the system cannot yet prove.
Teams inheriting a system
“Why is it like this?”
Engineering Memory. The reasoning behind the current state survives the person who made it.
FAQ
Frequently asked questions
- Is this an observability tool?
- No. Observability answers what happened. Broks Forge answers why the system is the way it is, what evidence supports it, and what to do next. They model different objects, and running both is sensible.
- Is Brok a chatbot? Does it use an LLM?
- Brok is not a chatbot and contains no language model. It resolves your question to one of 25 engineering intents by deterministic phrase scoring, then composes the answer from real database rows. Ask something the record cannot answer and it refuses.
- Do I have to use a specific framework?
- No. Agents are registered by HTTP endpoint. LangChain, LlamaIndex, a custom FastAPI service, a serverless function — if it is callable over REST, it works. There is no SDK to adopt.
- Is it in my production request path?
- No. Broks Forge is not a proxy or a gateway. It calls your agent endpoint during an evaluation and never otherwise, so it adds no latency to your traffic.
- Is it open source? What does it cost?
- Open source under Apache 2.0 and self-hostable with Docker Compose. No licence fee, no per-seat cost. You pay for the infrastructure you run it on and any model calls your evaluations make.
- How mature is it?
- Early. It is a complete, tested system — 499 backend tests running against real PostgreSQL — but it is a young project without the production track record or support organization of the commercial tools it is compared with.
- What is the minimum useful setup?
- An agent, a dataset, a prompt and one evaluation. That already produces observations, evidence, a graph, an AI Git timeline and answerable questions. The habit that matters most is writing one honest sentence on every version.
- Why does it say “unknown” instead of “healthy”?
- Because nothing has measured that artifact. Absence of failure is not evidence of health, and the platform refuses to imply otherwise. Run an evaluation and it becomes a real verdict.
Give your AI system an engineering record.
Self-hosted, open source, and running in about fifteen minutes — most of it waiting on the first Docker build.
New to the category?What is an AI Engineering Operating System?