The Evidence Is Becoming the Product

2026-09-20 — OpenJev’s calibration tables and a 176-setting harness study make evaluation practice visible—while the day’s other stories show how much consequential behaviour still hides in defaults, side effects and waivers.

The day in view

OpenJev published browser-local scores alongside measured gaps against Jev’s reported result. A new harness-design study compared 176 matched coding-agent settings and identified context management as a decisive factor when budgets tighten. Together, the two releases put calibration and comparative evidence in view. They also sharpen the contrast with systems whose consequential behaviour sits in defaults, undocumented side effects, or opaque release processes.
Altior — our view today

Read more from [Altior](https://altior.digital/blog/) or explore the [prompt library](https://altior.digital/blog/prompts/).

OpenAI / ChatGPT / Codex

OpenAI internal-repository breach writeup reaches the front page

A Hacktron writeup reported a compromise of OpenAI internal repositories through a chain involving a heap overflow and an SSO misconfiguration. The account is a report of the incident rather than a vendor postmortem, so the technical sequence should be treated as reported until independently corroborated. Its significance is not limited to one exploit class. Repository access, identity configuration and memory-safety failures can combine into a supply-chain event when they are evaluated in isolation instead of as one path to privileged code.

Our takeOur take: breach narratives are useful only when their controls, assumptions and chain of failure can be checked. A front-page claim is not a substitute for a verifiable postmortem.

Source

Anthropic / Claude

Claude Code adopts AGENTS.md as a fallback instruction file

Claude Code’s changelog records AGENTS.md as a fallback instruction file. That is a small implementation change with a larger operational consequence: a shared convention is becoming a control plane for agent behaviour. Teams will increasingly rely on a repository file to establish what an agent may do, where it should look, and which local rules prevail. Once that file carries behavioural authority, its origin, review history and integrity become part of the system’s security model rather than documentation housekeeping.

Our takeOur take: AGENTS.md is becoming an interface worth governing. Treat edits as behavioural changes, not merely copy changes, and make provenance visible to reviewers.

Source

Zhipu / GLM

ZCode was reported silently uploading complete Git history

Researcher Ferstar reported that ZCode, Zhipu’s coding agent, packaged complete workspaces into encrypted archives and uploaded them to Aliyun OSS. The reported contents included Git history, LFS cache, reflogs and global configuration files, with the private key held server-side. If confirmed, the finding makes the important question larger than whether a tool touches the current working tree. A coding agent’s effective data boundary includes every archive, cache and configuration path it collects by default.

Our takeOur take: undocumented workspace collection is an operational risk, even when encryption is present. Teams need an inspectable account of what leaves a developer machine and why.

Source

OpenClaw

OpenClaw v2026.9.5 ships stable under a written soak waiver

OpenClaw v2026.9.5 reached stable after a three-day slip. Its public release evidence recorded an operator-approved waiver for blocked soak-only lanes, while non-soak validation was recorded as green. The same evidence file showed zero ClawHub packages and no bootstrap evidence. The release also carries a cross-provider messaging default change, so operators have both a concrete configuration consequence and a documented qualification limit to assess before broad adoption.

Our takeOur take: the waiver is more useful because it is written down. Visible qualification limits let operators decide deliberately; silent exceptions turn into surprises after rollout.

Source

Other

OpenJev brings browser-local scoring and public calibration tables

OpenJev, an independent imitation of Jev, runs small open models locally in the browser as decision scorers. Its site publishes calibration tables comparing its own measured results with Jev’s published number on a shared subset. The headline is not that the imitation matches the original; it does not. The useful move is making the gap explicit. Readers can see the model choice, the evaluation subset and the distance between an observed result and a claimed reference point.

Our takeOur take: this is what comparative evidence looks like. A visible gap can build more trust than a clean score with no calibration, method or reference beside it.

Source

Microsoft executive calls AI scraping the largest theft of labour in human history

TechCrunch reported that newly unredacted filings in the New York Times copyright case included an internal Microsoft characterisation of AI scraping as “the largest theft of labor in human history.” The report says the material is drawn from litigation briefs and that some underlying exhibits remain sealed, so the quotations need their legal context. The story nevertheless moves the data-rights argument from abstract policy language into documented corporate communications and the evidence record of a live dispute.

Our takeOur take: strong language does not resolve the legal question. It does make provenance, licensing and evidence trails central operational issues for AI builders and buyers.

Source

Reported military near-miss puts hallucinated intelligence into an operational setting

CNN reported that the US military had a close call after AI was used for a hallucinated intelligence report. The account places a familiar model failure in a setting where an incorrect report can shape real decisions rather than merely produce a poor answer. That is the important boundary. Benchmark performance and conversational fluency do not show whether a system has appropriate verification, escalation and human review at the point where an operational decision is made.

Our takeOur take: high-consequence deployments need evaluation at the decision boundary. Test the report, the reviewer and the escalation path—not just the model’s response.

Source

Harness-design study tests 176 matched coding-agent settings

A new empirical study tested 176 matched settings across coding-agent harness designs. Its reported findings are practical: context management matters most as budgets become constrained; staged rule-based elision outperformed more elaborate recoverable-elision machinery; and planning shifts from an accuracy scaffold for weaker models toward a cost-saving tool as models improve. The study does not reduce agent performance to a single model score. It isolates operating choices that can materially change outcome, cost and reliability.

Our takeOur take: harness design belongs in the product conversation. Comparative experiments expose where a system’s performance comes from and which defaults deserve scrutiny.

Source

ALTIOR AI ADVANTAGE

Keep reading

Build with evidence.

Build AI systems with controls that can survive review.

The Altior blog The prompt library