Local green is not evidence.

/team/cred returned 503. google-cloud-secret-manager was installed in the local venv and absent from the control-plane image. Fixed by adding the dependency and a build-time import guard, so a missing client fails the image build instead of the first request.

Under production least-privilege, the public service could not write Secret Manager. Nothing on a workstation enforces that boundary, so it never appeared until the real IAM did. Public now forwards to the gated /team/cred over authenticated service-to-service with a run.invoker grant; read and dispatch stay forbidden on the public surface.

A tab collapse ate the mode=control-plane branch, so bare rr went back to SSH instead of POST /rr-task. An ordinary logic bug, recorded because the ladder is the ladder.

The dispatcher defaulted to heimdall-long-job. The deployed job was heimdall-maintainer-job. A default only diverges once a real named job exists, which happens exactly once, in production.

CloudRunJobRunner.build_request dropped base_env, so the dispatched job ran without the per-team credential it was supposed to carry. A pass-through that works in a local runner can be silently dropped by a different runner. The per-team credential and minted token are now threaded into the per-execution env override. The standing guard is the isolation oracle: bin/falsify rr-multitenant-isolation --assert-score 1.0, where every mutant is a real attack and the run fails unless every one is killed.

A task subprocess failed. Its stderr was captured. Nobody surfaced it. No error, no retry, no trail — the task was simply consumed. This is the shape that produced most of the later mysteries, and it is now a rule: on a nonzero exit, write the scrubbed stderr tail, retry where retrying is correct, and never consume a failure quietly.

The drain enumeration missed queued tasks, so the warm tick drained nothing and looked healthy doing it.

Cloud Run throttles CPU off the request path. The background tick ran on that path and starved. There is no local equivalent to test against. Fixed with --no-cpu-throttling and min-instances=1.

The per-execution env override carried PORT, K_SERVICE, K_REVISION and K_CONFIGURATION. Cloud Run injects those itself and rejects a RunJob request that sets them. They are stripped now.

The maintainer armed itself by reading a .maintainer.enabled file. A cold container has no such file and never will. The gate is an env override with a defined fresh-home behaviour instead.

The control plane ships as two images: the service that ticks and the job that executes. A bin/lib change rebuilt into one of them produced a job image carrying an old allowlist while the service moved on. Any change to shared code is a change to both images.

The budget meter fail-closed when it found no usage history. A cold container always has no usage history, so the meter refused to run the thing it was metering. Fresh home is now a defined state, not an error.

The archetype. The planning directory was built by joining a repo slug — owner/name — onto the current working directory, because on a laptop the CWD is always a writable checkout, so the join always landed somewhere real. In the container the CWD is /app, the read-only image root, and there is no checkout at all. os.makedirs() raised PermissionError at startup. A slug is not a path. resolve_workspace() now reuses an existing local directory unchanged, shallow-clones a bare slug into a writable workspace, and refuses a traversing or malformed slug loudly before anything is joined.

issue_queue.ingest had zero callers. The queue was never populated. The function was correct, tested, and unreachable — wiring that "obviously" runs, with nothing in the deployed path actually running it. gh issue list now feeds ingest_many.

This one cost the most time. Jobs failed with an 876-line traceback pointing at code the job image did not contain. The job image was byte-verified correct. cp_worker.run_job ignored HEIMDALL_JOB_RUNNER, so any service instance booting up would claim queued jobs and execute them in-process, using the gated service's own stale code instead of dispatching them to the job. The traceback was real. It was just from the wrong machine. Resume now re-dispatches through the configured runner and never runs in-process when the runner is remote, with a five-minute grace for young queued jobs, a two-hour running-orphan lease with one reclaim, and a loud resumed-via= line so the answer to "who actually ran this" is in the log.

Unconditional project IAM bindings need --condition=None. Without it, a binding-narrowing pass broke them without complaining.

An unbounded list_names scan over Firestore blocked the tick thread. Reads are bounded now, with prefix pushdown, a page cap, and a 55-second watchdog per step.

A destructive set-env-vars during a direct go-live run dropped RR_TENANT_AUTHZ and TEAM_CRED_STORE, which disabled the drain. The deploy reported success.

Legacy revision-less Firestore records were unclaimable, and compare-and-swap exhaustion was conflated with an empty queue. A backed-up queue looked idle. Exhaustion is loud now.

The fix cycle emitted every log line you would expect from a fix cycle and never called claude. It looked like it was working because it was narrating work it never did. Headless invocation, a loud fix_attempt telemetry node, and an explicit HEIMDALL_FIX_WITH_CLAUDE gate landed together.

gh pr create ran before git commit and git push, so the pull request it opened contained nothing. The fix commits onto heimdall/issue/<id> under bot identity, pushes with the token in the environment (--force-with-lease, never to main), and only then creates the PR. The test asserts the branch reached origin, and that a push failure does not produce a dangling PR.

CLAUDE_CONFIG_DIR was dropped across the handler-to-fix-child environment chain, so headless claude could not find its provisioned credential and fell back to an interactive OAuth login prompt, inside a container with no human in front of it.

The setup token was stored as the entire decorated invocation string rather than the key inside it. Ingestion never stripped the decoration, so every fix attempt authenticated with garbage. A shared claude_cred shape oracle now runs at both ends, client and server, and a malformed credential is refused rather than stored.

The container had neither pytest nor the target repo's own dependencies, so a correct fix could not be proven. The same run pushed a branch carrying __pycache__ and virtualenv files. Dependencies are bootstrapped before the evidence run; non-source paths are stripped from the pushed tree, and the test asserts every junk path is absent.

Evidence ran the target repo's entire test suite, so an unrelated co-resident failure blocked a correct fix. The gate is now the issue's named gating-test node, extracted injection-safely from the issue body. The whole suite is advisory.

With GH_CONFIG_DIR unset, gh fell back to the App JWT instead of the installation token, and PRs were authored under the App identity rather than the bot. One cleaned installation token, an isolated GH_CONFIG_DIR, no fallback.

The PR-bot test still encoded the pre-#21 contract, so it went red on correct behaviour. A test that describes the old world is a liability with a green checkmark. Rewritten against a hermetic bare origin with a bot-identity branch assertion.

With #28 fixed, the next run failed loudly for the first time: not a git repository. gh pr create had been running with the agent's working directory, which has no .git, rather than the clone. It had probably been failing that way for a while. Passing --repo <slug> makes gh independent of the working directory, and the call runs with cwd set to the clone. The falsifier reproduces the exact run-15 failure and proves the same non-git directory succeeds once --repo is passed.