Editor and native data efficiency plan

Status: implemented through package 6 (2026-09-26, macOS recording host). Packages 1–5 are implemented and qualified. Package 6's MultiMesh readback consumer is implemented and measured: the recorded inventory failure resolved as two darwin stage-tool regressions (fixed), the campaign numbers are frozen in fixtures/efficiency/evidence/, and the macOS kqueue notification adapter runs live. Package 7's receipt/lock question closed as a measured decision to retain the policy (archived evaluation). Publication followed through the successive gododin-devx-2026-09-26.1–.3 reseats. Remaining: package 6/7's failure matrix and end-to-end timing comparisons and the Windows/Linux notification adapters — owned by the DevX plan ledger (DX-0). The 2026-09-20 efficiency record contains no Windows/Linux checks; that is not a claim that the project has never been verified there. See the platform reconciliation for the owner's later verification report. Per-package notes are inline below.

Status reconciled 2026-09-26: this plan is partially implemented, not complete. Packages 1–5 should not be restarted. Package 6 needs source-rebuild reconciliation and campaign evidence; package 7 needs the matched publication, integrated failure/recovery checks and complete workflow measurements. Native notifications and the receipt/lock decision remain open. The DevX completion plan, DX-0 orders these residuals with the subsequent consumer work; the requirements below remain authoritative.

This document plans the communication, catalog access, build iteration, and native data improvements discussed after the receipt/catalog migration. The remaining implementation uses the current handoff prompt. This work is independent of Godot Agent Kit and must add no dependency on that repository.

The active implementation plan owns product sequencing and C1–C5 verification. This plan owns the bounded work packages below. Existing behavior remains governed by the architecture, Editor/Build PRD, Native PRD, and version policy. Planned changes do not qualify a platform, replace an accepted decision, or establish a performance improvement.

Intended result

Use the representation appropriate to each boundary:

Binary Variant encoding is selected for private control messages because Godot already implements its GDScript endpoint. It preserves admitted value types and avoids text parsing, but decoding still constructs values and may allocate. It is neither GDI nor a universal zero-copy format. GDI is Gododin's own mapped file format, not a Godot feature. Packed-array borrowing uses public native engine interfaces; pointers, live Objects, and Node identities do not become portable across processes. No shared-memory service, engine fork, native editor addon dependency, or user-installed codec/watcher is required.

Starting evidence and reading order

The inspected baseline is commit 6fbe3f11b2ca02e9ce4b345d1f90cde4e28b3bf8. Reconcile these observations with the executing checkout before editing:

Exploratory measurements on the recording macOS arm64 host on 2026-09-19 used the then-current exact engine, warm headless runs, and repeated decoder batches. A status payload decoded in about 18.7 µs as compact JSON and 4.2 µs as binary Variant; the native-basic catalog decoded in about 401 µs and 123 µs respectively. The binary messages were larger: 2,076 versus 1,670 bytes for status and 43,176 versus 31,494 bytes for that catalog. A CLI catalog dump plus GDScript parsing cost about 4.8 ms in a separate warm probe. These observations motivate work; they measure neither a complete reload improvement nor an Odin encoder. The probe inputs inherited JSON's numeric representation. Establish reproducible baselines in package 1 rather than relying on disposable local probe files.

At execution, follow AGENTS.md's required read order first. Then read only the affected contracts: ADRs 0045 and 0057 for bulk/parity and reload; ADRs 0059/0060 and the Odin API guide for native value/lifetime work; ADR 0066 for catalog and format policy; and the architecture's decision index for ownership. Resolve compiler/engine identities through current repository authorities, never an ambient executable or a historical receipt.

Work packages

Complete these in order, retaining resumable evidence at each boundary. The implementation session owns all packages; existing subsystem owners below remain authoritative. Record a concrete reason if measurement closes a conditional optimization without a code change.

1. Establish a reproducible baseline

Extend existing focused harnesses with timings/counters that attribute cost:

• Idle editor with a small project and a representative large source tree: editor frame time, CPU, scans, files hashed, bytes hashed, and child launches.
• Ordinary Odin body edit, declaration/property edit, and GDScript/scene edit: save-to-observed editor result, stage durations, cache hits, allocations, and time in synchronization/activation. Measure game restart separately.
• Status and catalog work: encode/decode time, bytes, temporary allocation, process overhead, validation, and compatibility/diff output equivalence.
• Existing bulk workloads: foreign calls, Odin allocation, staging/copy counts, engine work, and final observed output. Keep kernel and engine costs distinct.

Record revision, hardware, engine/compiler/SDK identities, profile, renderer, input size, cold/warm conditions, repetitions, and p50/p95 for latency. Separate instrumented counters from uninstrumented timing. Define noise bounds from repeated baselines before comparing changes; do not invent a percentage win. Use disposable projects and owned processes so measurements preserve user work.

Done when another session can reproduce the workloads from committed harnesses and retained inputs without this conversation or a local build/ directory.

2. Specify and implement the control protocol

Record a new ADR before switching production readers. (Implemented 2026-09-20: ADR 0067, the controlprotocol/ codec with quick-tier unit tests, and the fixtures/control_protocol/ bidirectional conformance fixture wired into the locked tier.) Supersede only the applicable parts of ADR 0066 decisions 4/7 and any other affected contracts; retain TOML receipts, GDI catalog authority, and JSON review output. Update the relevant PRD, architecture, and active plan together when behavior lands.

Implement a small Odin codec compiled into the existing CLI. Start with null, bool, signed 64-bit integers, finite floating-point values where required, UTF-8 strings, byte buffers, arrays, and string-keyed dictionaries. Add packed numeric types only for an actual message that needs them. Do not round-trip through JSON to encode/decode binary messages. Keep existing identity types; unsigned identifiers outside signed 64-bit range need an explicit lossless representation, not a float conversion.

Use the exact target's var_to_bytes/bytes_to_var format with object encoding and decoding disabled. Specify the admitted subset, width flags, byte order, padding, limits, errors, and independent application protocol/schema versions. Do not assume typed containers or arbitrary Godot releases share a stable wire format. Object/RID/Callable values and executable object reconstruction are outside this control contract.

Specify a bounded length-prefixed frame around each payload, request IDs, event sequence numbers, startup identity negotiation, and error/terminal responses. The handshake binds the verified CLI, SDK/target, project, profile, and editor session. Exact names and numeric limits belong in one protocol authority with conformance fixtures, not duplicated in this plan. Stdout is protocol-only; drain and bound stderr separately.

Both endpoints must handle partial reads/writes, multiple frames per read, EOF mid-frame, cancellation, unknown versions/types, integer overflow, oversized lengths, nesting/count limits, malformed UTF-8, and late replies. Validate before unbounded allocation. Integrate differential fixtures against the actual engine in both directions, including integer precision above 2^53, boundary values, empty containers, Unicode, and invalid payloads. Supplement valid round trips with malformed-input/fuzz coverage appropriate to a new binary parser.

Done when the codec works without a Godot runtime dependency in the CLI and the exact-target executable proves bidirectional conformance. A decoder microbenchmark alone does not close the package.

3. Move editor orchestration and catalog decisions onto the child

Replace repeated process launches/state polling with one verified CLI child. (Implemented 2026-09-20 on macOS: gododin editor-agent serves one addon lifetime over the ADR 0067 protocol — status, build, sync, and compat requests with coalesced status events and per-request backpressure — reusing the build coordinator and publication installer; catalog compat decides stored-property continuity by direct GDI record reads; gododin check walks the identity index directly instead of reconstructing a JSON tree; the addon drains frames on a worker thread, recovers the real loaded identity from the Host latch on child restart, and still owns the public reload call. Verified live: the agent probe (hello/status/build/compat-refusal/ping/unknown-op/terminal) is pinned in the locked tier, and a headless authoring editor performed edit → build → sync → compat → GODODIN_NATIVE_RELOADED in one process with a zero-owner teardown. Two exact-target facts shaped the addon reader: execute_with_pipe(..., false) opens a nonblocking pipe whose reads return error 14 both when idle and after child exit, so EOF is inferred from process death on an empty pipe. The addon's whole-tree scan remains until package 4.) owned by each addon lifetime, using the public nonblocking pipe API. Reuse the existing project coordination/locking, snapshot, build, sync, and publication owners; do not introduce a second build authority or a machine-wide daemon. Other CLI invocations must still observe and respect the same accepted state.

Push bounded status, diagnostic, and accepted-build events. Coalesce replaceable progress updates; preserve terminal outcomes and publication ordering. Bound bytes and work per editor frame as well as queued messages, so a large response cannot turn nonblocking I/O into a main-thread stall. EOF, addon disable, partial startup, child failure, and editor exit release pipes/watchers and owned subprocesses. Restart with bounded backoff and reconcile durable state; never mistake a restarted sequence or old event for a new accepted build.

Move compatibility checks into the CLI's validating GDI reader. Compare the actual loaded artifact with the candidate artifact, including existing property identity/alias, public-name, type, and storage rules, and return a small decision plus actionable reasons. Replace the API diff's reconstructed JSON model with direct record traversal while preserving its full existing semantics. Keep catalog dump for explicit review and any consumers that truly need JSON.

Retain the loaded and candidate publication identities through the decision and activation acknowledgement. Pin the immutable files while views are used; release mappings before eviction/replacement, including Windows file-lifetime constraints. Never retain an Odin string beyond its mapped owner. A child restart must recover the real loaded identity, not assume it equals the latest accepted build. Missing old evidence yields a useful refusal.

The addon still owns the public Godot reload call. It checks the decision's identities against current accepted/synced state immediately before activation and reports the actual outcome. Superseding saves/builds invalidate stale decisions. Preserve stored-property restoration, unsaved work, recovery-mode independence, failure reporting, and process-lifetime running-game images.

Done when normal activation performs no blocking OS.execute or full catalog JSON reconstruction, repeated actions reuse the child, and failure/reconnect cases preserve the existing reload and fail-closed guarantees.

4. Replace editor-thread tree hashing with owned change tracking

Put source observation in the CLI child. (Implemented 2026-09-20 on macOS: the watcher/ observer runs on the child's watcher thread — a per-file size/mtime census with digests only for files whose entry moved, so idle cost is a metadata walk and same-size atomic saves still supersede; bursts settle for 350 ms before one coalesced source_changed event, startup reconciliation primes the census without emitting, missing configured roots are skipped like the historical scan. The addon dropped its whole-tree scan, DirAccess walk, and Godot filesystem-signal hookup entirely; it only requests the build. Verified with the watcher unit test (edit/same-size/delete/re-create/idle-cost) and a live headless editor cycle: edit → child observation → build → sync → compat → GODODIN_NATIVE_RELOADED.) (Extended 2026-09-26 on macOS: the notification seam landed — watcher/notify_darwin.odin watches every directory and regular file below the configured roots with one kqueue (EVFILT_VNODE; file watches cover in-place same-size saves, directory watches cover atomic save/rename and directory moves), bounded at 16,384 entries with honest degradation to the reported polling mode beyond the bound, on kevent error, or on queue EOF. An event is only a wake-up: the census stays ground truth, refresh re-derives the watch set after every observed change, and the caller retains a bounded 10-tick periodic reconciliation. notify_fallback.odin keeps non-darwin hosts on the explicitly reported polling mode; the ReadDirectoryChangesW and inotify adapters remain desktop workstream work under ADR 0065. Verified with the extended watcher unit test (idle wake-free budget, write wake, refresh of a new directory, in-place same-size wake, census reconciliation) and a live child probe: fixtures/editor_agent/run.sh's driver grew --watch-seconds, and a mid-session authored edit produced kind=source_changed mode=notification through the real child protocol. Reproduce the live cycle with the commands in this plan's package 3 note.) Use native filesystem notifications through narrow platform adapters compiled into the existing distribution; coalesce bursts, retain canonical path/exclusion rules, and hash affected inputs outside the editor thread. Include configured roots outside res://.

Notifications are invalidation hints, not proof of freshness. Handle atomic save/rename, delete/recreate, rapid same-size edits, directory moves, root replacement, queue overflow, suspend/resume, and watcher loss. Run an initial reconciliation and reconcile after gaps; retain bounded periodic reconciliation where the platform's delivery guarantee requires it. A degraded watcher may use explicitly reported polling in the child, without restoring main-thread scans. Actual build snapshots and run/export admission still verify their inputs.

Implement portable ownership and narrow macOS/Windows/Linux adapters as the available toolchains permit. macOS execution proceeds locally; missing host qualification stays pending under Phase 11 and does not justify silently using an unqualified adapter or deferring all portable work.

Done when idle work no longer scales as whole-tree hashing every poll, the same-size/atomic-save cases rebuild correctly, and watcher failure cannot accept stale code. Measure idle CPU and frame impact alongside save latency.

5. Reduce repeated build and verification work

Split Host object caching from project-image object caching. (Implemented 2026-09-20 on macOS: the compile stage now stores host.o and project-image.o under independent coordinator-owned cache keys. The Host key covers the authority Host source, the generated runtime subtree identity, the toolchain, the implementation identity, profile, target, entry symbol, and Host defines — deliberately not the authored input identity or the whole generated bundle, whose receipt changes whenever authored bytes change. The image key keeps the full input identity. The compile stage restores, verifies, and publishes each object independently, so a project body edit recompiles the image while reusing the Host, and a Host/toolchain/generated-runtime change still invalidates it. Proven by fixtures/stage_cache/run.sh against real builds — cold publication, warm reuse of both objects, and Host reuse with image recompilation across an authored edit — pinned in the locked tier. The receipt/lock extent reduction evaluation owns the digest-web policy; its requested timing breakdown is recorded under package 7.) Derive each key from its actual source, generated inputs, toolchain, target, profile, ABI, defines, and flags. A project-body change may reuse the Host only when its inputs are unchanged; a Host/ABI/toolchain change must invalidate it. Preserve artifact validation, corruption refusal, bounded eviction, concurrent build coordination, and deterministic linkage. Prove warm reuse and each relevant invalidation rather than asserting two filenames imply independence.

Measure parsing/generation and verification separately before adding another cache. Reuse normalized models only behind a complete input/schema identity when measured cost warrants it; avoid maintaining a second semantic model.

The receipt/lock extent reduction work package remains the sole authority for digest-web redesign. Supply its requested timing breakdown and execute that linked work as the separately decided subpackage: record its own ADR if measurements justify changing the policy. Do not duplicate its checklist here or remove trust-boundary checks as an incidental cache optimization. A measured decision to retain the current policy closes that evaluation without pretending a reduction shipped.

Done when an ordinary project-only edit reuses a valid Host object, cache faults cannot promote stale artifacts, and measured verification findings have a recorded disposition under the linked package.

6. Complete a concrete native bulk improvement

Use the existing examples and the Native PRD's remaining packed staging/read directions to select a real consumer, such as MultiMesh readback. (Implemented 2026-09-20 on macOS; reconciled and measured 2026-09-26 on the same host. The named MultiMesh read direction has its consumer: the pure subsystem stream.verify_buffer checks every instance, every float, bitwise against the producer oracle plus the buffer digest, and the authored Stream_Field.readback consumes the engine's own MultiMesh.get_buffer() copy as an ADR 0060 borrowed []f32 view (two foreign calls, zero Gododin copies, zero Gododin allocations). The 2026-09-20 rebuild blocker resolved as two darwin-only stage-tool regressions from the Linux batch, not a generation/inventory drift: check_stage passed -minimum-os-version to odin check (rejected by the compiler), and the installer scan compared Mach-O's underscore-prefixed symbols against unprefixed names, so the recorded MultiMeshInstance3D cast message was a false positive of that scan. Both are fixed in cmd/gododin-stage/main.odin; the campaign itself then exposed and fixed a fixture-only oracle bug (get_buffer returns the interleaved layout verbatim, not the converted Transform3D order) and a driver contradiction (digest equality demanded across advancing frames). Measured 2026-09-26, windowed Forward+/Metal, 200 rounds p50: capacity 1000 (48 KB) engine copy 7.50 ms / native verify 264 us; capacity 4000 (192 KB) engine copy 6.71 ms / native verify 969 us; zero-owner teardown after both. Record: fixtures/efficiency/evidence/2026-09-26-macos-arm64-readback.json (repository: fixtures/efficiency/evidence/2026-09-26-macos-arm64-readback.json); reproduce with fixtures/multimesh_stream/readback.sh --cli <checkout-cli> --odin <odin> --godot <godot> [--capacity N] [--rounds N]. Remaining for this package: locked-tier and installed-authority reruns ride the DX-0 release reseat; the readback numbers cover the verification path only, not the timed frame path.) Record the type, direction, member position, workload size, and current cost before extending anything. Prefer batched typed/native operations, const packed-input views, and retained result/staging storage. Do not route native work through the new process codec or generate a general-purpose serialization layer.

Implement the consumer and the required reusable conversion/staging support together. Metadata-determined operations belong in the generator, and a helper must hide recurring ownership, lifetime, batching, or validation work. Preserve Main/Render roles, precision, copy-on-write rules, engine-storage lifetime, invalidation checks, and symmetric cleanup. Workers own their snapshots and cannot retain callback borrows or Godot authority.

Measure foreign calls, Odin allocations, bytes copied, and output correctness against the prior path. Account separately for unavoidable engine/GPU copies, readback stalls, and engine allocations. Prove empty/error inputs, repeated use, owner destruction, and reload/teardown. If the selected path is already optimal, record that evidence and examine the next named consumer gap; do not invent a copy or allocation reduction. Broader authored-value expansion remains governed by its existing admission work, not this transport migration.

Done when at least one remaining named consumer gap is implemented and measured, or a workload-backed audit shows the selected remaining directions need no additional Gododin mechanism and records their disposition explicitly.

7. Qualify the integrated path and update standing documentation

Use focused parser/protocol/cache/lifetime checks during development. (Status 2026-09-20, macOS recording host. Green: the repository quick tier passes end to end with the new packages in the CLI closure (GODODIN_TEST_PASS tier=quick, 112.4 s, all phases), the control-protocol conformance fixture (37/37 bidirectional), the editor-agent live probe, the stage-cache split proof, the compat decision unit test, the watcher unit test, and the live headless editor cycle edit → observation → build → sync → compat → GODODIN_NATIVE_RELOADED. The efficiency after-record (fixtures/efficiency/evidence/2026-09-20-macos-arm64-after-note.json) states the structural reductions and keeps the microbenchmarks comparative. Pending: the locked tier's sealed-release authentication gates (exact external-Odin public first-CLI path failed downstream reauthentication) require a new release identity — the sealed gododin-dev6-formats-2026-09-19.2 binary predates today's source closure (controlprotocol/, watcher/) exactly as ADR 0066's migration needed its own reseat; the three new locked fixtures were run standalone green and wire into the tier once the release is repacked (tools/release/pack_prebuilt.sh + lock reseat + installed-consumer requalification). The receipt/lock extent-reduction evaluation receives its requested quick-tier timing breakdown: total 112.4 s, of which phase one's generation determinism + docgen rederive is 59.9 s, phase five's frozen evidence checks 29.1 s, and the production-CLI block (including the sealed release authentication that the reduction would touch at trust boundaries) 14.9 s — recorded in docs/plans/archive/2026-09-19-receipt-lock-extent-reduction.md's terms; the redesign itself remains its own decision. Windows/Linux hosts: all new code is either portable or behind the ADR 0065 adapter boundary; no Windows/Linux check has been executed and none is claimed.) and tests/run.sh with an explicit version-matched compiler for routine tiers. Batch SDK publication, installed-consumer validation, editor reload/property restoration, affected runtime/rendering/export checks, and end-to-end workloads under the active plan's C1–C5 policy. Reuse unaffected receipts; a new artifact needs matching evidence for the behaviors it changes.

Exercise rapid edits, failure then repair, stale/superseded decisions, child crash/restart, pipe saturation, queued cancellation, addon disable/re-enable, watcher loss, corrupted caches/catalogs, and teardown with work outstanding. Check fresh attachment and recovery when native project code fails to build.

Compare package 1's workloads using the same inputs and reported noise bounds. Require the structural cost reductions above and no unexplained end-to-end regression. Report codec, idle editor, build, reload, and runtime results separately; never promote one microbenchmark into a whole-product claim.

Update standing architecture, PRDs, API/CLI documentation, relevant decisions, parity rows, and receipts with implemented facts. Keep this plan and its prompt marked completed while linking the resulting authorities, or retire them together once every package is accounted for. Pending real-host evidence gets its own explicit platform record and reproduction command. macOS completion does not wait for the owner to visit Windows/Linux, and does not claim their runtime qualification or close C5.

Primary references

• Godot binary serialization describes the facility; exact-target source and executable conformance own the actual subset, flags, and layout used by this implementation.
• Nonblocking process pipes provide the public editor endpoint; prove partial-I/O behavior on each host.
• PackedByteArray exposes offset decoding in GDScript, not a generic mapped-file/object view.
• The locked GDExtension interface (repository: vendor/godot/4.8-dev6/gdextension_interface.h) is the inspected baseline for native packed-array pointers. At execution, resolve the active target through the version policy rather than copying this historical source link into a new pin.