DEVELOPMENT DOCUMENTATION / SUPPORT & TROUBLESHOOTING

Know the boundary before you depend on it.

Repository evidence is scoped. Development progress is not a production support commitment.

Recorded hosted acceptance

The repository records CI run 36006258279 at 50b12df: locked Rust checks passed on Linux, macOS and Windows (excluding relay tests on Windows); M1 real-socket acceptance passed on Linux and macOS; M2 accelerated/fault and three actual 300-second rotation plans passed on Linux. The M7 harness suite also passed in that run. These are dated results, not a new test run or complete cluster/adapter certification.

The M1 pre-body admission flake was traced to a harness ordering race, not a relay defect. At 4aab5c4, held requests wait for the relay's 100 Continue before the ninth request must receive 429 ADMISSION_LIMIT. The repository records six passing hosted M1 jobs across Linux and macOS in three attempts of probe run 36101014774, plus 30 of 30 local acceptance runs under load. M7 also passed on all three probe attempts. There is still no Windows M1 acceptance, and credential creation/import are unsupported there.

That probe was not uniformly green: one attempt hit an MCP isolation quiesce 503, and another failed two fixture-supervision tests in the Linux Rust job. These newly tracked intermittents remain open; passing M1 and M7 jobs do not establish complete workspace or adapter acceptance. No runtime acceptance was rerun for this website review.

Client exit codes

The implemented tunnel-client mapping distinguishes operator actions. Read the diagnostic alongside the process status.

Current main maps AUTHORIZATION_STALE to exit 4, changed from exit 3 in older builds. It means a queued frame outlived its stream authorization window; the reconnect loop already treats this as retryable. It does not prove that earlier dispatched operations had no effect.

With reconnect enabled, retryable failures normally produce lifecycle events rather than process exits. A finite attempt limit, disabled reconnect or terminal cause ends the process with the last cause's status. Do not infer operation completion from a reconnect or an exit code.

Configuration rejected

Use client config check or relay check-serve-config, not the legacy bootstrap schema checker. Check unknown fields, endpoint scheme, namespace characters and rotation timing. A dry run does not verify referenced files.

Certificate or connection failure

Run local doctor. Check the device key/certificate pairing, expiry and CA trust; verify the relay hostname and mTLS listener. Do not bypass validation or swap in test credentials.

Local doctor now compares the profile device UUID with the certificate identity and reports CREDENTIAL_DEVICE_MISMATCH, exit 3. A healthy local result still does not prove relay admission. A device-identity refusal during control registration is a non-retryable CREDENTIAL_ERROR, exit 3; a catalog credential that is not yet valid remains retryable. Cluster ingress now sends the typed identity refusal for an unknown key and maps forwarded owner refusals, but this cluster path has unit evidence, not a production-cluster process gate. Older owners' generic refusals remain retryable rather than being misclassified as terminal.

A protocol-major mismatch closes with PROTOCOL_UNSUPPORTED and becomes a non-retryable PROTOCOL_ERROR, exit 1. This does not establish general mixed-version upgrade support. Runtime tests were not rerun for this website review.

Credential import recovery

credentials import refuses expired certificates and warns for certificates whose validity starts in the future. It checks both destinations and installs without replacing existing files. With hard-link support, a crash between installing the certificate and CA can leave only one installed; retry the same import to complete the pair. On filesystems without hard links, the direct-write fallback can leave a partial file after a crash, which the next import refuses by name. Inspect that refusal rather than assuming the two-file install is crash-atomic. Renewal and self-service enrollment remain unimplemented.

Before a device certificate expires

Renewal is deferred in this alpha. Replacing the certificate by hand is not a workaround: the catalog retains the first certificate's expiry. Re-enroll as a new device before expiry: create a new key and CSR in a new profile, have the issuer sign it for a new device UUID, then import it. Register the new device with tunnel-relay add-device, its services with add-service, and permissions with set-grant. Set the new profile's device and export IDs, switch the connector and every cloud caller to those IDs, then revoke-device the old device. Existing caller URLs do not automatically migrate. Use the setup and catalog workflow; this is new enrollment, not seamless renewal.

Consumer refusals

Missing, invalid, expired or unknown consumer credentials receive 401 UNAUTHENTICATED. A recognized consumer lacking the required token scope receives 403 ACCESS_DENIED; undiscoverable resources remain hidden behind the route's 404 response. Diagnose the error code and execution status together, and do not treat an HTTP failure as proof that earlier work never ran.

Operational surfaces

No audit log, client status command or network doctor is supplied in this alpha. Relay logs go to stderr; retention and rotation are external responsibilities. Treat logs as sensitive: connect --json lifecycle events include local socket addresses and deadlines, even though they carry no payloads or credentials.

Private metrics

Current main optionally serves Prometheus text at GET /metrics on a separate listener. Set the top-level relay option below; metrics are off by default.

metrics_bind = "127.0.0.1:9464"

This plain-HTTP endpoint is unauthenticated: restrict it to private scrapers. Configuration checks reject public, wildcard and consumer/device listener addresses. The public listeners do not expose metrics. Scrapes contain aggregate tunnel_relay_ counters and gauges with fixed labels, not device identifiers, paths, endpoints or payloads. Counters reset with the process. Only one snapshot runs at a time, with a two-second deadline; concurrent scrapes receive 503. Latency, RTT and rotation-duration measurements are not supplied. Older binaries reject metrics_bind as an unknown field: remove it before a rollback.

Readiness

/livez checks process liveness only. Cluster /readyz checks admission conditions. Current main's single-relay readiness also uses a cached, read-only Redis authority check, refreshed once per second with a five-second deadline. Failure can take about six seconds to appear as 503; readiness returns after a successful check. The HTTP request itself does not contact Redis. Run-binding, continuity and persistence refusals remain fenced. Older builds, including the recorded Fly smoke build 77bfd28, reported ready without this check. Neither endpoint proves an end-to-end capability works.

Redis connection diagnostics

Current main reports bounded connection stages and failure classes in serve, activate-first-incarnation, provision-catalog, recovery-observe and recover. Older binaries may report only a generic connection failure.

Redis catalog connection failed; stage=connection_establishment lane=3/6 class=timeout

This example means the initial connection and two lanes opened, then lane three failed. The optional lane number identifies one of six authority connections opened after the initial connection. Recovery commands use the prefix recovery Redis connection failed;. These diagnostics use fixed words rather than URLs, hostnames, credentials or Redis reply text.

Each startup or recovery connection gets a 10-second connection budget covering DNS, TCP, TLS and AUTH; subsequent commands have a 2-second deadline. This is not a 10-second bound for the whole startup. Current main also gives live authority-lane reconnects 10 seconds, outside the lane lock. Other callers still fail closed within their own two-second deadline while reconnection runs. This replaces the older two-second whole-reconnect limit that could fail on cold DNS. Source and scoped regression evidence were reviewed, not rerun as deployment acceptance.

Redis restart and backup

Current main supports one narrow case: a single non-cluster relay's Redis restarting in place from its own durable AOF, retaining acknowledged writes. It does not cover replica promotion, restored backups or replaced volumes. Cluster Redis restarts and uncertain authority still require the recovery workflow with an external signing authority that is not shipped.

Automatic continuity: the optional top-level relay setting redis_restart_continuity_seconds accepts 1 to 60 seconds and is rejected with [cluster]. The relay must remain running across the Redis restart. It periodically records a token and can rebind the namespace to the new Redis run only when continuity and durability checks pass. Both startup and ongoing checks require appendonly yes, appendfsync always and no-appendfsync-on-rewrite no, with permission for CONFIG GET. The Fly template sets the interval to 5 seconds; this is not a global default.

Do not use continuity with failover or restore. A recent backup or replica may hold the token yet omit later acknowledged revocations. Never weaken the persistence settings at runtime: detection has an interval-plus-deadline window, and a crash inside it can still lose recent writes. A stopped or failing token loop cannot indefinitely authorize a later restart. These checks do not replace operator control of Redis's storage history.

Operator attestation: if the relay did not remain running, or continuity was not enabled, first verify that the same Redis restarted from its own data directory with no restore, volume replacement or replica promotion. Only then use the following command with the same single-relay configuration:

tunnel-relay rebind-redis-run --config /absolute/path/relay.toml --redis-restarted-in-place

The flag is an operator declaration, not an automated proof that data survived. The command preserves the namespace and incarnation and refuses an unbound namespace or cluster configuration. It does not clear a running relay's previously detected continuity mismatch. Do not use it to bypass a restore or lost-data refusal; follow externally approved recovery or provision a genuinely new deployment instead.

Repository process evidence covers in-place restart/crash and selected lost-data refusals using shipped binaries; it is not hosted deployment acceptance or a new test run here. Devices may lose sessions and reconnect, and owner leases/backoff can delay resumed service. Cached readiness has a detection delay and does not replace an end-to-end operation.

Redis backups still use operator tooling. Historical Redis 8.4.0 dogfood evidence found AOF transaction replay failures with the default user disabled and a separate relay ACL user. No least-privilege ACL profile is validated. Test your exact Redis version, persistence and ACL configuration without assuming a saved AOF restores authorization records or weakening authentication. Preserve membership and recovery state.

Reconnect limits

After this process's own session ends, OWNER_BUSY is retried for up to 60 seconds; on a fresh process's first attempt it exits 7. A relay crash can retain ownership for the lease duration, up to 30 seconds. Current main's owner-local device control loop sends WebSocket Pings every 10 seconds and evicts after 30 seconds without any inbound frame, followed by a disconnect hand-off bounded at 5 seconds. These constants are not configurable. The client reconnects on its next attempt, not necessarily at exactly 30 seconds.

Repository evidence records an owner-local cut-path reconnect and a healthy idle session surviving the check. Older relays and devices attached through a non-owner cluster relay are not covered by this fix and can still retain stale ownership; an operator-managed relay restart interrupts its other sessions too. Reconnect does not restore in-flight operations or authorize replay.

Current main retains and coalesces control-socket WebSocket Pongs without a critical-response deadline. A full queue defers the Pong instead of ending the session after five seconds, and reserved OPEN capacity is preserved. Pongs can still arrive late or be replaced; this does not guarantee remote liveness under saturation. Source and regression evidence are not complete reconnect acceptance.

Delayed terminal acknowledgement

Current main treats an expired STREAM_FORGET revalidation window as retryable TRANSPORT_ERROR only when all proof checks except the final acknowledgement hold. A fresh session starts with an empty journal; it does not replay old operations. Contradictory cursors, byte counts, terminals or identities remain terminal protocol errors. The five-second window bounds pending proof retention, not a reconnect deadline or a guarantee that a side effect did not occur. Scoped regression evidence exists; broader acceptance remains separate.

Rotation admission

Current main holds new stream OPENs and finite unary echoes during QUIESCE through COMMIT instead of immediately refusing ordinary rotation admission. Held work is not dispatched and uses no extra socket. The hold is at most 1.5 seconds, shortened by configured deadlines, and bounded to eight requests per device (also subject to stream limits), 64 per tenant and 256 per relay. Unary bodies are capped at 64 KiB each. After commit, or an abort that resumes the old carrier, normal admission resumes. A disconnected consumer drops its held work.

A full or expired hold returns 503 ROTATION_FREEZE with retry guidance and no dispatch. Filesystem admission uses its own error schema and creates no session; its shared client maps this 503 to retryable backend-unavailable. Fault and recovery states retain their separate refusals. Retry only known not-dispatched work; reconcile ambiguous effects first.

Cluster upgrade requirement: all relays need the same version and identical operation_timeout and cluster.peer_idle_timeout_seconds. Startup does not enforce this. Older ingress relays do not understand the new private refusal and can report an unknown outcome; a shorter ingress deadline can expire before the owner's hold. Mixed-version and mismatched-timeout support remain open.

Historical dogfood and the Fly smoke build 77bfd28 saw rotation-time PEER_UNAVAILABLE or RESOURCE_EXHAUSTED refusals before this change. Those observations are not measurements of the new hold. Scoped source and regression evidence do not establish seamless rotation for every capability; runtime acceptance was not rerun here.

Current main resumes connector OPEN admission at rotation COMMIT, including the retiring phase, to match the owner. This addresses a separate intermittent 503 DEVICE_REJECTED caused by the connector refusing new work as draining after the owner had resumed admission. Admission closes again if the active data carrier is lost, including during retirement. Source and focused regression tests cover these decisions; they were not rerun for this website review.

Remaining limit: the connector's writer stays frozen until ROTATE_COMPLETE even though new operations can be admitted earlier. If retirement stalls, output may wait for forced completion at the overlap deadline plus handshake grace, 30 + 10 seconds with defaults. This is a source-derived bound, not a measured latency guarantee. A shorter consumer timeout can leave an unknown operation outcome rather than an explicit refusal. Do not replay side effects merely because the response timed out. Complete rotation acceptance remains open.

Admission and cluster limits

After a burst of stream releases, the owner may admit a consumer before the connector has retired those streams from its own capacity count. The consumer can receive a WebSocket 101 upgrade followed by a Close without a typed reason when the connector refuses the OPEN. This remains unresolved; an upgrade alone does not prove that work was admitted by the device.

Current main fixes a stale cached owner epoch after a device reconnects to the same relay: peer routing now binds to the freshly read owner generation rather than an older cached one. Scoped production-gate evidence supports the fix, but some peer-ingress refusals still end without response headers and can fail the shared HTTP/3 connection, interrupting other multiplexed streams. The routing fix removes a common trigger, not the remaining refusal defect. Complete cluster acceptance remains open; these runtime gates were not rerun for this website review.

Queue-saturation tests pin socket buffers only in the fixture. Production kernel buffers add OS-bounded capacity beyond the relay's configured application queues; the test's buffer sizes are not production limits. A graceful WebSocket Close also follows already-buffered records, so cancellation under saturation can take seconds while queued work continues. Do not assume immediate cancellation or blindly replay interrupted operations.

Echo retention limits

Completed finite echoes now have explicit retention cleanup, addressing the earlier failure after 128 requests in one session. Other exits, including RESET, cancellation, authorization invalidation and an expired or disconnected consumer request, can still retain entries. They can exhaust capacity or break a rotation roster. A client reporting an active session is not proof that it can admit new work; automatic replacement of a retention-exhausted session remains planned. Do not increase limits or blindly repeat interrupted operations as a substitute for resolving their outcomes.

Connected but access denied

Check the authenticated principal's tenant membership and device/service/operation grant, local export configuration and policy revision. Some inaccessible resources deliberately appear as not found rather than leaking their existence.

Catalog changes and revocation

The catalog-change CLI adds records and replaces or revokes grants while the relay serves. It does not add a second tenant or renew an existing device certificate. A dry run checks local inputs, not current authority state. Refusals exit 1; a second stop signal can abandon a writing command with exit 130 and an unknown outcome.

Revocation is not retroactive cancellation of completed effects. A grant already admitted can remain authorized for up to five seconds. Device/credential revocation is also detected by session maintenance, which checks up to 64 sessions every 500 ms. That schedule and Redis latency are not an instantaneous global cutoff. Repository evidence measured one device's revocation and terminal credential refusal on reconnect; the live-session close for credential-only revocation was not measured. No revocation runtime tests were rerun for this website review.

Control-path stalls and authorization expiry

The connector buffers stream input while authorization refresh is unconfirmed. If authorization expires, it discards that buffered input even when transport acknowledgements have already removed it from relay replay. Stall tolerance remains limited; transport acknowledgement is not application completion.

Corrected measurements: the earlier claim that a two-second stall lost an echo used an experiment whose actual stall lasted at least five seconds. In the corrected Linux-container experiment, exact two- and three-second stalls each delivered 3 of 3 records; four- and 5.5-second stalls each delivered 2 of 3. These small samples are not a timeout guarantee, and they do not fully explain the earlier hosted failures.

Current main fixes unanswered echo waiters when the connector sends FIN or RESET: the relay produces DEVICE_CLOSED or DEVICE_RESET, with execution unknown for dispatched records and not_dispatched for queued records. Scoped container evidence now records consumer closure as a lapsed stall ends instead of waiting out the old 30-second timeout. This is an echo-path fix, not proof that every capability or network fault delivers a prompt error. Reconcile unknown side effects before retrying.

The connector also skips already-invalidated streams during expiry checks; a focused regression test covers one expiry across five ticks. Internal StreamAuthCounters expose unconfirmed streams, refreshes in flight, buffered input and expired streams in connection status and harness diagnostics, not a new CLI command. Harness hold fixes and these product fixes do not establish complete transport acceptance. Runtime tests were not rerun for this website review; hosted verification remains separate.

Interrupted operation

Distinguish not-dispatched from unknown outcome. Reconcile the application state before retrying writes, clicks or prompts. Do not treat a transport acknowledgement or HTTP 202 as completion.

Reporting a problem

Capture the source revision, platform, selected profile, redacted identifiers, phase and counters. Exclude tokens, private keys, tool payloads, screenshots of private data and user file contents.