Nightly Refresh¶

Operating model¶

Recurring nightly refresh and lifecycle recovery are separate worker planes.

.github/workflows/nightly.yml selects only recurring accounts.
.github/workflows/connection-lifecycle.yml polls non-ready onboarding and recovery work independently.

The intended nightly source is public.amazon_recurring_eligibility: current derived readiness, provider status verified, no revocation, an active manifest, and recurring_enabled=true. A provider-verified account is not enough. A blocked or awaiting-credentials account must not enter the nightly matrix, but it must remain visible to lifecycle recovery.

The live scheduled selector is still legacy. See Live scheduled selector below (FR-021 / FR-022). Do not treat eligibility as what scheduled runs use until that cutover condition is met.

There is no default seller. Every execution command receives an explicit seller key. Amazon onboarding and nightly execution do not require QBO, Ads, daily inventory, or an external sheet.

Live scheduled selector¶

Scheduled nightly-refresh uses --source legacy. .github/workflows/nightly.yml pins schedule events to legacy and defaults workflow_dispatch to legacy. SQP submit and harvest call scripts/list_active_sellers.py --source legacy so they share that list instead of a hardcoded seller id set.

legacy is the live selector because the eligibility cutover is not yet true for every active production account. Switching scheduled work to eligibility today would drop ecomhd_us and culinary_couture_us from recurring Amazon work with no operator-visible signal.

Do not change the scheduled selector until each account it would drop has been proven against the current standard (FR-022). The accountable release owner records before/after matrices, then updates the scheduled default in nightly.yml and this section together. A silent cutover is not allowed.

The intended steady state remains public.amazon_recurring_eligibility (--source eligibility) after that proof. compatibility and legacy stay as documented rollback modes below.

Owners¶

Boundary	Action owner	Receipt
Lifecycle queue, stage retry, stale lease	Lifecycle worker / platform operations	Fenced lifecycle stage receipt
Nightly selection and seller plan	Platform operations	Selector matrix plus per-seller plan
Amazon collector failure	Amazon ingestion owner	Sanitized source/stage reason and coverage
Forecast or recommendations	Forecast owner	Accepted fingerprint and run reference
Runtime feed publication/readback	Console platform owner	Publication and authenticated readback IDs
Selector cutover or rollback	Accountable release/incident owner	Before/after matrices and workflow receipt
Production deploy or migration	Accountable release owner	Separate release receipt

Read-only preflight¶

Run these from the repository root. They query selection and plans but execute no pipeline stage:

python3 scripts/list_active_sellers.py \
  --source eligibility \
  --matrix

python3 scripts/connection_lifecycle/runner.py \
  --manifest-id <manifest-uuid> \
  --plan

python3 scripts/nightly_refresh.py \
  --seller <eligible-runtime-seller-id> \
  --plan

Expected receipts:

The selector prints {"include":[...]} with only derived-ready recurring sellers; an empty array is valid and must not fall back to legacy accounts.
The lifecycle plan prints the same stored and compiled registry fingerprint.
The nightly plan contains only the explicit seller and never substitutes another account.

To inspect a non-eligible seller without making it runnable:

python3 scripts/list_active_sellers.py \
  --source eligibility \
  --seller <runtime-seller-id> \
  --verify-only \
  --matrix

The --seller flag is rejected unless --verify-only is present.

Staged eligibility cutover¶

Do not cut over until migration 0100 passes disposable apply/reapply and rollback checks, reconciliation dry-run is approved, both workflow YAML files validate, the runtime feed is integrated with the lifecycle console_feed claim, and the independent acceptance suite passes.

Stage 0: compatibility observation¶

Compare the normal derived matrix with the bounded migration allowlist:

python3 scripts/list_active_sellers.py --source eligibility --matrix \
  > /private/tmp/nightly-eligibility.json
python3 scripts/list_active_sellers.py --source compatibility --matrix \
  > /private/tmp/nightly-compatibility.json

Expected difference: only explicitly reviewed, unexpired legacy allowlist rows. Revoked, disabled, inactive, Culinary identity-blocked, and Keto awaiting-credentials accounts must not appear.

Stage 1: credential and plan verification¶

An authorized release owner may dispatch a verify-only workflow after the read-only local preflight is green:

gh workflow run nightly.yml \
  -f selector=eligibility \
  -f verify_only=true

Expected receipt: the accounts job succeeds; every matrix row passes credential and plan verification; no Run nightly refresh step executes; no provider, database, sheet, accounting, or deployment write occurs.

Stage 2: scheduled selector cutover¶

After the approved matrix is saved and the prior selector value is recorded, the accountable release owner sets the scheduled default:

gh variable set STICKY_NIGHTLY_SELECTOR --body eligibility

Expected receipt: the next scheduled accounts job uses eligibility; its matrix matches the approved set; each seller emits a pipeline-execution-<seller> artifact. Freshness remains a separate signal and cannot turn a failed execution green.

Stage 3: observation¶

For each included seller, preserve:

accounts-job matrix;
explicit seller plan;
first failed stage or completed per-stage receipts;
accepted forecast/recommendation fingerprints;
tenant runtime publication and authenticated readback IDs; and
final workflow conclusion.

Do not add a seller to recurring execution to repair onboarding. Recovery owns non-ready accounts.

Manual recurring run¶

This command performs production work and requires explicit authority:

python3 scripts/nightly_refresh.py --seller <eligible-runtime-seller-id>

For GitHub Actions, the authorized release/incident owner may dispatch:

gh workflow run nightly.yml \
  -f selector=eligibility \
  -f verify_only=false \
  -f skip_external_writes=true

skip_external_writes=true suppresses accounting and external exports but still allows internal provider/database work. It is not a dry-run. Use --plan or verify_only=true for no-write verification.

Receipts and failure ownership¶

The nightly executor writes .pipeline-status/<seller>-<phase>.json and one public.pipeline_step_receipts row per step via scripts/run_receipts.py, including failures, timeouts, and skipped steps. Each failed receipt stores a plain-language cause, an owner, and a sanitized error excerpt. A failed step makes the run's overall result failed (ok: false); a later summary job cannot turn that into success. GitHub uploads pipeline-execution-<seller>. A failed dependency blocks downstream work rather than letting it measure stale inputs.

Lifecycle receipts are database records created under a current lease and fencing token. Required success also needs the current stage input fingerprint and query-backed postcondition. Provider coverage may be queried_nonzero or queried_zero; not_attempted is not success. Optional failures retain age, last error, and retry time without removing readiness.

Start diagnosis at the first failed boundary:

Confirm which selector and seller matrix were used.
Open the seller execution receipt and identify the first failure; later blocked stages are consequences.
Compare its input fingerprint with the current expected fingerprint.
For lifecycle work, inspect lease owner, lease expiry, fencing token, sanitized reason code, and retry time.
Reproduce with the explicit seller in --plan mode before authorizing work.

Never log a credential, raw provider error, raw response body, database URL, or authorization header. Receipts use stable reason codes and opaque source IDs.

Incident recovery¶

Interrupted or stale lifecycle worker¶

Do not complete the stage manually. Wait for the old lease to expire; the recovery worker may then create a higher fencing token. Confirm the old token is rejected by stage completion and publication-pointer advancement. The recovery workflow polls independently:

python3 scripts/connection_lifecycle/runner.py --poll-once

That command writes production state and is limited to the lifecycle worker or an explicitly authorized incident owner. It claims at most one run. A missing claim is a successful no-op, not permission to use nightly eligibility as the recovery queue.

Revoked or disabled credential¶

Revocation must immediately invalidate current evidence and remove the seller from every selector, including compatibility and legacy rollback modes. Stop any manual retry, preserve the sanitized invalidation receipt, and route secure reconnection to the authorized account operator. Never bypass the kill switch by changing raw.sellers.is_active or the nightly selector.

Runtime feed interruption¶

An object upload without fenced pointer advancement and authenticated readback is not published. Preserve the previous pointer. Retry only under the current console_feed claim; the content-addressed object makes the same-input retry idempotent. A stale worker cannot advance the pointer.

Rollback¶

Selector rollback is reversible and does not delete lifecycle evidence.

Preferred rollback: bounded compatibility¶

gh variable set STICKY_NIGHTLY_SELECTOR --body compatibility

Expected receipt: derived-ready sellers plus only active, verified, non-revoked, unexpired allowlist sellers. Use this while repairing a lifecycle evidence gap.

Emergency rollback: legacy registry¶

gh variable set STICKY_NIGHTLY_SELECTOR --body legacy

Expected receipt: active legacy registry sellers, still excluding any revoked Amazon connection. This restores pre-cutover selection only; it does not let any route, operator, workflow, or stage assert ready.

Before either rollback, save the current selector and matrices. After changing it, run the verify-only workflow and compare the actual matrix with the approved rollback set. If it differs, stop; do not dispatch recurring execution.

Restore normal selection after remediation and a green acceptance rerun:

gh variable set STICKY_NIGHTLY_SELECTOR --body eligibility

Migration rollback, data deletion, workflow dispatch, and deployment are separate release actions and are not authorized by this runbook.

Required verification commands¶

python3 -m unittest tests.test_connection_lifecycle_acceptance -v
python3 -m unittest \
  tests.test_connection_lifecycle_store \
  tests.test_connection_lifecycle_amazon \
  tests.test_tenant_runtime_feed \
  tests.test_seller_pipeline -v

cd hd
bunx tsc --noEmit
node --test \
  src/lib/amazon/connection-store.test.ts \
  src/lib/company-settings.test.ts \
  src/lib/runtime-feed.test.ts

Validate .github/workflows/nightly.yml and .github/workflows/connection-lifecycle.yml with actionlint. All gates must be green before cutover; a release recommendation is not permission to deploy.