Test a shadow policy

Evaluate a candidate policy in shadow — live or by historical replay — and see where it would diverge from the active policy, without changing any real decision. Shadow evaluation is structurally non-authoritative: it cannot alter the active verdict.

Prerequisites

• An authenticated customer session on a hosted or embedded EVE service (shadow is authenticated).
• Historical decisions to replay, or live traffic to observe.
• No production credentials in examples; use the authenticated pilot session.

Installation

Shadow evaluation runs in the EVE service and is reached through the authenticated shadow API (saas/routers/shadow_router.py). No client package publishes shadow.

Runnable code

Conceptually, register a candidate policy for shadow, replay historical decisions against it, and read the signed divergence report:

# 1) Submit a candidate policy for shadow evaluation (authenticated).
# 2) Replay historical decisions or observe live traffic; the ACTIVE decision is
#    unchanged — shadow only records what the candidate WOULD have decided.
# 3) Read the signed divergence report:
#    {agreement_permille, divergence_by_class, critical_regressions}

Rates are carried as integer permille so the signed report stays inside the supported canonicalization domain.

Expected result

• The active decision is unchanged for every request — shadow never alters the live verdict.
• The report shows agreement (in permille), divergences grouped by class, and any critical regressions the candidate would introduce.
• A "promote" step is dry-run only and does not activate the candidate as a production policy.

Evidence output

signed shadow report v2 (jcs-1) with divergence_by_class
shadow report {agreement_permille, divergence_by_class, critical_regressions}

Verification

The shadow report is signed over jcs-1 canonicalization; verify it offline in Python, Node, or the browser (see verify-evidence-offline.md).

Failure example

Divergence estimates are modeled effects from replay or sampling — they are explicitly not observed production outcomes. Reading them as measured live impact would overstate what shadow proves.

Production considerations

• Use shadow to gate a candidate policy before any change to the active one; treat critical regressions as blockers.
• Promotion from shadow is dry-run only; activating a policy is a separate, deliberate step outside shadow.
• Keep rates as integer permille to stay within the signed canonicalization domain.

Limitations

• Shadow evaluation is authenticated pilot-ready (PILOT_READY). It is structurally non-authoritative and cannot alter the active verdict; promotion is dry-run only and never activates a production policy.
• Divergence estimates are modeled effects from replay/sample, explicitly not observed production outcomes.

Next step

Once a candidate looks safe, exercise it adversarially with run-a-red-team-suite.md, and verify all reports with verify-evidence-offline.md.