Demo: shadow evaluation — candidate ALLOW→BLOCK divergence, no production change

Authenticated customers can evaluate candidate policies in shadow — live or by historical replay — and see divergences without changing the authoritative decision. This demo replays a set of past decisions against a stricter candidate policy, surfaces a case where the active policy ALLOWed but the candidate would BLOCK, and confirms the authoritative verdict is unchanged.

Synthetic replay fixtures. No production credentials.

Readiness

• Authenticated shadow-policy evaluation: authenticated pilot-ready (PILOT_READY).
• Policy replay + signed divergence reports: PILOT_READY.
• Python; hosted and embedded(service) modes.

Setup

# demo_shadow_evaluation.py — synthetic fixtures only
from core.eve_sdk import EVE

IDENT = {"tenant_id": "acme", "principal_id": "agent-1", "session_id": "sess-1"}
production_changes = []  # must remain empty — shadow cannot change production

# Synthetic historical decisions to replay (already-decided; no re-execution of tools).
history = [
    {"tool": "loan_approval", "arguments": {"amount": 1000},
     "context": {"credit_score": 760, "debt_to_income": 0.15}, "active_action": "ALLOW"},
    {"tool": "loan_approval", "arguments": {"amount": 40000},
     "context": {"credit_score": 610, "debt_to_income": 0.44}, "active_action": "ALLOW"},
    {"tool": "loan_approval", "arguments": {"amount": 90000},
     "context": {"credit_score": 590, "debt_to_income": 0.51}, "active_action": "ALLOW"},
]

eve = EVE(policy="lending_v1", mode="embedded")

Code

# Candidate policy is stricter (lower DTI ceiling). Evaluate it in SHADOW.
shadow = eve.shadow(candidate_policy="lending_v1_strict")   # non-authoritative

report = shadow.replay(history)   # replays decisions; does NOT re-run any tool

# Surface ALLOW (active) -> BLOCK (candidate) divergences.
divergences = [d for d in report.divergences
               if d.active_action == "ALLOW" and d.candidate_action == "BLOCK"]
print("ALLOW->BLOCK divergences:", len(divergences))
for d in divergences:
    print("  ", d.arguments, "candidate:", d.candidate_action, d.reason_codes)

# The authoritative decision is unchanged: shadow cannot alter the active verdict.
assert report.authoritative_changed is False
assert len(production_changes) == 0, "shadow must not change production"

Expected decision / divergence

The active policy remains authoritative throughout; the candidate verdicts are modeled, not applied.

Expected evidence

A signed shadow report (v2, jcs-1) with divergence classes and modeled-effect estimates:

{"report": "shadow_v2",
 "agreement_permille": 333,
 "divergence_by_class": {"allow_to_block": 2},
 "critical_regressions": 0,
 "authoritative_changed": false,
 "certificate": {"signature": "ed25519-...", "canon": "jcs-1"}}

Rates are carried as integer permille (jcs-1 supported value domain).

Verification command

from core.eve_sdk import verify_evidence
v = verify_evidence("shadow_report", report.certificate, expected_tenant="acme")
assert v["valid"], v.get("reason")
print("verify:", v["valid"])   # -> True

Zero-side-effect assertion

production_changes is empty and report.authoritative_changed is False. Replay reads past decisions and computes candidate verdicts; it re-runs no tool and never activates a policy. The active verdict is untouched.

Cleanup

production_changes.clear()   # nothing was added; symmetry only

No external state created.

Limitations

• Shadow evaluation is structurally non-authoritative and cannot alter the active verdict; promotion is dry-run only and never activates a production policy.
• Divergence estimates are modeled effects from replay/sample, explicitly not observed production outcomes.
• PILOT_READY, not production.