Healthcare Eval Harness

1---2name: healthcare-eval-harness3description: Patient safety evaluation harness for healthcare application deployments. Automated test suites for CDSS accuracy, PHI exposure, clinical workflow integrity, and integration compliance. Blocks deployments on safety failures.4origin: Health1 Super Speciality Hospitals — contributed by Dr. Keyur Patel5version: "1.0.0"6---7 8# Healthcare Eval Harness — Patient Safety Verification9 10Automated verification system for healthcare application deployments. A single CRITICAL failure blocks deployment. Patient safety is non-negotiable.11 12> **Note:** Examples use Jest as the reference test runner. Adapt commands for your framework (Vitest, pytest, PHPUnit, etc.) — the test categories and pass thresholds are framework-agnostic.13 14## When to Use15 16- Before any deployment of EMR/EHR applications17- After modifying CDSS logic (drug interactions, dose validation, scoring)18- After changing database schemas that touch patient data19- After modifying authentication or access control20- During CI/CD pipeline configuration for healthcare apps21- After resolving merge conflicts in clinical modules22 23## How It Works24 25The eval harness runs five test categories in order. The first three (CDSS Accuracy, PHI Exposure, Data Integrity) are CRITICAL gates requiring 100% pass rate — a single failure blocks deployment. The remaining two (Clinical Workflow, Integration) are HIGH gates requiring 95%+ pass rate.26 27Each category maps to a Jest test path pattern. The CI pipeline runs CRITICAL gates with `--bail` (stop on first failure) and enforces coverage thresholds with `--coverage --coverageThreshold`.28 29### Eval Categories30 31**1. CDSS Accuracy (CRITICAL — 100% required)**32 33Tests all clinical decision support logic: drug interaction pairs (both directions), dose validation rules, clinical scoring vs published specs, no false negatives, no silent failures.34 35```bash36npx jest --testPathPattern='tests/cdss' --bail --ci --coverage37```38 39**2. PHI Exposure (CRITICAL — 100% required)**40 41Tests for protected health information leaks: API error responses, console output, URL parameters, browser storage, cross-facility isolation, unauthenticated access, service role key absence.42 43```bash44npx jest --testPathPattern='tests/security/phi' --bail --ci45```46 47**3. Data Integrity (CRITICAL — 100% required)**48 49Tests clinical data safety: locked encounters, audit trail entries, cascade delete protection, concurrent edit handling, no orphaned records.50 51```bash52npx jest --testPathPattern='tests/data-integrity' --bail --ci53```54 55**4. Clinical Workflow (HIGH — 95%+ required)**56 57Tests end-to-end flows: encounter lifecycle, template rendering, medication sets, drug/diagnosis search, prescription PDF, red flag alerts.58 59```bash60tmp_json=$(mktemp)61npx jest --testPathPattern='tests/clinical' --ci --json --outputFile="$tmp_json" || true62total=$(jq '.numTotalTests // 0' "$tmp_json")63passed=$(jq '.numPassedTests // 0' "$tmp_json")64if [ "$total" -eq 0 ]; then65  echo "No clinical tests found" >&266  exit 167fi68rate=$(echo "scale=2; $passed * 100 / $total" | bc)69echo "Clinical pass rate: ${rate}% ($passed/$total)"70```71 72**5. Integration Compliance (HIGH — 95%+ required)**73 74Tests external systems: HL7 message parsing (v2.x), FHIR validation, lab result mapping, malformed message handling.75 76```bash77tmp_json=$(mktemp)78npx jest --testPathPattern='tests/integration' --ci --json --outputFile="$tmp_json" || true79total=$(jq '.numTotalTests // 0' "$tmp_json")80passed=$(jq '.numPassedTests // 0' "$tmp_json")81if [ "$total" -eq 0 ]; then82  echo "No integration tests found" >&283  exit 184fi85rate=$(echo "scale=2; $passed * 100 / $total" | bc)86echo "Integration pass rate: ${rate}% ($passed/$total)"87```88 89### Pass/Fail Matrix90 91| Category | Threshold | On Failure |92|----------|-----------|------------|93| CDSS Accuracy | 100% | **BLOCK deployment** |94| PHI Exposure | 100% | **BLOCK deployment** |95| Data Integrity | 100% | **BLOCK deployment** |96| Clinical Workflow | 95%+ | WARN, allow with review |97| Integration | 95%+ | WARN, allow with review |98 99### CI/CD Integration100 101```yaml102name: Healthcare Safety Gate103on: [push, pull_request]104 105jobs:106  safety-gate:107    runs-on: ubuntu-latest108    steps:109      - uses: actions/checkout@v4110      - uses: actions/setup-node@v4111        with:112          node-version: '20'113      - run: npm ci114 115      # CRITICAL gates — 100% required, bail on first failure116      - name: CDSS Accuracy117        run: npx jest --testPathPattern='tests/cdss' --bail --ci --coverage --coverageThreshold='{"global":{"branches":80,"functions":80,"lines":80}}'118 119      - name: PHI Exposure Check120        run: npx jest --testPathPattern='tests/security/phi' --bail --ci121 122      - name: Data Integrity123        run: npx jest --testPathPattern='tests/data-integrity' --bail --ci124 125      # HIGH gates — 95%+ required, custom threshold check126      # HIGH gates — 95%+ required127      - name: Clinical Workflows128        run: |129          TMP_JSON=$(mktemp)130          npx jest --testPathPattern='tests/clinical' --ci --json --outputFile="$TMP_JSON" || true131          TOTAL=$(jq '.numTotalTests // 0' "$TMP_JSON")132          PASSED=$(jq '.numPassedTests // 0' "$TMP_JSON")133          if [ "$TOTAL" -eq 0 ]; then134            echo "::error::No clinical tests found"; exit 1135          fi136          RATE=$(echo "scale=2; $PASSED * 100 / $TOTAL" | bc)137          echo "Pass rate: ${RATE}% ($PASSED/$TOTAL)"138          if (( $(echo "$RATE < 95" | bc -l) )); then139            echo "::warning::Clinical pass rate ${RATE}% below 95%"140          fi141 142      - name: Integration Compliance143        run: |144          TMP_JSON=$(mktemp)145          npx jest --testPathPattern='tests/integration' --ci --json --outputFile="$TMP_JSON" || true146          TOTAL=$(jq '.numTotalTests // 0' "$TMP_JSON")147          PASSED=$(jq '.numPassedTests // 0' "$TMP_JSON")148          if [ "$TOTAL" -eq 0 ]; then149            echo "::error::No integration tests found"; exit 1150          fi151          RATE=$(echo "scale=2; $PASSED * 100 / $TOTAL" | bc)152          echo "Pass rate: ${RATE}% ($PASSED/$TOTAL)"153          if (( $(echo "$RATE < 95" | bc -l) )); then154            echo "::warning::Integration pass rate ${RATE}% below 95%"155          fi156```157 158### Anti-Patterns159 160- Skipping CDSS tests "because they passed last time"161- Setting CRITICAL thresholds below 100%162- Using `--no-bail` on CRITICAL test suites163- Mocking the CDSS engine in integration tests (must test real logic)164- Allowing deployments when safety gate is red165- Running tests without `--coverage` on CDSS suites166 167## Examples168 169### Example 1: Run All Critical Gates Locally170 171```bash172npx jest --testPathPattern='tests/cdss' --bail --ci --coverage && \173npx jest --testPathPattern='tests/security/phi' --bail --ci && \174npx jest --testPathPattern='tests/data-integrity' --bail --ci175```176 177### Example 2: Check HIGH Gate Pass Rate178 179```bash180tmp_json=$(mktemp)181npx jest --testPathPattern='tests/clinical' --ci --json --outputFile="$tmp_json" || true182jq '{183  passed: (.numPassedTests // 0),184  total: (.numTotalTests // 0),185  rate: (if (.numTotalTests // 0) == 0 then 0 else ((.numPassedTests // 0) / (.numTotalTests // 1) * 100) end)186}' "$tmp_json"187# Expected: { "passed": 21, "total": 22, "rate": 95.45 }188```189 190### Example 3: Eval Report191 192```193## Healthcare Eval: 2026-03-27 [commit abc1234]194 195### Patient Safety: PASS196 197| Category | Tests | Pass | Fail | Status |198|----------|-------|------|------|--------|199| CDSS Accuracy | 39 | 39 | 0 | PASS |200| PHI Exposure | 8 | 8 | 0 | PASS |201| Data Integrity | 12 | 12 | 0 | PASS |202| Clinical Workflow | 22 | 21 | 1 | 95.5% PASS |203| Integration | 6 | 6 | 0 | PASS |204 205### Coverage: 84% (target: 80%+)206### Verdict: SAFE TO DEPLOY207```