Analyze Test Run

1---2name: analyze-test-run3description: "Analyze a GitHub Actions integration test run and produce a skill invocation report with failure root-cause issues. TRIGGERS: analyze test run, skill invocation rate, test run report, compare test runs, skill invocation summary, test failure analysis, run report, test results, action run report"4license: MIT5metadata:6  author: Microsoft7  version: "1.0.6"8---9 10# Analyze Test Run11 12Downloads artifacts from a GitHub Actions integration test run, generates a summarized skill invocation report, and files GitHub issues for each test failure with root-cause analysis.13 14## When to Use15 16- Summarize results of a GitHub Actions integration test run17- Calculate skill invocation rates for the skill under test18- For azure-deploy tests: track the full deployment chain (azure-prepare → azure-validate → azure-deploy)19- Compare skill invocation across two runs20- File issues for test failures with root-cause context21 22## Input23 24| Parameter | Required | Description |25|-----------|----------|-------------|26| **Run ID or URL** | Yes | GitHub Actions run ID (e.g. `22373768875`) or full URL |27| **Comparison Run** | No | Second run ID/URL for side-by-side comparison |28 29## MCP Tools30 31All tools use `owner: "microsoft"` and `repo: "GitHub-Copilot-for-Azure"` as fixed parameters. `method` selects the operation within the tool.32 33| Tool | `method` | Key Parameter | Purpose |34|------|----------|---------------|---------|35| `actions_get` | `get_workflow_run` | `resource_id`: run ID | Fetch run status and metadata |36| `actions_list` | `list_workflow_run_artifacts` | `resource_id`: run ID | List all artifacts for a run |37| `actions_get` | `download_workflow_run_artifact` | `resource_id`: artifact ID | Get a temporary download URL for an artifact ZIP |38| `get_job_logs` | — | `run_id` + `failed_only: true` | Retrieve job logs when artifact content is inaccessible |39| `search_issues` | — | `query`: search string | Find existing open issues before creating new ones |40| `create_issue` | — | `title`, `body`, `labels`, `assignees` | File a new GitHub issue for a test failure |41 42## Workflow43 44### Phase 1 — Download & Parse45 461. Extract the numeric run ID from the input (strip URL prefix if needed)472. Fetch run metadata using the MCP `actions_get` tool:48   ```javascript49   actions_get({ method: "get_workflow_run", owner: "microsoft", repo: "GitHub-Copilot-for-Azure", resource_id: "<run-id>" })50   ```513. List artifacts using the MCP `actions_list` tool, then download each relevant artifact:52   ```javascript53   // List artifacts54   actions_list({ method: "list_workflow_run_artifacts", owner: "microsoft", repo: "GitHub-Copilot-for-Azure", resource_id: "<run-id>" })55   // Download individual artifacts by ID56   actions_get({ method: "download_workflow_run_artifact", owner: "microsoft", repo: "GitHub-Copilot-for-Azure", resource_id: "<artifact-id>" })57   ```58   The download returns a temporary URL. Fetch the ZIP archive from that URL and extract it locally. If the environment restricts outbound HTTP (e.g. AWF sandbox), record in the analysis report that artifact content was unavailable and fall back to job logs via the `get_job_logs` MCP tool.59 604. Locate these files in the downloaded artifacts:61   - `junit.xml` — test pass/fail/skip/error results62   - `*-SKILL-REPORT.md` — generated skill report with per-test details63   - `agent-metadata-*.md` files — raw agent session logs per test64 65   > ⚠️ **Note:** If artifact ZIP files cannot be downloaded due to network restrictions, or if downloaded files cannot be extracted, use the `get_job_logs` MCP tool to identify test failures and produce a best-effort analysis from whatever data is accessible.66 67### Phase 2 — Build Summary Report68 69Produce a markdown report with four sections. See [report-format.md](references/report-format.md) for the exact template.70 71**Section 1 — Test Results Overview**72 73Parse `junit.xml` to build:74 75| Metric | Value |76|--------|-------|77| Total tests | count from `<testsuites tests=…>` |78| Executed | total − skipped |79| Skipped | count of `<skipped/>` elements |80| Passed | executed − failures − errors |81| Failed | count of `<failure>` elements |82| Test Pass Rate | passed / executed as % |83 84Include a per-test table with name, duration (from `time` attribute, convert seconds to `Xm Ys`), and Pass/Fail result.85 86**Section 2 — Skill Invocation Rate**87 88Read the SKILL-REPORT.md "Per-Test Case Results" sections. For each executed test determine whether the skill under test was invoked.89 90The skills to track depend on which integration test suite the run belongs to:91 92**azure-deploy integration tests** — track the full deployment chain:93 94| Skill | How to detect |95|-------|---------------|96| `azure-prepare` | Mentioned as invoked in the narrative or agent-metadata |97| `azure-validate` | Mentioned as invoked in the narrative or agent-metadata |98| `azure-deploy` | Mentioned as invoked in the narrative or agent-metadata |99 100Build a per-test invocation matrix (Yes/No for each skill) and compute rates:101 102| Skill | Invocation Rate |103|-------|----------------|104| azure-deploy | X% (n/total) |105| azure-prepare | X% (n/total) |106| azure-validate | X% (n/total) |107| Full skill chain (P→V→D) | X% (n/total) |108 109> The azure-deploy integration tests exercise the full deployment workflow where the agent is expected to invoke azure-prepare, azure-validate, and azure-deploy in sequence. This three-skill chain tracking is **specific to azure-deploy tests only**.110 111**All other integration tests** — track only the skill under test:112 113| Skill | Invocation Rate |114|-------|----------------|115| {skill-under-test} | X% (n/total) |116 117For non-deploy tests (e.g. azure-prepare, azure-ai, azure-kusto), only track whether the primary skill under test was invoked. Do not include azure-prepare/azure-validate/azure-deploy chain columns.118 119**Section 3 — Report Confidence & Pass Rate**120 121Extract from SKILL-REPORT.md:122- Skill Invocation Success Rate (from the report's statistics section)123- Overall Test Pass Rate (from the report's statistics section)124- Average Confidence (from the report's statistics section)125 126**Section 4 — Comparison** (only when a second run is provided)127 128Repeat Phase 1–3 for the second run, then produce a side-by-side delta table. See [report-format.md](references/report-format.md) § Comparison.129 130### Phase 3 — File Issues for Failures131 132For every test with a `<failure>` element in `junit.xml`:133 1341. Read the failure message and file:line from the XML1352. Read the actual line of code from the test file at that location1363. Read the `agent-metadata-*.md` for that test from the artifacts1374. Read the corresponding section in the SKILL-REPORT.md for context on what the agent did1385. Determine root cause category:139   - **Skill not invoked** — agent bypassed skills and used manual commands140   - **Deployment failure** — infrastructure or RBAC error during deployment141   - **Timeout** — test exceeded time limit142   - **Assertion mismatch** — expected files/links not found143   - **Quota exhaustion** — Azure region quota prevented deployment1446. Search for existing open issue before creating a new one using the `search_issues` MCP tool:145   ```javascript146   search_issues({147     owner: "microsoft", repo: "GitHub-Copilot-for-Azure",148     query: "Integration test failure: {skill} in:title is:open"149   })150   ```151   Match criteria: an open issue whose title and body describe a similar problem. If a match is found, skip issue creation for this failure and note the existing issue number(s) in the summary report.1527. If no existing issue was found, create a GitHub issue using the `create_issue` MCP tool, assign the label with the name of the skill, and assign it to the code owners listed in .github/CODEOWNERS file based on which skill it is for:153 154```javascript155create_issue({156  owner: "microsoft", repo: "GitHub-Copilot-for-Azure",157  title: "Integration test failure: <skill> – <keywords> [<root-cause-category>]",158  labels: ["bug", "integration-test", "test-failure", "<skill>"],159  body: "<body>",160  assignees: ["<codeowners>"]161})162```163 164   **Title format:** `Integration test failure: {skill} – {keywords} [{root-cause-category}]`165   - `{keywords}`: 2-4 words from the test name — app type (function app, static web app) + IaC type (Terraform, Bicep) + trigger if relevant166   - `{root-cause-category}`: one of the categories from step 5 in brackets167 168Issue body template — see [issue-template.md](references/issue-template.md).169 170> ⚠️ **Note:** Do NOT include the Error Details (JUnit XML) or Agent Metadata sections in the issue body. Keep issues concise with the diagnosis, prompt context, skill report context, and environment sections only.171> ⚠️ **Note:** Do NOT create issues for skill invocation test failures.172 173> For azure-deploy integration tests, include an "azure-deploy Skill Invocation" section showing whether azure-deploy was invoked (Yes/No), with a note that the full chain is azure-prepare → azure-validate → azure-deploy. For all other integration tests, include a "{skill} Skill Invocation" section showing only whether the primary skill under test was invoked.174 175## Error Handling176 177| Error | Cause | Fix |178|-------|-------|-----|179| `no artifacts found` | Run has no uploadable reports | Verify the run completed the "Export report" step |180| `HTTP 404` on `actions_get` | Invalid run ID or no access | Check the run ID and ensure the MCP token has repo access |181| `rate limit exceeded` | Too many GitHub API calls | Wait and retry; reduce concurrent MCP tool calls |182| Artifact ZIP download blocked | AWF sandbox restricts outbound HTTP to blob storage | Use `get_job_logs` MCP tool to get failure details from job logs; produce best-effort analysis from metadata |183 184## References185 186- [report-format.md](references/report-format.md) — Output report template187- [issue-template.md](references/issue-template.md) — GitHub issue body template