feat: add windows-log-analysis Copilot skill (LLM sub-skills)

[timmy-wright]

feat: add windows-log-analysis Copilot skill (LLM sub-skills)

Summary

Adds a Copilot CLI skill for diagnosing Windows AKS node issues from log bundles produced by collect-windows-logs.ps1. This skill uses LLM sub-skill markdown files that instruct AI agents how to analyze each log category.

It also adds a skill to save markdown to disk because my agent kept having so many issues with this task.

Why LLM sub-skills instead of scripts?

• Resilient to format changes — LLM reads raw files instead of brittle regex/column parsing
• Discovers novel issues — not limited to hard-coded patterns
• Parallel dispatch — each sub-skill runs as an independent sub-agent
• Domain knowledge preserved — HCS error codes, HNS failure modes, CSE error codes, known bugs from GitHub issues all encoded as analyst instructions

Architecture

SKILL.md (orchestrator)
├── common-reference.md (encoding, thresholds, error codes, dispatch guidance)
└── sub-skills/
    ├── analyze-containers.md    # Pod restarts, crash-loops, readiness
    ├── analyze-termination.md   # Stuck Terminating pods, zombie HCS, Defender file locks
    ├── analyze-images.md        # Dangling images, mutable tags, GC failures, snapshot bloat
    ├── analyze-disk.md          # C: drive free space trends
    ├── analyze-hcs.md           # Host Compute Service: lifecycle tracking, error codes, vmcompute health
    ├── analyze-hns.md           # Host Network Service: endpoints, LBs, CNI, DNS, WFP/VFP
    ├── analyze-kubeproxy.md     # kube-proxy: HNS policy sync, DSR, port range conflicts, SNAT
    ├── analyze-kubelet.md       # Node conditions, lease renewal, clock skew, cert rotation
    ├── analyze-memory.md        # Physical RAM, pagefile, OOM, process memory
    ├── analyze-crashes.md       # WER reports, minidumps, BSODs, unexpected reboots
    ├── analyze-csi.md           # CSI proxy, SMB/Azure Files mounts, Azure Disk
    ├── analyze-gmsa.md          # gMSA/CCG authentication, Kerberos, credential specs
    ├── analyze-gpu.md           # nvidia-smi, DirectX device plugin, Xid errors
    ├── analyze-bootstrap.md     # CSE flow, WINDOWS_CSE_ERROR codes (0-83), bootstrap config
    ├── analyze-extensions.md    # Azure VM extension execution errors
    └── analyze-services.md      # Windows service health, node versions, OS info

What the skill detects

Sub-Skill	Key Detections
containers	Crash-looping containers (≥10 restarts), pods not Ready
termination	Zombie HCS containers, orphaned shims, containerd reinstall without drain, Defender file lock interference
images	Dangling images from mutable tags, containerd GC failure (k8s#116020), snapshot accumulation
disk	C: drive free space with cross-snapshot trend analysis
hcs	vmcompute memory/handle leaks, HCS operation duration degradation, creation storms, 30+ error codes
hns	Endpoint leaks (IP exhaustion), LB count drops after HNS reset, stale LB rules, WFP filter accumulation
kubeproxy	DSR degraded policies, excluded port range conflicts with NodePort, stale LB rules, SNAT exhaustion
kubelet	NotReady/DiskPressure/MemoryPressure, lease renewal failures, clock skew, certificate rotation
memory	Physical RAM exhaustion, pagefile misconfiguration, per-process working set analysis
crashes	Application crashes (kubelet, containerd, shim), BSODs, WER correlation with service events
csi	CSI proxy crashes, stale SMB global mappings, credential rotation failures, named pipe version mismatches
gmsa	CCG plugin errors, Kerberos ticket failures, domain controller connectivity, credential spec validation
gpu	nvidia-smi parsing, Xid error classification, ECC memory errors, DirectX device plugin scheduling
bootstrap	CSE execution timeline, 83 WINDOWS_CSE_ERROR codes, bootstrap config validation, service startup ordering
extensions	VM extension exit codes with curl progress false-positive filtering
services	12 critical AKS service health checks, service PID cross-reference, start type validation

Orchestrator features

• Symptom-based dispatch — common-reference.md includes a dispatch table so agents pick the right 3-5 sub-skills instead of running all 16
• Synthesis decision tree — SKILL.md provides a full decision tree for combining findings across sub-skills
• 17 root cause chains — maps symptoms → checks → root causes (e.g., disk pressure → images → mutable tags)
• Timeline correlation — instructions for building cross-sub-skill event timelines from anchor events
• Consistent structure — all sub-skills use identical sections: Purpose, Input Files, Analysis Steps, Findings Format, Known Patterns, Cross-References

Key research that informed the sub-skills

• HCS: vmcompute memory leak from failed exec probes (hcsshim#1680), notification timeout deadlocks, 30+ error codes from Microsoft docs
• HNS: endpoint leak → IP exhaustion, LB disappearance (k/k#110849), stale LB rules (k/k#112836)
• kube-proxy: DSR degraded policies (Windows-Containers#79), port exhaustion (Port exhaustion on Windows node makes DNS stop working on Windows Pods AKS#1375), slow sync (k/k#109162)
• CSI: stale SMB global mappings (csi-driver-smb#219), credential rotation (Azuredisk-csi daemonset pod Failed to connect to csi-proxy v1beta with error: open \\.\\pipe\\csi-proxy-filesystem-v1beta1: The system cannot find the file specified. AKS#2693)
• Images: containerd GC failure due to ImageFsStats sparse-file bug (k8s#116020)
• Defender: Update-DefenderPreferences missing path exclusions for C:\ProgramData\containerd and containerd-shim-runhcs-v1.exe

Files changed

• .github/skills/windows-log-analysis/SKILL.md — orchestrator with decision tree and root cause chains
• .github/skills/windows-log-analysis/sub-skills/*.md — 16 sub-skills + common reference (3,337 lines total)
• .github/skills/windows-log-analysis/.gitignore

[Copilot]

Pull request overview

Adds a new GitHub Copilot skill under .github/skills/windows-log-analysis/ to help diagnose Windows AKS node issues from log bundles produced by staging/cse/windows/debug/collect-windows-logs.ps1, including an accompanying Python analyzer script.

Changes:

• Introduces SKILL.md with a log-bundle reference guide and troubleshooting playbooks.
• Adds analyze-windows-logs.py to scan multi-snapshot bundles, trend key metrics, and emit prioritized findings.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 10 comments.

File	Description
.github/skills/windows-log-analysis/SKILL.md	Skill definition and reference guide for interpreting collected Windows node logs
.github/skills/windows-log-analysis/analyze-windows-logs.py	Python 3 analyzer for automated triage of collected Windows log bundles

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

[Copilot]

*_services.csv is exported by collect-windows-logs.ps1 via Export-Csv without -Encoding, which defaults to UTF-16LE on Windows PowerShell. Marking it as UTF-8 here will cause parsers to mis-decode the file; update the encoding (and ideally the pattern to <ts>_services.csv for consistency with other entries).

Suggested change

	| `*_services.csv` | UTF-8 | Service status timeline used for kubelet crash/restart and clock skew checks |
	| `<ts>_services.csv` | UTF-16-LE with BOM | Service status timeline used for kubelet crash/restart and clock skew checks |

[Copilot]

File-dispatch mapping seems incomplete: kubectl-describe-nodes.log is consumed by analyze-kubelet.md (node conditions/taints/events) as well as GPU analysis, but the table only routes it to analyze-gpu. Add analyze-kubelet here to avoid skipping kubelet triage when this file is present.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

[Copilot]

This section says the table is sourced from windowscsehelper.ps1 and later references a “full code table”, but the table omits several defined codes (e.g., 18, 42, 44, 51, 63, 69). To avoid misdiagnosis, either (a) include the missing codes/ranges, or (b) label this as a partial list of common codes and link readers to parts/windows/windowscsehelper.ps1 for the authoritative set.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 6 comments.

[Copilot]

kubectl-describe-nodes.log is marked as UTF-8, but in collect-windows-logs.ps1 it is produced via PowerShell redirection (kubectl ... > file), which writes UTF-16LE (“Unicode”) by default on Windows PowerShell 5.1. Please update the expected encoding (or note that it may be UTF-16LE) so the analysis doesn’t mis-decode the file.

[Copilot]

This skill references <ts>-aks-info.log, but collect-windows-logs.ps1 does not generate any *-aks-info.log file (it generates kubectl-describe-nodes.log / kubectl-get-nodes.log instead). Either update the collector to emit this file, or update the skill docs/file discovery table to use the actual bundle filenames.

[Copilot]

kubectl-get-nodes.log is marked as UTF-8, but it is produced by collect-windows-logs.ps1 via PowerShell redirection (kubectl ... > file), which writes UTF-16LE by default on Windows PowerShell 5.1. Please update the encoding guidance (or mention it may be UTF-16LE).

Suggested change

	| `kubectl-get-nodes.log` | UTF-8 | `kubectl get nodes -o wide` output |
	| `kubectl-get-nodes.log` | UTF-16-LE with BOM or UTF-8 | `kubectl get nodes -o wide` output; often UTF-16-LE when collected via Windows PowerShell 5.1 redirection (`>`) |

[Copilot]

This references <ts>-aks-info.log, but the log bundle produced by collect-windows-logs.ps1 doesn’t include such a file. Consider switching this input to kubectl-describe-nodes.log (which the collector does generate) or documenting how <ts>-aks-info.log is expected to be produced.

[Copilot]

This references <ts>-aks-info.log, but the Windows bundle collector in this repo (collect-windows-logs.ps1) does not generate it. Either add it to the collector, or update this skill to rely on the existing kubectl-describe-nodes.log/kubectl-get-nodes.log outputs to extract versions and node YAML.

Suggested change

	| `<ts>-aks-info.log` | UTF-16-LE with BOM | `kubectl describe node` + node YAML, component versions |
	| `kubectl-describe-nodes.log` | UTF-8 or UTF-16-LE with BOM | `kubectl describe node` output for node conditions, taints, addresses, and event history |
	| `kubectl-get-nodes.log` | UTF-8 or UTF-16-LE with BOM | `kubectl get nodes` output (including wide/YAML forms when present) for Kubernetes versions and node object details |

[Copilot]

kubectl-describe-nodes.log is marked as UTF-8, but it’s produced via PowerShell redirection in collect-windows-logs.ps1 (Windows PowerShell 5.1), which writes UTF-16LE by default. Update the encoding expectations so GPU analysis can actually parse the file in real bundles.

Suggested change

	| `kubectl-describe-nodes.log` | UTF-8 | `kubectl describe node` — resource capacity/allocatable including GPU |
	| `kubectl-describe-nodes.log` | UTF-16-LE with BOM | `kubectl describe node` — resource capacity/allocatable including GPU |