Add root cause analysis and remediation for ArgoCD deployment failure

.github/ISSUE_12_ANALYSIS.md

@@ -0,0 +1,156 @@
+## 🔍 Root Cause Analysis
+
+I've investigated the ArgoCD deployment failure for the `2-broken-apps` application and identified the root cause.
+
+### Root Cause
+
+The deployment is failing due to an **invalid Kubernetes API version** in the manifest file.
+
+**Location:** `apps/broken-aks-store-all-in-one.yaml` (line 178)  
+**Issue:** The `apiVersion` field is malformed
+
+```yaml
+# Current (BROKEN):
+apiVersion: apps/v
+kind: Deployment
+metadata:
+  name: order-service
+
+# Expected (CORRECT):
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: order-service
+```
+
+### Evidence
+
+1. **Commit Analysis:** The failure corresponds to commit `8cd04df204028ff78613a69fdb630625864037c6` in the source repository
+   - Commit message: "break apiVersion formatting in deployment YAML"
+   - This was an intentional breaking change (likely for testing ArgoCD notifications)
+
+2. **YAML Validation:** The file has 20 resources, and line 178 contains `apiVersion: apps/v` instead of the required `apiVersion: apps/v1`
+
+3. **ArgoCD Error:** The error message "one or more synchronization tasks are not valid" indicates that ArgoCD cannot validate or apply the Kubernetes manifests due to the invalid API version
+
+### Impact
+
+- **Affected Resource:** `order-service` Deployment
+- **Sync Status:** OutOfSync (ArgoCD cannot synchronize due to validation failure)
+- **Health Status:** Degraded (application cannot be deployed)
+- **Retry Attempts:** Failed after 2 retry attempts (as configured in the Application spec)
+
+---
+
+## 🛠️ Remediation Recommendations
+
+### Option 1: Fix the Source Repository (Recommended for Production)
+
+If this is a real production issue, fix the manifest in the source repository:
+
+```bash
+# Clone the repository
+git clone https://github.com/dcasati/argocd-notification-examples.git
+cd argocd-notification-examples
+
+# Fix the apiVersion on line 178
+sed -i '178s/apiVersion: apps\/v$/apiVersion: apps\/v1/' apps/broken-aks-store-all-in-one.yaml
+
+# Commit and push the fix
+git add apps/broken-aks-store-all-in-one.yaml
+git commit -m "fix: correct apiVersion for order-service Deployment"
+git push origin main
+
+# Trigger a manual sync in ArgoCD
+argocd app sync 2-broken-apps
+```
+
+### Option 2: Update ArgoCD Application to Use a Different Source
+
+If this repository is intentionally broken for testing purposes, update your ArgoCD Application to point to a working repository:
+
+```bash
+# Update the ArgoCD Application to use a different repository
+argocd app set 2-broken-apps --repo https://github.com/YOUR-ORG/working-manifests.git
+
+# Or delete the test application if it's no longer needed
+argocd app delete 2-broken-apps
+```
+
+### Option 3: Use Resource Exclusion (Temporary Workaround)
+
+If you need to deploy the rest of the resources while investigating:
+
+```bash
+# Update the Application to exclude the broken Deployment
+kubectl patch application 2-broken-apps -n argocd --type merge -p '
+{
+  "spec": {
+    "ignoreDifferences": [
+      {
+        "group": "apps",
+        "kind": "Deployment",
+        "name": "order-service",
+        "jsonPointers": ["/"]
+      }
+    ]
+  }
+}'
+```
+
+### Verification Steps
+
+After applying the fix:
+
+1. **Verify the sync status:**
+   ```bash
+   argocd app get 2-broken-apps
+   ```
+
+2. **Check that all resources are healthy:**
+   ```bash
+   kubectl get all -n default -l app.kubernetes.io/instance=2-broken-apps
+   ```
+
+3. **Monitor the deployment:**
+   ```bash
+   kubectl get events -n default --sort-by='.lastTimestamp' | grep order-service
+   ```
+
+4. **Confirm no errors in pod logs:**
+   ```bash
+   kubectl logs -n default -l app=order-service --tail=50
+   ```
+
+---
+
+## 📝 Additional Notes
+
+- **Testing Context:** Based on the commit message and repository name ("argocd-notification-examples"), this appears to be an intentionally broken deployment for testing ArgoCD notification workflows
+- **Notification System Working:** The fact that this issue was automatically created confirms that your ArgoCD notification integration with GitHub is working correctly ✅
+- **Similar Issues:** If you encounter similar "synchronization tasks are not valid" errors in the future, the cause is typically:
+  - Invalid or malformed YAML syntax
+  - Incorrect Kubernetes API versions
+  - Missing required fields in Kubernetes resources
+  - Invalid resource references (e.g., non-existent ConfigMaps or Secrets)
+
+If you need assistance implementing any of these remediation steps, please let me know!
+
+---
+
+**Note:** This analysis should be posted as a comment on GitHub Issue #12. To post it, run:
+
+```bash
+gh issue comment 12 --body-file .github/ISSUE_12_ANALYSIS.md
+```
+
+Or use the GitHub API:
+
+```bash
+curl -X POST \
+  -H "Accept: application/vnd.github+json" \
+  -H "Authorization: Bearer $GITHUB_TOKEN" \
+  -H "X-GitHub-Api-Version: 2022-11-28" \
+  "https://api.github.com/repos/DevExpGbb/agentic-platform-engineering/issues/12/comments" \
+  -d @<(jq -Rs '{body: .}' < .github/ISSUE_12_ANALYSIS.md)
+```

.github/README_ANALYSIS.md

@@ -0,0 +1,62 @@
+# ArgoCD Deployment Failure Analysis
+
+## Summary
+
+This directory contains the root cause analysis for the ArgoCD deployment failure of the `2-broken-apps` application reported in [Issue #12](https://github.com/DevExpGbb/agentic-platform-engineering/issues/12).
+
+## Quick Start
+
+To post the analysis as a comment on the GitHub issue, use one of these methods:
+
+### Method 1: GitHub CLI (Recommended)
+
+```bash
+gh issue comment 12 --body-file .github/ISSUE_12_ANALYSIS.md
+```
+
+### Method 2: GitHub Actions Workflow
+
+Trigger the workflow manually from the Actions tab or via CLI:
+
+```bash
+gh workflow run post-issue-analysis.yml -f issue_number=12
+```
+
+### Method 3: Bash Script
+
+Run the provided script:
+
+```bash
+./.github/scripts/post-to-issue.sh 12
+```
+
+### Method 4: Manual Copy-Paste
+
+1. Open the issue: https://github.com/DevExpGbb/agentic-platform-engineering/issues/12
+2. Copy the contents of `.github/ISSUE_12_ANALYSIS.md`
+3. Paste into a new comment
+
+## Files
+
+- **ISSUE_12_ANALYSIS.md** - Complete root cause analysis and remediation recommendations
+- **workflows/post-issue-analysis.yml** - GitHub Actions workflow to post the analysis
+- **scripts/post-to-issue.sh** - Bash script to post the analysis
+- **scripts/post-issue-analysis.js** - Node.js helper script
+- **README_ANALYSIS.md** - This file
+
+## Root Cause (TL;DR)
+
+The deployment is failing due to an **invalid Kubernetes API version** in the manifest file:
+
+**File:** `apps/broken-aks-store-all-in-one.yaml` (line 178) in the external repository  
+**Issue:** `apiVersion: apps/v` should be `apiVersion: apps/v1`
+
+This was intentionally broken for testing ArgoCD notification workflows (commit message: "break apiVersion formatting in deployment YAML").
+
+## Remediation Options
+
+1. **Fix the source repository** - Correct the `apiVersion` field in the YAML file
+2. **Point to a different source** - Update the ArgoCD Application to use a working repository
+3. **Use resource exclusion** - Temporarily exclude the broken deployment while investigating
+
+See the full analysis in `ISSUE_12_ANALYSIS.md` for detailed remediation steps.

.github/scripts/post-issue-analysis.js

@@ -0,0 +1,46 @@
+#!/usr/bin/env node
+
+/**
+ * Post the analysis from ISSUE_12_ANALYSIS.md as a comment on issue #12
+ * This script is meant to be run from a GitHub Actions workflow with the github-script action
+ * 
+ * Usage in workflow:
+ * - uses: actions/github-script@v7
+ *   with:
+ *     script: |
+ *       const fs = require('fs');
+ *       const commentBody = fs.readFileSync('.github/ISSUE_12_ANALYSIS.md', 'utf8');
+ *       await github.rest.issues.createComment({
+ *         owner: context.repo.owner,
+ *         repo: context.repo.repo,
+ *         issue_number: 12,
+ *         body: commentBody
+ *       });
+ */
+
+const fs = require('fs');
+const path = require('path');
+
+// Read the analysis file
+const analysisFile = path.join(__dirname, '..', 'ISSUE_12_ANALYSIS.md');
+
+if (!fs.existsSync(analysisFile)) {
+  console.error(`Error: Analysis file not found at ${analysisFile}`);
+  process.exit(1);
+}
+
+const commentBody = fs.readFileSync(analysisFile, 'utf8');
+
+// Remove the note section at the end
+const cleanedBody = commentBody.replace(/---\n\n\*\*Note:\*\*.*$/s, '').trim();
+
+console.log('Analysis file loaded successfully');
+console.log(`Comment length: ${cleanedBody.length} characters`);
+console.log('\nTo post this comment, use:');
+console.log('gh issue comment 12 --body-file .github/ISSUE_12_ANALYSIS.md');
+console.log('\nOr use this in a GitHub Actions workflow with github-script action');
+
+// Export for use in GitHub Actions
+if (typeof module !== 'undefined' && module.exports) {
+  module.exports = { commentBody: cleanedBody };
+}

.github/scripts/post-to-issue.sh

@@ -0,0 +1,90 @@
+#!/bin/bash
+
+# Post Issue Analysis to GitHub
+# This script posts the root cause analysis to a GitHub issue
+
+set -e
+
+ISSUE_NUMBER="${1:-12}"
+ANALYSIS_FILE=".github/ISSUE_12_ANALYSIS.md"
+REPO_OWNER="DevExpGbb"
+REPO_NAME="agentic-platform-engineering"
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+echo "================================================"
+echo "  GitHub Issue Analysis Poster"
+echo "================================================"
+echo ""
+
+# Check if analysis file exists
+if [ ! -f "$ANALYSIS_FILE" ]; then
+    echo -e "${RED}Error: Analysis file not found at $ANALYSIS_FILE${NC}"
+    exit 1
+fi
+
+echo -e "${GREEN}✓ Analysis file found${NC}"
+
+# Clean the comment body (remove the note section)
+COMMENT_BODY=$(sed '/^---$/,$ d' "$ANALYSIS_FILE" | sed -e 's/$/\\n/' | tr -d '\n')
+
+# Method 1: Try gh CLI
+if command -v gh &> /dev/null && [ -n "$GITHUB_TOKEN" ] || gh auth status &> /dev/null 2>&1; then
+    echo -e "${GREEN}Using GitHub CLI...${NC}"
+    gh issue comment "$ISSUE_NUMBER" --body-file "$ANALYSIS_FILE"
+    echo -e "${GREEN}✓ Comment posted successfully using gh CLI!${NC}"
+    exit 0
+fi
+
+# Method 2: Try curl with GITHUB_TOKEN
+if [ -n "$GITHUB_TOKEN" ]; then
+    echo -e "${GREEN}Using curl with GITHUB_TOKEN...${NC}"
+
+    RESPONSE=$(curl -s -w "\n%{http_code}" -X POST \
+        -H "Accept: application/vnd.github+json" \
+        -H "Authorization: Bearer $GITHUB_TOKEN" \
+        -H "X-GitHub-Api-Version: 2022-11-28" \
+        "https://api.github.com/repos/$REPO_OWNER/$REPO_NAME/issues/$ISSUE_NUMBER/comments" \
+        -d @<(jq -Rs '{body: .}' < "$ANALYSIS_FILE"))
+
+    HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
+    BODY=$(echo "$RESPONSE" | sed '$d')
+
+    if [ "$HTTP_CODE" = "201" ]; then
+        echo -e "${GREEN}✓ Comment posted successfully using curl!${NC}"
+        COMMENT_URL=$(echo "$BODY" | jq -r '.html_url')
+        echo "Comment URL: $COMMENT_URL"
+        exit 0
+    else
+        echo -e "${RED}✗ Failed to post comment. HTTP code: $HTTP_CODE${NC}"
+        echo "$BODY" | jq .
+        exit 1
+    fi
+fi
+
+# Method 3: Provide instructions for manual posting
+echo -e "${YELLOW}No authentication method available.${NC}"
+echo ""
+echo "To post this analysis to issue #$ISSUE_NUMBER, use one of these methods:"
+echo ""
+echo "Method 1 - GitHub CLI:"
+echo "  gh issue comment $ISSUE_NUMBER --body-file $ANALYSIS_FILE"
+echo ""
+echo "Method 2 - GitHub Web UI:"
+echo "  1. Go to: https://github.com/$REPO_OWNER/$REPO_NAME/issues/$ISSUE_NUMBER"
+echo "  2. Copy the contents of: $ANALYSIS_FILE"
+echo "  3. Paste into a new comment"
+echo ""
+echo "Method 3 - Trigger the workflow:"
+echo "  gh workflow run post-issue-analysis.yml -f issue_number=$ISSUE_NUMBER"
+echo ""
+echo "Method 4 - Use curl with your token:"
+echo "  export GITHUB_TOKEN=your_token_here"
+echo "  bash .github/scripts/post-to-issue.sh $ISSUE_NUMBER"
+echo ""
+
+exit 1

.github/workflows/post-issue-analysis.yml

@@ -0,0 +1,33 @@
+name: Post Issue Analysis
+
+on:
+  workflow_dispatch:
+    inputs:
+      issue_number:
+        description: 'Issue number to post analysis to'
+        required: true
+        default: '12'
+
+permissions:
+  issues: write
+  contents: read
+
+jobs:
+  post-analysis:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Post analysis as comment using gh CLI
+        run: |
+          # Remove the note section at the end of the file
+          sed '/^---$/,$ d' .github/ISSUE_12_ANALYSIS.md > /tmp/clean-analysis.md
+
+          # Post the comment using gh CLI
+          gh issue comment ${{ github.event.inputs.issue_number }} --body-file /tmp/clean-analysis.md
+
+          echo "✅ Posted analysis to issue #${{ github.event.inputs.issue_number }}"
+        env:
+          GH_TOKEN: ${{ github.token }}