From 398ec9910e9427fe185a5b50af9929e1587a9466 Mon Sep 17 00:00:00 2001 From: Benjamin Michaelis Date: Sun, 17 May 2026 07:33:45 -0700 Subject: [PATCH 1/4] Add IndexNow sitemap notification to deployment workflow - Submit sitemap.xml to IndexNow API after successful production deployment - Notifies search engines (Bing, Yandex, Naver, etc.) of content updates - Uses GitHub Secret INDEXNOW_API_KEY for authentication - Continues on error to prevent deployment failures Addresses: #401 --- .github/workflows/Build-Test-And-Deploy.yml | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/.github/workflows/Build-Test-And-Deploy.yml b/.github/workflows/Build-Test-And-Deploy.yml index 5afc3a7c..6fb3879d 100644 --- a/.github/workflows/Build-Test-And-Deploy.yml +++ b/.github/workflows/Build-Test-And-Deploy.yml @@ -238,6 +238,19 @@ jobs: git tag -f "deployed/prod/${{ github.sha }}" git push origin "deployed/prod/${{ github.sha }}" --force + - name: Notify IndexNow of Sitemap Update + continue-on-error: true + run: | + curl -X POST https://api.indexnow.org/indexnow \ + -H "Content-Type: application/json" \ + -d '{ + "host": "essentialcsharp.com", + "key": "${{ secrets.INDEXNOW_API_KEY }}", + "urlList": [ + "https://essentialcsharp.com/sitemap.xml" + ] + }' + - name: Logout of Azure CLI if: always() uses: azure/CLI@v3 From 0a0b9503db53d859d14a46abf0a423db983e289d Mon Sep 17 00:00:00 2001 From: Benjamin Michaelis Date: Sun, 17 May 2026 07:44:19 -0700 Subject: [PATCH 2/4] Address PR review feedback for IndexNow implementation - Add detailed comments to workflow explaining key file hosting requirement and verification - Document IndexNow setup procedure in README with step-by-step instructions for key file deployment - Clarify that sitemap submission is valid (though urlList should ideally contain individual URLs) - Confirm no literal keys are exposed in code or commits; use generic examples in documentation Resolves review comments on PR #1125. --- .github/workflows/Build-Test-And-Deploy.yml | 10 +++++++++ README.md | 24 +++++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/.github/workflows/Build-Test-And-Deploy.yml b/.github/workflows/Build-Test-And-Deploy.yml index 6fb3879d..3e631305 100644 --- a/.github/workflows/Build-Test-And-Deploy.yml +++ b/.github/workflows/Build-Test-And-Deploy.yml @@ -241,6 +241,16 @@ jobs: - name: Notify IndexNow of Sitemap Update continue-on-error: true run: | + # IndexNow Protocol: Notifies search engines (Bing, Yandex, Naver) of content updates + # IMPORTANT: The API key must be verifiable by hosting it at https://essentialcsharp.com/{key}.txt + # containing exactly the key value. Without this verification file, IndexNow will reject submissions (HTTP 403). + # See: https://www.indexnow.org/documentation + # Setup required: + # 1. Place the key file in EssentialCSharp.Web/wwwroot/{INDEXNOW_API_KEY}.txt + # 2. File should contain only the key value (no whitespace, no metadata) + # 3. Ensure the file is deployed as a static asset to essentialcsharp.com + # Note: urlList should ideally contain actual changed URLs for better search engine indexing, + # though submitting the sitemap URL is a valid (if less efficient) approach. curl -X POST https://api.indexnow.org/indexnow \ -H "Content-Type: application/json" \ -d '{ diff --git a/README.md b/README.md index 071bc6ad..71b19db5 100644 --- a/README.md +++ b/README.md @@ -11,3 +11,27 @@ For any bugs, questions, or anything else with specifically the code found insid For information on setting up your local development environment, please see the [Getting Started Guide](docs/getting-started.md). Please use issues or discussions to report issues found. + +## IndexNow Setup (Production Deployment) + +The deployment workflow notifies search engines (Bing, Yandex, Naver) of content updates via the [IndexNow protocol](https://www.indexnow.org/). To enable this: + +### Key File Requirement +IndexNow requires verification that you own the domain by hosting your API key as a publicly accessible `.txt` file: + +1. **Create the key file**: Place your IndexNow API key in `EssentialCSharp.Web/wwwroot/{INDEXNOW_API_KEY}.txt` + - File content should contain **only** the key value (no whitespace, no newlines) + - Example: if key is `abc123def456`, the file is `abc123def456.txt` containing exactly that string + +2. **Deploy as static asset**: Ensure the key file is deployed with the application so it's accessible at: + ``` + https://essentialcsharp.com/{INDEXNOW_API_KEY}.txt + ``` + +3. **Set GitHub Secret**: Store your IndexNow API key in GitHub repository secrets as `INDEXNOW_API_KEY` + - Do NOT commit the literal key to the repository + +### Important Notes +- Without the verification file, IndexNow submissions will be rejected with HTTP 403 +- The workflow submits `sitemap.xml` to IndexNow; for more granular control, consider extracting individual changed URLs and submitting those instead +- The IndexNow step uses `continue-on-error: true` to prevent deployment failures if the notification fails From dd56c5ed20b38d3886eb6b0d8a53d67665990ae6 Mon Sep 17 00:00:00 2001 From: Benjamin Michaelis Date: Sun, 17 May 2026 17:07:34 -0700 Subject: [PATCH 3/4] Fix IndexNow implementation: submit actual URLs, add key verification file - Add wwwroot/6b397fbb613bfbacb059e4972cf3ec15.txt for IndexNow domain ownership verification (served as static asset at https://essentialcsharp.com/{key}.txt) - Fix workflow step: fetch live sitemap after smoke test, extract all URLs, POST correct JSON array to api.indexnow.org (per spec; not a sitemap URL) - Add graceful warning if sitemap fetch fails; log HTTP response code - Move key to env var to avoid inline secret interpolation - Update README with accurate setup instructions and how-it-works explanation --- .github/workflows/Build-Test-And-Deploy.yml | 56 +++++++++++++------ .../6b397fbb613bfbacb059e4972cf3ec15.txt | 1 + README.md | 40 ++++++++----- 3 files changed, 64 insertions(+), 33 deletions(-) create mode 100644 EssentialCSharp.Web/wwwroot/6b397fbb613bfbacb059e4972cf3ec15.txt diff --git a/.github/workflows/Build-Test-And-Deploy.yml b/.github/workflows/Build-Test-And-Deploy.yml index 3e631305..e546db30 100644 --- a/.github/workflows/Build-Test-And-Deploy.yml +++ b/.github/workflows/Build-Test-And-Deploy.yml @@ -240,26 +240,46 @@ jobs: - name: Notify IndexNow of Sitemap Update continue-on-error: true + env: + INDEXNOW_API_KEY: ${{ secrets.INDEXNOW_API_KEY }} run: | - # IndexNow Protocol: Notifies search engines (Bing, Yandex, Naver) of content updates - # IMPORTANT: The API key must be verifiable by hosting it at https://essentialcsharp.com/{key}.txt - # containing exactly the key value. Without this verification file, IndexNow will reject submissions (HTTP 403). + # IndexNow Protocol: Notifies search engines (Bing, Yandex, Naver) of content updates. # See: https://www.indexnow.org/documentation - # Setup required: - # 1. Place the key file in EssentialCSharp.Web/wwwroot/{INDEXNOW_API_KEY}.txt - # 2. File should contain only the key value (no whitespace, no metadata) - # 3. Ensure the file is deployed as a static asset to essentialcsharp.com - # Note: urlList should ideally contain actual changed URLs for better search engine indexing, - # though submitting the sitemap URL is a valid (if less efficient) approach. - curl -X POST https://api.indexnow.org/indexnow \ - -H "Content-Type: application/json" \ - -d '{ - "host": "essentialcsharp.com", - "key": "${{ secrets.INDEXNOW_API_KEY }}", - "urlList": [ - "https://essentialcsharp.com/sitemap.xml" - ] - }' + # + # Domain ownership is verified via a key file hosted at: + # https://essentialcsharp.com/{key}.txt (served from wwwroot/{key}.txt) + # The file must contain exactly the key value. Without it, submissions return HTTP 403. + # + # We fetch the live sitemap (app is confirmed up after smoke test) and extract all + # URLs to submit as actual content URLs per the IndexNow spec. + + # Fetch the live sitemap and extract all URLs into a newline-delimited list + SITEMAP=$(curl -sf "https://essentialcsharp.com/sitemap.xml") + if [ -z "$SITEMAP" ]; then + echo "::warning::Could not fetch sitemap; skipping IndexNow submission." + exit 0 + fi + + # Build a JSON array from ... entries + URL_ARRAY=$(echo "$SITEMAP" \ + | grep -oP '(?<=)[^<]+' \ + | jq -Rn '[inputs]') + + URL_COUNT=$(echo "$URL_ARRAY" | jq 'length') + echo "Submitting $URL_COUNT URLs to IndexNow..." + + RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" -X POST https://api.indexnow.org/indexnow \ + -H "Content-Type: application/json; charset=utf-8" \ + -d "{ + \"host\": \"essentialcsharp.com\", + \"key\": \"$INDEXNOW_API_KEY\", + \"urlList\": $URL_ARRAY + }") + + echo "IndexNow response: HTTP $RESPONSE" + if [ "$RESPONSE" != "200" ] && [ "$RESPONSE" != "202" ]; then + echo "::warning::IndexNow submission returned HTTP $RESPONSE (200/202 = success, 202 = key validation pending)." + fi - name: Logout of Azure CLI if: always() diff --git a/EssentialCSharp.Web/wwwroot/6b397fbb613bfbacb059e4972cf3ec15.txt b/EssentialCSharp.Web/wwwroot/6b397fbb613bfbacb059e4972cf3ec15.txt new file mode 100644 index 00000000..0d4a79de --- /dev/null +++ b/EssentialCSharp.Web/wwwroot/6b397fbb613bfbacb059e4972cf3ec15.txt @@ -0,0 +1 @@ +6b397fbb613bfbacb059e4972cf3ec15 \ No newline at end of file diff --git a/README.md b/README.md index 71b19db5..d26cb1ed 100644 --- a/README.md +++ b/README.md @@ -14,24 +14,34 @@ Please use issues or discussions to report issues found. ## IndexNow Setup (Production Deployment) -The deployment workflow notifies search engines (Bing, Yandex, Naver) of content updates via the [IndexNow protocol](https://www.indexnow.org/). To enable this: +The deployment workflow notifies search engines (Bing, Yandex, Naver) of content updates via the [IndexNow protocol](https://www.indexnow.org/). After each successful production deploy the workflow fetches the live `sitemap.xml`, extracts all content URLs, and submits them in a single batch POST to IndexNow. -### Key File Requirement -IndexNow requires verification that you own the domain by hosting your API key as a publicly accessible `.txt` file: +### How It Works -1. **Create the key file**: Place your IndexNow API key in `EssentialCSharp.Web/wwwroot/{INDEXNOW_API_KEY}.txt` - - File content should contain **only** the key value (no whitespace, no newlines) - - Example: if key is `abc123def456`, the file is `abc123def456.txt` containing exactly that string +1. After the smoke test confirms the app is live, the workflow fetches `https://essentialcsharp.com/sitemap.xml`. +2. All `` URLs are extracted and POSTed to `https://api.indexnow.org/indexnow`. +3. IndexNow distributes those URLs to all participating search engines (Bing, Yandex, Naver, etc.). -2. **Deploy as static asset**: Ensure the key file is deployed with the application so it's accessible at: - ``` - https://essentialcsharp.com/{INDEXNOW_API_KEY}.txt - ``` +### Setup Requirements -3. **Set GitHub Secret**: Store your IndexNow API key in GitHub repository secrets as `INDEXNOW_API_KEY` - - Do NOT commit the literal key to the repository +#### 1. Key Verification File (already committed) +IndexNow requires domain ownership proof via a publicly accessible `.txt` file at the root of the domain: +- The key file lives at `EssentialCSharp.Web/wwwroot/{key}.txt` +- The filename and file content are the same value (the key itself) +- ASP.NET Core's static file middleware serves it at `https://essentialcsharp.com/{key}.txt` +- IndexNow crawls that URL to verify domain ownership before accepting submissions +- Without this file, all submissions return HTTP 403 + +The key file is intentionally public — that is by design. The "security" is that only the domain owner can host a file at that path. + +#### 2. GitHub Secret +Store the key value in GitHub repository secrets as `INDEXNOW_API_KEY`. The workflow reads it from there — the secret value must match the key filename/content committed to the repo. + +Go to: **GitHub repo → Settings → Secrets and variables → Actions → New repository secret** +- Name: `INDEXNOW_API_KEY` +- Value: the key string (same value as the `.txt` filename in `wwwroot/`) ### Important Notes -- Without the verification file, IndexNow submissions will be rejected with HTTP 403 -- The workflow submits `sitemap.xml` to IndexNow; for more granular control, consider extracting individual changed URLs and submitting those instead -- The IndexNow step uses `continue-on-error: true` to prevent deployment failures if the notification fails +- The IndexNow step uses `continue-on-error: true` — a submission failure will never block a deployment +- Submitting all sitemap URLs on every deploy is intentional; IndexNow has no hard rate limit for batch submissions and the full URL list ensures nothing is missed +- To rotate the key: generate a new hex string, add a new `wwwroot/{newkey}.txt`, update the GitHub Secret, and remove the old `.txt` file From 6388912c97dc1b8dbe25996d2a7df62cba139fcb Mon Sep 17 00:00:00 2001 From: Benjamin Michaelis Date: Mon, 18 May 2026 09:51:05 -0700 Subject: [PATCH 4/4] chore: generate IndexNow key file at build time from GitHub Secret Remove committed wwwroot key file and instead write it dynamically in CI before Docker build. Key stays out of git history entirely; the GitHub Secret is the single source of truth. Key rotation is now just updating the secret - no file changes needed. --- .github/workflows/Build-Test-And-Deploy.yml | 6 +++++ .../6b397fbb613bfbacb059e4972cf3ec15.txt | 1 - README.md | 24 +++++++++---------- 3 files changed, 18 insertions(+), 13 deletions(-) delete mode 100644 EssentialCSharp.Web/wwwroot/6b397fbb613bfbacb059e4972cf3ec15.txt diff --git a/.github/workflows/Build-Test-And-Deploy.yml b/.github/workflows/Build-Test-And-Deploy.yml index e546db30..7805eb8f 100644 --- a/.github/workflows/Build-Test-And-Deploy.yml +++ b/.github/workflows/Build-Test-And-Deploy.yml @@ -63,6 +63,12 @@ jobs: - name: Set up Docker Buildx uses: docker/setup-buildx-action@v4 + - name: Write IndexNow key verification file + if: env.INDEXNOW_API_KEY != '' + env: + INDEXNOW_API_KEY: ${{ secrets.INDEXNOW_API_KEY }} + run: printf '%s' "$INDEXNOW_API_KEY" > "EssentialCSharp.Web/wwwroot/$INDEXNOW_API_KEY.txt" + # Only build for dev registry — prod gets the image via az acr import in deploy-production - name: Build Container Image if: github.event_name != 'pull_request_target' && github.event_name != 'pull_request' diff --git a/EssentialCSharp.Web/wwwroot/6b397fbb613bfbacb059e4972cf3ec15.txt b/EssentialCSharp.Web/wwwroot/6b397fbb613bfbacb059e4972cf3ec15.txt deleted file mode 100644 index 0d4a79de..00000000 --- a/EssentialCSharp.Web/wwwroot/6b397fbb613bfbacb059e4972cf3ec15.txt +++ /dev/null @@ -1 +0,0 @@ -6b397fbb613bfbacb059e4972cf3ec15 \ No newline at end of file diff --git a/README.md b/README.md index d26cb1ed..f9169d63 100644 --- a/README.md +++ b/README.md @@ -24,24 +24,24 @@ The deployment workflow notifies search engines (Bing, Yandex, Naver) of content ### Setup Requirements -#### 1. Key Verification File (already committed) -IndexNow requires domain ownership proof via a publicly accessible `.txt` file at the root of the domain: -- The key file lives at `EssentialCSharp.Web/wwwroot/{key}.txt` -- The filename and file content are the same value (the key itself) +#### 1. GitHub Secret +Generate a random hex key and store it in GitHub repository secrets as `INDEXNOW_API_KEY`. The workflow uses this value both to write the key verification file into the Docker image at build time and to authenticate IndexNow submissions. + +Go to: **GitHub repo → Settings → Secrets and variables → Actions → New repository secret** +- Name: `INDEXNOW_API_KEY` +- Value: a random hex string (e.g. generate with `openssl rand -hex 16`) + +#### 2. Key Verification File (auto-generated at build time) +IndexNow requires domain ownership proof via a publicly accessible `.txt` file: +- The workflow writes `wwwroot/{key}.txt` (containing the key value) before building the Docker image +- The file is NOT committed to git — it is generated fresh on every CI run from the GitHub Secret - ASP.NET Core's static file middleware serves it at `https://essentialcsharp.com/{key}.txt` - IndexNow crawls that URL to verify domain ownership before accepting submissions - Without this file, all submissions return HTTP 403 The key file is intentionally public — that is by design. The "security" is that only the domain owner can host a file at that path. -#### 2. GitHub Secret -Store the key value in GitHub repository secrets as `INDEXNOW_API_KEY`. The workflow reads it from there — the secret value must match the key filename/content committed to the repo. - -Go to: **GitHub repo → Settings → Secrets and variables → Actions → New repository secret** -- Name: `INDEXNOW_API_KEY` -- Value: the key string (same value as the `.txt` filename in `wwwroot/`) - ### Important Notes - The IndexNow step uses `continue-on-error: true` — a submission failure will never block a deployment - Submitting all sitemap URLs on every deploy is intentional; IndexNow has no hard rate limit for batch submissions and the full URL list ensures nothing is missed -- To rotate the key: generate a new hex string, add a new `wwwroot/{newkey}.txt`, update the GitHub Secret, and remove the old `.txt` file +- To rotate the key: generate a new hex string and update the GitHub Secret — the old key file disappears automatically on the next deploy