Merge pull request #9 from AtlasAnalyticsLab/fix/jekyll-sitemap-2026-02-08

Fix/jekyll sitemap 2026 02 08

Author: chengjiahao1234

DEVELOPMENT.md

@@ -570,6 +570,7 @@ front matter...
    ```
    - **Purpose:** Tells search engines where to find the sitemap
    - **Impact:** SEO - affects how search engines crawl the site
+   - **Note:** `/sitemap.xml` is generated automatically by the `jekyll-sitemap` plugin (configured in `_config.yml`). Do not maintain a manual `sitemap.xml` file.
    - **Priority:** 🔴 CRITICAL
 
 3. **`CNAME`** (line 19)
@@ -615,6 +616,7 @@ If you need to change the website URL:
 #### Step 1: Update Critical Configuration
 - [ ] Update `_config.yml` → `url:` field
 - [ ] Verify `robots.txt` → `Sitemap:` line (generated from `{{ site.url }}{{ site.baseurl }}`)
+- [ ] Verify `/sitemap.xml` is generated (jekyll-sitemap) and includes key pages
 - [ ] Update or remove `CNAME` file if using custom domain
 
 #### Step 2: Test Locally
@@ -768,9 +770,10 @@ headline: "Text with [link](OPENINGS_LINK)"
 **Security & Protection:**
 - ✅ Enhanced 404 page with navigation buttons and Bootstrap icons
 - ✅ Comprehensive `robots.txt` with crawler access control
-  - Allows: Googlebot, Bingbot, Slurp with 10-second crawl delay
-  - Blocks: MJ12bot, AhrefsBot, SemrushBot, DotBot, PetalBot, DataForSeoBot
-  - Restricts: `/images/`, `/assets/`, `/css/`, `/js/` directories
+  - Allows: major search engines (Googlebot, Bingbot, Slurp, DuckDuckBot, etc.)
+  - Blocks: known heavy scraper / SEO bots (AhrefsBot, SemrushBot, MJ12bot, DotBot, PetalBot, DataForSeoBot)
+  - Avoids `Crawl-delay` (ignored by Googlebot and may trigger Search Console warnings)
+  - Restricts: internal build artifacts only (`/_site/`, `/bin/`)
 - ✅ Apache security configuration (`.htaccess`)
   - Directory browsing disabled
   - Security headers (X-Frame-Options, X-XSS-Protection, etc.)
@@ -968,7 +971,7 @@ headline: "Text with [link](OPENINGS_LINK)"
    - Test theme compatibility
 
 3. **Security Review:**
-   - Review `robots.txt` blocked crawlers
+   - Review `robots.txt` blocked scraper list (and confirm sitemap URL)
    - Update `.htaccess` security headers
    - Check GitHub Pages security settings
 

Gemfile

@@ -10,6 +10,7 @@
 # Key Dependencies:
 #   - jekyll 4.4.1+: Static site generator
 #   - jekyll-scholar: BibTeX bibliography support
+#   - jekyll-sitemap: Automatic sitemap.xml generation for search engines
 #   - webrick 1.9+: Ruby web server for local development
 #
 # Installation:
@@ -29,5 +30,6 @@ gem "jekyll", "4.4.1"
 # gem "github-pages", "~> 232", group: :jekyll_plugins
 
 gem "jekyll-scholar", group: :jekyll_plugins
+gem "jekyll-sitemap", group: :jekyll_plugins
 gem "webrick", "~> 1.9"
 gem "wdm", ">= 0.1.0" if Gem.win_platform?

README.md

@@ -328,7 +328,7 @@ bundle install
 
 ## Security Features
 
-- ✅ **Crawler Protection:** `robots.txt` controls search engine access
+- ✅ **Crawler Protection:** `robots.txt` allows major search engines and blocks known heavy scraper bots
 - ✅ **Custom 404 Page:** User-friendly error handling with navigation
 - ✅ **DDoS Protection:** GitHub Pages + Cloudflare CDN
 - ✅ **Security Headers:** Content security and XSS protection

_config.yml

@@ -34,6 +34,10 @@ include:
   - _pages
   - robots.txt
 
+plugins:
+  - jekyll-scholar
+  - jekyll-sitemap
+
 sass:
   sass_dir: _sass
 

_pages/aboutwebsite.md

@@ -2,7 +2,6 @@
 title: "About the website"
 layout: textlay
 excerpt: "About the website."
-sitemap: false
 permalink: /aboutwebsite.html
 ---
 <!--

_pages/allnews.md

@@ -2,7 +2,6 @@
 title: "News"
 layout: textlay
 excerpt: "Atlas Analytics Lab at Concordia University."
-sitemap: false
 permalink: /allnews.html
 ---
 <!--

_pages/contact.md

@@ -2,7 +2,6 @@
 title: "Atlas Analytics Lab - Contact"
 layout: textlay
 excerpt: "Ways to reach the Atlas Analytics Lab."
-sitemap: false
 permalink: /contact/
 ---
 <!--

_pages/funding.md

@@ -2,7 +2,6 @@
 title: "Atlas Analytics Lab - Funding"
 layout: textlay
 excerpt: "Atlas Analytics Lab -- Funding."
-sitemap: false
 permalink: /funding/
 ---
 <!--

_pages/gallery.md

@@ -2,7 +2,6 @@
 title: "Lab Life"
 layout: gallerylay
 excerpt: "Atlas Analytics Lab at Concordia University."
-sitemap: false
 permalink: /gallery/
 ---
 <!--

_pages/home.md

@@ -2,7 +2,6 @@
 title: "Atlas Analytics Lab - Home"
 layout: homelay
 excerpt: "Atlas Analytics Lab at Concordia University."
-sitemap: false
 permalink: /
 ---
 <!--