IndexNow API Hammering Fix

Problem Fixed

Your Eleventy site was hammering the Bing IndexNow API by submitting ALL 343 URLs on every build, regardless of whether the content had actually changed. This was causing potential penalties and rate limiting issues.

Solution Implemented

The SEO automation plugin has been fixed to:

1. Smart Change Detection

• Only submits URLs for pages that have actually changed
• Uses comprehensive content hashing including:
  • Page title and description
  • File modification time and size
  • Tags and date metadata
  • Template information

2. Rate Limiting Protection

• Minimum 5-minute interval between IndexNow submissions
• Tracks last submission timestamp
• Prevents rapid successive submissions

3. URL Limits

• Maximum 1000 URLs per submission (down from 10,000)
• Only changed/new pages are submitted
• Prevents API abuse

How It Works Now

Normal Operation (Recommended)

npm run build

Result: Only changed pages will be submitted to IndexNow

Force Full Submission (Use Sparingly)

set FORCE_FULL_INDEXNOW_SUBMISSION=true && npm run build

Result: Submits ALL URLs (use only when needed, e.g., after major site restructure)

Environment Variables

Add these to your .env file for control:

# Core settings
ENABLE_INDEXNOW=true
ENABLE_CHANGE_TRACKING=true

# Rate limiting (optional)
FORCE_FULL_INDEXNOW_SUBMISSION=false

# Debugging (optional)
VERBOSE_LOGGING=true

What You'll See Now

Typical Build Output (No Changes)

============================================================
  Change Tracking
============================================================
[SEO] ℹ️ INFO: No changes detected since last build
[SEO] ℹ️ INFO: No changes detected - skipping IndexNow submission

Build Output (With Changes)

============================================================
  Change Tracking
============================================================
[SEO] ℹ️ INFO: Detected 3 changes
[SEO] ℹ️ INFO: Found 3 changed pages, submitting 3 URLs

============================================================
  IndexNow URL Submission
============================================================
[SEO] ℹ️ INFO: Preparing to submit 3 URLs
[SEO] ✅ SUCCESS: IndexNow submission successful (HTTP 200)
[SEO] ℹ️ INFO: Submitted 3 URLs to Bing, Yandex, and other search engines

Benefits

1. No More API Hammering: Only changed content is submitted
2. Better SEO: Search engines receive relevant updates, not spam
3. Rate Limiting Protection: Prevents accidental API abuse
4. Detailed Logging: See exactly what changed and why
5. Emergency Override: Can force full submission when needed

Files Modified

• _seo-automation/index.js - Main plugin logic fixed
• Added rate limiting and change tracking improvements
• Enhanced logging for better visibility

Testing

1. First build after fix: Will submit all URLs (creating baseline)
2. Second build (no changes): Should submit 0 URLs
3. Build after editing a post: Should submit only changed URLs

Monitoring

Check these files to monitor the system:

• _seo-logs/last-indexnow-submission.json - Last submission info
• _seo-logs/snapshots/latest.json - Current site state
• _seo-logs/changes-YYYY-MM-DD.jsonl - Change history
• _seo-logs/audit-YYYY-MM-DD.json - SEO audit reports

The fix ensures your site behaves properly during builds and prevents penalties from search engines due to excessive API usage.