SEO audits with Crawl4AI, Python and an LLM

This post outlines an audit method for a website's SEO health and content validity. It combines Crawl4AI, Python and a large language model (LLM) guided by a custom system instruction.

The components of the audit engine

The process uses four components:

• Crawl4AI is the data harvester. It is an asynchronous Python web crawler that gathers website data. It can deep crawl to collect URLs, HTML structure, text content and links, and save the output as structured Markdown files.

  Configuration might look like this:

  # (Illustrative snippet from a crawl script)
  from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
  from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
  from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
  
  TARGET_DEPTH = 3 # Depth of crawl
  md_generator = DefaultMarkdownGenerator()
  config = CrawlerRunConfig(
      deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=TARGET_DEPTH),
      markdown_generator=md_generator,
      cache_mode=CacheMode.BYPASS # Ensures fresh content
  )
  # ... rest of crawler setup ...

• Python and Jupyter notebooks are the orchestrator and processor. Python scripts start the crawls and process the data. An aggregation script, for example, can consolidate the crawled Markdown files.

  An aggregation script might include:

  # (Illustrative snippet from an aggregation script)
  from pathlib import Path
  
  def clean_markdown_content(content):
      # Example: Regex to remove common boilerplate
      content = re.sub(r"[Skip to content].*?Main Menu.*?\n# ", "\n# ", content, flags=re.DOTALL | re.MULTILINE)
      content = re.sub(r"\nRelated Posts:.*?Copyright © \d{4}.*?All Rights Reserved\.\s*$", "", content, flags=re.DOTALL | re.MULTILINE)
      return content.strip()
  
  all_md_files = list(Path("./crawled_output_folder").rglob('*.md'))
  with open("aggregated_site_content.md", "w", encoding="utf-8") as outfile:
      for md_file_path in sorted(all_md_files):
          # ... logic to write filename as header, then cleaned content ...
          # with open(md_file_path, "r", encoding="utf-8") as infile:
          #     original_content = infile.read()
          #     cleaned_content = clean_markdown_content(original_content)
          #     outfile.write(f"## START OF FILE {md_file_path.name}\n\n")
          #     outfile.write(cleaned_content)
          #     outfile.write(f"\n\n--- END OF FILE {md_file_path.name} ---\n\n")
          pass # Simplified for brevity
          

• An advanced LLM, such as Google Gemini, is the analyst and strategist. It processes the aggregated text and performs the SEO analysis, guided by a System Instruction.
• The LLM-generated System Instruction is the analyst's guide. It is a detailed document, ideally generated by an LLM, that defines the scope, methodology and reporting structure for the SEO audit.

---

Meta-prompting for an expert AI analyst

"Meta-prompting" (asking an LLM to write its own instructions for a complex task) is useful for several reasons:

• An LLM that defines an expert role often considers nuances you might overlook at first.
• The LLM understands how it processes information best, so the structure and detail it writes are often tuned for its own execution.
• It lets you define expert roles for many tasks quickly.

What makes a good system instruction?

An effective System Instruction for an SEO audit should include:

• A clear role and goal. An example snippet:

  "You are an elite SEO strategist and technical auditor. Your primary goal is to analyze provided scraped website data and produce a prioritized, actionable SEO audit report."

• Defined input expectations. What data format and content will it receive?

  "You will receive scraped data. This data should ideally include URLs, HTML Content (as Markdown), Text Content, HTTP Status Codes (if available), Page Titles, Meta Descriptions, Header Tags."

• Core analysis pillars and detailed checkpoints. These set the areas of investigation: technical SEO, on-page content, E-E-A-T and LLM-specific factors. An example snippet for the technical SEO pillar:

  "1. Technical SEO Fitness
  * Crawlability and Indexability
  * Robots.txt Analysis. Are LLM-specific user-agents handled?
  * XML Sitemap Review
  * Canonicalization Assessment"

• A prioritization framework. How to classify the severity of issues, for example P1 Critical, P2 Important and P3 Best Practice.
• A specific output format. The structure of the final report: an executive summary, detailed findings and a prioritized action plan table.
• Guiding principles. The overall philosophy: user-first, data-driven and E-E-A-T centric.

Prompting an LLM to write the system instruction

To have an LLM generate these instructions, define its role as the creator of the instructions. For example:

"You are a world-class expert in designing operational frameworks for AI agents. Create a comprehensive 'System Instruction' document for an AI whose task will be to perform a full SEO audit of a website based on scraped Markdown data. This AI should act as a top-tier SEO consultant. The System Instruction must cover all critical aspects of modern SEO, including technical factors, on-page content, E-E-A-T, and considerations for LLM-based search. Define the AI's role, expected input data, core analysis pillars with detailed checkpoints, a prioritization framework, the desired output report structure, and guiding principles. Ensure the instructions are detailed and thorough, as this document will be the AI's sole guide."

The LLM then generates the detailed System Instruction, which guides the AI that performs the audit.

---

The workflow from data to insight

1. Define the target and scope. Specify the website and depth for the audit.
2. Craft the System Instruction with meta-prompting. Use an LLM to generate the comprehensive System Instruction for an SEO audit.
3. Deploy Crawl4AI with Python. Run the crawler script to gather website data, outputting Markdown files.
4. Consolidate and prepare the data with Python. Aggregate the crawled data. The example script showed how to clean and combine files, perhaps adding delimiters like "--- START OF FILE filename.md ---" and "--- END OF FILE filename.md ---" around each file's content for clarity when presenting it to the LLM.
5. Engage the LLM analyst. Provide the aggregated Markdown content and the AI-generated System Instruction to an LLM instance.
6. Receive the audit report. The LLM processes the data against the System Instruction and outputs the SEO audit.

---

Outcomes beyond speed

This workflow offers these benefits:

• The audit can show when a site that claims expertise, for example in SEO services, has basic SEO flaws, E-E-A-T gaps or low-quality content. It might flag findings such as "Widespread Duplicate & Near-Duplicate Content" or "Content Quality & E-E-A-T Concerns: A substantial portion of the blog content exhibits patterns consistent with AI-generation without sufficient human oversight."
• Systemic issues such as problematic URL structures or unoptimized archive pages become clear. An audit might note: "Critical Technical Flaws: Broken internal links, suboptimal URL structure for blog posts (date-based)."
• Guided by E-E-A-T principles, the LLM can flag content that lacks depth or unique insight.