The AI Scraper Playbook
— Every Step Exposed

What the attacker does

Fetches robots.txt, sitemap.xml, and the root index.html. Uses automated link-following to map the full URL structure of the site within minutes without triggering any alerts.

What the attacker does

Launches a full headless Chromium instance configured to mimic genuine user browser headers precisely. Rotates IP addresses through residential proxy networks to avoid rate-limiting detection.

What the attacker does

Unlike legacy scrapers, Firecrawl loads the full React/Next.js page, waits for networkidle, and captures all dynamically rendered content including lazy-loaded prices and AJAX-injected inventory indicators.

What the attacker does

The rendered page Markdown is passed to an LLM with a structured extraction schema. The LLM semantically identifies price, product name, SKU, and inventory regardless of DOM structure changes — making traditional selector obfuscation worthless without full polymorphic rendering.

What the attacker does

Runs an asynchronous batch crawl across the entire 40,000 SKU catalog in parallel. Results stream into a structured database or directly into a price-matching engine or LLM training pipeline.

Three ways the attacker profits from your data

• Competitor pricing engine: Harvested prices feed automated repricing systems. Every time you lower a price, theirs drops within minutes — eliminating pricing strategy permanently.
• LLM training dataset: Product descriptions, reviews, and images bundled into a training dataset. The attacker's AI is now powered by your content — without licence or payment.
• Resale to data brokers: Structured catalog data sold to market intelligence firms, comparison engines, or counterfeit operators. Your IP circulates permanently.