The Scraper That Fixes Itself

The Problem With Web Scrapers

Every e-commerce operator needs competitive pricing data. They also know that scrapers break every time the target site updates its CSS classes, restructures its product grid, or migrates to a new frontend framework. A scraper that works today is a scraper that's broken next month. The industry solution is either expensive SaaS ($200-$2,000/month for tools like Prisync or Competera) or engineering time rebuilding broken scrapers. Both are recurring costs for what should be a solved problem.

How Self-Healing Works

Scalpel uses an LLM exactly once: to generate the initial scraping script from the target site's HTML. After that, the script runs on a cron schedule with zero LLM cost. Pure BeautifulSoup/Scrapy, deterministic, fast. When the script breaks (and it will), the self-healing loop kicks in:

Error detection catches the failure
The broken HTML and old selectors go to the LLM
LLM generates new selectors
Sandbox validates the extraction against the expected schema
If valid, the fixed script auto-deploys
If not, it alerts the operator

The LLM cost is near-zero because healing events are rare (sites change layouts quarterly, not daily) and each healing call is a single prompt with structured output.

Architecture

Two services, one database: Web dashboard (Next.js 14): Add competitors, view price changes, configure alerts. Supabase auth, Stripe billing. Scraper service (Python FastAPI): Orchestrates scraping jobs, runs self-healing, delivers alerts via email (Resend), Slack, or webhook. Built-in intelligence for Shopify (/products.json) and WooCommerce (Store API) stores. Generic HTML scraper with LLM-generated selectors for everything else. Store type auto-detection on URL input.

Pricing Model

Three tiers, all monthly:

Starter ($29): 5 competitors, daily scraping, email alerts
Growth ($49): 25 competitors, hourly scraping, Slack + webhook
Pro ($99): Unlimited competitors, real-time scraping, API access, CSV export

Stack: Next.js 14, FastAPI, Supabase (PostgreSQL + Auth), Stripe, Resend, Railway + Vercel deployment. 16 tests passing, 23 commits.