Every e-commerce operator needs competitive pricing data. They also know that scrapers break every time the target site updates its CSS classes, restructures its product grid, or migrates to a new frontend framework. A scraper that works today is a scraper that's broken next month.The industry solution is either expensive SaaS ($200-$2,000/month for tools like Prisync or Competera) or engineering time rebuilding broken scrapers. Both are recurring costs for what should be a solved problem.
How Self-Healing Works
Scalpel uses an LLM exactly once: to generate the initial scraping script from the target site's HTML. After that, the script runs on a cron schedule with zero LLM cost. Pure BeautifulSoup/Scrapy, deterministic, fast.When the script breaks (and it will), the self-healing loop kicks in:
Error detection catches the failure
The broken HTML and old selectors go to the LLM
LLM generates new selectors
The fixed script redeploys
If generation fails, it alerts the operator
What's missing, and I should say so plainly: there is no sandbox check that the repaired selector pulls the right price rather than a plausible-looking wrong one. That's the hard part, and the prototype doesn't do it yet.The LLM cost is near-zero because healing events are rare (sites change layouts quarterly, not daily) and each healing call is a single prompt with structured output.
Architecture
Two services, one database:Web dashboard (Next.js 14): Add competitors, view price changes, configure alerts. Supabase auth, Stripe billing.Scraper service (Python FastAPI): Orchestrates scraping jobs, runs self-healing, delivers alerts via email (Resend), Slack, or webhook.Built-in intelligence for Shopify (/products.json) and WooCommerce (Store API) stores. Generic HTML scraper with LLM-generated selectors for everything else. Store type auto-detection on URL input.
