Universal Scraping Architect
FreeBuild robust data extraction pipelines with ease.
Free · Opens the source repo
What Universal Scraping Architect does
The Universal Scraping Architect is designed to facilitate the creation of comprehensive data-extraction pipelines tailored for various web scraping and data extraction tasks. It supports multiple modes of extraction, including API-driven approaches using Firecrawl, local Python scripts for traditional file handling, and hybrid pipelines that combine both methods. This flexibility allows developers and data engineers to choose the most efficient strategy based on the source of their data and the complexity of the extraction task.
The skill emphasizes the importance of intelligent routing and validation throughout the extraction process. Users can specify whether to use Firecrawl for dynamic web content or local Python scripts for static files, ensuring that the chosen method aligns with the data's characteristics. Additionally, the skill incorporates token-budget tracking to help manage API quotas effectively, which is crucial when dealing with large-scale data extraction projects.
Validation is a core feature of this skill, with built-in mechanisms to ensure that the extracted data meets quality standards before it is delivered. Users are encouraged to run validation scripts on their outputs to catch errors or missing information, thereby preventing the shipment of flawed data. The skill also provides guidance on best practices, such as avoiding brittle selectors and respecting web scraping etiquette, which are essential for maintaining the integrity of both the data and the scraping process.
This skill is particularly beneficial for developers and data scientists looking to streamline their data extraction workflows. By providing robust templates and clear guidelines, the Universal Scraping Architect helps users avoid common pitfalls in web scraping and data handling, resulting in more reliable and maintainable data pipelines.
When to use it
Use this skill when you need to extract data from various sources, whether it's web pages, local files, or APIs, and require a robust validation process.
When not to use it
This skill may not be suitable for simple scraping tasks that do not require validation or for highly interactive scraping scenarios that need user emulation.
What you can build with it
Extracting Data from a Public API
Use the API-Driven mode to scrape data from a public URL that requires dynamic content handling.
Processing Local Files
Utilize Local Python mode for extracting data from sensitive or private files like PDFs and Excel spreadsheets.
Combining Web and Local Data
Implement a Hybrid Pipeline to scrape data from a website and then clean and structure it using local Python libraries.
How to install Universal Scraping Architect
View source1. Install with the skills CLI
npx skills add alirezarezvani/claude-skills/universal-scraping-architect --agent claude-code2. Or install it manually
Download the skill folder and drop it into ~/.claude/skills/ for all projects, or .claude/skills/ to scope it to one repo. Restart Claude Code so it picks up the new skill.
Anthropic's agentic coding CLI, and the reference implementation of Agent Skills. Drop a skill folder into ~/.claude/skills and Claude Code loads it automatically whenever a task matches the skill's description. Claude Code docs
Inside SKILL.md
Written by alirezarezvaniUniversal Scraping Architect
Design complete, robust data-extraction pipelines with intelligent routing, validation, and token-budget tracking — not brittle one-off scripts.
Dependency Notice: BYOK (Bring Your Own Key) pattern for Firecrawl; API keys must only be loaded via environment variables. Per-script dependencies:
| Script | Dependencies | Exact CLI |
|---|---|---|
scripts/validate_extraction.py | stdlib only | python3 scripts/validate_extraction.py output.json --json |
scripts/firecrawl_example.py | firecrawl, requests (template; --sample runs offline) | python3 scripts/firecrawl_example.py --sample |
scripts/local_bs4_example.py | beautifulsoup4, pandas (template; --sample runs offline) | python3 scripts/local_bs4_example.py --sample |
Before Starting
Check for context first:
If project-context.md exists, read it before asking questions. Determine the target data format, scale of extraction, and deployment environment before writing any code.
How This Skill Works
This skill supports 3 extraction modes based on intelligent routing:
Mode 1: API-Driven (Firecrawl)
Use when the source is a public URL, heavily dynamic (JS/SPA), requires search-first discovery, or involves bulk crawling across a domain.
Mode 2: Local Python (Traditional)
Use when extracting from local files (PDF, Excel, CSV), the data is private/sensitive, or the target is a simple static HTML page where Firecrawl is overkill.
Mode 3: Hybrid Pipeline
Use when Firecrawl handles URL discovery/web extraction, but local Python (Pandas) is required to clean, normalize, and structure the output before saving.
The Extraction Pipeline
When executing a scraping task, always follow this sequence:
- Route the Approach: Explicitly state whether Firecrawl or Local Python is being used and why.
- Track Budgets: Estimate Firecrawl API quotas or LLM token context limits before executing large jobs.
- Extract Safely: Implement checkpointing for multi-page jobs. Handle pagination and dynamic layouts gracefully. Start from the editable runner templates —
scripts/firecrawl_example.py(Mode 1) orscripts/local_bs4_example.py(Mode 2); run each with--samplefirst to see the expected summary shape without network access. - Validate & Clean: Run
python3 scripts/validate_extraction.py extracted_output.json --jsonon every extraction result before delivering it. It exits 0 only on{"status": "ok"};warning(empty output) orerror(malformed JSON) exit 1 — fix and re-extract, never ship unvalidated data. Beyond this structural gate, also check required fields and duplicates against the pipeline spec before delivering. - Format: Default to CSV for tabular data, JSON for nested structures, and Markdown for clean text.
Proactive Triggers
Surface these issues WITHOUT being asked when you notice them in context:
- Hardcoded API Keys → Flag immediately and rewrite to use
os.getenv('FIRECRAWL_API_KEY'). - Private Data Leakage → If the user asks to send local, sensitive files to an external API, flag the privacy risk and suggest Mode 2 (Local Python).
- Missing Pagination → If the target implies hundreds of records but no pagination logic is requested, flag it and add checkpointing.
Output Artifacts
| When you ask for... | You get... |
|---|---|
| "Scrape this site" | A fully validated Python extraction script with routing logic and error handling. |
| "Get data from this table" | A clean CSV/JSON dataset with a summary log of row counts and empty values. |
| "Crawl these docs" | A Markdown deliverable chunked for LLM token limits. |
Anti-Patterns
- Brittle Selectors: Never use highly nested CSS selectors (e.g.,
div > span > ul > li:nth-child(3)). Use data attributes or robust structural anchors. - Ignoring Etiquette: Never scrape without checking
robots.txtor implementing sensible rate limits. - No Validation: Never blindly write scraped data to a file without checking if the array is empty or missing critical keys.
Related Skills
- data-cleaning: Use when the scraped data requires complex statistical normalization or deduplication.
- browser-automation: Use for highly interactive scraping requiring user emulation (clicks, logins) where Firecrawl is insufficient.
Frequently asked questions about Universal Scraping Architect
Similar skills
WinMD API Search
Easily find and explore Windows desktop APIs.
WebMCPify
Transform any web app into an agent-ready platform.
Phoenix Tracing
Instrument LLM applications with OpenInference tracing.
Foundry Hosted Agent CopilotKit
Guidance for developing agentic web apps on Azure.
Power Automate Foundation
Connect AI agents to Power Automate seamlessly.
Power Automate Flow Builder
Efficiently build and deploy Power Automate flows programmatically.