YUQI / WEB DATA PIPELINE · Custom development based on Firecrawl

Turn websites into
continuously updated data sources

From URL lists and site crawling to clean Markdown and JSON. Yuqi plans a repeatable data workflow for websites, documentation, product catalogs and announcements, with traceable sources and defined delivery formats.

Prototype notice This is a static introduction. Consoles, task records, output samples and logs are simulated locally. No real crawl requests are made and no working backend is implied.
8engine capability areas
checked individually
2standard output formats
Markdown / JSON
4pipeline stages
Source → Delivery
3deployment options
Cloud / Self-host / Hybrid
Collection console DEMO · No real requests

01 / Target URL (sample)

https://

02 / Collection scope

03 / Output format

Status: ready (demo) 0%
OUTPUT / MARKDOWNSample data
# Product description (sample)
Run the demo workflow to see a simulated extraction result. 
ROBOTS Respected by default LOG Traceable tasks RAG Data preparation ≠ Knowledge base
SCROLL / Explore the pipeline All interfaces · Data · Logs are simulated
Web sources → URL lists / Sitemaps / Domain scope
Collection & extraction → Fetching / Content / Structured fields
Cleaning & preparation → Rules / Deduplication / QA / Changes
Delivery & integration → Packages / Task records / Knowledge-base connector
Firecrawl web data collection and preparation — Chinese-language solution illustration
Firecrawl web data collection and preparation · Illustration in Chinese, not a delivered interface

Interfaces, figures and prices are examples. Actual features, licensing and delivery scope are confirmed for each project.

01 / CAPABILITY MAP · Official sources reviewed

Included engine features and custom implementation

Firecrawl is the collection engine. These groups distinguish engine features from custom work. Availability depends on the version, deployment path and supporting services. This distinction defines implementation effort and acceptance scope.

SOURCE / firecrawl · GitHub + Docs

Single-page Scrape

Convert a URL to Markdown, HTML, screenshots or structured JSON, with content-only options.

Official capability

Site-wide Crawl

Discover pages through sitemaps and recursive links, with limits, discovery depth and path filters.

Official capability

URL discovery with Map

Enumerate site URLs to confirm the collection scope before crawling.

Official capability

Asynchronous Batch

Process URL batches asynchronously and retrieve results through status checks; capacity depends on configuration.

Official capability

Content & output formats

Markdown and HTML output, plus schema-based structured JSON extraction.

Official capability

Task status & callbacks

Status polling, watchers and webhooks for started, page and completed events, with signature verification.

Official capability

Search & result extraction

Search can return extracted result content; providers and self-hosted configuration need validation.

Requires configuration

Model-assisted structure

Extract JSON against a schema. Configure model providers, file parsing and advanced extraction for the chosen deployment.

Verify per deployment

Site-specific cleaning

Define content selectors and remove navigation, footers, ads and template noise for each source.

Custom development

Content-level deduplication

Define content hashes, similarity thresholds and canonical merging, with human review for exceptions.

Custom development

Scheduled incremental updates

Set schedules, detect changes and selectively recollect changed pages for an updatable data source.

Custom development

Data quality checks

Check field completeness, missing values and abnormal pages, producing reviewable QA results.

Custom development

Task orchestration & audit

Track runs, retries, retention and access records to trace each delivery.

Custom development

Packaging & versions

Define Markdown and JSON folders, export conventions, snapshots and change lists.

Custom development

Downstream integration

Define write interfaces, field mappings and chunk metadata. Knowledge retrieval is a separate implementation.

Custom development

Access & collection policies

Review robots.txt and site terms, allowlists, collection rates and access audit requirements.

Custom development
Sources reviewed (2026-11): Official repository firecrawl/firecrawl、Official documentation and Crawl documentation . The official product documents Search, Scrape, Interact, Parse, Agent, Crawl, Map, Batch Scrape and Monitor. Scrape supports Markdown, HTML, screenshots and structured JSON; deployment availability varies. Key boundary: Business cleaning and content deduplication require explicit rules; URL crawling parameters alone do not implement them. ignoreQueryParameters avoids URL variants with different query parameters, rather than content duplicates. A completed crawl does not guarantee every page was covered. Business cleaning, content deduplication and update governance are separately scoped work.

02 / DATA PIPELINE · Pipeline visualization

Web sources → Extraction → Cleaning & preparation → Delivery & integration

Each stage has reviewable inputs and outputs. The engine handles collection and extraction; site-specific cleaning is custom work. Delivery defines the recipient, format and update cadence.

FLOW / 04 STAGES
STAGE 01 / SOURCE

Web sources

Agree what to collect and exclude: URL lists, domains, path filters, update frequency and access requirements.

URL list Sitemap Path rules Source ledger
STAGE 02 / COLLECT

Collection & extraction

Collect single pages, sites or URL batches. Render JavaScript pages where needed and return Markdown, HTML or schema-based JSON.

Single page Site crawl Content extraction JSON schema
STAGE 03 / CLEAN · Custom layer

Cleaning & preparation

Remove navigation, footers and ads; deduplicate by content hash or similarity; validate fields; mark new and changed pages for incremental updates.

Cleaning rules Content-level deduplication Quality checks Change markers
STAGE 04 / DELIVER

Deliver data / Connect a knowledge base

Deliver Markdown and JSON packages with a task ledger, or write to an agreed downstream system. Building the knowledge base is outside this stage.

Data package Task records Write interface Update plan
[DEMO EVENT] DEMO · Waiting for a simulated pipeline event; no real task is running.
Data collection ≠ A working RAG knowledge base

Collection produces usable source material. Retrieval and question answering still require the separate knowledge-base work shown here.

Scope boundary

This service: collection & preparation

01Source management : URL lists, domain and path rules
02Collection & extraction : Content, Markdown and structured fields
03Cleaning & deduplication : Site rules, QA and change markers
04Delivery & records : Packages, task ledger and update plan
05Optional connector : Send data to a customer-designated system

Separate work: RAG knowledge base

01Chunking & metadata : Chunk strategy, source attribution and timestamps
02Embeddings : Model selection and batch processing
03Storage & indexing : Vector storage, retrieval indexes and access boundaries
04Retrieval & generation : Recall, reranking, citations and prompts
05Evaluation & operations : Answer quality, refresh handling and monitoring

RAG DATA READINESS · Data preparation

Prepare material for RAG, with a clear handoff boundary

We prepare source material for downstream ingestion. These six deliverables remain part of collection and preparation; they do not constitute a knowledge base.

READY / 06 ITEMS
READY 01

Traceable sources

Include source_url, site identifiers and collection timestamps for attribution and review.

READY 02

Content structure

Clean Markdown retains heading levels for downstream semantic chunking.

READY 03

Structured fields

JSON fields such as name, SKU, specifications and update time can serve as retrieval metadata.

READY 04

Deduplication & versions

Merge duplicate pages and retain version markers to avoid repeated ingestion.

READY 05

Incremental change markers

Identify new, changed and invalid pages to support batch updates.

READY 06

Delivery contract

Agree folder conventions, batch identifiers and write interfaces for downstream ingestion.

Scope reminder: This is data preparation only. Chunking, embeddings, vector stores, indexing, reranking, citations and evaluation require a separate RAG implementation. No completed knowledge base is claimed.

03 / PRODUCT SURFACE · Interface prototype

Daily work revolves around tasks, outputs and rules

Four local prototype views show task filtering, output formats and rule toggles. They illustrate a potential delivery interface and have no connection to a live service or database.

DEMO UI / 04 VIEWS
Data preparation workspace All data is simulated
Filter (demo) Sample ledger
Task ID Source (sample domain) Mode Pages Output Status Time
Total 0 sample records · No live tasks Ledger fields can be adapted to the customer system
RAW / Page content (sample) Sample data
# Raw page noise (illustration) 
Home  Products  Docs  Pricing  Sign in  ▸
 ## PX-200 industrial gateway description 
PX-200 supports redundant links and remote maintenance…
Related reading | Products | Copyright ©
Ads / Breadcrumbs / Footer / Cookie notice 
CLEAN / Markdown output (sample) Sample data
# PX-200 industrial gateway 
"category": "Network equipment" 
"sku": "PX-200"

## Specifications
- Redundant link switchover: < 200 ms
- Operating temperature: -20 to 65 °C
- Remote maintenance: supported

 --- source: docs.example.com/product/px-200
--- crawled_at: 2026-11-15 09:42 (sample) 
Keep clean content and source metadata Confirm output fields with the customer
SCHEMA / Structured extraction (sample) Sample data
{
  "type": "object",
  "properties": {
    "product_name": { "type": "string" },
    "sku":          { "type": "string" },
    "specs":        { "type": "array" },
    "updated_at":   { "type": "string" }
  }
}
RESULT / Single-page extraction (sample) Sample data
{
  "product_name": "PX-200 industrial gateway" ,
  "sku": "PX-200",
  "specs": [
    "Redundant links < 200 ms" ,
    "Operating temperature -20 to 65 °C" 
  ],
  "source_url": "docs.example.com/…",
  "updated_at": "Sample timestamp" 
}
Define fields for the business; schema-based JSON extraction is an engine capability Field validation is custom work
These are site-specific implementation rules, rather than an included business workflow Toggle sample rule states

04 / DEPLOYMENT · Deployment

Three delivery paths, chosen by data boundaries

Deployment determines the data path, operating responsibility and license obligations. These are options; this page performs no deployment.

OPTION / 03 PATHS
PATH 01 / CLOUD API

Firecrawl Cloud API

Use the managed API without operating the infrastructure. Cloud also provides capabilities beyond the default open-source stack.

  • Fits: fast validation, varied sources and fluctuating demand
  • Data path: processed by a third-party service
  • Trade-off: metered usage and data-boundary review
PATH 02 / SELF-HOST

Self-hosted Docker Compose

Run the selected release using SELF_HOST.md and its Compose file. Plan customer-side storage and separately review external sites, models and proxies. The customer operates supporting services.

  • Fits: infrastructure control and an operations team
  • Components: API / Workers / NuQ PostgreSQL and supporting services
  • Trade-off: upgrades, scaling and monitoring
PATH 03 / HYBRID

Hybrid: separate collection from ingestion

Choose cloud or self-hosted extraction, then keep cleaning, deduplication, task records and delivery in the customer environment.

  • Fits: existing data platforms and gradual migration
  • Boundary: explicit ownership of cleaning and scheduling
  • Trade-off: clear interfaces and data contracts
License & policy review: The main repository uses AGPL-3.0,Some SDK and UI packages use MIT; verify each package ; official deployment references include SELF_HOST.md、docker-compose.yaml and the Self-Hosting Guide. Components and variables must match the selected release. Firecrawl respects robots.txtby default. Users remain responsible for site policies, privacy obligations and terms of use. Review AGPL-3.0 obligations before providing a modified network service ; this page is not legal advice.

05 / DELIVERABLES · Deliverables

The deliverable is a repeatable data workflow with defined handover assets

Sources, rules, task records and update plans have reviewable outputs. Final implementation and acceptance scope must be agreed for the project.

OUTPUT / 06 ITEMS

Source list & collection configuration

URL lists, domains, path filters and collection cadence.

Site cleaning rules

Content selectors, excluded noise, field mappings and exception handling.

Structured dataset

Markdown and JSON, with source URLs and collection timestamps.

Deduplication & QA results

Duplicate decisions, changed pages and review items for spot checks.

Task records & execution logs

Run status, page counts, failure reasons and rerun history.

Refresh & integration plan

Incremental updates, folder conventions and downstream write interfaces.

06 / WORK SCENARIOS · Business scenarios

From copying pages to maintaining fresh data

Common starting points include scattered product documentation, recurring notice reviews and stale knowledge-base material. Stock photos illustrate collaboration, rather than customer projects.

SCENE / 03 FRAMES
SCENE 01 / SOURCE REVIEWStock photo of a team reviewing information sources and scope
Define sources & scope

Agree included pages, excluded paths and update frequency before running tasks.

SCENE 02 / QUALITY CHECKStock photo of a team reviewing data preparation results
Review extraction & cleaning

Check noisy pages, duplicates and missing fields, then record exceptions as rules.

SCENE 03 / HANDOVERStock photo of a developer reviewing a data integration environment
Handover & integration

Deliver agreed packages and task records, or connect a customer-designated system.

SCENE 04 / OPERATIONSStock photo of code in a working environment
Recurring updates

Bring new and changed pages into the dataset on an agreed schedule. Update governance is custom implementation work.

Stock photos illustrate collaboration and are not Yuqi customer-site photographs. Interfaces, task data and logs are simulated.

07 / VALUE · Less manual preparation

Replace repetition with a maintainable data process

This comparison describes workflow changes, not measured business outcomes. Results depend on source structure, content quality and the agreed update cadence.

BEFORE / AFTER

Manual preparation: common starting point

01Open, copy and paste page by page with inconsistent formats
02Navigation, ads and footers mixed into content and removed manually
03Duplicates are hard to identify and outdated pages remain after redesigns
04No execution history to trace problems
05Updates depend on reminders while knowledge material becomes stale

Collection workflow: proposed approach

01Use agreed URL lists and path rules consistently
02Site-specific cleaning Remove template noise
03Content deduplication + Change markers to process changed pages
04Task ledger & logs for review and reruns
05Scheduled refreshes for a maintainable data source
NOTE Proposed approach, not a performance guarantee SCOPE RAG knowledge-base construction excluded DEMO All page data is simulated ACTION No real bulk collection

NEXT STEP / Discuss your requirements

Start with sources and update frequency,
then plan deployment and integration

Begin with a URL list. Agree source structure, cleaning, deduplication and formats, then define implementation scope. RAG knowledge-base construction is evaluated separately.