Back to work

Cloud infrastructure

Crawl.io

Distributed crawling. Reproducible infrastructure.

  • Distributed web crawling for LLM training, RAG, and search datasets.
  • Decoupled AWS services for scheduling, crawling, URL discovery, and storage.
  • Reproducible deployments through Terraform.
  • URL deduplication, crawl limits, filtering, and robots.txt compliance.

Decoupled by design

  • Separate scheduling, crawling, discovery, and storage across AWS services.
  • Provision infrastructure with Terraform.
Crawl.io fan-out architecture with EC2 coordinators, an SQS work queue, parallel Lambda workers, and storage in S3 and RDS
SQS distributes crawl jobs across parallel Lambda workers. A storage worker saves datasets in S3 and metadata in RDS.

From a URL to a dataset

  • Load crawl configuration and normalize, validate, and deduplicate URLs in the coordinator.
  • Fetch pages, parse content, and extract links in parallel Lambda workers.
  • Send discovered links back to the coordinator and processed results to a separate storage worker.
Crawl.io workflow from the CLI and configuration loader through the EC2 coordinator, SQS work, discovery and result queues, Lambda workers, RDS, and S3
The complete workflow, including the discovery queue that feeds newly found URLs back to the coordinator.

Observable and testable

  • Use structured logging and backend-independent metrics.
  • Keep deterministic pytest tests and deployment troubleshooting documentation.

Built with

  • Python
  • httpx
  • BeautifulSoup4
  • Protego
  • Pydantic
  • SQLAlchemy
  • AWS EC2 / Lambda / SQS / S3 / RDS
  • Terraform
  • pytest
next projectAutonomous QA agent