Marco Cippitelli

ADIE: What About Declarative Web Scraping?

With over half a decade of web scraping experience, I couldn't help but wonder if there wasn't a cleaner way to scrape the web, and this is what I came up with.

ADIE Screenshot

Overview

Agnostic Data Ingestion Engine (ADIE) is a high-performance, declarative web scraping and data ingestion engine built in Rust. It is engineered specifically for open and public data ingestion, serving as a robust, systems-level alternative to heavy, brittle, and ad-hoc imperative scraping routines.

ADIE is not an browser-automation tool for simulating user interactions or handling authenticated sessions. Instead, it is a backend-first engine designed to manage the entire lifecycle of public data extraction—from asynchronous fetching to structured persistence—while prioritizing system stability, concurrency control, and observability.

Key Features

Concurrency & Performance

  • Async Runtime: Built on tokio for non-blocking I/O and efficient task scheduling.
  • Resource Management: Implements tokio::sync::Semaphore for domain-level rate limiting, ensuring polite scraping and preventing resource exhaustion.
  • Thread Safety: Utilizes Arc and Mutex primitives to maintain data integrity across concurrent execution threads.

Data Pipeline (ETL)

  • Extraction: Declarative engine designed to handle simple XML/HTML structures, JavaScript-hydrated sites and API outputs.
  • Transformation: Strongly-typed data mapping using Rust’s `struct` system to ensure schema consistency.
  • Load: Modular storage integration with SQLite, MongoDB, MySQL and PostgreSQL for flexible, high-volume data ingestion.

Observability & TUI

  • Real-time Monitoring: Custom TUI (Terminal User Interface) to track motor status, execution cycles, and health metrics.
  • Structural Logging: Segmented logging system for granular debugging of concurrent processes.

Environment & Stack

  • Language: Rust (Edition 2024).
  • Operating System: Currently developed on and optimized for Debian GNU/Linux 13 (Trixie).
  • Supported Databases: SQLite, MySQL, PostgreSQL, MongoDB.

Development Principles

  • Sovereignty: Designed as a proprietary tool to maintain full control over the scraping logic and data lifecycle.
  • Resilience: Focus on long-term uptime and defensive error handling to manage the volatility of live web targets.
  • Efficiency: Minimal footprint compared to browser-based automation tools like Puppeteer or Selenium.

At its core, ADIE shifts the paradigm from writing complex extraction code to writing simple configurations. By using a human-readable YAML Domain-Specific Language (DSL), it allows you to define what data to collect, completely decoupling your data schema from the underlying network and parsing mechanics.

You focus entirely on the public data structure; ADIE handles the execution safely and at scale.