Data engineering · science · architecture

IngestThis

Working notes for people who move data for a living. Pipelines, lakehouses, table formats, and the architecture underneath them.

Read the archiveJoin the Slack
Articles
475+
Topics
Iceberg, pipelines, AI
Submissions
Open
Price
Free

Latest

All articles
  1. 01Mastering Apache Iceberg v3 Deletion Vectors for High-Throughput Streaming IngestApache Iceberg v3 deletion vectors for high-throughput streaming ingest: how bitmaps and Puffin files fix CDC write amplification and read decay.2026-08-19
  2. 02The Decoupled Data Lakehouse: Multi-Engine Freedom with Open REST CatalogsThe decoupled data lakehouse: multi-engine freedom with open REST catalogs, credential vending, and an estate that outlives its tools.2026-08-19
  3. 03The Five Layers of an Agentic LakehouseThe five layers of an agentic lakehouse: Storage, Catalog, Semantic, Gateway, and Agent Surface, and how one question travels through all of them.2026-08-19
  4. 04Goal-Directed Data Quality Agents: Anomaly Quarantine on Apache IcebergGoal-directed data quality agents that watch Apache Iceberg tables, detect anomalies, and quarantine suspect data safely with snapshot isolation and branches.2026-08-19
  5. 05Managing the TCO of Agentic Analytics: Token Budgets, Query Throttles, and the Economics of AutonomyManaging the total cost of agentic analytics: token budgets, query throttles, unit economics, and the FinOps discipline that keeps AI spend under control.2026-08-19
  6. 06Metric Contracts in 2026: Standardizing Business Logic Across Multi-Agent FrameworksMetric contracts in 2026: versioned, testable definitions of business logic that let multi-agent frameworks compute revenue identically, with OSI interchange.2026-08-19
  7. 07Policy-Aware Lakehouse Telemetry: Building Auditable AI Records on Apache Iceberg for the EU AI Act EraPolicy-aware lakehouse telemetry for the EU AI Act era: build auditable, immutable, governed AI decision records on Apache Iceberg.2026-08-19
  8. 08Query Routing at Machine Scale: Multi-Engine Workload Distribution for the Agentic LakehouseQuery routing at machine scale for the agentic lakehouse: engine selection, acceleration substitution, admission control, and placement across multi-engine estates.2026-08-19

Reference shelf

Long reads from around the web
Semantic Layer

The Semantic Layer: Definitive Guide

A comprehensive guide to the Semantic Layer — how it creates a single source of truth for metrics, powers headless BI, and makes AI agents answer business questions accurately.

ReadApache Polaris

Apache Polaris: The Catalog Standard for Lakehouses and AI

How Apache Polaris is emerging as the universal Iceberg catalog standard, enabling multi-engine interoperability and governed AI access across the lakehouse ecosystem.

ReadTable Formats

What Are Table Formats and Why Were They Needed?

The origin story of open table formats — the problems with Hive, why Apache Iceberg, Delta Lake, and Hudi were created, and what they unlock for modern data platforms.

ReadDremio

What Is Dremio?

A clear-eyed breakdown of what Dremio is, how its semantic layer, query federation, Reflections, and Apache Arrow Flight power the Intelligent Lakehouse Platform.

ReadApache Iceberg

What Apache Iceberg Native Actually Means

Not all 'Iceberg support' is equal. This piece breaks down what it means to be genuinely Apache Iceberg native versus bolt-on, and why it matters for your lakehouse.

ReadOpen Source

Open Source and the Data Lakehouse

How the Apache Software Foundation's open-source projects — Iceberg, Arrow, Parquet, Polaris — form the modular foundation of the modern open data lakehouse.

ReadAgentic AI

What Is Agentic Analytics?

Agentic AI is reshaping how organizations interact with data. This guide explains agentic analytics, the role of the semantic layer, and why query performance matters for AI agents.

ReadData Lakehouse

Definitive Guide to the Data Lakehouse

The complete, authoritative guide to the Data Lakehouse architecture — what it is, why it supersedes the data warehouse + data lake combination, and how to build one.

ReadAI & Performance

How Dremio Keeps Agentic Analytics Fast Without Manual Tuning

How Dremio's layered autonomous performance architecture — Reflections, caching, vectorized execution — handles unpredictable AI agent query patterns at interactive speed.

Read

Elsewhere

Pitch an idea: alex@ingestthis.com