Data engineering · science · architecture

IngestThis

Working notes for people who move data for a living. Pipelines, lakehouses, table formats, and the architecture underneath them.

Read the archiveJoin the Slack
Articles
500+
Topics
Iceberg, pipelines, AI
Submissions
Open
Price
Free

Latest

All articles
  1. 01The Agent Is Now a Named Coworker, and It Needs a File FormatNamed, persistent agents need a file format. Open Agent Profile, Buzz, Grok Bot, and Hermes Bot Mode show why a portable agent identity matters.2026-08-24
  2. 02Your Agent Should Answer the Phone: A Field Guide to AI Gateways on Slack, Discord, Telegram, Signal, and TeamsA field guide to AI gateways on Slack, Discord, Telegram, Signal, and Teams: architecture, auth, cost, and the failure modes that matter.2026-08-24
  3. 03Graphs in AI Engineering Have Solved Three Problems. The Fourth Is the Plan.Knowledge graphs, GraphRAG, and LangGraph solved three problems. The fourth is the work itself: a reviewable graph of bounded agentic loops.2026-08-24
  4. 04The Hidden Cost of Tiny Iceberg CommitsTrace what one tiny Iceberg commit writes, then model hourly, per-minute, and per-second cadences so streaming costs become arithmetic, not adjectives.2026-08-24
  5. 05Deletion Vectors vs Position Deletes vs Equality Deletes: The Iceberg Delete Story in 2026Position deletes, equality deletes, and deletion vectors compared from the Iceberg spec: what each writes, how readers apply it, and when to use which.2026-08-24
  6. 06Iceberg Is Becoming a Library, Not Just a Table FormatIceberg is turning from a JVM table format into a library other systems embed. What that shift changes for engines, catalogs, and the spec itself.2026-08-24
  7. 07Iceberg Is Escaping the JVM: Why Rust, Go, Python and C++ Implementations MatterRust, Go, Python, and C++ Iceberg implementations change who can write the format. Why multi-language clients matter more than another JVM engine.2026-08-24
  8. 08The Iceberg REST Catalog Compatibility Test: One Suite of Operations Every Platform Should PassOne suite of REST catalog operations every Iceberg platform should pass. What sameness means, where implementations diverge, and how to test it.2026-08-24

Reference shelf

Long reads from around the web
Semantic Layer

The Semantic Layer: Definitive Guide

A comprehensive guide to the Semantic Layer — how it creates a single source of truth for metrics, powers headless BI, and makes AI agents answer business questions accurately.

ReadApache Polaris

Apache Polaris: The Catalog Standard for Lakehouses and AI

How Apache Polaris is emerging as the universal Iceberg catalog standard, enabling multi-engine interoperability and governed AI access across the lakehouse ecosystem.

ReadTable Formats

What Are Table Formats and Why Were They Needed?

The origin story of open table formats — the problems with Hive, why Apache Iceberg, Delta Lake, and Hudi were created, and what they unlock for modern data platforms.

ReadDremio

What Is Dremio?

A clear-eyed breakdown of what Dremio is, how its semantic layer, query federation, Reflections, and Apache Arrow Flight power the Intelligent Lakehouse Platform.

ReadApache Iceberg

What Apache Iceberg Native Actually Means

Not all 'Iceberg support' is equal. This piece breaks down what it means to be genuinely Apache Iceberg native versus bolt-on, and why it matters for your lakehouse.

ReadOpen Source

Open Source and the Data Lakehouse

How the Apache Software Foundation's open-source projects — Iceberg, Arrow, Parquet, Polaris — form the modular foundation of the modern open data lakehouse.

ReadAgentic AI

What Is Agentic Analytics?

Agentic AI is reshaping how organizations interact with data. This guide explains agentic analytics, the role of the semantic layer, and why query performance matters for AI agents.

ReadData Lakehouse

Definitive Guide to the Data Lakehouse

The complete, authoritative guide to the Data Lakehouse architecture — what it is, why it supersedes the data warehouse + data lake combination, and how to build one.

ReadAI & Performance

How Dremio Keeps Agentic Analytics Fast Without Manual Tuning

How Dremio's layered autonomous performance architecture — Reflections, caching, vectorized execution — handles unpredictable AI agent query patterns at interactive speed.

Read

Elsewhere

Pitch an idea: alex@ingestthis.com