No Application Code
PipelineDB enables realtime data processing using only SQL. Our continuous query planner and execution engine handle all of the complexity of running realtime computation on streams.
PostgreSQL Compatible
PipelineDB is built into the PostgreSQL core and can be used as a drop in replacement for PostgreSQL without making any application code changes. It comes pre-installed with PostGIS and supports all other extensions too.
No ETL
PipelineDB eliminates the ETL layer. Stream granular data directly into the database and continuously distill it with the SQL queries you’ve declared.
Efficient and Sustainable
PipelineDB stores only the output of continuous queries, which are incrementally updated as data is ingested. The database’s size is independent of the amount of data ingested over time.
How It Works
Continuous views are the core abstraction of PipelineDB. You can think about them as very high-throughput, realtime, incrementally updated materialized views. The most important property of a continuous view is that only its output is actually stored in the database.
Stream Buffer: Tuples inserted into a stream are stored in a concurrent shared memory circular buffer.
SELECT count(*) FROM stream
| count |
|---|
| 0 |
Worker: Worker processes read micro-batches from the stream buffer, aggregate them into partial results and send them to combiner processes.
SELECT sum(count) FROM view, worker
| count |
|---|
| 10 + 5 |
Combiner: Combiner processes take partial results from workers and continuously merge them with on-disk tuples.