Highlights
In-memory Data Management
HyPer relies on in-memory data management without the ballast of traditional database systems caused by DBMS-controlled page structures and buffer management. SQL table definitions are transformed into simple vector-based virtual memory representations – which constitutes a column oriented physical storage scheme.
Efficient Snapshotting
OLAP query processing is separated from mission-critical OLTP transaction processing by forking virtual memory snapshots. Thus, no concurrency control mechanisms are needed – other than the hardware-assisted transparent VM management – to separate the two workload classes.
Data-centric Code Generation
Transactions and queries are specified in SQL or a PL/SQL-like scripting language and are efficiently compiled into efficient LLVM assembly code.
No compromises
HyPer's transaction processing is fully ACID-compliant. Queries are specified in SQL-92 plus some extensions from subsequent standards.
Team
Head: Prof. Alfons Kemper, Prof. Thomas Neumann
Ph.D. Students: Robert Brunel, Jan Finis, Florian Funke, Moritz Kaufmann, Viktor Leis, Tobias Mühlbauer, Wolf Rödiger, Manuel Then
Publications
Foundational HyPer publications
| Venue | Publication | Link |
|---|---|---|
| ICDE 2011 |
HyPer: A Hybrid OLTP&OLAP Main Memory Database System Based on Virtual Memory Snapshots Alfons Kemper and Thomas Neumann, 2011. |
IEEE Xplore Report (pdf) |
| VLDB 2011 |
Efficiently Compiling Efficient Query Plans for Modern Hardware Thomas Neumann, 2011. |
List of all publications
| Venue | Publication | Link |
|---|---|---|
| VLDB 2014 |
Engineering High-Performance Database Engines Thomas Neumann, 2014. |
|
| DaMoN 2014 |
Heterogeneity-Conscious Parallel Query Execution: Getting a better mileage while driving faster! Tobias Mühlbauer, Wolf Rödiger, Robert Seilbeck, Alfons Kemper, Thomas Neumann, 2014. |
|
| SIGMOD 2014 |
Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Framework for the Many-Core Age Viktor Leis, Peter Boncz, Alfons Kemper, Thomas Neumann, 2014. |
|
| SIGMOD 2014 |
One DBMS for all: the Brawny Few and the Wimpy Crowd (Demonstration) Tobias Mühlbauer, Wolf Rödiger, Robert Seilbeck, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2014. |
|
| DEBULL |
Compiling Database Queries into Machine Code Thomas Neumann, Viktor Leis, Data Engineering Bulletin, March 2014. |
|
| ICDE 2014 |
Locality-Sensitive Operators for Parallel Main-Memory Database Clusters Wolf Rödiger, Tobias Mühlbauer, Philipp Unterbrunner, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2014. |
|
| ICDE 2014 |
Exploiting Hardware Transactional Memory in Main-Memory Databases Viktor Leis, Alfons Kemper, Thomas Neumann, 2014. Best Paper Award |
|
| PVLDB 2013, VLDB 2014 |
Instant Loading for Main Memory Databases Tobias Mühlbauer, Wolf Rödiger, Robert Seilbeck, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2013. |
|
| CIKM 2013 |
RWS-Diff: Flexible and Efficient Change Detection in Hierarchical Data Jan Finis, Martin Raiber, Nikolaus Augsten, Robert Brunel, Alfons Kemper, Franz Färber, 2013. |
|
| IMDM 2013 |
An Evaluation of Strict Timestamp Ordering Concurrency Control for Main-Memory Database Systems Stephan Wolf, Henrik Mühe, Alfons Kemper, Thomas Neumann, 2013. |
|
| IMDM 2013 |
Massively Parallel NUMA-aware Hash Joins Harald Lang, Viktor Leis, Martina-Cezara Albutiu, Thomas Neumann, Alfons Kemper, 2013. |
|
| DEBULL |
Transaction Processing in the Hybrid OLTP&OLAP Main-Memory Database System HyPer Alfons Kemper, Thomas Neumann, Jan Finis, Florian Funke, Viktor Leis, Henrik Mühe, Tobias Mühlbauer, Wolf Rödiger, IEEE Computer Society Data Engineering Bulletin, Special Issue on "Main Memory Databases", 2013. |
Issue |
| DanaC 2013 |
ScyPer: Elastic OLAP Throughput on Transactional Data Tobias Mühlbauer, Wolf Rödiger, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2013. |
ACM DL |
| SIGMOD 2013 |
DeltaNI: An Efficient Labeling Scheme for Versioned Hierarchical Data Jan Finis, Robert Brunel, Alfons Kemper, Thomas Neumann, Franz Faerber, Norman May, 2013. |
|
| BTW 2013 |
Extending the MPSM Join Martina-Cezara Albutiu, Alfons Kemper, Thomas Neumann, 2013. |
|
| BTW 2013 |
ScyPer: A Hybrid OLTP&OLAP Distributed Main Memory Database System for Scalable Real-Time Analytics (Demonstration) Tobias Mühlbauer, Wolf Rödiger, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2013. |
|
| CIDR 2013 |
Executing Long-Running Transactions in Synchronization-Free Main Memory Database Systems Henrik Mühe and Alfons Kemper and Thomas Neumann, 2013. |
|
| ICDE 2013 |
CPU and Cache Efficient Management of Memory-Resident Databases Holger Pirk, Florian Funke, Martin Grund, Thomas Neumann, Ulf Leser, Stefan Manegold, Alfons Kemper, Martin Kersten, 2013. |
|
| ICDE 2013 |
The Adaptive Radix Tree: ARTful Indexing for Main-Memory Databases Viktor Leis, Alfons Kemper and Thomas Neumann, 2013. |
|
| VLDB 2012 |
Massively Parallel Sort-Merge Joins in Main Memory Multi-Core Database Systems Martina-Cezara Albutiu, Alfons Kemper and Thomas Neumann, 2012. |
|
| VLDB 2012 |
Compacting Transactional Data in Hybrid OLTP&OLAP Databases Florian Funke, Alfons Kemper, Thomas Neumann, 2012. |
|
| DEBULL |
HyPer: Adapting Columnar Main -Memory Data Management for Transactional AND Query Processing Alfons Kemper, Thomas Neumann, Florian Funke, Viktor Leis, Henrik Mühe, Bulletin of the Technical Committee on Data Engineering, March 2012, Vol. 35, No. 1, pp. 46–51. |
Issue |
| Technical Report |
Massively Parallel Sort-Merge Joins in Main Memory Multi-Core Database Systems Martina-Cezara Albutiu, Alfons Kemper and Thomas Neumann, Technical Report, TUM-I121, March, 16, 2012. |
pdf, pptx |
| EDBT 2012 |
The Mainframe Strikes Back: Elastic Multi-Tenancy Using Main Memory Database Systems On A Many-Core Server Henrik Mühe, Alfons Kemper and Thomas Neumann, 2012. |
|
| VLDB 2011 |
HyPer-sonic Combined Transaction AND Query Processing Florian Funke and Alfons Kemper and Thomas Neumann, 2011. |
|
| VLDB 2011 |
Efficiently Compiling Efficient Query Plans for Modern Hardware Thomas Neumann, 2011. |
|
| DBTest 2011 |
The mixed workload CH-benCHmark Dagstuhl "Robust Query Processing" Breakout Group "Workload Management", 2011. |
|
| DaMoN 2011 |
How to Efficiently Snapshot Transactional Data: Hardware or Software Controlled? Henrik Mühe and Alfons Kemper and Thomas Neumann, 2011. |
|
| Datenbank Spektrum |
HyPer: Die effiziente Reinkarnation des Schattenspeichers in einem Hauptspeicher-DBMS Florian Funke and Alfons Kemper and Henrik Mühe and Thomas Neumann, 2011. |
Datenbank Spektrum |
| BTW 2011 |
Benchmarking Hybrid OLTP&OLAP Database Systems Florian Funke and Alfons Kemper and Thomas Neumann, 2011. |
|
| ICDE 2011 |
HyPer: A Hybrid OLTP&OLAP Main Memory Database System Based on Virtual Memory Snapshots Alfons Kemper and Thomas Neumann, 2011. |
IEEE Xplore |
| Technical Report |
HyPer - Hybrid OLTP&OLAP High Performance Database System Alfons Kemper and Thomas Neumann, Technical Report, TUM-I1010, May, 19, 2010. |
Presentations/Mentions
| Date | Venue |
|---|---|
| May 21, 2010 | Colloquium of the Chair of Database Systems |
| May 26, 2010 | "Grundlagen von Datenbanken " Workshop (GvDB, Bad Helmstedt) |
| June 26, 2010 | IBM Böblingen |
| July 22, 2010 | Inaugural Lecture ("Antrittsvorlesung" Thomas Neumann) |
| August 13, 2010 | IBM Almaden Research |
| August 24, 2010 | HP Labs Palo Alto |
| August 30, 2010 | SAP Labs Palo Alto |
| September 1, 2010 | Greenplum (See Florian Waas' Blog about the presentation) |
| September 3, 2010 | Oracle Redwood Shores |
| September 13, 2010 | Keynote at the VLDB BIRTE Workshop |
| September 30, 2010 | IBM DB2 Community Meeting, Böblingen |
| October 1, 2010 | SAP Walldorf |
| March 3, 2011 | BTW 2011, Presentation |
| April 12, 2011 | ICDE 2011, Poster |
| May 30, 2011 | Humboldt Universität Berlin |
| June, 2011 | "Grundlagen von Datenbanken" Workshop (Tirol, Austria) |
| October 26, 2011 | HyPer-sonic: Combined Transaction AND Query Processing, HPTS 2011, Slides |
| November 18, 2011 | Skalierbarkeit ODER Virtualisierung at FGDB Herbsttreffen, Potsdam |
| December 2, 2011 | HyPer-sonic Combined Transaction AND Query Processing at HIPERFIT Workshop, Kopenhagen |
| June 13, 2012 | Oracle Labs Research – Tea Time Talk |
| June 20, 2012 | HyPer and its Scale-Out at Software AG |
| October 11, 2012 | IBM DB2 Community Meeting, Böblingen |
| November 2, 2012 | GI FG-DB Workshop Scalable Analytics |
| January 4, 2013 | Join Processing and Indexing in Multi-Core Main-Memory Databases, Oracle Labs |
| March 11–15, 2013 | ScyPer: A Hybrid OLTP&OLAP Distributed Main Memory Database System for Scalable Real-Time Analytics (Poster), BTW |
| April 5, 2013 | The Adaptive Radix Tree, University of Sydney |
| June 10, 2013 | 2. Deutsches Community Treffen für GPUs in Datenbanken, TU Ilmenau |
| July 15, 2013 | Microsoft Research Faculty Summit 2013, Slides (Thomas Neumann) |
| August 16, 2013 | IBM Almaden Research |
| September 24, 2013 | Hardware Transactional Memory on Haswell, HPTS 2013 |
| Januar 31, 2014 | HyPer: one DBMS for all, New England Database Summit 2014, Slides (pdf), Slides (Keynote '09), Abstract |
Summary
The HyPer prototype demonstrates that it is indeed possible to build a main-memory database system that achieves world-record transaction processing throughput and best-of-breed OLAP query response times in one system in parallel on the same database state. The two workloads of online transaction processing (OLTP) and online analytical processing (OLAP) present different challenges for database architectures. Currently, users with high rates of mission-critical transactions have split their data into two separate systems, one database for OLTP and one so-called data warehouse for OLAP. While allowing for decent transaction rates, this separation has many disadvantages including data freshness issues due to the delay caused by only periodically initiating the Extract Transform Load-data staging and excessive resource consumption due to maintaining two separate information systems. We present an efficient hybrid system, called HyPer, that can handle both OLTP and OLAP simultaneously by using hardware-assisted replication mechanisms to maintain consistent snapshots of the transactional data. HyPer is a main-memory database system that guarantees the full ACID properties for OLTP transactions and executes OLAP query sessions (multiple queries) on arbitrarily current and consistent snapshots. The utilization of the processor-inherent support for virtual memory management (address translation, caching, copy-on-write) yields both at the same time: unprecedentedly high transaction rates as high as 100000 per second and very fast OLAP query response times on a single system executing both workloads in parallel. The performance analysis is based on a combined TPC-C and TPC-H benchmark. We have developed the novel hybrid OLTP&OLAP database system HyPer that is based on snapshotting transactional data via the virtual memory management of the operating system. In this architecture the OLTP process owns the database and periodically (e.g., in the order of seconds or minutes) forks an OLAP process. This OLAP process constitutes a fresh transaction consistent snapshot of the database. Thereby, we exploit operating systems functionality to create virtual memory snapshots for new, cloned processes. In Unix, for example, this is done by creating a child process of the OLTP process via the fork system call. The forked child process obtains an exact copy of the parent processes address space. This virtual memory snapshot that is created by the fork-operation will be used for executing a session of OLAP queries. These queries can be executed in parallel threads or serially, depending on the system resources or client requirements. In essence, the virtual memory snapshot mechanism constitutes a OS/hardware supported shadow paging mechanism as proposed decades ago for disk-based database systems. However, the original proposal incurred severe costs as it had to be software-controlled and it destroyed the clustering on disk. Neither of these drawbacks occurs in the virtual memory snapshotting as clustering across RAM pages is not an issue. Furthermore, the sharing of pages and the necessary copy-on-update/write is managed by the operating system with effective hardware support of the MMU (memory management unit) via the page table that translates VM addresses to physical pages and traps necessary replication (copy-on-write) actions. Therefore, the page replication is extremely efficiently done in 2μs as we measured in a micro-benchmark. HyPer's OLTP throughput is better than VoltDB's published TPC-C performance and HyPer's OLAP query response times are superior to MonetDB's query response times. It should be emphasized that HyPer can match (or beat) these two best- of-breed transaction (VoltDB) and query (MonetDB) processing engines at the same time by performing both workloads in parallel on the same database state. HyPer's performance is due to the following design:
- HyPer relies on in-memory data management without the ballast of traditional database systems caused by DBMS-controlled page structures and buffer management. SQL table definitions are transformed into simple vector-based virtual memory representations – which constitutes a column oriented physical storage scheme.
- OLAP query processing is separated from mission-critical OLTP transaction processing by forking virtual memory snapshots. Thus, no concurrency control mechanisms are needed – other than the hardware-assisted transparent VM management – to separate the two workload classes.
- Transactions and queries are specified in SQL or a PL/SQL-like scripting language and are efficiently compiled into efficient LLVM assembly code.
- HyPer's transaction processing is fully ACID-compliant. Queries are specified in standard SQL-92 plus some extensions from following standard SQL standards.
- HyPer relies on logical logging where, in essence, the invocation parameters of the stored (transaction) procedures are logged via a high-speed network.
Contact
Contact us (see Team for emails) if you are interested in a thesis, student job or even a Ph.D. position!
Technische Universität MünchenInstitut für Informatik
Lehrstuhl III: Datenbanksysteme (I3)
Boltzmannstraße 3
85748 Garching bei München
Germany