Exokernel: An Operating System Architecture for
Application-Level Resource Management
Dawson R. Engler, M. Frans Kaashoek, and James O’Toole Jr.
M.I.T. Laboratory for Computer Science
Cambridge, MA 02139, U.S.A
{engler, kaashoek, james}@lcs .mit. edu
Abstract
Traditional operating systems limit the performance, flexibility, and
functionality of applications by fixing the interface and implemen-
tation of operating system abstractions such as interprocess com-
munication and virtual memory. The exokernel operating system
architecture addresses this problem by providing application-level
management of physical resources. In the exokernel architecture, a
small kernel securely exports all hardware resources through a low-
level interface to untrusted library operating systems. Library op-
erating systems use this interface to implement system objects and
policies. This separation of resource protection from management
allows application-specific customization of traditional operating
system abstractions by extending, specializing, or even replacing
libraries.
We have implemented a prototype exokernel operating system.
Measurements show that most primitive kernel operations (such
as exception handling and protected control transfer) are ten to 100
times faster than in Ultrix, a mature monolithic UNIX operating sys-
tem. In addition, we demonstrate that an exokemel allows applica-
tions to control machine resources in ways not possible in traditional
operating systems. For instance, virtual memory and interprocess
communication abstractions are implemented entirely within an
application-level library. Measurements show that application-level
virtual memory and interprocess communication primitives are five
to 40 times faster than Ultrix’s kernel primitives. Compared to
state-of-the-art implementations from the literature, the prototype
exokemel system is at least five times faster on operations such as
exception dispatching and interprocess communication.
1 Introduction
Operating systems define the interface between applications and
physical resources. Unfortunately, this interface can significantly
limit the performance and implementation freedom of applications.
Traditionally, operating systems hide information about machine
resources behind high-level abstractions such as processes, tiles,
address spaces and interprocess communication. These abstrac-
tions define a virtual machine on which applications execute; their
implementation cannot be replaced or modified by untrusted appli-
cations. Hardcoding the implementations of these abstractions is
Tins research was supported m part by the Advanced Research Projects Agency under
contract NOOO14-94-1-0985 and by a NSF National Young Investigator Awwd.
Permission to make digital/hard copy of part or all of this work for personal
or classroom use is granted without fee provided that copies are not made
or distributed for profit or commercial advantage, the copyright notice, the
title of the publication and its date appear, and notice is given that
copying is by permission of ACM, Inc. To copy otherwise, to republish, to
post on servers, or to redistribute to tists, requires prior specific permission
andior a fee,
SIGOPS ’95 12/95 CO, USA
ID 1995 ACM 0-89791 -715-419510012... $3.50
inappropriate for three main reasons: it denies applications the ad-
vantages of domain-specific optimizations, it discourages changes
to the implementations of existing abstractions, and it restricts the
flexibility of application builders, since new abstractions can only
be added by awkward emulation on top of existing ones (if they can
be added at all).
We believe these problems can be solved through application-
level (i.e., untrusted) resource management. To this end, we have
designed a new operating system architecture, exokernel, in which
traditional operating system abstractions, such as virtual memory
(VM) and interprocess communication (IPC), are implemented en-
tirely at application level by untrusted software. In this architecture,
a minimal kernel-which we call an exokernel—securely multi-
plexes available hardware resources. Library operating systems,
working above the exokemel interface, implement higher-level ab-
stractions. Application writers select libraries or implement their
own. New implementations of library operating systems are incor-
porated by simply relinking application executable.
Substantial evidence exists that applications can benefit greatly
from having more control over how machine resources are used
to implement higher-level abstractions. Appel and Li [5] reported
that the high cost of general-purpose virtual memory primitives
reduces the performance of persistent stores, garbage collectors, and
distributed shared memory systems. Cao et al. [10] reported that
application-level control over file caching can reduce application
running time by L$5~0. Harty and Cheriton [26] and Krueger et
al. [30] showed how application-specific virtual memory policies
can increase application performance. Stonebraker [47] argued
that inappropriate tile-system implementation decisions can have a
dramatic impact on the performance of databases. Thekkath and
Levy [50] demonstrated that exceptions can be made an order of
magnitude faster by deferring signal handling to applications.
To provide applications control over machine resources, an ex-
okemel defines a low-level interface. The exokemel architecture is
founded on and motivated by a single, simple, and old observation:
the lower the level of a primitive, the more efficiently it can be
implemented, and the more latitude it grants to implementors of
higher-level abstractions.
To provide an interface that is as low-level as possible (ideally,
just the hardware interface), an exokemel designer has a single
overriding goal: to separate protection from management. For
instance, an exokemel should protect framebuffers without under-
standing windowing systems and disks without understanding file
systems. One approach is to give each application its own virtual
machine [17]. As we discuss in Section 8, virtual machines can
have severe performance penalties. Therefore, an exokemel uses a
different approach: it exports hardware resources rather than emu-
lating them, which allows an efficient and simple implementation.
An exokemel employs three techniques to export resources securely.
First, by using secure bindings, applications can securely bind to
machine resources and handle events. Second, by using visible re -
251
Application-Level Resource Management
Dawson R. Engler, M. Frans Kaashoek, and James O’Toole Jr.
M.I.T. Laboratory for Computer Science
Cambridge, MA 02139, U.S.A
{engler, kaashoek, james}@lcs .mit. edu
Abstract
Traditional operating systems limit the performance, flexibility, and
functionality of applications by fixing the interface and implemen-
tation of operating system abstractions such as interprocess com-
munication and virtual memory. The exokernel operating system
architecture addresses this problem by providing application-level
management of physical resources. In the exokernel architecture, a
small kernel securely exports all hardware resources through a low-
level interface to untrusted library operating systems. Library op-
erating systems use this interface to implement system objects and
policies. This separation of resource protection from management
allows application-specific customization of traditional operating
system abstractions by extending, specializing, or even replacing
libraries.
We have implemented a prototype exokernel operating system.
Measurements show that most primitive kernel operations (such
as exception handling and protected control transfer) are ten to 100
times faster than in Ultrix, a mature monolithic UNIX operating sys-
tem. In addition, we demonstrate that an exokemel allows applica-
tions to control machine resources in ways not possible in traditional
operating systems. For instance, virtual memory and interprocess
communication abstractions are implemented entirely within an
application-level library. Measurements show that application-level
virtual memory and interprocess communication primitives are five
to 40 times faster than Ultrix’s kernel primitives. Compared to
state-of-the-art implementations from the literature, the prototype
exokemel system is at least five times faster on operations such as
exception dispatching and interprocess communication.
1 Introduction
Operating systems define the interface between applications and
physical resources. Unfortunately, this interface can significantly
limit the performance and implementation freedom of applications.
Traditionally, operating systems hide information about machine
resources behind high-level abstractions such as processes, tiles,
address spaces and interprocess communication. These abstrac-
tions define a virtual machine on which applications execute; their
implementation cannot be replaced or modified by untrusted appli-
cations. Hardcoding the implementations of these abstractions is
Tins research was supported m part by the Advanced Research Projects Agency under
contract NOOO14-94-1-0985 and by a NSF National Young Investigator Awwd.
Permission to make digital/hard copy of part or all of this work for personal
or classroom use is granted without fee provided that copies are not made
or distributed for profit or commercial advantage, the copyright notice, the
title of the publication and its date appear, and notice is given that
copying is by permission of ACM, Inc. To copy otherwise, to republish, to
post on servers, or to redistribute to tists, requires prior specific permission
andior a fee,
SIGOPS ’95 12/95 CO, USA
ID 1995 ACM 0-89791 -715-419510012... $3.50
inappropriate for three main reasons: it denies applications the ad-
vantages of domain-specific optimizations, it discourages changes
to the implementations of existing abstractions, and it restricts the
flexibility of application builders, since new abstractions can only
be added by awkward emulation on top of existing ones (if they can
be added at all).
We believe these problems can be solved through application-
level (i.e., untrusted) resource management. To this end, we have
designed a new operating system architecture, exokernel, in which
traditional operating system abstractions, such as virtual memory
(VM) and interprocess communication (IPC), are implemented en-
tirely at application level by untrusted software. In this architecture,
a minimal kernel-which we call an exokernel—securely multi-
plexes available hardware resources. Library operating systems,
working above the exokemel interface, implement higher-level ab-
stractions. Application writers select libraries or implement their
own. New implementations of library operating systems are incor-
porated by simply relinking application executable.
Substantial evidence exists that applications can benefit greatly
from having more control over how machine resources are used
to implement higher-level abstractions. Appel and Li [5] reported
that the high cost of general-purpose virtual memory primitives
reduces the performance of persistent stores, garbage collectors, and
distributed shared memory systems. Cao et al. [10] reported that
application-level control over file caching can reduce application
running time by L$5~0. Harty and Cheriton [26] and Krueger et
al. [30] showed how application-specific virtual memory policies
can increase application performance. Stonebraker [47] argued
that inappropriate tile-system implementation decisions can have a
dramatic impact on the performance of databases. Thekkath and
Levy [50] demonstrated that exceptions can be made an order of
magnitude faster by deferring signal handling to applications.
To provide applications control over machine resources, an ex-
okemel defines a low-level interface. The exokemel architecture is
founded on and motivated by a single, simple, and old observation:
the lower the level of a primitive, the more efficiently it can be
implemented, and the more latitude it grants to implementors of
higher-level abstractions.
To provide an interface that is as low-level as possible (ideally,
just the hardware interface), an exokemel designer has a single
overriding goal: to separate protection from management. For
instance, an exokemel should protect framebuffers without under-
standing windowing systems and disks without understanding file
systems. One approach is to give each application its own virtual
machine [17]. As we discuss in Section 8, virtual machines can
have severe performance penalties. Therefore, an exokemel uses a
different approach: it exports hardware resources rather than emu-
lating them, which allows an efficient and simple implementation.
An exokemel employs three techniques to export resources securely.
First, by using secure bindings, applications can securely bind to
machine resources and handle events. Second, by using visible re -
251
source revocation, applications participate in a resource revocation
protocol. Third, by using an abortprotocol, an exokernel can break
secure bindings of uncooperative applications by force.
We have implemented a prototype exokemel system based on
secure bindings, visible revocation, and abort protocols. It includes
an exokemel (Aegis) and an untrusted library operating system
(ExOS). We use this system to demonstrate several important prop-
erties of the exokemel architecture: (1) exokemels can be made
efficient due to the limited number of simple primitives they must
provide; (2) low-level secure multiplexing of hardware resources
can be provided with low overhead; (3) traditional abstractions,
such as VM and IPC, can be implemented efficiently at application
level, where they can be easily extended, specialized, or replaced;
and (4) applications can create special-purpose implementations of
abstractions, tailored to their functionality and performance needs.
In practice, our prototype exokernel system provides applica-
tions with greater flexibility and better performance than mono-
lithic and microkemel systems. Aegis’s low-level interface allows
application-level software such as ExOS to manipulate resources
very efficiently. Aegis’s protected control transfer is almost seven
times faster than the best reported implementation [33]. Aegis’s
exception dispatch is five times faster than the best reported imple-
mentation [50]. On identical hardware, Aegis’s exception dispatch
and control transfer are roughly two orders of magnitude faster than
in Ultrix 4.2, a mature monolithic system.
Aegis also gives ExOS (and other application-level software)
flexibility that is not available in microkemel-based systems. For
instance, virtual memory is implemented at application level, where
it can be tightly integrated with distributed shared memory systems
and garbage collectors. Aegis’s efficient protected control trans-
fer allows applications to construct a wide array of efficient IPC
primitives by trading performance for additional functionality. In
contrast, microkemel systems such as Amoeba [48], Chorus [43],
Mach [2], and V[15] do not allow untrusted application software to
define specialized IPC primitives because virtual memory and mes-
sage passing services are implemented by the kernel and trusted
servers. Similarly, many other abstractions, such as page-table
structures and process abstractions, cannot be modified in micro-
kemels. Finally, many of the hardware resources in microker-
nel systems, such as the network, screen, and disk, are encapsu-
lated in heavyweight servers that cannot be bypassed or tailored
to application-specific needs. These heavyweight servers can be
viewed as fixed kernel subsystems that run in user-space.
This paper focuses on the exokemel architecture design and how
it can be implemented securely and efficiently. Section 2 provides
a more detailed case for exokemels. Section 3 discusses the issues
that arise in their design. Section 4 overviews the status of our proto-
type and explains our experimental methodology. Sections 5 and 6
present the implementation and summarize performance measure-
ments of Aegis and ExOS. Section 7 reports on experiments that
demonstrate the flexibility of the exokemel architecture. Section 8
summarizes related work and Section 9 concludes.
2 Motivation for Exokernels
Traditionally, operating systems have centralized resource manage-
ment via a set of abstractions that cannot be specialized, extended,
or replaced. Whether provided by the kernel or by trusted user-
level servers (as in microkemel-based systems), these abstractions
are implemented by privileged software that must be used by all
applications, and therefore cannot be changed by untrusted soft-
ware. Typically, the abstractions include processes, files, address
spaces, and interprocess communication. In this section we discuss
the problems with general-purpose implementations of these ab-
stractions and show how the exokernel architecture addresses these
problems.
2.1 The Cost of Fixed High-Level Abstractions
The essential observation about abstractions in traditional operating
systems is that they are overly general. Traditional operating sys-
tems attempt to provide all the features needed by all applications.
As previously noted by Lampson and Sproul [32], Anderson et
al. source revocation, applications participate in a resource revocation
protocol. Third, by using an abortprotocol, an exokernel can break
secure bindings of uncooperative applications by force.
We have implemented a prototype exokemel system based on
secure bindings, visible revocation, and abort protocols. It includes
an exokemel (Aegis) and an untrusted library operating system
(ExOS). We use this system to demonstrate several important prop-
erties of the exokemel architecture: (1) exokemels can be made
efficient due to the limited number of simple primitives they must
provide; (2) low-level secure multiplexing of hardware resources
can be provided with low overhead; (3) traditional abstractions,
such as VM and IPC, can be implemented efficiently at application
level, where they can be easily extended, specialized, or replaced;
and (4) applications can create special-purpose implementations of
abstractions, tailored to their functionality and performance needs.
In practice, our prototype exokernel system provides applica-
tions with greater flexibility and better performance than mono-
lithic and microkemel systems. Aegis’s low-level interface allows
application-level software such as ExOS to manipulate resources
very efficiently. Aegis’s protected control transfer is almost seven
times faster than the best reported implementation [33]. Aegis’s
exception dispatch is five times faster than the best reported imple-
mentation [50]. On identical hardware, Aegis’s exception dispatch
and control transfer are roughly two orders of magnitude faster than
in Ultrix 4.2, a mature monolithic system.
Aegis also gives ExOS (and other application-level software)
flexibility that is not available in microkemel-based systems. For
instance, virtual memory is implemented at application level, where
it can be tightly integrated with distributed shared memory systems
and garbage collectors. Aegis’s efficient protected control trans-
fer allows applications to construct a wide array of efficient IPC
primitives by trading performance for additional functionality. In
contrast, microkemel systems such as Amoeba [48], Chorus [43],
Mach [2], and V[15] do not allow untrusted application software to
define specialized IPC primitives because virtual memory and mes-
sage passing services are implemented by the kernel and trusted
servers. Similarly, many other abstractions, such as page-table
structures and process abstractions, cannot be modified in micro-
kemels. Finally, many of the hardware resources in microker-
nel systems, such as the network, screen, and disk, are encapsu-
lated in heavyweight servers that cannot be bypassed or tailored
to application-specific needs. These heavyweight servers can be
viewed as fixed kernel subsystems that run in user-space.
This paper focuses on the exokemel architecture design and how
it can be implemented securely and efficiently. Section 2 provides
a more detailed case for exokemels. Section 3 discusses the issues
that arise in their design. Section 4 overviews the status of our proto-
type and explains our experimental methodology. Sections 5 and 6
present the implementation and summarize performance measure-
ments of Aegis and ExOS. Section 7 reports on experiments that
demonstrate the flexibility of the exokemel architecture. Section 8
summarizes related work and Section 9 concludes.
2 Motivation for Exokernels
Traditionally, operating systems have centralized resource manage-
ment via a set of abstractions that cannot be specialized, extended,
or replaced. Whether provided by the kernel or by trusted user-
level servers (as in microkemel-based systems), these abstractions
are implemented by privileged software that must be used by all
applications, and therefore cannot be changed by untrusted soft-
ware. Typically, the abstractions include processes, files, address
spaces, and interprocess communication. In this section we discuss
the problems with general-purpose implementations of these ab-
stractions and show how the exokernel architecture addresses these
problems.
2.1 The Cost of Fixed High-Level Abstractions
The essential observation about abstractions in traditional operating
systems is that they are overly general. Traditional operating sys-
tems attempt to provide all the features needed by all applications.
As previously noted by Lampson and Sproul [32], Anderson et
al. [4] and Massalin and PO [36], general-purpose implementations
of abstractions force applications that do not need a given feature to
pay substantial overhead costs. This longstanding problem has be-
come more important with explosive improvements in raw hardware
performance and enormous growth in diversity of the application
software base. We argue that preventing the modification of the
implementation of these high-level abstractions can reduce the per-
formance, increase the complexity, and limit the functionality of
application programs.
Fixed high-level abstractions hurt application performance be-
cause there is no single way to abstract physical resources or to
implement an abstraction that is best for all applications. In im-
plementing an abstraction, an operating system is forced to make
trade-offs between support for sparse or dense address spaces,
read-intensive or write-intensive workloads, etc. Any such trade-
off penalizes some class of applications. For example, relational
databases and garbage collectors sometimes have very predictable
data access patterns, and their performance suffers when a general-
purpose page replacement strategy such as LRU is imposed by
the operating system. The performance improvements of such
application-specific policies can be substantial; Cao et al. [1 O] mea-
sured that application-controlled file caching can reduce application
running time by as much as 45%.
Fixed high-level abstractions hide information from applica-
tions. For instance, most current systems do not make low-level
exceptions, timer interrupts, or raw device I/O directly available to
application-level software. Unfortunately, hiding this information
makes it difficult or impossible for applications to implement their
own resource management abstractions. For example, database im-
plementors must struggle to emulate random-access record storage
on top of tile systems [47]. As another example, implementing
lightweight threads on top of heavyweight processes usually re-
quires compromises in correctness and performance, because the
operating system hides page faults and timer interrupts [4]. In such
cases, application complexity increases because of the difficuky of
getting good performance from high-level abstractions.
Fixed high-level abstractions limit the jimctionality of appli-
cations, because they are the only available interface between ap-
plications and hardware resources. Because all applications must
share one set of abstractions, changes to these abstractions occur
rarely, if ever. This may explain why few good ideas from the
last decade of operating systems research have been adopted into
widespread use: how many production operating systems support
scheduler activations [4], multiple protection domains within a sin-
gle address space [11 ], efficient IPC [33], or efficient and flexible ‘
virtual memory primitives [5, 26, 30]?
2.2 Exokernels: An End-to-End Argument
The familiar “end-to-end” argument applies as well to low-level
operating system software as it does to low-level communication
protocols [44], Applications know better than operating systems
what the goal of their resource management decisions shordd be
and therefore, they should be given as much control as possible over
252
protocol. Third, by using an abortprotocol, an exokernel can break
secure bindings of uncooperative applications by force.
We have implemented a prototype exokemel system based on
secure bindings, visible revocation, and abort protocols. It includes
an exokemel (Aegis) and an untrusted library operating system
(ExOS). We use this system to demonstrate several important prop-
erties of the exokemel architecture: (1) exokemels can be made
efficient due to the limited number of simple primitives they must
provide; (2) low-level secure multiplexing of hardware resources
can be provided with low overhead; (3) traditional abstractions,
such as VM and IPC, can be implemented efficiently at application
level, where they can be easily extended, specialized, or replaced;
and (4) applications can create special-purpose implementations of
abstractions, tailored to their functionality and performance needs.
In practice, our prototype exokernel system provides applica-
tions with greater flexibility and better performance than mono-
lithic and microkemel systems. Aegis’s low-level interface allows
application-level software such as ExOS to manipulate resources
very efficiently. Aegis’s protected control transfer is almost seven
times faster than the best reported implementation [33]. Aegis’s
exception dispatch is five times faster than the best reported imple-
mentation [50]. On identical hardware, Aegis’s exception dispatch
and control transfer are roughly two orders of magnitude faster than
in Ultrix 4.2, a mature monolithic system.
Aegis also gives ExOS (and other application-level software)
flexibility that is not available in microkemel-based systems. For
instance, virtual memory is implemented at application level, where
it can be tightly integrated with distributed shared memory systems
and garbage collectors. Aegis’s efficient protected control trans-
fer allows applications to construct a wide array of efficient IPC
primitives by trading performance for additional functionality. In
contrast, microkemel systems such as Amoeba [48], Chorus [43],
Mach [2], and V[15] do not allow untrusted application software to
define specialized IPC primitives because virtual memory and mes-
sage passing services are implemented by the kernel and trusted
servers. Similarly, many other abstractions, such as page-table
structures and process abstractions, cannot be modified in micro-
kemels. Finally, many of the hardware resources in microker-
nel systems, such as the network, screen, and disk, are encapsu-
lated in heavyweight servers that cannot be bypassed or tailored
to application-specific needs. These heavyweight servers can be
viewed as fixed kernel subsystems that run in user-space.
This paper focuses on the exokemel architecture design and how
it can be implemented securely and efficiently. Section 2 provides
a more detailed case for exokemels. Section 3 discusses the issues
that arise in their design. Section 4 overviews the status of our proto-
type and explains our experimental methodology. Sections 5 and 6
present the implementation and summarize performance measure-
ments of Aegis and ExOS. Section 7 reports on experiments that
demonstrate the flexibility of the exokemel architecture. Section 8
summarizes related work and Section 9 concludes.
2 Motivation for Exokernels
Traditionally, operating systems have centralized resource manage-
ment via a set of abstractions that cannot be specialized, extended,
or replaced. Whether provided by the kernel or by trusted user-
level servers (as in microkemel-based systems), these abstractions
are implemented by privileged software that must be used by all
applications, and therefore cannot be changed by untrusted soft-
ware. Typically, the abstractions include processes, files, address
spaces, and interprocess communication. In this section we discuss
the problems with general-purpose implementations of these ab-
stractions and show how the exokernel architecture addresses these
problems.
2.1 The Cost of Fixed High-Level Abstractions
The essential observation about abstractions in traditional operating
systems is that they are overly general. Traditional operating sys-
tems attempt to provide all the features needed by all applications.
As previously noted by Lampson and Sproul [32], Anderson et
al. source revocation, applications participate in a resource revocation
protocol. Third, by using an abortprotocol, an exokernel can break
secure bindings of uncooperative applications by force.
We have implemented a prototype exokemel system based on
secure bindings, visible revocation, and abort protocols. It includes
an exokemel (Aegis) and an untrusted library operating system
(ExOS). We use this system to demonstrate several important prop-
erties of the exokemel architecture: (1) exokemels can be made
efficient due to the limited number of simple primitives they must
provide; (2) low-level secure multiplexing of hardware resources
can be provided with low overhead; (3) traditional abstractions,
such as VM and IPC, can be implemented efficiently at application
level, where they can be easily extended, specialized, or replaced;
and (4) applications can create special-purpose implementations of
abstractions, tailored to their functionality and performance needs.
In practice, our prototype exokernel system provides applica-
tions with greater flexibility and better performance than mono-
lithic and microkemel systems. Aegis’s low-level interface allows
application-level software such as ExOS to manipulate resources
very efficiently. Aegis’s protected control transfer is almost seven
times faster than the best reported implementation [33]. Aegis’s
exception dispatch is five times faster than the best reported imple-
mentation [50]. On identical hardware, Aegis’s exception dispatch
and control transfer are roughly two orders of magnitude faster than
in Ultrix 4.2, a mature monolithic system.
Aegis also gives ExOS (and other application-level software)
flexibility that is not available in microkemel-based systems. For
instance, virtual memory is implemented at application level, where
it can be tightly integrated with distributed shared memory systems
and garbage collectors. Aegis’s efficient protected control trans-
fer allows applications to construct a wide array of efficient IPC
primitives by trading performance for additional functionality. In
contrast, microkemel systems such as Amoeba [48], Chorus [43],
Mach [2], and V[15] do not allow untrusted application software to
define specialized IPC primitives because virtual memory and mes-
sage passing services are implemented by the kernel and trusted
servers. Similarly, many other abstractions, such as page-table
structures and process abstractions, cannot be modified in micro-
kemels. Finally, many of the hardware resources in microker-
nel systems, such as the network, screen, and disk, are encapsu-
lated in heavyweight servers that cannot be bypassed or tailored
to application-specific needs. These heavyweight servers can be
viewed as fixed kernel subsystems that run in user-space.
This paper focuses on the exokemel architecture design and how
it can be implemented securely and efficiently. Section 2 provides
a more detailed case for exokemels. Section 3 discusses the issues
that arise in their design. Section 4 overviews the status of our proto-
type and explains our experimental methodology. Sections 5 and 6
present the implementation and summarize performance measure-
ments of Aegis and ExOS. Section 7 reports on experiments that
demonstrate the flexibility of the exokemel architecture. Section 8
summarizes related work and Section 9 concludes.
2 Motivation for Exokernels
Traditionally, operating systems have centralized resource manage-
ment via a set of abstractions that cannot be specialized, extended,
or replaced. Whether provided by the kernel or by trusted user-
level servers (as in microkemel-based systems), these abstractions
are implemented by privileged software that must be used by all
applications, and therefore cannot be changed by untrusted soft-
ware. Typically, the abstractions include processes, files, address
spaces, and interprocess communication. In this section we discuss
the problems with general-purpose implementations of these ab-
stractions and show how the exokernel architecture addresses these
problems.
2.1 The Cost of Fixed High-Level Abstractions
The essential observation about abstractions in traditional operating
systems is that they are overly general. Traditional operating sys-
tems attempt to provide all the features needed by all applications.
As previously noted by Lampson and Sproul [32], Anderson et
al. [4] and Massalin and PO [36], general-purpose implementations
of abstractions force applications that do not need a given feature to
pay substantial overhead costs. This longstanding problem has be-
come more important with explosive improvements in raw hardware
performance and enormous growth in diversity of the application
software base. We argue that preventing the modification of the
implementation of these high-level abstractions can reduce the per-
formance, increase the complexity, and limit the functionality of
application programs.
Fixed high-level abstractions hurt application performance be-
cause there is no single way to abstract physical resources or to
implement an abstraction that is best for all applications. In im-
plementing an abstraction, an operating system is forced to make
trade-offs between support for sparse or dense address spaces,
read-intensive or write-intensive workloads, etc. Any such trade-
off penalizes some class of applications. For example, relational
databases and garbage collectors sometimes have very predictable
data access patterns, and their performance suffers when a general-
purpose page replacement strategy such as LRU is imposed by
the operating system. The performance improvements of such
application-specific policies can be substantial; Cao et al. [1 O] mea-
sured that application-controlled file caching can reduce application
running time by as much as 45%.
Fixed high-level abstractions hide information from applica-
tions. For instance, most current systems do not make low-level
exceptions, timer interrupts, or raw device I/O directly available to
application-level software. Unfortunately, hiding this information
makes it difficult or impossible for applications to implement their
own resource management abstractions. For example, database im-
plementors must struggle to emulate random-access record storage
on top of tile systems [47]. As another example, implementing
lightweight threads on top of heavyweight processes usually re-
quires compromises in correctness and performance, because the
operating system hides page faults and timer interrupts [4]. In such
cases, application complexity increases because of the difficuky of
getting good performance from high-level abstractions.
Fixed high-level abstractions limit the jimctionality of appli-
cations, because they are the only available interface between ap-
plications and hardware resources. Because all applications must
share one set of abstractions, changes to these abstractions occur
rarely, if ever. This may explain why few good ideas from the
last decade of operating systems research have been adopted into
widespread use: how many production operating systems support
scheduler activations [4], multiple protection domains within a sin-
gle address space [11 ], efficient IPC [33], or efficient and flexible ‘
virtual memory primitives [5, 26, 30]?
2.2 Exokernels: An End-to-End Argument
The familiar “end-to-end” argument applies as well to low-level
operating system software as it does to low-level communication
protocols [44], Applications know better than operating systems
what the goal of their resource management decisions shordd be
and therefore, they should be given as much control as possible over
252
A ‘pplicatiOnsA/ Www \ I DSM \
Figure 1: An example exokemel-based system consisting of a thin
exokemel veneer that exports resources to library operating systems
through secure bindings. Each library operating system implements
its own system objects and policies. Applications link against stan-
dard libraries (e.g., WWW, POSIX, and TCP libraries for Web ap-
plications) or against specialized libraries (e.g., a distributed shared
memory library for parallel applications).
those decisions. Our solution is to allow traditional abstractions to
be implemented entirely at application level.
To provide the maximum opportunity for application-level re-
source management, the exokernel architecture consists of a thin
exokemel veneer that multiplexes and exports physical resources
securely through a set of low-level primitives. Library operating
systems, which use the low-level exokemel interface, implement
higher-level abstractions and can define special-purpose implemen-
tations that best meet the performance and functionality goals of
applications (see Figure 1). (For brevity, we sometimes refer to
“library operating system” as “application.”) This structure allows
the extension, specialization and even replacement of abstractions.
For instance, page-table structures can vary among library operat-
ing systems: an application can select a library with a particular
implementation of a page table that is most suitable to its needs.
To A ‘pplicatiOnsA/ Www \ I DSM \
Figure 1: An example exokemel-based system consisting of a thin
exokemel veneer that exports resources to library operating systems
through secure bindings. Each library operating system implements
its own system objects and policies. Applications link against stan-
dard libraries (e.g., WWW, POSIX, and TCP libraries for Web ap-
plications) or against specialized libraries (e.g., a distributed shared
memory library for parallel applications).
those decisions. Our solution is to allow traditional abstractions to
be implemented entirely at application level.
To provide the maximum opportunity for application-level re-
source management, the exokernel architecture consists of a thin
exokemel veneer that multiplexes and exports physical resources
securely through a set of low-level primitives. Library operating
systems, which use the low-level exokemel interface, implement
higher-level abstractions and can define special-purpose implemen-
tations that best meet the performance and functionality goals of
applications (see Figure 1). (For brevity, we sometimes refer to
“library operating system” as “application.”) This structure allows
the extension, specialization and even replacement of abstractions.
For instance, page-table structures can vary among library operat-
ing systems: an application can select a library with a particular
implementation of a page table that is most suitable to its needs.
To the best of our knowledge, no other secure operating system
architecture allows applications so much useful freedom.
This paper demonstrates that the exokemel architecture is an ef-
fective way to address the problems listed in Section 2.1. Many of
these problems are solved by simply moving the implementation of
abstractions to application level, since conflicts between application
needs and available abstractions can then be resolved without the
intervention of kernel architects. Furthermore, secure multiplexing
does not require complex algorithms; it mostly requires tables to
track ownership. Therefore, the implementation of an exokernel can
be simple. A simple kernel improves reliability and ease of main-
tenance, consumes few resources, and enables quick adaptation to
new requirements (e.g., gigabit networking). Additionally, as is
true with RISC instructions, the simplicity of exokemel operations
allows them to be implemented efficiently.
2.3 Library Operating Systems
The implementations of abstractions in library operating systems
can be simpler and more specialized than in-kernel implementa-
tions, because library operating systems need not multiplex a re-
source among competing applications with widely different de-
mands. In addition, since libraries are not trusted by an exokemel,
they are free to trust the application. For example, if an application
passes the wrong arguments to a library, only that application will
be affected. Finally, the number of kernel crossings in an exokemel
system can be smaller, since most of the operating system runs in
the address space of the application.
Library operating systems can provide as much portability and
compatibility as is desirable. Applications that use an exokemel
interface directly will not be portable, because the interface will in-
clude hardware-specific information. Applications that use library
operating systems that implement standard interfaces (e.g., POSIX)
will be portable across any system that provides the same interface.
An application that runs on an exokemel can freely replace these
library operating systems without any special privileges, which sim-
plifies the addition and development of new standards and features.
We expect that most applications will use a handful of available
library operating systems that implement the popular interfaces;
only designers of more ambitious applications will develop new
library operating systems that fit their needs. Library operating
systems themselves can be made portable by designing them to use
a low-level machine-independent layer to hide hardware details.
Extending or specializing a library operating system might be
considerably simplified by modular design. It is possible that object-
oriented programming methods, overloading, and inheritance can
provide useful operating system service implementations that can
be easily specialized and extended, as in the VM++ library [30].
To reduce the space required by these libraries, support for shared
libraries and dynamic linking will bean essential part of a complete
exokemel-based system.
As in microkemel systems, an exokemel can provide backward
compatibility in three ways: one, binary emulation of the operating
system and its programs; two, by implementing its hardware ab-
straction layer on top of an exokemel; and three, re-implementing
the operating system’s abstractions on top of an exokemel.
3 Exokernel Design
The challenge for an exokemel is to give library operating systems
maximum freedom in managing physical resources while protecting
them from each othen a programming error in one library operat-
ing system should not affect another library operating system. To
achieve this goal, an exokemel separates protection from manage-
ment through a low-level interface.
In separating protection from management, an exokemel per-
forms three important tasks: (1) tracking ownership of resources,
(2) ensuring protection by guarding all resource usage or binding
points, and (3) revoking access to resources. To achieve these
tasks, an exokemel employs three techniques. First, using secure
bindings, library operating systems can securely bind to machine
resources. Second, visible revocation allows library operating sys-
tems to participate in a resource revocation protocol. Third, an
abort protocol is used by an exokemel to break secure bindings of
uncooperative library operating systems by force.
In this section, we enumerate the central design principles of
the exokemel architecture. Then, we discuss in detail the three
techniques that we use to separate protection from management.
3.1 Design Principles
An exokemel specifies the details of the interface that library op-
erating systems use to claim, release, and use machine resources.
This section articulates some of the principles that have guided
our efforts to design an exokemel interface that provides library
operating systems the maximum d%= of control.
Securely expose hardware. The central tenet of the exoker-
nel architecture is that the kernel should provide secure low-level
253
Figure 1: An example exokemel-based system consisting of a thin
exokemel veneer that exports resources to library operating systems
through secure bindings. Each library operating system implements
its own system objects and policies. Applications link against stan-
dard libraries (e.g., WWW, POSIX, and TCP libraries for Web ap-
plications) or against specialized libraries (e.g., a distributed shared
memory library for parallel applications).
those decisions. Our solution is to allow traditional abstractions to
be implemented entirely at application level.
To provide the maximum opportunity for application-level re-
source management, the exokernel architecture consists of a thin
exokemel veneer that multiplexes and exports physical resources
securely through a set of low-level primitives. Library operating
systems, which use the low-level exokemel interface, implement
higher-level abstractions and can define special-purpose implemen-
tations that best meet the performance and functionality goals of
applications (see Figure 1). (For brevity, we sometimes refer to
“library operating system” as “application.”) This structure allows
the extension, specialization and even replacement of abstractions.
For instance, page-table structures can vary among library operat-
ing systems: an application can select a library with a particular
implementation of a page table that is most suitable to its needs.
To A ‘pplicatiOnsA/ Www \ I DSM \
Figure 1: An example exokemel-based system consisting of a thin
exokemel veneer that exports resources to library operating systems
through secure bindings. Each library operating system implements
its own system objects and policies. Applications link against stan-
dard libraries (e.g., WWW, POSIX, and TCP libraries for Web ap-
plications) or against specialized libraries (e.g., a distributed shared
memory library for parallel applications).
those decisions. Our solution is to allow traditional abstractions to
be implemented entirely at application level.
To provide the maximum opportunity for application-level re-
source management, the exokernel architecture consists of a thin
exokemel veneer that multiplexes and exports physical resources
securely through a set of low-level primitives. Library operating
systems, which use the low-level exokemel interface, implement
higher-level abstractions and can define special-purpose implemen-
tations that best meet the performance and functionality goals of
applications (see Figure 1). (For brevity, we sometimes refer to
“library operating system” as “application.”) This structure allows
the extension, specialization and even replacement of abstractions.
For instance, page-table structures can vary among library operat-
ing systems: an application can select a library with a particular
implementation of a page table that is most suitable to its needs.
To the best of our knowledge, no other secure operating system
architecture allows applications so much useful freedom.
This paper demonstrates that the exokemel architecture is an ef-
fective way to address the problems listed in Section 2.1. Many of
these problems are solved by simply moving the implementation of
abstractions to application level, since conflicts between application
needs and available abstractions can then be resolved without the
intervention of kernel architects. Furthermore, secure multiplexing
does not require complex algorithms; it mostly requires tables to
track ownership. Therefore, the implementation of an exokernel can
be simple. A simple kernel improves reliability and ease of main-
tenance, consumes few resources, and enables quick adaptation to
new requirements (e.g., gigabit networking). Additionally, as is
true with RISC instructions, the simplicity of exokemel operations
allows them to be implemented efficiently.
2.3 Library Operating Systems
The implementations of abstractions in library operating systems
can be simpler and more specialized than in-kernel implementa-
tions, because library operating systems need not multiplex a re-
source among competing applications with widely different de-
mands. In addition, since libraries are not trusted by an exokemel,
they are free to trust the application. For example, if an application
passes the wrong arguments to a library, only that application will
be affected. Finally, the number of kernel crossings in an exokemel
system can be smaller, since most of the operating system runs in
the address space of the application.
Library operating systems can provide as much portability and
compatibility as is desirable. Applications that use an exokemel
interface directly will not be portable, because the interface will in-
clude hardware-specific information. Applications that use library
operating systems that implement standard interfaces (e.g., POSIX)
will be portable across any system that provides the same interface.
An application that runs on an exokemel can freely replace these
library operating systems without any special privileges, which sim-
plifies the addition and development of new standards and features.
We expect that most applications will use a handful of available
library operating systems that implement the popular interfaces;
only designers of more ambitious applications will develop new
library operating systems that fit their needs. Library operating
systems themselves can be made portable by designing them to use
a low-level machine-independent layer to hide hardware details.
Extending or specializing a library operating system might be
considerably simplified by modular design. It is possible that object-
oriented programming methods, overloading, and inheritance can
provide useful operating system service implementations that can
be easily specialized and extended, as in the VM++ library [30].
To reduce the space required by these libraries, support for shared
libraries and dynamic linking will bean essential part of a complete
exokemel-based system.
As in microkemel systems, an exokemel can provide backward
compatibility in three ways: one, binary emulation of the operating
system and its programs; two, by implementing its hardware ab-
straction layer on top of an exokemel; and three, re-implementing
the operating system’s abstractions on top of an exokemel.
3 Exokernel Design
The challenge for an exokemel is to give library operating systems
maximum freedom in managing physical resources while protecting
them from each othen a programming error in one library operat-
ing system should not affect another library operating system. To
achieve this goal, an exokemel separates protection from manage-
ment through a low-level interface.
In separating protection from management, an exokemel per-
forms three important tasks: (1) tracking ownership of resources,
(2) ensuring protection by guarding all resource usage or binding
points, and (3) revoking access to resources. To achieve these
tasks, an exokemel employs three techniques. First, using secure
bindings, library operating systems can securely bind to machine
resources. Second, visible revocation allows library operating sys-
tems to participate in a resource revocation protocol. Third, an
abort protocol is used by an exokemel to break secure bindings of
uncooperative library operating systems by force.
In this section, we enumerate the central design principles of
the exokemel architecture. Then, we discuss in detail the three
techniques that we use to separate protection from management.
3.1 Design Principles
An exokemel specifies the details of the interface that library op-
erating systems use to claim, release, and use machine resources.
This section articulates some of the principles that have guided
our efforts to design an exokemel interface that provides library
operating systems the maximum d%= of control.
Securely expose hardware. The central tenet of the exoker-
nel architecture is that the kernel should provide secure low-level
253
