A Survey of Practical Formal Methods for Security

The amount of research publications within the area of applying FM towards cyber security challenges is significant. Therefore, several constraints have been placed on the choice of research publications to be considered within this survey. The first important constraint is the recency of the research reported, considering the landscape of the past decade, limiting the publication date to be no earlier than 2012. Furthermore, all of the research work needs to be published in scientific venues such as journals, conferences, or workshops. The next constraint is focus on computer-based tool supported FM, i.e., only FM with tools that can provide computer-based analysis and often guide users on performing this analysis are considered. This consideration is to focus more on the FM that could be potentially applied outside of academia, bringing the benefits of the formal security analysis to industry. This goes hand-in-hand with our focus on the applied FM, searching for research publications, where a tool-supported FM is utilised to deal with a concrete cyber security problem. Hence, this survey does not focus on theoretical advances of FM in security or proposed processes that briefly mention use of FM, such as theoretical approaches to model checking algorithms, specification of hyperproperties, and similar. Furthermore, our survey does not cover the approach to security commonly referred to as provable security. This refers to a mathematical approach to analysing the security of cryptographic mechanisms or systems. The approach considers the system in the context of an attacker model and expresses the security requirements within that model as a limitation on what the attacker should be able to achieve. A proof consists of establishing that the attacker would need to break a known hard problem (such as the Quadratic Residuosity Problem [105]) to break the security of the system. Thus, the security of the system is reduced to the difficulty of the underpinning hard problem. This approach is typically used within the field of cryptography rather than secure systems, and so falls outside the scope of our survey. We point the reader to Reference [28] providing the report within the area of FM in cryptography. Finally, we constrain our search to research that considers aspects of security explicitly, and not as a by-product of safety or correctness. The search for the research publications was carried out as a cross database search using Google Scholar, while focusing on research papers, excluding research abstracts or extended abstracts.

As this survey provides readers with a quick overview of the research conducted, we have further decided to categorise different research publications by the industry (domain) on which they focus as well as the level of abstraction on which FM is utilised. In this way, researchers and also potentially industrial users can quickly find the area of their interest within this survey. Furthermore, we classify the research based on the cyber security problem classification as elicited from the discovered research papers and inspired by existing literature [208].

This section presents a history of impactful research works within FM in security over the past 40 years. We choose four case studies where formal methods have been applied to secure systems:

The Needham-Schroeder Public-key Protocol. The Needham-Schroeder Public-Key Protocol is a transport-level protocol for communication between network devices [190], providing mutual authentication between two parties in a network. The protocol is visualised in Figure 1(a). Simple and well known, it has become a popular benchmark for testing security protocol verification technology. We discuss it here because it is an important security protocol that nevertheless contained a significant error. This error was found through formal modelling and analysis.

Lowe [165] showed that, contrary to its intention, the protocol fails to ensure authentication. In particular, he demonstrated that an intruder can impersonate an agent A during a run of the protocol. The impersonator tricks another agent B into thinking that they are talking to A.

The protocol uses public key cryptography. Each agent A possesses a public key, which any other agent can get from a server. A also possesses a secret key that is the inverse of its public key. Any agent can encrypt a message using A’s public key, but only A can decrypt it, ensuring secrecy. The protocol also uses nonces: random numbers coined for single runs of the protocol.

Lowe encoded the protocol in CSP [123] and analysed it by use of CSP’s model checker, FDR [103]. Lowe did not direct FDR where to look in the protocol for a vulnerability—he simply modelled the intruder capabilities, but the exhaustive search carried out by the model checker found an attack in spite of this.

Suppose that I (an intruder) is a network user who can take part in network sessions. I can also intercept messages and inject new ones, but is not able to decrypt messages without the key. I can produce a new message in two circumstances: if I invents the nonce or if I already understands the message’s contents. This intruder can also replay complete encrypted messages, even without understanding the contents [213]. This approach is commonly known as the Dolev-Yao model [85].

The attack involves two simultaneous runs of the protocol, as shown in Figure 1(b). A establishes a valid session with I. At the same time, I impersonates A to establish a fake session with B. The flawed run of the protocol could be explained as follows: A sends a message with nonce $N_A$to I, who decrypts the message with I’s secret key. I relays the message to B, pretending to B that A is communicating. B sends $N_B$in response, encrypted for A, and so I relays this encrypted nonce to A. A decrypts $N_B$and confirms it to I, who learns it. I re-encrypts $N_B$and returns it to B, which convinces B that A is the other party. At the end of the attack, B falsely believes that A is the communication partner and that only A and B know $N_A$and $N_B$. This shows that the protocol is insecure. Protocol analysts call this a man-in-the-middle attack. Here, it has been discovered automatically.

Mondex. The Mondex application consists of smart cards with electronic purses (wallets) for use in the electronic commerce [235]. Customers use Mondex smart cards for low-value, cash-like transactions that need no third-party involvement. The Bank of England (the financial regulator in this instance) considered the requirements for Mondex to be security-critical: Mondex must have no implementation or design bugs that could allow electronic counterfeiting. So the developers certified Mondex to the highest standard available at the time. This was ITSEC Level E6 [129], equal to Common Criteria Evaluation Assurance Level 7 [57].¹ Mondex was the first commercial product to achieve ITSEC Level E6 (EAL7).

Reference [235] further describes the development of the Mondex application, with its abstract and concrete models. The abstract model describes the world of electronic purses: Atomic transactions exchange value and the abstract model expresses their required security properties. The concrete model is the purse design and the message protocol for value exchange.

The design team used the Z notation [233, 263] to specify both models. They proved that the concrete model is formally a refinement of the abstract one. This means that the concrete model respects all the abstract security requirements. The abstract model and its security properties is often easier to understand than the concrete model. Developers wrote manual proofs, believing that no efficient automated tools existed for such a large task. Instead, proof steps were type-checked using the fuzz ² and Formaliser tools [97]. Proofs were also checked by independent external evaluators.

There were four principal security properties:

The design team changed a secondary protocol after the proof revealed a bug. A detailed account of the project is given in Reference [265]. Mondex has proved to be a dependably secure system, guaranteed by its formal development.

Tokeneer. The Tokeneer system was designed and developed by the US National Security Agency (NSA) [30]. ³ It provides secure access to an enclave of workstations with controlled physical entry. Access control requires biometric checks and security tokens. These tokens describe a user’s permitted actions within a particular visit to the enclave.

Developers needed to assure the security properties. They did this by conformance with the Common Criteria Evaluation Assurance Level 5 [57]. They also needed to show they could do this in a cost-effective way. NSA invited bids to use FM to develop a component of the Tokeneer system and then monitored this experiment to measure the effort and skills needed to perform the development.

Praxis (a UK company) won the contract and wrote a formal specification in Z [233, 263], formally refining the specification to a SPARK program. SPARK is a subset of Ada with an accompanying tool-set [29]. They proved key system properties and the absence of runtime errors, using traditional methods to develop extra software. These extra Ada programs provided interfaces with peripherals.

The project required 260 person-days, three people part-time, and nine months’ elapsed time. It produced about 10K lines of SPARK code with about 16.5K contracts. About 200 loc were written on average per day during the implementation phase, with about 40 loc through the entire project. A further 3.5K lines of standard Ada code were produced, with about 200 loc per day in the implementation phase or 90 loc throughout the project. System testing took about 4% of the project effort, much smaller than usual.

Two defects were found in Tokeneer. One was found using formal analysis, another was found by code inspection.⁴ The testing team discovered two in-scope failures: missing items in the user manual.

The task set by NSA was to conform to Common Criteria EAL5. The Tokeneer development actually exceeded EAL5 requirements in several areas: configuration control, fault management, and testing. Although the main body of the core development work was carried out to EAL5, the development areas covering specification, design, implementation, correspondence were accomplished to EAL6 and EAL7. Why? Because it was cheaper!

The seL4 Microkernel. The third-generation microkernel seL4 provides abstractions for virtual address spaces, threads, and inter-process communication. It provides an explicit memory management model and capabilities for authorisation. There is a guarantee that the binary code of the ARM version of the seL4 microkernel is a correct implementation.⁵ seL4 meets its abstract specification and does nothing else. In particular, the seL4 ARM binary meets the classic security properties of integrity and confidentiality.

The seL4 micro-kernel has a formal proof of its C code against its abstract specification [144]. This proof is machine-checked in Isabelle/HOL [194]. This assumes correctness of boot code, cache management, hardware, and hand-written assembly code.

The developers claim seL4 to be the only verified general-purpose operating system (OS) kernel. An operational model of the system forms as an abstract specification. A Haskell program prototypes the kernel. This prototype provides an automatic translation into Isabelle/HOL. The Isabelle code is then an executable, design-level specification of the kernel. This is hand-coded in C to form a high-performance C implementation of seL4. Refinement proofs link the specifications and the C code. Developers proved that attackers cannot subvert the kernel, not even if they use buggy encodings, spurious calls, or buffer overflow attacks.

Here, we provide a set of common terms and definitions used throughout this article. This is particularly important, since the fields of formal verification and security developed independently for many years, and hence, some terms are overloaded and have slightly different meanings, depending on the context in which they are used. For example, in security (particularly cryptography), a certificate refers to a document that is used to bind an entity to a cryptographic key. However, within FM, certificate is used as a proof of correctness of a system or protocol.

Throughout this survey, on the security side, we use terms such as authentication to refer to the process of identifying and validating whether a user (an entity or individual) accessing a system is who the user claims to be. This is in contrast with authorisation, which is the process of allowing a user access to a system based on their identity. We furthermore often find security protocols designed to provide specific properties such as isolation, which is a design principle in which processes are separated and given privileged access to shared resources, e.g., shared memory (typically using techniques such as containerisation or virtualisation) or non-repudiation, ensuring that it is not possible to deny a validity of a statement or a message, especially in terms of its authorship. We would also like to point out the difference between anonymity where an identity of a user or a process shall not be disclosed and confidentiality where the content encoded as a block of data shall not be disclosed.

On the formal methods side it could be said that systems under consideration typically comprise a set of coordinated processes, which are program instances defining a set of instructions that are executed by one or more threads. We think of processes as being active entities in a system, as opposed to programs that are passive entities. Formal frameworks to describe the behaviour of processes include CSP, CCS, ACP, $\pi$-calculus, and so on. Processes typically implement protocols, i.e., a set of rules for transmission of data, and may synchronise over shared memory or communicate over a channel, which is an abstraction of a physical communication network. Shared memory implementations are increasingly complex due to the use of intermediate processor caches and may implement many different consistency models [6]. Similarly, one may place many different assumptions on a channel, e.g., FIFO ordering of messages; whether the channel guarantees integrity, availability, and confidentiality; whether the channel is error-free; whether message types can be distinguished, and so on.

In general, verification is with respect to a specification, which is an abstract (formal or informal) description of the allowable behaviours of an entity, e.g., hardware, a system, computer program, data structure, and so on. Formal verification often proceeds with respect to a model of a system, which provides a precise formal description of an entity, capturing the key characteristics of the entity being modelled. One must ensure that every feature described by a model is an actual feature of the entity. Different models of the same entity may be developed, depending on the properties that are of interest; a computer program for example may be modelled by relations between pre/post state; traces of states; functions between inputs and outputs, and so on. A model may describe behavioural functionality, protocols, and so on. In FM, one typically develops models at several levels of abstraction, with precise descriptions of the relationship between these levels. FM for security also requires a model of an attacker, e.g., the Dolev-Yao model, which is used in the context of communicating systems.

Within hardware verification, the term co-verification is used to prove that system software executes correctly on a representation of the underlying hardware design. It enables integration of software with hardware, before any physical devices (e.g., chips or boards) are available. The aim of verification is to ensure that it meets its implementation (of a specification), i.e., the physical manifestation of an entity. In some instances, one may refer to an implementation as a model that provides enough detail about an entity for the corresponding physical entity to be readily obtained.

To finalise the definitions it is important to revisit our classifications of FM. The first category we classify is theorem proving, where a proof of correctness of a system is provided in symbolic logic. This survey focuses on automated theorem proving, where proof assistants are utilised to generate the proofs. This method utilises a system of logic trying to determine if a statement $\varphi$follows from a set of statements $\mathrm{\Gamma }=\{{\psi }_1,\dots ,{\psi }_n\}$. The second class we consider is model checking where a finite state model of a system is utilised to systematically search the entire state space of this model (i.e., all possible system states contained in the model) for detection of counter-examples to statements about the system expressed as properties, for example in linear temporal logic as $◻\diamond p$stating that a property p holds in infinitely many states. As with theorem proving, we focus on automated model checking. Finally, we define a class of lightweight formal methods where we place methods that do not provide exhaustive analysis. A simple example could be static code analysis, where the code is analysed while not running to determine if it breaks predefined rules. Another example could be VDM [38], where the system is modelled in a formal modelling language and properties are expressed as contracts, i.e., pre, post conditions and system invariants, for example inv t == t.issueTime $<$t.expirationTime stating that universally t must be issued before it can expire. To perform the check, the model is animated and only specific scenarios are set (often those considered critical), however, VDM can also utilise combinatorial testing combining input paths to generate large amount of tests, hence significantly increasing the test coverage.

Since FM in security are applied across many domains, we structure the scope of the survey by presenting a categorisation based on a domain and a level of abstraction. This is done to provide a systematic overview of the wide field of FM in security. The labels for the four domains we have selected are:

As presenting the four domains would only separate the FM in security research by the field of application, we further present five levels of abstraction at which the formal verification is carried out. These levels of abstraction are:

This categorisation allows us to systematically review the state-of-the-art-research and provide an overview based on this. To provide a clear overview of FM in security, we further apply a third dimension, defining the type of the FM used, i.e., model checking, theorem proving, and lightweight FM. This provides a quantitative overview of different research works within Figure 2.

The sections within this survey follow a logical organisation, where the research works are grouped together by the type of the application, system, protocol, implementation, or hardware that they are applied to. Within this grouping, the research works are further organised into paragraphs following logical categorisation. As an example, the first paragraph could consider works related to manufacturing, while the next paragraph considers works related to industrial control. In both cases the research is aimed at industrial domain but with a different scope. Each section within the survey represents a single domain, where we present a systematic summary of research works belonging to different levels of abstraction. As this survey attempts to categorise large amount of research works, each of them is only presented briefly focusing on the analysed problem and utilised FM technology. For more details on individual research works, we point the reader to a technical report that this survey is based on [149].

Financial computing, including banking systems, independent budgeting applications, and mobile payment applications, is a rapidly developing field. This section provides an overview of how FM have been used to analyse the security of banking mobile applications, alternative currencies, such as cryptocurrencies, smart contracts, banking backend systems, electronic trading systems, payment protocols, cryptocurrency hardware, and wallets. On a different level of abstraction, it could be seen within this section that the system level consists of most research works. This section also mentions the legal challenges when applying FM to financial systems. These legal challenges arise from limited access to these systems and a certain level of avoidance of publication of potential vulnerabilities by vendors of these systems, i.e., often making it difficult for researchers to get deep insight into these systems.

The method most used to analyse security within the financial domain is model checking, and authors apply different model checkers to specific problems. This could be attributed to the fact that the different entities whose security is being analysed lend themselves well to be modelled in a state transition representation and also that their state space is sufficiently limited to be analysed without issues such as the state space explosion. The cyber security topics present within the financial section are shown in Figure 3.

Application. Nowadays, banks not only provide mobile applications, but whole alternative currencies are being developed. This rapid growth provides many opportunities for use of FM on an application level.

An application of FM to banking apps could be found in Reference [63], where the authors analyze security of apps from 15 leading UK banks discovering several vulnerabilities. The authors proposed a correction to one of the flaws, which was formally verified using ProVerif.

Another widespread security threat in banking is malware. In this area, the authors of Reference [152] analysed Android banking applications using Krakatau byte-code tool⁶ to generate the Java byte-code of the application, further translating it to Calculus of Communicating Systems (CCS) [181]. The authors then dispatched the CCS model to the Concurrency Workbench of New Century (CWB-NC) model checker [69] searching for malware properties. The authors have accomplished 98% malware detection rate. Similar approach was utilised by the authors of Reference [126] focusing on the banking SMS messages.

In the alternative currency area, the authors of Reference [249] analyse the Electrum Bitcoin wallet by creating a model in ASLan++ [253] of the two-factor authentication utilised by the wallet. The authors have uncovered potential vulnerabilities using the Cl-Atse protocol analyser [248].

Smart or programmatic contracts are also an important aspect of modern financial landscape. The automated contract enforcement requires an implementation that shall be free of vulnerabilities. This led to use of FM for creation of certified contract languages [17] and certified virtual machine byte-code [199].

A specific survey has been carried out in this area, providing more detail on utilisation of FM [180].

System. The financial systems of today could be categorised as classic systems such as SWIFT inter-bank network system and new systems often introduced by smaller players or by regulatory pressures [259] that tend to push the sector to move faster.

Most of the FM works related to classic systems are decades old with few works such as use of BAN logic for analysis of mobile payment system [7] or use of SPIN tool for analysis of ATM systems [197] or internet payment systems [272]. In Reference [216] the authors show legal barriers faced by proponents of FM in the domain.

Within the area of Electronic Trading Systems (ETSs) there is a trend of application of FM to decentralised systems such as blockchain-based cryptocurrencies. These works range from verifying algorithms in a cryptocurrency platform [268], analysis of Ethereum smart contracts [121], to verification of the blockchain system as a whole [87]. Other works have considered building models of Europay Master Visa standard by use of automata learning techniques [2] or use of lightweight formal specification in fraud detection [209].

Protocols. Protecting financial transactions is a work well suited for FM. Specifically in the case of Near Field Communication (NFC), a short-range radio communication technology, utilised in contactless payments. This method has several vulnerabilities [175], where the authors of Reference [168] propose and verify an NFC protocol suing the Scyther tool [73] addressing several vulnerabilities. In similar fashion, in Reference [5] the authors propose a protocol for securing of NFC payments and analyse it using the FDR model checker. Authors of Reference [11] have focused on analysis of security of NFC enabled forgery protection also utilising the FDR model checker, discovering several potential attacks and providing mitigating measures. In the field of authentication protocols the authors of Reference [266] verified the mutual authentication properties of a secure electronic protocol using SPIN tool, discovering several vulnerabilities. Similarly in Reference [118] the authors have analysed a biometric transaction authentication protocol using ProVerif [41] and proposed fixes to discovered vulnerabilities.

Finally, the authors of Reference [44] have proposed a secure SMS based protocol for mobile payments and analysed it against several security properties using AVISPA [21].

Implementation. Since vulnerabilities in financial software could lead to financial losses, use of FM can provide a substantial benefit in this area. Smartphone applications are often used as a gateway to financial services. The authors of Reference [127] utilised static analysis tools, discovering that financial applications from developed countries contain less vulnerabilities than those from developing countries. Similarly, the authors of Reference [241] have statically analysed over 10,000 Android applications to compare the security of financial applications with the rest, which led to a discovery of a worrisome trend where the analysed applications have gained more vulnerabilities within a span of two years.

The authors of Reference [98] have modelled 80% of EMV2, a successor to EMV, in VDM to provide a formal model for the implementation and analyse security attributes of EMV2. The authors have further attempted to code generate parts of EMV2 to Java directly from the VDM model.

Another aspect of financial software is use of open APIs. The authors of Reference [94] have modelled a financial grade OpenID API as a set of theorems, discovering several vulnerabilities and proposing fixes to these. Finally, the authors of Reference [16] have analysed bitcoin contracts using UPPAAL, determining the secure time to live within the contract protocol. Similarly, other types of smart contracts are being utilised [35]. The authors of Reference [199] have verified Ethereum smart contracts such as the ERC20 token contract [90] using K-framework’s reachability logic theorem prover [234], discovering that the token implementation that diverges form the ERC20 specification contains vulnerabilities.

Hardware. In the area of contactless payments, the NFC hardware can pose security challenges. To address one of these challenges, the authors of Reference [62] have introduced a scheme to prevent relay attacks based on a distance bounding protocol [46] and verify this scheme using ProVerif.

Cryptocurrency is often associated with a specific hardware, where in several cases FM were used for security improvement. To this end, the authors of Reference [22] have proposed a device for approval of security critical operations. The authors have verified a property of deterministic start of the device using an SMT solver. Similarly, the authors of Reference [171] have utilised theorem proving to check an unforgeability property of a hardware wallet to answer a question: what if the manufacturer of the wallet cannot be trusted? Finally, the authors of Reference [19] have also attempted to prove security properties of several hardware wallets using theorem proving, showing that the wallets were secure only under specific assumptions.

Industrial processes are a backbone of a modern society, as they provide control not only for production of necessary goods, but also utilities such as electricity and water treatment. This section provides an overview of how FM have been utilised in security analysis of automotive control applications, robotic applications, PLC software, industrial communication protocols such as Modbus and OPC UA, SCADA systems, and hardware devices underpinning industrial computing. An interesting note is that the research works are distributed uniformly across the different levels of abstraction, demonstrating that all aspects of industrial computing have been scrutinised using FM to provide either security analysis or security assurances. In the industrial application of FM the problem is often considered domain-specific, i.e., cyber security properties are based on whether the considered industrial system is, for example, an automotive controller or a water treatment plant.

As within the financial section, the most-used FM to analyse the security properties is model checking. This could once again be attributed to the nature of the problem where, for example, PLC programs and industrial processes lend themselves to be easily modelled using state transition systems. As some of the industrial computing is complex, the problems are often modelled more abstractly to avoid the state space explosion problem. Within the hardware level of abstraction, however, in industrial computing, theorem proving is often the FM of choice, as it allows description of the hardware in more detail. The cyber security topics within the industrial section are shown in Figure 4.

Application . Industrial applications are often used to control critical processes, hence FM could provide strong assurance of security. The authors of Reference [120] utilised model checking based on automated translation of automotive ECU applications to CSP [122] and subsequently dispatching the model to the FDR model checker. Discovered counter examples were then provided to the implementation team.

Industry of the future utilises interconnectivity and robotics. In Reference [193] the authors have analysed a security of an application controlling a cap attaching robot by creation of a model using Maude [179] and two attacker models, discovering possible attack vectors. Similarly, the authors of Reference [254] analysed security of applications based on the Robotic Operating System [205], expressing the security properties in CTL and utilising the UPPAAL model checker [159]. The authors then automatically generate the implementation C++ code.

The authors of Reference [177] have focused on security analysis of a water treatment SCADA system by use of the system logs. The application behaviour was modelled as timed automata, while the security properties have been expressed in timed temporal logic [12], providing 100% attack detection rate based on log data. The authors of Reference [273] have successfully utilised the Z3SMT solver [80] for PLC malware detection, while the authors of Reference [146] used the NuSMV model checker [65] for automated detection of intrusion code, demonstrating the usefulness of FM in this area.

System . Cyber attacks against industrial systems could have severe consequences [258]. To mitigate this, the authors of Reference [211] have created a formal model in ASLan++ of a real-world industrial control system and carried out attacks utilising a Cyber Physical Dolev-Yao attacker [210]. The analysis with help of CL-Atse analyser has discovered seven out of eight possible attacks. Similarly, the authors of Reference [82] mitigate the cyber attacks by proposing a formally verified security framework for industrial control systems. The verification was carried out using ProVerif and proved the security aspects of the system utilising the framework. Furthermore, the authors of Reference [114] verified a PLC program using timed automata and UPPAAL ensuring that the program was not compromised. In a similar fashion the authors of Reference [255] have modelled a water-level control system in timed automata utilising the PAT model checker [237] to successfully verify security recovery mechanisms of the system.

Within the area of connectivity the authors of Reference [214] have demonstrated formalisation and analysis of firewall rules using the Z3 SMT solver, lowering the errors in firewall configurations. In Reference [150] the authors have utilised TLA+ [158] to ensure effectiveness of mitigations strategies against several cyber attacks, while the authors of Reference [246] have used combinatorial testing within VDMJ [161] to generate 145 million tests for a formal model expressed in VDM-SL [160] of an industrial control system, providing assurance for several security properties.

Protocols. Industrial communication protocols carry critical data, requiring a high level of security. The authors of Reference [224] have formally analysed security of the Modbus/TCP using Coloured Petri Nets [138] combined with Formal Component Analysis [201] discovering a possible attack. Similarly, the authors of Reference [187] modelled the Modbus protocol as a Dynamic State Machine [186], automatically translating it to Promela for verification using the SPIN model checker. The authors have uncovered a possible man-in-the-middle attack. Another protocol, OPC UA, has been analysed by the authors of Reference [203] using ProVerif finding vulnerabilities in the authentication sub-protocol. In similar fashion, the authors of Reference [86] analysed several Modbus and OPC UA authenticity and integrity properties using TAMARIN theorem prover [176], discovering the necessity for secure channels. In Reference [15] the authors have formally analysed the authentication properties of DNP3 protocol utilising the CPN state space analysis tool [137], discovering a potential for replay attack. Finally, the authors of Reference [49] have analysed an authenticated CAN protocol using ProVerif, discovering that limited use of cryptography allows for a replay attack with partially modified data.

Implementation. Since industrial software often consists of a large codebase it could be difficult to be formally analysed in its entirety. Over the years, several approaches to code analysis have been presented; for example, the authors of Reference [18] use a dialect of UML, SysML-Sec to model of a large industrial codebase, iteratively refining the model and automatically translating it to $\pi$-calculus using the TTool [88] and analysing it using ProVerif. The authors propose for this approach to be integrated to the software development process.

In the aeronautics industry the authors of Reference [70] have created a formally verified implementation of unmanned aerial vehicle using several tools; for example, the jKind model checker [99], satisfying correctness of components and Isabelle/HOL theorem prover [194] to assure that system execution semantics matches the model. This approach has successfully prevented cyber attacks against the vehicle.

As PLCs are the backbone of industrial automation, a lot of focus have been put into ensuring that the PLC programs satisfy security properties. For example, in Reference [223] the authors use state transition diagrams of PLC programs as a basis for formal model dispatched to NuSMV model checker, while expressing the security properties in LTL. Similarly, the authors of Reference [247] utilised Petri Nets to develop software falsification detection by translating Petri Nets to Promela and dispatching it to the SPIN model checker, while modelling the falsification properties in LTL.

Hardware . FM for hardware verification is a well established field with specification languages such as Verilog [243] and VHDL [188]. The authors of Reference [124] focus on co-verification of Intellectual Property Blocks (IPs) for use within System on a Chip (SoC) architectures, considering technologies such as secure boot and concurrency in a time-of-check-to-time-of-use considerations [48]. The authors utilised semi-automatic co-verification methodology using a toolchain comprised of Boogie [31] as intermediate verification language, through Corral software verifier [157] and SMACK [206] for bit-precise checking with an ultimate goal of producing secure SoCs. The authors of Reference [162] consider that all software layers could be compromised and have developed an application-specific hardware monitor based on a formally analysed C code and a junction box validated in a hardware description language with a goal to monitor the hardware controller for malicious activity. The authors model their hardware monitor in Frama-C [74] with Jessie plugin, allowing for automatic deductive verification using Why [95].

Within the area of integrated circuits, the authors of Reference [164] consider the trustworthiness of hardware using Proof Carrying Code (PCC) [189], utilising Coq to derive theorems for the hardware descriptions annotated with PCC. This has been later extended in Reference [110] to a notion of a Proof Carrying Hardware (PCH), utilised to verify security of IPs supplied by untrusted vendors by extending the VHDL and utilising Coq to carry out the verification of security theorems. Finally, the authors of Reference [3] consider the possibility of hardware trojans being injected during manufacturing and utilise the nuXmv model checker [56], while specifying the hardware properties in LTL to detect these trojans.

Consumer computing such as use of personal computer, smartphones, and underlying connected services is an integral part of modern life. Consumer computing has often been characterised as of less critical nature than for example industrial systems, however, this view is changing as society introduces more digital technologies to everyday life. This section provides an overview of use of FM in analysis of cyber security of consumer computing, ranging from consumer electronics for fitness equipment, mobile operating systems, web browsers, consumer Internet of Things (IoT) devices to commodity hardware for devices such as personal electricity meters. An interesting fact within the consumer domain is significant use of so called lightweight FM, utilised often not only on the application level of abstraction, but also considering implementation and hardware. One challenge in formal analysis of consumer systems is a rapid nature of evolution of these systems, where the competition in consumer markets often forces fast adoption of new technologies.

Once again the most utilised FM is model checking. It could be argued that this is due to the significant model checking experience gained in other domains. Many consumer computing entities are, however, complex, interconnected chains of services, which could explain wider utilisation of lightweight FM, especially as some of these are used directly to build chains used to construct the consumer computing entities. It should also, however, be stated that theorem proving has also been utilised on all the different levels of abstraction within the consumer computing domain. The cyber security topics present within the consumer section are shown in Figure 5.

Application . FM thrive in checking consumer applications for malware. A practical definition of malware (one that can be used to classify executable files) intersects the perspective shift FM advocate: focus on “what” is computed instead of the “how.” Quoting Reference [147]: “any (formal) definition of the concept of malware depends on the definition of the concept of software system correctness.” Also, a majority of malware is the product of tools generating variants of known vulnerabilities/attacks or known malware. The authors of Reference [66] show variants are easy to hide syntactically, but not semantically.

Model checking-based approaches provide malicious behaviour semantic signatures by providing counterexamples. Recent approaches as in References [229, 231] extract push-down automata as models. A promising area is the application of the techniques to the realm of the Android operating system [230]. Although successful, the FM techniques provide no panacea to consumer malware protection. The malware game advances with discovery of zero-day (latent) vulnerabilities. FM have been argued to avoid these vulnerabilities in the first place [174], but practically, new malicious behaviours are expected to appear, thus the problem becomes to learn malicious behaviours. There are several proofs of concept where FM leverage the signature learning either in terms of or using push-down automata reachability in the process [75, 76, 167]. There are a few works where theorem proving is applied in malware [227], but the number of publications is small, and it is difficult to ascertain if there is an effective gain from it. Perhaps model checking is more appropriate to the domain due to its non-interactive nature, since malware is inherently a game between attackers and an algorithm.

System . Recently, we have seen the adoption of the software marketplace paradigm to prevent attacks and malware to reach consumer systems. For instance, Android enforces permissions at the application level, which could, however, lead to privilege escalation [78]. The authors of Reference [26] propose a tool-based approach, called COVERT, for compositional analysis of Android inter-app permission leakage. COVERT assesses the security of a system as a whole by generating Alloy specifications [131] and analysed more than 500 real-world applications, confirming the findings previously found within References [23, 91], showing that many Android applications are over-privileged. The authors of Reference [25] have moved towards analysing the permission protocol itself, identifying design flaws where two applications can apply the same custom permission, resulting in first installed application being able to access the resources of the second.

Operating systems usually contain an underlying security model amenable to FM checking. In Reference [83] the authors have verified a proposed access and integrity control for a Linux-like OS, using Alloy and Event-B [4]. While the authors have experienced scalability issues with Alloy, the analysis has uncovered bugs that could become more serious if discovered in the implementation phase. Another example is Reference [170], where the authors verify security policies in the form of invariants annotating the code of ExpressOS, a secure OS alternative to Android. The authors utilise automated theorem provers and report the verification overhead (added annotations) was roughly 2.8% source code.

Consumer IoT systems such as smart home devices pose security vulnerabilities and have been widely verified using FM. The authors of Reference [153] use model checking tools within AVISPA and by BAN logic to verify a framework ensuring anonymity, authentication, and integrity in smart home environments. In Reference [182] the authors have developed the IoTRiskAnalyzer tool used to help engineers apply the most fitting security policies. This has been achieved using a Markov Decision Process [202], formalisation of risk properties as probabilistic CTL formulas, and verification using the PRISM model checker [156]. Car manufacturers are also taking advantage of connected devices, especially smartphones. For instance, the authors of Reference [53] have developed a smartphone-based immobiliser with formally verified protocol using ProVerif against a Dolev-Yao attacker model to ensure strong guarantees of security requirements.

Protocols . The area of consumer communication protocols covers text and multimedia communication lending itself to formal verification of security.

In Reference [71] the authors have created a formal model of the Signal protocol in terms of predicates and theorems and have applied theorem proving, resulting in improvements in the use of the protocol’s random generator.

Security of the consumer communication systems often depends on the mechanisms introduced in the Needham-Schroeder [190] authentication protocol and the Denning Sacco protocol [81] for secret key distribution. The authors of Reference [60] have created a simplified model of the Needham-Schroeder NPSK protocol and the Denning Sacco protocol and expressed security properties using LTL. The authors have provided an efficient model for model checking of the security properties using the Spin model checker.

Implementation . The recent adoption of FM tools by large technology companies has shaken up the field. If in the past FM were tied to niche safety critical domains (e.g., aerospace, railway, medical) and fields with significant governmental regulation, the current panorama shows that the future brings the usage of FM tools in the daily practice of software engineering. No matter the intention behind the usage of FM tools, the outcome has demonstrated a contribution to increasingly secure implementations. According to recent reports, when a developer commits a code modification to one of the large technology companies’ codebases, a static analysis tool is invoked and a code review is provided. The author of Reference [196] describes the process as continuous reasoning, and any change to a Facebook product is analysed by the Infer static analysis tool, which checks “small theorems” on large codebases. This approach has been shown to improve the security of the company’s own codebase and library implementations (e.g., OpenSSL). The same is reported about the software engineering practice inside Google [215], although it is not clear whether FM is used by Project Zero, its elite security team. However, the authors of Reference [24] report on how numerous security vulnerabilities were fixed by applying FUDGE, a static analysis tool based on fuzzing developed in house.

Particularly targeted to the security domain, the authors of Reference [84] report a static analysis tool, Zoncolan, in collaboration with the Facebook App Security team. Zoncolan uses abstract interpretation to analyse and issue security alerts for the implementations of the applications in the company’s codebase: Messenger, WhatsApp, Instagram, or Facebook. This level of application of FM shows the implementations of software used by millions of consumers has been swept by a FM tool.

FM is also being applied to secure implementations of web browsers, which are designed with security in mind because they mediate a vast amount of personal information (e.g., credentials, banking details). Nevertheless, due to a large attack surface of a web browser, attacks are possible, and implementation flaws are not uncommon. In a bolder move, the authors of Reference [133] propose a new browser, QUARK, that follows the “kernel architecture”⁷ of modern browsers, but QUARK’s kernel is formally verified. The formal verification yields to the Coq theorems to assert properties as tab non-interference, or cookie confidentiality and integrity. According to work in Reference [96], the price to pay for such a prime example of functional correctness verification (above airline runtime error-free level) is 25% increase in overhead, affecting performance. Increasing browsers’ security risks, browser extensions can spy and exploit users as demonstrated in Reference [109]. In Reference [218] the authors report on a verified design of an experimental browser using the Maude tool and rewriting logic, and the authors of Reference [184] show that x86 native code executed by arbitrary clients conforms with a predefined sandbox policy when using Google Chrome’s Native Client service.

Hardware . In contrast to critical-system hardware (e.g., fly-by-wire hardware) attackers cannot be prevented from physical access to the consumer hardware, which provides a large attack surface. Modern consumer hardware provides hardware-level protections for critical software components. An example of this is ARM TrustZone [192], providing separation between trusted and rich software providing potentially untrusted interfaces. The authors of Reference [93] propose verification of hardware security properties by use of information flow control at the level of the Hardware Description Language (HDL) such as SecVeriLog [271]. The authors create SecVeriLogBL, an extension to SecVeriLog, by adding new types for security labels defined in SecVeriLog. This allows for static analysis at the design time, providing a lightweight verification with small effects on the hardware performance. To demonstrate this approach, the authors have designed an implementation of TrustZone, including 10+ security bugs. Similarly the authors of Reference [163] present a formally defined hardware security enforcement for x86 architecture. In this setting, the software relies on underlying hardware for security enforcement; for example, memory paging features of an x86 CPU. The authors note that incorrect implementations of hardware enforcement policies often lead to vulnerabilities [140]. The authors use Coq to model the architecture and the Coq theorem prover to prove the soundness of the security policy.

In the area of commodity hardware, the authors of Reference [239] have used model checking to determine possible attacks on smart meters, which are considered critical devices [143]. The authors have created a model of the smart meter using rewriting logic, formal definition of the attacker’s actions, and used the Maude tool to check that the attacker’s actions are not able to break the security invariants. The discovered attacks were then mapped to an implementation of a smart meter, the SEGMeter, to investigate the practicality of these attacks. The authors determined that many attacks discovered by the model checker are indeed practical, despite the model being abstract and not specifically refined towards the SEGMeter implementation.

Today, hardware is often packaged as an SoC, which security may be verified using the combination of integrated theorem proving and model checking proposed in Reference [111]. Due to the hierarchical nature of SoCs, the authors propose that the design expressed in HDL is decomposed into sub-modules and security specifications into sub-specifications. The sub-specifications are then verified using the Cadence IFV model checker [1]. These verified sub-specifications are then used as proven lemmas in the Coq theorem prover [242], removing the need to prove these lemmas by hand. This simplifies the model checking as well by providing only a small specification to the model checker, avoiding the state space explosion. The authors extend their method by automated code conversion from HDL to verifiable specification [112]. SoC complexity increases in Multi Processor SoCs (MPSoC), where multiple processors exchange data via Network on a Chip (NoC) routers. The authors of Reference [220] have used unbounded model checking to verify security properties of an NoC, which was practical due to the highly sequential behaviour of NoCs. The authors formalise the security and functionality correctness properties using LTL and use the CIP unbounded model checker [155] to verify them. As a proof of concept, the authors have analysed six different router implementations, determining the feasibility of their approach for NoC security analysis in early design stages.

Enterprise and large corporate computing is the backbone of large international business. In recent years, there is a trend in enterprise computing to utilise cloud solutions, while still often operating on-premises (local) data centers. These data centers and cloud clusters are utilised for a plethora of enterprise tasks such as virtualisation of collaboration platforms, company management, and hosting of corporate web portals. This section provides an overview of utilisation of FM to address security challenges of enterprise computing, ranging from secure data storage through virtualisation and software-defined networking security to strong authentication using hardware tokens. As enterprises are larger entities, changes are often slower and need to be well managed. To this end, the FM have been utilised as a booster in cloud adoption by enterprises, as several FM-based solutions have been proposed to enable enterprises’ secure switch from local data centers to federated cloud solutions.

Similarly to previous sections, model checking is the most used tool in formal analysis of security in enterprise computing. Theorem proving is, however, not far behind, especially within analysis of hardware such as Trusted Platform Module chips within enterprise servers. Lightweight FM have also been significantly utilised at the implementation level of abstraction, since they are often provided as plugins to software development environments, making them easily accessible. The cyber security topics present within the enterprise section are shown in Figure 6.

Application. Enterprise applications often process and store data critical for an organisation. Nowadays, such data is carried by Software-Defined Networking (SDN). The authors of Reference [226] have created a verification platform for applications utilising SDN, consisting of a modelling language that could be automatically translated and dispatched to PRISM, SPIN, and Alloy model checkers. Any counterexamples are then displayed in the tool. Similarly to SDN, Service-Oriented Architectures (SOAs) are often used within enterprise applications, increasing the interconnectivity of these applications. In Reference [20] the authors present a platform for security assessment of SOAs utilising a formal specification language and several model checkers, namely, CL-Atse and OFMC. The authors uncovered an issue with SAML-SSO integration in Google Apps.

Enterprise data are often stored within large relational databases. In this context, the authors of Reference [61] have proposed a method for secure outsourcing of databases to untrusted servers, building upon the notion of Verifiable Databases [36], and utilised theorem proving to demonstrate their method as secure. Another often used technology in the cloud is virtualisation. The authors of Reference [222] introduce a formal analysis scheme for security of Xen hypervisor, consisting of model checking and static analysis, successfully rediscovering a known vulnerability. Finally, we would like to point to the work within Reference [72] describing how a leading cloud provider utilises FM for security of their services, noting that the benefits of FM are important to their customers.

System. Nowadays, large enterprises can either host their own infrastructure, fully utilise the cloud, or partially combine their infrastructure with the cloud, leading to federated cloud systems [183]. The authors of Reference [270] have proposed a method for analysis of federated cloud behaviour utilising CPN and CPN tools [137], creating several models for security analysis. Similarly, the authors of Reference [256] have used Z with Z/EVES theorem prover to formally analyse a data exchange system against confidentiality and integrity properties, while also generating tests utilising the domain theory [39]. In Reference [135], the authors present a formal approach to analysis of firewall rules and cloud topology based on Mobile Ambients [55] and the non-interfering Boxed Ambients calculus [51]. As cloud computing is often built utilising shared resources, the authors of Reference [169] have built an offline framework for formal analysis of network isolation properties, ensuring isolation among shared resources. This has been carried out by use of first order logic and the constraint satisfaction solver Sugar [240]. Similarly, the authors of Reference [173] have proposed a security framework for cloud complexity management (agent) system [113], utilising the Z/EVES theorem prover to analyse several cloud security properties within a NIST [43] cloud reference architecture. Also, in Reference [232], the authors have proposed a broker solution for automatically pairing cloud services with customers while managing the cloud complexity. An important part of the broker is finding a service satisfying customers security requirements, defined in first-order relational logic [130] using KODKOD finite model finder [244].

Several works also consider virtualisation within the cloud system. For example, the authors of Reference [117] have proposed a formal framework for analysis of security and trust in virtualised system, combining hardware and software models expressed in CSP# [238] dispatched to the PAT model checker discovering a subtle bug in a real-world cloud system. Similarly, the authors of Reference [42] have proposed a security subsystem for change analysis within virtualised infrastructures in relation to security policies by utilisation of graphs and graph transformations dispatched to GROOVE model checker [102].

Protocols. Enterprise computing is moving towards the cloud, creating need for secure communication protocols, benefiting from FM analysis.

Amazon cloud services [13] use the s2n [14], the open source implementation of the TLS protocol, utilising FM to prove its correctness. For example, in Reference [64] the authors demonstrate that the Hashed based Message Authentication Code (HMAC) utilised by the protocol is indistinguishable from a random generator using Cryptol specification language [89] describing the HMAC, which was then dispatched to Coq theorem prover and the results are connected with the implementation by use of Software Analysis Workbench [100]. In the Microsoft cloud, the authors of Reference [136] have developed a tool for analysis of network protocols to assist with the task of network policy maintenance within data centers by use of the Z3 SMT solver, providing an important security tooling for Azure cloud services. Since the cloud services are often accessed remotely, the authors of Reference [151] have utilised Alloy analyser to find vulnerabilities in the SAML protocol [125].

Nowadays, clouds collect data from small-footprint IoT devices [185], which prompted the authors of Reference [141] to propose a lightweight mutual authentication protocol and verify it against several attacks using OFMC and CL-AtSe. Similarly, the authors of Reference [212] have proposed a mobile authentication scheme verified using ProVerif. The IoT devices could take advantage of the 5G networks, where in Reference [33] the authors use Tamarin, finding an issue with the authentication sub-protocol, while the authors of Reference [8] have analysed the authentication framework protocol [269] and the mobile ethernet protocol [128] by expressing them in the CSP, which was subsequently dispatched to the FDR model checker for analysis against mutual authentication properties.

Implementation . Enterprise computing is often composed of many applications implemented using different technologies. For example, the authors of Reference [195] have created a static code analysis tool for PHP plugins, phpSafe, that was then utilised to discover over 580 vulnerabilities in several PHP plugins. In similar fashion, the authors of Reference [267] have created a tool utilising invariant analysis [116] for malicious behaviour detection, noting the high effectiveness of logic flaws in several web applications.

Hypervisors are an important part of enterprise computing. To the end of security implementation of hypervisors, the authors of Reference [252] have created a framework for implementation of security verified hypervisors based on behavioural contracts and verified using FRAMA-C [225] for static analysis of the behavioural contracts. Similarly, the authors of Reference [251] have created a hypervisor framework for verification of memory integrity within single guest hypervisors utilising the CBMC model checker [67] for automated analysis of most of the codebase.

Hardware. Enterprise computing requires significant cloud hardware infrastructure and assurances such as data confidentiality and computational security. Customers often consider a cloud provider as an untrusted entity, where the administrators themselves could pose a security threat [217]. In this regard, the authors of Reference [219] created a cloud isolation system, separating the user data from administrators and limiting the operations that the administrators could take against a user’s virtual machine, utilising a hardware module, that the authors named Trusted Cloud Module (TCM), which provides a limited set of interfaces to the cloud administrator, manages encryption keys, and provides secure storage for the user. The module is built from off-the-shelf hardware components using the Scyther verification tool. Similarly, the basis of any trusted computing is the Trusted Platform Module (TPM) co-processor providing secure storage and computing environment. Unfortunately, the security of platforms using TPM is often not formally verified leading to vulnerabilities [50]. To mitigate this, the authors of Reference [27] have proposed TRUSTFOUND, a formal modelling framework for model checking utilising a Trusted CSP#, an extension of CSP#, and LS${}^2$[77], where the PAT model checker is used for verification and detecting six implied assumptions and two severe logic flaws.

Sometimes, to provide strong authentication, small One Time Password (OTP) generation hardware is used by enterprises to authenticate users towards cloud services [45]. One such device is Yubikey, a USB OTP generator. In Reference [154], the authors have formally analysed the security of the Yubikey OTP and also a security of the Hardware Security Module (HSM) . Another challenge in this area is addressing CPU side-channel attacks. One of these attacks is a timing channel attack, where an attacker, possibly a virtual machine, could determine the algorithm executed by another virtual machine in a shared environment. To solve this, the authors of Reference [92] have proposed Timing Compartments, an isolation scheme implemented in hardware isolating timing information between parties sharing the resources. The scheme was checked by information flow analysis using SecVerilog.

Our survey covers more than a decade of the use of FM in security. It reveals the rich variety of formal specification languages and their tools, theorem provers, model checkers, and verification frameworks. We have recorded more than 40 different specification languages and more than 40 different verification tools. These include the following:

Specification languages . AADL (Architecture Analysis & Design Language) [18, 70], ASF (Anonymous Secure Framework) [153], ASLan++ (AVANTSSAR Specification Language) [20], BAN logic [52, 228], Boogie [31], Boxed Ambients [135], CASM (ASM-based SL for compilers) [252], CCS (Calculus of Communicating Systems) [152], COVERT (compositional analysis of Android apps) [26], CSP (Communicating Sequential Systems) [120], CSP$\mathrm{\#}$(shared variables CSP) [237], CTL (Computation tree temporal logic) [229], Cloud Calculus [135], Cryptol [89], Dynamic State Machine [187], ERC20 token contracts [199], Event-B [83], HLPSL (High Level Protocol Specification Language) [44], Hoare logic [108], LS2 (Logic of Secure Systems) [27], LTL (linear-time temporal logic) [266], Markov Decision Process [182], Petri nets [15], $\pi$-calculus [40], PlusCal [9], Promela [172], RTL (real-time logic) [110], SPDL (Security Protocol Description Language) [168], SysML-Sec [18], TLA+ (Temporal Logic of Actions) [72], Trusted CSP$\mathrm{\#}$[27], überSpark [252], VDM [98], Verilog [164], VHDL [111], VML [226], vTRUST [117], XMHF (eXtensible and Modular Hypervisor Framework) [251], Z [265].

Model checkers . AVISPA (Automated Validation of Internet Security Protocols and Applications) [21], Alloy [83], CBMC (Bounded Model Checker for C and C++) [58], CWB-NC (Concurrency Workbench of New Century) [152], Cadence IFV (RTL block-level verifier) [111], FDR [103], GROOVE [102], jKind [99], NuSMV [65], OFMC (on-the-fly model checker) [34], PAT (Process Analysis Toolkit for CSP$\mathrm{\#}$) [237], PRISM (probabilistic model checker) [182], SATMC (SAT-based model checker for security protocols) [44], SPIN [247], TRUSTFOUND [27], UPPAAL [177], UVHM (formal analysis scheme for hypervisors) [251].

Theorem provers . Coq [72], Isabelle/HOL [144], K-framework [199], TAMARIN [154], Why [95].

Verification tools and frameworks. AndroBugs (Framework For Android Vulnerability Scanning) [241], Cl-Atse (protocol analyser) [141], FUDGE (Fuzz driver generator) [24], Frama-C [252], Krakatau [152], Maude (rewrite engine) [193], MobSF (mobile security framework) [127], phpSAFE [195], ProVerif [40], Quark [134], SAW (Software Analysis Workbench) [72], SMACK [72], SecGuru [136], SecVeriLog [92], Sugar (SAT-based) [169], TTool (translator from SysML-Sec to $\pi$-calculus) [18], Z3 [22].

This shows how research and application in FM for security has developed since Burrows et al.’s seminal paper [52]. Our survey concentrated in particular on the practical application of these techniques, especially on an industrial scale. Today, it seems inconceivable that a company would produce a commercial secure system without subjecting it to formal analysis. We also suspect that hackers use formal techniques to crack supposedly secure systems.

As a final statement, we need to acknowledge that a survey provides a snapshot in time within a developing field. It is therefore necessary to return to a survey work every decade, something we are planning on doing. Despite this shortcoming, it is the opinion of the authors that a survey work is an important part of the research, as it provides a starting point and a direction indicator for new and experienced practitioners looking for works under the large field of formal methods in security.