An Opinionated Survey of Functional Web Development

(Published on 2017-05-28.)

Intro

TL;DR: In this article I provide an overview of the frameworks and libraries available for creating websites in statically-typed functional programming languages.

I recommend you now skip directly to the next section, but if you're interested in some context and don't mind a rant, feel free to read on. :-)

<Rant mode>

When compared to native desktop application development, web development just sucks. Native development is relatively simple with toolkits such as Qt, GTK+ and others: You have convenient widget libraries, and you can describe your entire application, from interface design to all behavioural aspects, in a single programming language. You're also largely free to structure code in whichever way makes most sense. You can describe what a certain input field looks like, what happens when the user interacts with it and what will happen with the input data, all succinctly in a single file. There are even drag-and-drop UI builders to speed up development.

Web development is the exact opposite of that. There are several different technologies you're forced to work with even when creating the most mundane website, and there's a necessary but annoying split between code that runs on the server and code that runs in the browser. Creating a simple input field requires you to consider and maintain several ends:

The back end (server-side code) that describes how the input field interacts with the database.
Some JavaScript code to describe how the user can interact with the input field.
Some CSS to describe what the input field looks like.
And then there's HTML to act as a glue between the above.

In many web development setups, all four of the above technologies are maintained in different files. If you want to add, remove or modify an input field, or just about anything else on a page, you'll be editing at least four different files with different syntax and meaning. I don't know how other developers deal with this, but the only way I've been able to keep these places synchronized is to just edit one or two places, test if it works in a browser, and then edit the other places accordingly to fix whatever issues I find. This doesn't always work well: I don't get a warning if I remove an HTML element somewhere and forget to also remove the associated CSS. Heck, in larger projects I can't even tell whether it's safe to remove or edit a certain line of CSS because I have no way to know for sure that it's not still being used elsewhere. Perhaps this particular case can be solved with proper organization and discipline, but similar problems exist with the other technologies.

Yet despite that, why do I still create websites in my free time? Because it is the only environment with high portability and low friction - after all, pretty much anyone can browse the web. I would not have been able to create a useful "Visual Novel Database" any other way than through a website. And the entire purpose of Manned.org is to provide quick access to man pages from anywhere, which is not easily possible with native applications.

</Rant mode>

Fortunately, I am not the only one who sees the problems with the "classic" development strategy mentioned above. There are many existing attempts to improve on that situation. A popular approach to simplify development is the Single-page application (SPA). The idea is to move as much code as possible to the front end, and keep only a minimal back end. Both the HTML and the entire behaviour of the page can be defined in the same language and same file. With libraries such as React and browser support for Web components, the split between files described above can be largely eliminated. And if JavaScript isn't your favorite language, there are many alternative languages that compile to JavaScript. (See The JavaScript Minefield for an excellent series of articles on that topic).

While that approach certainly has the potential to make web development more pleasant, it has a very significant drawback: Performance. For some applications, such as web based email clients or CRM systems, it can be perfectly acceptable to have a megabyte of JavaScript as part of the initial page load. But for most other sites, such as this one, or the two sites I mentioned earlier, or sites like Wikipedia, a slow initial page load is something I consider to be absolutely unacceptable. The web can be really fast, and developer laziness is not a valid excuse to ruin it. (If you haven't seen or read The Website Obesity Crisis yet, please do so now).

I'm much more interested in the opposite approach to SPA: Move as much code as possible to the back end, and only send a minimal amount of JavaScript to the browser. This is arguably how web development has always been done in the past, and there's little reason to deviate from it. The difference, however, is that people tend to expect much more "interactivity" from web sites nowadays, so the amount of JavaScript is increasing. And that is alright, so long as the JavaScript doesn't prevent the initial page from loading quickly. But this increase in JavaScript does amplify the "multiple files" problem I ranted about earlier.

So my ideal solution is a framework where I can describe all aspects of a site in a single language, and organize the code among files in a way that makes sense to me. That is, I want the same kind of freedom that I get with native desktop software development. Such a framework should run on the back end, and automatically generate efficient JavaScript and, optionally, CSS for the front end. As an additional requirement (or rather, strong preference), all this should be in a statically-typed language - because I am seemingly incapable of writing large reliable applications with dynamic typing - and in a language from functional heritage - because programming in functional languages has spoiled me.

I'm confident that what I describe is possible, and it's evident that I'm not the only person to want this, as several (potential) solutions like this do indeed exist. I've been looking around for these solutions and have experimented with a few that looked promising. This article provides an overview of what I have found so far.

OCaml

My adventure began with OCaml. It's been a few years since I last used OCaml for anything, but development on the language and its ecosystem have all but halted. Real World OCaml has been a great resource to get me up to speed again.

Ocsigen

For OCaml there is one project that has it all: Ocsigen. It comes with an OCaml to JavaScript compiler, a web server, several handy libraries, and a framework to put everything together. Its syntax extension allows you to mix front and back end code, and you can easily share code between both ends. The final result is a binary that runs the server and a JavaScript file that handles everything on the client side.

The framework comes with an embedded DSL with which you can conveniently generate HTML without actually typing HTML. And best of all, this DSL works on both the client and the server: On the server side it generates an HTML string that can be sent to the client, and running the same code on the client side will result in a DOM element that is ready to be used.

Ocsigen makes heavy use of the OCaml type system to statically guarantee the correctness of various aspects of the application. The HTML DSL ensures not only that the generated HTML well-formed, but also prevents you from incorrectly nesting certain elements and using the wrong attributes on the wrong elements. Similarly, an HTML element generated on the server side can be referenced from client side code without having to manually assign a unique ID to the element. This prevents accidental typos in the ID naming and guarantees that the element that the client side code refers to actually exists. URL routing and links to internal pages are also checked at compile time.

Ocsigen almost exactly matches what I previously described as the perfect development framework. Unfortunately, it has a few drawbacks:

The generated JavaScript is quite large, a bit over 400 KiB for an hello world. In my brief experience with the framework, this also results in a noticeably slower page load. I don't know if it was done for performance purposes, but subsequent page views are per default performed via in-browser XHR requests, which do not require that all the JavaScript is re-parsed and evaluated, and is thus much faster. This, however, doesn't work well if the user opens pages in multiple tabs or performs a page reload for whatever reason. And as I mentioned, I care a lot about the initial page loading time.
The framework has a steep learning curve, and the available documentation is by far not complete enough to help you. I've found myself wondering many times how I was supposed to use a certain API and have had to look for example code for enlightenment. At some point I ended up just reading the source code instead of going for the documentation. What doesn't help here is that, because of the heavy use of the type system to ensure code correctness, most of the function signatures are far from intuitive and are sometimes very hard to interpret. This problem is made even worse with the generally unhelpful error messages from the compiler. (A few months with Rust and its excellent error messages has really spoiled me on this aspect, I suppose).
I believe they went a bit too far with the compile-time verification of certain correctness properties. Apart from making the framework harder to learn, it also increases the verbosity of the code and removes a lot of flexibility. For instance, in order for internal links to be checked, you have to declare your URLs (or services, as they call it) somewhere central such that the view part of your application can access it. Then elsewhere you have to register a handler to that service. This adds boilerplate and enforces a certain code structure. And the gain of all this is, in my opinion, pretty small: In the 15 years that I have been building web sites, I don't remember a single occurrence where I mistyped the URL in an internal link. I do suppose that this feature makes it easy to change URLs without causing breakage, but there is a trivial counter-argument to that: Cool URIs don't change. (Also, somewhat ironically, I have found more dead internal links on the Ocsigen website than on any other site I have visited in the past year, so perhaps this was indeed a problem they considered worth fixing. Too bad it didn't seem to work out so well for them).

Despite these drawbacks, I am really impressed with what the Ocsigen project has achieved, and it has set a high bar for the future frameworks that I will be considering.

Haskell

I have always seen Haskell as that potentially awesome language that I just can't seem to wrap my head around, despite several attempts in the past to learn it. Apparently the only thing I was missing in those attempts was a proper goal: When I finally started playing around with some web frameworks I actually managed to get productive in Haskell with relative ease. What also helped me this time was a practical introductory Haskell reference, What I Wish I Knew When Learning Haskell, in addition to the more theoretical Learn You A Haskell for Great Good.

Haskell itself already has a few advantages when compared to OCaml: For one, it has a larger ecosystem, so for any task you can think of there is probably already at least one existing library. As an example, I was unable to find an actively maintained SQL DSL for OCaml, while there are several available for Haskell. Another advantage that I found where the much more friendly and detailed error messages generated by the Haskell compiler, GHC. In terms of build systems, Haskell has standardized on Cabal, which works alright most of the time. Packaging is still often complex and messy, but it's certainly improving as Stack is gaining more widespread adoption. Finally, I feel that the Haskell syntax is slightly less verbose, and more easily lends itself to convenient DSLs.

Despite Haskell's larger web development community, I could not find a single complete and integrated client/server development framework such as Ocsigen. Instead, there are a whole bunch of different projects focussing on either the back end or the front end. I'll explore some of them with the idea that, perhaps, it's possible to mix and match different libraries and frameworks in order to get the perfect development environment. And indeed, this seems to be a common approach in many Haskell projects.

Server-side

Let's start with a few back end frameworks.

Scotty

Scotty is a web framework inspired by Sinatra. I have no experience with (web) development in Ruby and have never used Sinatra, but it has some similarities to what I have been using for a long time: TUWF.

Scotty is a very minimalist framework; It does routing (that is, mapping URLs to Haskell functions), it has some functions to access request data and some functions to create and modify a response. That's it. No database handling, session management, HTML generation, form handling or other niceties. But that's alright, because there are many generic libraries to help you out there.

Thanks to its minimalism, I found Scotty to be very easy to learn and get used to. Even as a Haskell newbie I had a simple website running within a day. The documentation is appropriate, but the idiomatic way of combining Scotty with other libraries is through the use of Monad Transformers, and a few more examples in this area would certainly have helped.

Spock

Continuing with the Star Trek franchise, there's Spock. Spock is very similar to Scotty, but comes with type-safe routing and various other goodies such as session and state management, CSRF protection and database helpers.

As with everything that is (supposedly) more convenient, it also comes with a slightly steeper learning curve. I haven't, for example, figured out yet how to do regular expression based routing. I don't even know if that's still possible in the latest version - the documentation isn't very clear. Likewise, it's unclear to me what the session handling does exactly (Does it store something? And where? Is there a timeout?) and how that interacts with CSRF protection. Spock seems useful, but requires more than just a cursory glance.

Servant

Servant is another minimalist web framework, although it is primarily designed for creating RESTful APIs.

Servant distinguishes itself from Scotty and Spock by not only featuring type-safe routing, it furthermore allows you to describe your complete public API as a type, and get strongly typed responses for free. This also enables support for automatically generated documentation and client-side API wrappers.

Servant would be an excellent back end for a SPA, but it does not seem like an obvious approach to building regular websites.

Happstack / Snap / Yesod

Happstack, Yesod and Snap are three large frameworks with many auxiliary libraries. They all come with a core web server, routing, state and database management. Many of the libraries are not specific to the framework and can be used together with other frameworks. I won't go into a detailed comparison between the three projects because I have no personal experience with any of them, and fortunately someone else already wrote a comparison in 2012 - though I don't know how accurate that still is today.

So there are a fair amount of frameworks to choose from, and they can all work together with other libraries to implement additional functions. Apart from the framework, another important aspect of web development is how you generate the HTML to send to the client. In true Haskell style, there are several answers.

For those who prefer embedded DSLs, there are xhtml, BlazeHTML and Lucid. The xhtml package is not being used much nowadays and has been superseded by BlazeHTML, which is both faster and offers a more readable DSL using Haskell's do-notation. Lucid is heavily inspired by Blaze, and attempts to fix several of its shortcomings. Having used Lucid a bit myself, I can attest that it is easy to get started with and pretty convenient in use.

I definitely prefer to generate HTML using DSLs as that keeps the entire application in a single host language and with consistent syntax, but the alternative approach, templating, is also fully supported in Haskell. The Snap framework comes with Heist, which are run-time interpreted templates, like similar systems in most other languages. Yesod comes with Shakespeare, which is a type-safe templating system with support for inlining the templates in Haskell code. Interestingly, Shakespeare also has explicit support for templating JavaScript code. Too bad that this doesn't take away the need to write the JavaScript yourself, so I don't see how this is an improvement over some other JavaScript solution that uses JSON for communication with the back end.

Client-side

It is rather unusual to have multiple compiler implementations targeting JavaScript for the same source language, but Haskell has three of them. All three can be used to write front end code without touching a single line of JavaScript, but there are large philosophical differences between the three projects.

Fay

Fay compiles Haskell code directly to JavaScript. The main advantage of Fay is that it does not come with a large runtime, resulting small and efficient JavaScript. The main downside is that it only supports a subset of Haskell. The result is a development environment that is very browser-friendly, but where you can't share much code between the front and back ends. You're basically back to the separated front and back end situation in classic web development, but at least you can use the same language for both - somewhat.

Fay itself doesn't come with many convenient UI libraries, but Cinder covers that with a convenient HTML DSL and DOM manipulation library.

Fay is still seeing sporadic development activity, but there is not much of a lively community around it. Most people have moved on to other solutions.

GHCJS

GHCJS uses GHC itself to compile Haskell to a low-level intermediate language, and then compiles that language to JavaScript. This allows GHCJS to achieve excellent compatibility with native Haskell code, but comes, quite predictably, at the high cost of duplicating a large part of the Haskell runtime into the JavaScript output. The generated JavaScript code is typically measured in megabytes rather than kilobytes, which is (in my opinion) far too large for regular web sites. The upside of this high compatibility, of course, is that you can re-use a lot of code between the front and back ends, which will certainly make web development more tolerable.

The community around GHCJS seems to be more active than that of Fay. GHCJS integrates properly with the Stack package manager, and there are a whole bunch of libraries available.

Haste

Haste provides a middle ground between Fay and GHCJS. Like GHCJS, Haste is based on GHC, but it instead of using low-level compiler output, Haste uses a higher-level intermediate language. This results in good compatibility with regular Haskell code while keeping the output size in check. Haste has a JavaScript runtime of around 60 KiB and the compiled code is roughly as space-efficient as Fay.

While it should be possible to share a fair amount of code between the front and back ends, not all libraries work well with Haste. I tried to use Lucid within a Haste application, for example, but that did not work. Apparently one of its dependencies (probably the UTF-8 codec, as far as I could debug the problem) performs some low-level performance optimizations that are incompatible with Haste.

Haste itself is still being sporadically developed, but not active enough to be called alive. The compiler lags behind on the GHC version, and the upcoming 0.6 version has stayed unreleased and in limbo state for at least 4 months on the git repository. The community around Haste is in a similar state. Various libraries do exist, such as Shade (HTML DSL, Reactive UI), Perch (another HTML DSL), haste-markup (yet another HTML DSL) and haste-dome (yet another HTML DSL), but they're all pretty much dead.

Despite having three options available, only Haste provides enough benefit of code reuse while remaining efficient enough for the kind of site that I envision. Haste really deserves more love than it is currently getting.

More Haskell

In my quest for Haskell web development frameworks and tools, I came across a few other interesting libraries. One of them is Clay, a CSS preprocessor as a DSL. This will by itself not solve the CSS synchronisation problem that I mentioned at the start of this article, but it could still be used to keep the CSS closer to code implementing the rest of the site.

It also would not do to write an article on Haskell web development and not mention a set of related projects: MFlow, HPlayground and the more recent Axiom. These are ambitious efforts at building a very high-level and functional framework for both front and back end web development. I haven't spend nearly enough time on these projects to fully understand their scope, but I'm afraid of these being a bit too high level. This invariably results in reduced flexibility (i.e. too many opinions being hard-coded in the API) and less efficient JavaScript output. Axiom being based on GHCJS reinforces the latter concern.

Other languages

I've covered OCaml and Haskell now, but there are relevant projects in other languages, too:

PureScript

PureScript is the spiritual successor of Fay - except it does not try to be compatible with Haskell, and in fact intentionally deviates from Haskell at several points. Like Fay, and perhaps even more so, PureScript compiles down to efficient and small JavaScript.

Being a not-quite-Haskell language, sharing code between a PureScript front end and a Haskell back end is not possible, the differences are simply too large. It is, however, possible to go into the other direction: PureScript could also run on the back end in a NodeJS environment. I don't really know how well this is supported by the language ecosystem, but I'm not sure I'm comfortable with replacing the excellent quality of Haskell back end frameworks with a fragile NodeJS back end (or such is my perception, I admittedly don't have too much faith in most JavaScript-heavy projects).

The PureScript community is very active and many libraries are available in the Persuit package repository. Of note is Halogen, a high-level reactive UI library. One thing to be aware of is that not all libraries are written with space efficiency as their highest priority, the simple Halogen button example already compiles down to a hefty 300 KB for me.

Elm

Elm is similar to PureScript, but rather than trying to be a generic something-to-JavaScript compiler, Elm focuses exclusively on providing a good environment to create web UIs. The reactive UI libraries are well maintained and part of the core Elm project. Elm has a strong focus on being easy to learn and comes with good documentation and many examples to get started with.

Ur/Web

Ur/Web is an ML and Haskell inspired programming language specifically designed for client/server programming. Based on its description, Ur/Web is exactly the kind of thing I'm looking for: It uses a single language for the front and back ends and provides convenient methods for communication between the two.

This has been a low priority on my to-try list because it seems to be primarily a one-man effort, and the ecosystem around it is pretty small. Using Ur/Web for practical applications will likely involve writing your own libraries or wrappers for many common tasks, such as for image manipulation or advanced text processing. Nonetheless, I definitely should be giving this a try sometime.

(Besides, who still uses frames in this day and age? :-)

Opa

I'll be moving out of the functional programming world for a bit.

Opa is another language and environment designed for client/server programming. Opa takes a similar approach to "everything in PureScript": Just compile everything to JavaScript and run the server-side code on NodeJS. The main difference with other to-JavaScript compilers is that Opa supports mixing back end code with front end code, and it can automatically figure out where the code should be run and how the back and front ends communicate with each other.

Opa, as a language, is reminiscent of a statically-typed JavaScript with various syntax extensions. While it does support SQL databases, its database API seems to strongly favor object-oriented use rather than relational database access.

GWT

Previously I compared web development to native GUI application development. There is no reason why you can't directly apply native development structure and strategies onto the web, and that's exactly what GWT does. It provides a widget-based programming environment that eventually runs on the server and compiles the client-side part to JavaScript. I haven't really considered it further, as Java is not a language I can be very productive in.

Webtoolkit

In the same vein, there's Wt. The name might suggest that it is a web-based clone of Qt, and indeed that's what it looks like. Wt is written in C++, but there are wrappers for other languages. None of the languages really interest me much, however.

That said, if I had to write a web UI for a resource-constrained device, this seems like an excellent project to consider.

To conclude

To be honest, I am a bit overwhelmed at the number of options. On the one hand, it makes me very happy to see that a lot is happening in this world, and that alternatives to boring web frameworks do exist. Yet after all this research I still have no clue what I should use to develop my next website. I do like the mix and match culture of Haskell, which has the potential to form a development environment entirely to my own taste and with my own chosen trade-offs. On the other hand, the client-side Haskell solutions are simply too immature and integration with the back end frameworks is almost nonexistent.

Almost none of the frameworks I discussed attempt to tackle the CSS problem that I mentioned in the introduction, so there is clearly room for more research in this area.

There are a few technologies that I should spend more time on to familiarize myself with. Ur/Web is an obvious candidate here, but perhaps it is possible to create a Haskell interface to Wt. Or maybe some enhancements to the Haste ecosystem could be enough to make that a workable solution instead.