The state of sync in web apps is that sync sucks. GTalk loses messages inbetween desktop and mobile. Apple iCloud can’t merge back document replicas if those were edited concurrently. Google Docs needs some ungodly chemistry to resync after working offline. These are SNAFUs I personally experienced and I have also heard other folks complaining about Evernote, Skype, Dropbox, you name it.

Skipping collaborative apps, these days even a single user needs to sync his/her devices (see continuity). The device landscape is getting predominantly mobile, so an average user has multiple powerful devices with unreliable internet connections.

Popular solutions suck for one simple reason: common sense fails us in distributed systems. Proper sync needs some math like a GPS device needs some relativity formulas. Yes, that is just a box that says your x and y. No, it can’t work by common sense alone; it needs math.

In 2013, we released the 0.1 version of Swarm, which was, basically, some slap-together per-column last-write-wins solution, like Firebase and many others are using. We already had the experience of implementing a real-time collaborative editor and we knew that LWW was not enough. So we reimplemented Swarm from scratch (twice) to have proper op-based CRDT as its base.

The result was like magic.

TodoMVC: real-time + offline, the holiest grail

Switching from theory to practice, we decided to approach the standard TodoMVC app a bit more creatively to showcase all the interesting capabilities. Actually, the first Swarm+React TodoMVC was created by Andrey Popp on 15 July 2014, but that one was not idiomatic. The new demo can:

• synchronize in real-time (WebSocket),
• cache data at the client (WebStorage),
• load and work completely offline (Application Cache).

Importantly, it employs a linear CRDT collection (Vector). That is different from both Swarm v0.1 and the July demo which used a set-sorting hack to make collections sort-of linear (same as Firebase). That is particularly important because serious approach to collaborative editing requires advanced linear collections (text is linear).

The app employs Swarm models and React views. It is important to notice that the code is written like it is a local MVC app. The only network-aware call in client-side code is connect() which “plugs” the model into the server’s sync “outlet”. Later on, all the sync is done by Swarm in the background; update events are delivered to views using the popular on()/off() subscription pattern, and even that is hidden inside ReactMixin.

The app is optimized for instant load using the Holy Grail approach. All the views, models and even the router are shared between client and server. The server-side code prerenders React views so the browser gets static HTML (fast). The page is made live once scripts and data arrive (slower). Holy Grail is critical, considering that React.js is rather heavy. The resulting page load latency is quite close to RTT.

Another interesting part is offline work. Swarm is able of caching all the data in the browser’s local storage. Thanks to CRDT merge features, the app may work offline and synchronize to the server on reconnect. That relieves the developer from the necessity to workaround temporary network failures. Swarm syncs the entire chain of replicas: back-end storage to server-side objects to client-side objects to browser storage. Every step of that chain may fail - because of server restarts, intermittent connections or simply a user closing his/her laptop. All such interruptions are solved by the unified syncing protocol that re-syncs data on re-connection. Just for comparison, Meteor’s DDP lacks such versioning/resync features, which fact essentially turned it into yet-another rediscovery of RPC API.

Thanks to HTML5 Application Cache, the app essentially installs itself into the browser, so you may load and run it entirely offline. Thanks to CRDT, it can always resync later. Actually, it is often hard to tell whether the app is online or not. That provides some native-like app experience, except maybe for the fps aspect.

One more fine detail that is easy to miss. The app is able to sync by multiple channels at once. While other users and browsers are synced through the server, browser tabs sync by WebStorage events.

To explore the app’s collaborative features, please send the exact page link to your friend or open it in another browser.

What is the big picture?

Essentially, Swarm is a data event bus shared between client and server. We think that such a data/event bus will become a standard component of the stack, very much like a database is today. Such a bus does a great deal of work to overcome friction in distributed systems, especially as the client becomes a full-featured part of it. Our idea is not entirely new; see e.g. Apache Kafka. The new part is to extend that bus to the client using CRDT.

I remember the times when my fellow developers were leery of the CVS optimistic model. Back then, relativism of git was entirely unthinkable. On hearing of CRDTs, folks often ask about ACID which, most likely, they don’t actually need. We believe that real-time sync middleware will have a transformative effect on the domain of web/mobile app development. That effect would be comparable to the one TCP had on internetworking, containerization had on international logistics or git on version control systems. To arrive at the destination, we need to solve many difficult issues, like access control (more on that later) and reasonable consistency guarantees.

Once you have this kind of platform-level sync, features of user collaboration and continuity come virtually for free and server-side scaling becomes easy and robust.

So, feel free to play with the app, check it out on GitHub, try the lib and subscribe to our twitter account.