Talk: UIs are streaming DAGs (Getz 2022)

Dustin Getz 2022 May (released at Hytradboi 2022) https://twitter.com/dustingetz

Watching the video is more effective than reading the transcript. It’s only 10 minutes; we worked hard to present this in a tight way that is respectful of everyone's attention.

Welcome. In this video, I’m going to describe a new paradigm for modern full-stack web development that is competitive with React.js, GraphQL and Relay. At Hyperfiddle, we’ve been researching how functional programming can be applied to simplify web development. This is the result of that research.

As a problem statement, this here is the code we as web developers wish we could write:

There are client-side DOM effects and server-side queries. It feels like such a composition should be well-defined, but it’s not, because of how the web works. The network makes us split our composed program into two separate programs, breaking function composition.

This is basically an impedance mismatch. Your programming language is designed for single programs but your application as a whole is a client/server distributed system.

This impedance mismatch is the root cause of complexity in web development, and we think functional programming can solve it.

At Hyperfiddle, we designed a new programming language called Photon which is hosted by Clojure:

Here you can see in Photon we mark the client bits and the server bits.

, which defines a photon function, is a Clojure macro. It looks like a Clojure function, and acts like a Clojure function, but actually the macro is compiling the function body into something else...

We compile it into a flowchart, or DAG (Directed Acyclic Graph):

The DAG is abstract. It captures only the dependency flow of data through your functions without imposing any evaluation model. Throw away your typical call stack evaluation model, the rules here are different.

You could imagine interpreting the DAG in many ways.

We could use an async interpreter with promise-like chaining between steps.

We could do an event propagation network with events streaming through the DAG one after another.

In fact, such an event propagation network can be used to achieve efficient reactive rendering like React.js, where functions are efficiently recomputed only when their dependencies change.

Let’s tweak our vocabulary, instead of saying “reactive”, let’s say “streaming”.

Since DAGs essentially give you streaming, they are natively asynchronous.

And this is the key insight. If the computation is already streaming, inserting a small delay into the stream for a websocket hop, has no impedance with the streaming computation model.

The composition of steps in the DAG is still well-defined!

So what happens when you actually run this Photon program?

At compile time, Photon will color the AST, and separate out the client and server parts.

You now have two programs, and they run concurrently:

The purple program will run as far as it can. It doesn’t get very far, as it immediately needs a server result, so the program will park and await. It will NOT issue a request, it will NOT send a message to the server, there is no I/O at all, it will silently wait.

The red server program is running concurrently. Its dependencies are all satisfied, so it is able to run the query immediately and then stream the result to the client.

The client receives the result and unparks, continuing to render the view.

You can conceptualize photon as a reactive programming language. Unlike React.js, in Photon, the programming language itself is reactive.

Reactive . Reactive . Reactive -. Reactive DOM bindings are a special case of a more general reactive programming language.

In Photon, the entire client/server system is expressed in the same unified programming language. Both frontend and backend are reactive and streaming. And since they match, they compose just like we wanted them to in the problem statement. No more impedance mismatch!

Let’s be clear about the network traffic. The client sends nothing to the server. The client and server are like a sports team, they agree on the play in advance. The client knows where the ball is going to come, and the server throws the ball to where the client is going to be. The only network traffic is a single message.

Here’s a more interesting example with user input:

Highlighted is a DOM input. In React.js, input elements have a change callback. That is not the case here. This (dom/input) function, uses its return channel, to return a stream of values bound to the symbol , which is conceptually a stream.

Note that is in lexical scope of the server. How can lexical scope cross to the server?

Recall that photon functions are DAGs. When the input has a new value, the event will propagate and reflow the downstream DAG, sometimes crossing over the websocket:

Here’s a larger example, a CRUD grid component:

If you’re not a Clojure programmer don’t worry about it, the DAG is what matters, and we’ll show you the DAG in a moment.

There’s even a for loop driving this table, with row specific queries inside the loop:

The above naive, inlined code is actually exactly what you would write if you’re writing PHP and the database was co-located with your program!

There is zero web framework boilerplate, no coordination of GraphQL queries, no client-side ORM or client-side cache. The network is not visible in your program.

Here’s the DAG. I want you to look at the innermost circle. That’s the loop:

The contents of the loop are half purple and half red, it is both colors.

Photon can see this same as you. Photon understands the underlying primitives that make up the loop itself, and slices the loop in half at a more primitive layer. And you can see this in the pseudocode: the loop is in both places.

The loop body on the server is not blocked on any purple client dependencies! The server will enter the loop and run it as far as it can, and then stream the intermediate environment to the client. The client will catch the ball and resume execution in mid-flight, right in the middle of the loop.

This is stunning. When we first recognized this structure, it blew us away!

Dear reader, I want you to pause right now and think about this idea of slicing the DAG. Pause and convince yourself that this messy and tangled program, when presented as a DAG, is actually clean and organized. There are clear red and purple regions. There’s a clean cut between them. And there is always such a cut for any real world program no matter how big and complicated, even in the presence of loops.

Let’s talk about security for a second because this seems insecure, right? Actually, it is secure!

Look at this green line:

This is the security boundary. All the red logic is running on your trusted backend. Put your permission checks here. So the security model is equivalent to the backend for frontend pattern.

Photon is the backend-for-frontend pattern, except you don’t have to hand-code it, the compiler generates it automatically.

If it helps, you can mentally rotate this model into the shape of a REST API which has a security model you’re accustomed to. To get REST, all you need to do is slow down the streaming, by waiting for all the queries to finish, before sending a big batch response all at once.

The main question at this point is how far can the DAG abstraction scale. Can it do Functions? Higher order functions? Closures?

Yes! DAGs nest, you can embed a DAG inside a DAG:

Look here where we call this field function and pass it the two red parameters. The red parameters are not used by the client, they stay on the server. And that is clear from the static information available in the program. When you define a Photon function you’re defining a DAG, and when you compose Photon functions you’re composing DAGs. Next is closures.

Here’s a closure. Quickly, the big idea here is that closures are higher order DAGs, “DAG-as-a-value.” Closures essentially pack up a DAG, pass it around as a parameter, and then splice it in at the call site.

The operation I just called “splice” is the monadic join operation.

This computational model is called “self-adjusting computation”, and Jane Street has a helpful article explaining this.

Lets quickly recap what we’ve seen:

Compiler managed network. This is so important. The compiler will produce better netcode than you could ever achieve by hand, regardless of the composition depth or how dynamic your application is. You know how every SaaS tool lags these days? Framing it as a compiler optimization problem is how you solve that.

Backend for frontend pattern - gone. GraphQL - gone. Your whole web framework – gone.

Think about how we even conceptualize web development work.

Frontend team, backend team. Frontend language, backend language. Frontend framework, backend framework.

We baked the platform into our org chart, and then we shipped the org chart! Today’s orgs cannot even conceptualize that it could be different.

We’re hosted by Clojure/Script. By “hosted” we mean a language embedded in a language. Photon is just a Clojure library. When you adopt Photon, you’re not adopting some baby programming language with only 10 users, Closure is battle-hardened, has an incredible community, and is commercially supported.

Why else Clojure?

Metaprogramming is roughly equivalent to monads, and its applications are under-explored.

Most importantly, Clojure itself is hosted by JavaScript, JVM, etc. All your legacy code, we reach it. Want to use shell scripts as your backend? No problem.

We have a pilot with a series B startup, which is driving us through all the real world edge cases like loading states, security, pagination, screwy business rules, etc.

All right, why did we even do this?

We didn’t set out to build this. We set out to build a holy grail low-code application platform that scales to complex and messy real world requirements. We ended up here because it turns out that this is the minimum level of ambition that is useful.

So the question is really, what is the holy grail low-code tool that composable network makes possible? Well, do we have a killer demo for you, but we’re out of time. So to see the demo, follow me on Twitter here, from twitter you can get to the demo.

Follow us, DM us, post hype tweets, make introductions. To get in the technical alpha, DM me with your use case (we’re starting small). Help us bring this to the world!

I’m Dustin Getz and we are Hyperfiddle.