Binary instances for GADTs
(or: RTTI in Haskell)

Thursday, 15 June 2017, by Edsko de Vries.
Filed under coding.

In this blog post we consider the problem of defining Binary instances for GADTs such as

data Val :: * -> * where
  VI :: Int    -> Val Int
  VD :: Double -> Val Double

If you want to play along the full source code for the examples in this blog post can be found on github.

Failed attempt

The “obvious” way in which you might attempt to serialize and deserialize Val could look something like

instance Binary (Val a) where
  put (VI i) = putWord8 0 >> put i
  put (VD d) = putWord8 1 >> put d

  get = do
    tag <- getWord8
    case tag of
      0 -> VI <$> get -- Couldn't match type ‘a’ with ‘Int’
      1 -> VD <$> get -- Couldn't match type ‘a’ with ‘Double’
      _ -> error "invalid tag"

However, this does not work. The definition of put is type correct (but dubious), but the definition of get is not type correct. And actually this makes sense: we are claiming that we can define Binary (Val a) for any a; but if the tag is 0, then that a can only be Int, and if the tag is 1, then that a can only be Double.

One option is to instead give a Binary (Some Val) instance with Some defined as

data Some :: (* -> *) -> * where
  Exists :: forall f x. f x -> Some f

That is often independently useful, but is a different goal: in such a case we are discovering type information when we deserialize. That’s not what we’re trying to achieve in this blog post; we want to write a Binary instance that can be used when we know from the context what the type must be.

Working, but inconvenient

The next thing we might try is to introduce Binary instances for the specific instantiations of that a type variable:

instance Binary (Val Int) where
  put (VI i) = put i
  get = VI <$> get

instance Binary (Val Double) where
  put (VD d) = put d
  get = VD <$> get

Note that there is no need to worry about any tags in the encoded bytestring; we always know the type. Although this works, it’s not very convenient; for example, we cannot define

encodeVal :: Val a -> ByteString
encodeVal = encode

because we don’t have a polymorphic instance Binary (Val a). Instead we’d have to define

encodeVal :: Binary (Val a) => Val a -> ByteString
encodeVal = encode

but that’s annoying: we know that that a can only be Int or Double, and we have Binary instances for both of those cases. Can’t we do better?

Introducing RTTI

Although we know that a can only be Int or Double, we cannot take advantage of this information in the code. Haskell types are erased at compile time, and hence we cannot do any kind of pattern matching on them. The key to solving this problem then is to introduce some explicit runtime type information (RTTI).

We start by introducing a data family associating with each indexed datatype a corresponding datatype with RTTI:

data family RTTI (f :: k -> *) :: (k -> *)

For the example Val this runtime type information tells us whether we’re dealing with Int or Double:

data instance RTTI Val a where
  RttiValInt    :: RTTI Val Int
  RttiValDouble :: RTTI Val Double

For serialization we don’t need to make use of this:

putVal :: Val a -> Put
putVal (VI i) = put i
putVal (VD d) = put d

but for deserialization we can now pattern match on the RTTI to figure out what kind of value we’re expecting:

getVal :: RTTI Val a -> Get (Val a)
getVal RttiValInt    = VI <$> get
getVal RttiValDouble = VD <$> get

We’re now almost done: the last thing we need to express is that if we know at the type level that we have some RTTI available, then we can serialize. For this purpose we introduce a type class that returns the RTTI:

class HasRTTI f a where
  rtti :: RTTI f a

which we can use as follows:

instance HasRTTI Val a => Binary (Val a) where
  put = putVal
  get = getVal rtti

This states precisely what we described in words above: as long as we have some RTTI available, we can serialize and deserialize any kind of Val value.

The last piece of the puzzle is to define some instances for HasRTTI; right now, if we try to do encode (VI 1234) ghc will complain

No instance for (HasRTTI Val Int)

Fortunately, these instances are easily defined:

instance HasRTTI Val Int    where rtti = RttiValInt
instance HasRTTI Val Double where rtti = RttiValDouble

and the good news is that this means that whenever we construct specific Vals we never have to construct the RTTI by hand; ghc’s type class resolution takes care of it for us.

Taking stock

Instead of writing

encodeVal :: Binary (Val a) => Val a -> ByteString
encodeVal = encode

we can now write

encodeVal :: HasRTTI Val a => Val a -> ByteString
encodeVal = encode

While it may seem we haven’t gained very much, HasRTTI is a much more fine-grained constraint than Binary; from HasRTTI we can derive Binary constraints, like we have done here, but also other constraints that rely on RTTI. So while we do still have to carry these RTTI constraints around, those are – ideally – the only constraints that we still need to carry around. Moreover, as we shall see a little bit further down, RTTI also scales nicely to composite type-level structures such as type-level lists.

Another example: heterogeneous lists

As a second—slightly more involved—example, lets consider heterogeneous lists or n-ary products:

data NP (f :: k -> *) (xs :: [k]) where
  Nil  :: NP f '[]
  (:*) :: f x -> NP f xs -> NP f (x ': xs)

An example of such a heterogeneous list is

VI 12 :* VD 34.56 :* Nil :: NP Val '[Int, Double]

The type here says that this is a list of two Vals, the first Val being indexed by Int and the second Val being indexed by Double. If that makes zero sense to you, you may wish to study Well-Typed’s Applying Type-Level and Generic Programming in Haskell lecture notes.

As was the case for Val, we always statically know how long such a list is, so there should be no need to include any kind of length information in the encoded bytestring. Again, for serialization we don’t need to do anything very special:

putNP :: All Binary f xs => NP f xs -> Put
putNP Nil       = return ()
putNP (x :* xs) = put x >> putNP xs

The only minor complication here is that we need Binary instances for all the elements of the list; we guarantee this using the All type family (which is a minor generalization of the All type family explained in the same set of lecture notes linked above):

type family All p f xs :: Constraint where
  All p f '[]       = ()
  All p f (x ': xs) = (p (f x), All p f xs)

Deserialization however needs to make use of RTTI again. This means we need to define what we mean by RTTI for these heterogenous lists:

data instance RTTI (NP f) xs where
  RttiNpNil  :: RTTI (NP f) '[]
  RttiNpCons :: (HasRTTI f x, HasRTTI (NP f) xs)
             => RTTI (NP f) (x ': xs)

instance HasRTTI (NP f) '[] where
  rtti = RttiNpNil
instance (HasRTTI f x, HasRTTI (NP f) xs)
      => HasRTTI (NP f) (x ': xs) where
  rtti = RttiNpCons

In this case the RTTI gives us the shape of the list. We can take advantage of this during deserialization:

getNP :: All Binary f xs => RTTI (NP f) xs -> Get (NP f xs)
getNP RttiNpNil  = return Nil
getNP RttiNpCons = (:*) <$> get <*> getNP rtti

allowing us to give the Binary instance as follows:

instance (All Binary f xs, HasRTTI (NP f) xs)
      => Binary (NP f xs) where
  put = putNP
  get = getNP rtti

Serializing lists of `Val`s

If we use this Binary instance to serialize a list of Vals, we would end up with a type such as

decodeVals :: (HasRTTI (NP Val) xs, All Binary Val xs)
           => ByteString -> NP Val xs
decodeVals = decode

This All Binary Val xs constraint however is unfortunate, because we know that all Vals can be deserialized! Fortunately, we can do better. The RTTI for the (:*) case (RttiNpCons) included RTTI for the elements of the list. We made no use of that above, but we can make use of that when giving a specialized instance for lists of Vals:

putNpVal :: NP Val xs -> Put
putNpVal Nil       = return ()
putNpVal (x :* xs) = putVal x >> putNpVal xs

getNpVal :: RTTI (NP Val) xs -> Get (NP Val xs)
getNpVal RttiNpNil  = return Nil
getNpVal RttiNpCons = (:*) <$> get <*> getNpVal rtti

instance {-# OVERLAPPING #-} HasRTTI (NP Val) xs
      => Binary (NP Val xs) where
  put = putNpVal
  get = getNpVal rtti

This allows us to define

decodeVals :: HasRTTI (NP Val) xs => ByteString -> NP Val xs
decodeVals = decode

Note that this use of overlapping type classes instances is perfectly safe: the overlapping instance is fully compatible with the overlapped instance, so it doesn’t make a difference which one gets picked. The overlapped instance just allows us to be more economical with our constraints.

Here we can appreciate the choice of RTTI being a data family indexed by f; indeed the constraint HasRTTI f x in RttiNpCons is generic as possible. Concretely, decodeVals required only a single HasRTTI constraint, as promised above. It is this compositionality, along with the fact that we can derive many type classes from just having RTTI around, that gives this approach its strength.

Advanced example

To show how all this might work in a more advanced example, consider the following EDSL describing simple functions:

data Fn :: (*,*) -> * where
  Exp   :: Fn '(Double, Double)
  Sqrt  :: Fn '(Double, Double)
  Mod   :: Int -> Fn '(Int, Int)
  Round :: Fn '(Double, Int)
  Comp  :: (HasRTTI Fn '(b,c), HasRTTI Fn '(a,b))
        => Fn '(b,c) -> Fn '(a,b) -> Fn '(a,c)

If you are new to EDSLs (embedded languages) in Haskell, you way wish to watch the Well-Typed talk Haskell for embedded domain-specific languages. However, hopefully the intent behind Fn is not too difficult to see: we have a datatype that describes functions: exponentiation, square root, integer modules, rounding, and function composition. The two type indices of Fn describe the function input and output types. A simple interpreter for Fn would be

eval :: Fn '(a,b) -> a -> b
eval Exp          = exp
eval Sqrt         = sqrt
eval (Mod m)      = (`mod` m)
eval Round        = round
eval (g `Comp` f) = eval g . eval f

In the remainder of this blog post we will consider how we can define a Binary instance for Fn. Compared to the previous examples, Fn poses two new challenges:

The type index does not uniquely determine which constructor is used; if the type is (Double, Double) then it could be Exp, Sqrt or indeed the composition of some functions.
Trickier still, Comp actually introduces an existential type: the type “in the middle” b. This means that when we serialize and deserialize we do need to include some type information in the encoded bytestring.

RTTI for `Fn`

To start with, let’s define the RTTI for Fn:

data instance RTTI Fn ab where
  RttiFnDD :: RTTI Fn '(Double, Double)
  RttiFnII :: RTTI Fn '(Int, Int)
  RttiFnDI :: RTTI Fn '(Double, Int)

instance HasRTTI Fn '(Double, Double) where rtti = RttiFnDD
instance HasRTTI Fn '(Int, Int)       where rtti = RttiFnII
instance HasRTTI Fn '(Double, Int)    where rtti = RttiFnDI

For our DSL of functions, we only have functions from Double to Double, from Int to Int, and from Double to Int (and this is closed under composition).

Serializing type information

The next question is: when we serialize a Comp constructor, how much information do we need to serialize about that existential type? To bring this into focus, let’s consider the type information we have when we are dealing with composition:

data RttiComp :: (*,*) -> * where
  RttiComp :: RTTI Fn '(b,c) -> RTTI Fn '(a,b) -> RttiComp '(a,c)

Whenever we are deserializing a Fn, if that Fn happens to be the composition of two other functions we know RTTI about the composition; but since the “type in the middle” is unknown, we have no information about that at all. So what do we need to store? Let’s start with serialization:

putRttiComp :: RTTI Fn '(a,c) -> RttiComp '(a,c) -> Put

The first argument here is the RTTI about the composition as a whole, and sets the context. We can look at that context to determine what we need to output:

putRttiComp :: RTTI Fn '(a,c) -> RttiComp '(a,c) -> Put
putRttiComp rac (RttiComp rbc rab) = go rac rbc rab
  where
    go :: RTTI Fn '(a,c) -> RTTI Fn '(b,c) -> RTTI Fn '(a,b) -> Put
    go RttiFnDD RttiFnDD RttiFnDD = return ()

    go RttiFnII RttiFnII RttiFnII = return ()
    go RttiFnII RttiFnDI rAB      = case rAB of {}

    go RttiFnDI RttiFnII RttiFnDI = putWord8 0
    go RttiFnDI RttiFnDI RttiFnDD = putWord8 1

Let’s take a look at what’s going on here. When we know from the context that the composition has type Double -> Double, then we know that the types of both functions in the composition must also be Double -> Double, and hence we don’t need to output any type information. The same goes when the composition has type Int -> Int, although we need to work a bit harder to convince ghc in this case. However, when the composition has type Double -> Int then the first function might be Double -> Int and the second might be Int -> Int, or the first function might be Double -> Double and the second might be Double -> Int. Thus, we need to distinguish between these two cases (in principle a single bit would suffice).

Having gone through this thought process, deserialization is now easy: remember that we know the context (the RTTI for the composition):

getRttiComp :: RTTI Fn '(a,c) -> Get (RttiComp '(a,c))
getRttiComp RttiFnDD = return $ RttiComp RttiFnDD RttiFnDD
getRttiComp RttiFnII = return $ RttiComp RttiFnII RttiFnII
getRttiComp RttiFnDI = do
    tag <- getWord8
    case tag of
      0 -> return $ RttiComp RttiFnII RttiFnDI
      1 -> return $ RttiComp RttiFnDI RttiFnDD
      _ -> fail "invalid tag"

`Binary` instance for `Fn`

The hard work is now mostly done. Although it is probably not essential, during serialization we can clarify the code by looking at the RTTI context to know which possibilities we need to consider at each type index. For example, if we are serializing a function of type Double -> Double, there are three possibilities (Exp, Sqrt, Comp). We did something similar in the previous section.

putAct :: RTTI Fn a -> Fn a -> Put
putAct = go
  where
    go :: RTTI Fn a -> Fn a -> Put
    go r@RttiFnDD fn =
      case fn of
        Exp      -> putWord8 0
        Sqrt     -> putWord8 1
        Comp g f -> putWord8 255 >> goComp r (rtti, g) (rtti, f)
    go r@RttiFnII fn =
      case fn of
        Mod m    -> putWord8 0   >> put m
        Comp g f -> putWord8 255 >> goComp r (rtti, g) (rtti, f)
    go r@RttiFnDI fn =
      case fn of
        Round    -> putWord8 0
        Comp g f -> putWord8 255 >> goComp r (rtti, g) (rtti, f)

    goComp :: RTTI Fn '(a,c)
           -> (RTTI Fn '(b,c), Fn '(b,c))
           -> (RTTI Fn '(a,b), Fn '(a,b))
           -> Put
    goComp rAC (rBC, g) (rAB, f) = do
      putRttiComp rAC (RttiComp rBC rAB)
      go rBC g
      go rAB f

Deserialization proceeds along very similar lines; the only difficulty is that when we deserialize RTTI using getRttiComp we somehow need to reflect that to the type level; for this purpose we can provide a function

reflectRTTI :: RTTI f a -> (HasRTTI f a => b) -> b

It’s definition is beyond the scope of this blog post; refer to the source code on github instead. With this function in hand however deserialization is no longer difficult:

getAct :: RTTI Fn a -> Get (Fn a)
getAct = go
  where
    go :: RTTI Fn a -> Get (Fn a)
    go r@RttiFnDD = do
      tag <- getWord8
      case tag of
        0   -> return Exp
        1   -> return Sqrt
        255 -> goComp r
        _   -> error "invalid tag"
    go r@RttiFnII = do
      tag <- getWord8
      case tag of
        0   -> Mod <$> get
        255 -> goComp r
        _   -> error "invalid tag"
    go r@RttiFnDI = do
      tag <- getWord8
      case tag of
        0   -> return Round
        255 -> goComp r
        _   -> error "invalid tag"

    goComp :: RTTI Fn '(a,c) -> Get (Fn '(a,c))
    goComp rAC = do
      RttiComp rBC rAB <- getRttiComp rAC
      reflectRTTI rBC $ reflectRTTI rAB $
        Comp <$> go rBC <*> go rAB

We can define the corresponding Binary instance for Fn simply using

instance HasRTTI Fn a => Binary (Fn a) where
  put = putAct rtti
  get = getAct rtti

If desired, a specialized instance for HList Fn can be defined that relies only on RTTI, just like we did for Val (left as exercise for the reader).

Conclusion

Giving type class instances for GADTs, in particular for type classes that produce values of these GADTs (deserialization, translation from Java values, etc.) can be tricky. If not kept in check, this can result in a code base with a lot of unnecessarily complicated function signatures or frequent use of explicit computation of evidence of type class instances. By using run-time type information we can avoid this, keeping the code clean and allowing programmers to focus at the problems at hand rather than worry about type classes instances.

PS: Singletons

RTTI looks a lot like singletons, and indeed things can be set up in such a way that singletons would do the job. The key here is to define a new kind for the type indices; for example, instead of

data Val :: * -> * where
  VI :: Int    -> Val Int
  VD :: Double -> Val Double

we’d write something like

data U = Int | Double

data instance Sing (u :: U) where
  SI :: Sing 'Int
  SD :: Sing 'Double

data Val :: U -> * where
  VI :: Int    -> Val 'Int
  VD :: Double -> Val 'Double

instance SingI u => Binary (Val u) where
  put (VI i) = put i
  put (VD d) = put d

  get = case sing :: Sing u of
          SI -> VI <$> get
          SD -> VD <$> get

In such a setup singletons can be used as RTTI. Which approach is preferable depends on questions such as are singletons already in use in the project, how much of their infrastructure can be reused, etc. A downside of using singletons rather than a more direct encoding using RTTI as I’ve presented it in this blog post is that using singletons probably means that some kind of type level decoding needs to be introduced (in this example, a type family U -> *); on the other side, having specific kinds for specific purposes may also clarify the code. Either way the main ideas are the same.

Binary instances for GADTs (or: RTTI in Haskell)