JSON, interfaces, and go generate

Francesc Campoy

Developer, Advocate, and Gopher

Your mission

Your mission, should you choose to accept it, is to decode this message: 

{
    "name": "Gopher",
    "birthdate": "2009/11/10",
    "shirt-size": "XS"
}

into: 

type Person struct {
    Name string
    Born time.Time
    Size ShirtSize
}

Your mission (cont.)

Where ShirtSize is an enum (1): 

type ShirtSize byte

const (
    NA ShirtSize = iota
    XS
    S
    M
    L
    XL
)

(1): Go doesn't have enums.
In this talk I will refer to constants of integer types as enums.

Using a map

Using a map

Pros: very simple

Cons: too simple? we have to write extra code 

func (p *Person) Parse(s string) error {
    fields := map[string]string{}

    dec := json.NewDecoder(strings.NewReader(s))
    if err := dec.Decode(&fields); err != nil {
        return fmt.Errorf("decode person: %v", err)
    }

    // Once decoded we can access the fields by name.
    p.Name = fields["name"]

Parsing dates

Time format based on a "magic" date: 

Mon Jan 2 15:04:05 -0700 MST 2006

An example: 

package main

import (
	"fmt"
	"time"
)


func main() {
    now := time.Now()
    fmt.Printf("Standard format: %v\n", now)
    fmt.Printf("American format: %v\n", now.Format("Jan 2 2006"))
    fmt.Printf("European format: %v\n", now.Format("02/01/2006"))
    fmt.Printf("Chinese format: %v\n", now.Format("2006/01/02"))
}

Why that date?

Let's reorder: 

Mon Jan 2 15:04:05 -0700 MST 2006

into: 

01/02 03:04:05 PM 2006 -07:00 MST

which is:

1 2 3 4 5 6 7!

Parsing the birth date:

Since our input was: 

{
    "name": "Gopher",
    "birthdate": "2009/11/10",
    "shirt-size": "XS"
}

Parse the birth date: 

    born, err := time.Parse("2006/01/02", fields["birthdate"])
    if err != nil {
        return fmt.Errorf("invalid date: %v", err)
    }
    p.Born = born

Parsing the shirt size

Many ways of writing this, this is a pretty bad one: 

func ParseShirtSize(s string) (ShirtSize, error) {
    sizes := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}
    ss, ok := sizes[s]
    if !ok {
        return NA, fmt.Errorf("invalid ShirtSize %q", s)
    }
    return ss, nil
}

Use a switch statement, but a map is more compact.

Parsing the shirt size

Our complete parsing function: 

func (p *Person) Parse(s string) error {
    fields := map[string]string{}

    dec := json.NewDecoder(strings.NewReader(s))
    if err := dec.Decode(&fields); err != nil {
        return fmt.Errorf("decode person: %v", err)
    }

    // Once decoded we can access the fields by name.
    p.Name = fields["name"]

    born, err := time.Parse("2006/01/02", fields["birthdate"])
    if err != nil {
        return fmt.Errorf("invalid date: %v", err)
    }
    p.Born = born

    p.Size, err = ParseShirtSize(fields["shirt-size"])
    return err
}

Does this work?

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
	"time"
)

const input = `
{
    "name": "Gopher",
    "birthdate": "2009/11/10",
    "shirt-size": "XS"
}
`

type Person struct {
	Name string
	Born time.Time
	Size ShirtSize
}

type ShirtSize byte

const (
	NA ShirtSize = iota
	XS
	S
	M
	L
	XL
)

func (ss ShirtSize) String() string {
	sizes := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}
	s, ok := sizes[ss]
	if !ok {
		return "invalid ShirtSize"
	}
	return s
}

func ParseShirtSize(s string) (ShirtSize, error) {
	sizes := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}
	ss, ok := sizes[s]
	if !ok {
		return NA, fmt.Errorf("invalid ShirtSize %q", s)
	}
	return ss, nil
}

func (p *Person) Parse(s string) error {
	fields := map[string]string{}

	dec := json.NewDecoder(strings.NewReader(s))
	if err := dec.Decode(&fields); err != nil {
		return fmt.Errorf("decode person: %v", err)
	}

	// Once decoded we can access the fields by name.
	p.Name = fields["name"]

	born, err := time.Parse("2006/01/02", fields["birthdate"])
	if err != nil {
		return fmt.Errorf("invalid date: %v", err)
	}
	p.Born = born

	p.Size, err = ParseShirtSize(fields["shirt-size"])
	return err
}


func main() {
    var p Person
    if err := p.Parse(input); err != nil {
        log.Fatalf("parse person: %v", err)
    }
    fmt.Println(p)
}

Note: ShirtSize is a fmt.Stringer

JSON decoding into structs

JSON decoding into structs

Use tags to adapt field names: 

type Person struct {
    Name string    `json:"name"`
    Born time.Time `json:"birthdate"`
    Size ShirtSize `json:"shirt-size"`
}

But this doesn't fit: 

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
	"time"
)

const input = `
    {
        "name":"Gopher",
        "birthdate": "2009/11/10",
        "shirt-size": "XS"
    }
    `

type Person struct {
	Name string    `json:"name"`
	Born time.Time `json:"birthdate"`
	Size ShirtSize `json:"shirt-size"`
}

type ShirtSize byte

const (
	NA ShirtSize = iota
	XS
	S
	M
	L
	XL
)

func (ss ShirtSize) String() string {
	s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss]
	if !ok {
		return "invalid ShirtSize"
	}
	return s
}


func main() {
    var p Person
    dec := json.NewDecoder(strings.NewReader(input))
    if err := dec.Decode(&p); err != nil {
        log.Fatalf("parse person: %v", err)
    }
    fmt.Println(p)
}

Let's use an auxiliary struct type

Use string fields and do any decoding manually afterwards. 

    var aux struct {
        Name string
        Born string `json:"birthdate"`
        Size string `json:"shirt-size"`
    }

Note: the field tag for Name is not needed; the JSON decoder performs a case
insensitive match if the exact form is not found.

Let's use an auxiliary struct type (cont.)

The rest of the Parse function doesn't change much: 

func (p *Person) Parse(s string) error {
    var aux struct {
        Name string
        Born string `json:"birthdate"`
        Size string `json:"shirt-size"`
    }

    dec := json.NewDecoder(strings.NewReader(s))
    if err := dec.Decode(&aux); err != nil {
        return fmt.Errorf("decode person: %v", err)
    }

    p.Name = aux.Name
    born, err := time.Parse("2006/01/02", aux.Born)
    if err != nil {
        return fmt.Errorf("invalid date: %v", err)
    }
    p.Born = born
    p.Size, err = ParseShirtSize(aux.Size)
    return err
}

Can we do better?

Current solution

Repetition if other types have fields with:

Let's make the types smarter so json.Decoder will do all the work transparently.

Goal: json.Decoder should do all the work for me!

Meet Marshaler and Unmarshaler

Types satisfying json.Marshaler define how to be encoded into json. 

type Marshaler interface {
    MarshalJSON() ([]byte, error)
}

And json.Unmarshaler for the decoding part. 

type Unmarshaler interface {
    UnmarshalJSON([]byte) error
}

UnmarshalJSON all the things!

Let's make Person a json.Unmarshaler

Replace: 

func (p *Person) Parse(s string) error {

with: 

func (p *Person) UnmarshalJSON(data []byte) error {
    var aux struct {
        Name string
        Born string `json:"birthdate"`
        Size string `json:"shirt-size"`
    }

    dec := json.NewDecoder(bytes.NewReader(data))
    if err := dec.Decode(&aux); err != nil {
        return fmt.Errorf("decode person: %v", err)
    }
    p.Name = aux.Name
    // ... rest of function omitted ...

Let's make Person a json.Unmarshaler (cont.)

And our main function becomes: 

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"log"
	"strings"
	"time"
)

const input = `
{
	"name": "Gopher",
	"birthdate": "2009/11/10",
	"shirt-size": "XS"
}
`

type Person struct {
	Name string
	Born time.Time
	Size ShirtSize
}

type ShirtSize byte

const (
	NA ShirtSize = iota
	XS
	S
	M
	L
	XL
)

func (ss ShirtSize) String() string {
	s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss]
	if !ok {
		return "invalid ShirtSize"
	}
	return s
}

func ParseShirtSize(s string) (ShirtSize, error) {
	ss, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s]
	if !ok {
		return NA, fmt.Errorf("invalid ShirtSize %q", s)
	}
	return ss, nil
}

func (p *Person) UnmarshalJSON(data []byte) error {
	var aux struct {
		Name string
		Born string `json:"birthdate"`
		Size string `json:"shirt-size"`
	}

	dec := json.NewDecoder(bytes.NewReader(data)) // HL
	if err := dec.Decode(&aux); err != nil {
		return fmt.Errorf("decode person: %v", err)
	}
	p.Name = aux.Name
	// ... rest of function omitted ...
	born, err := time.Parse("2006/01/02", aux.Born)
	if err != nil {
		return fmt.Errorf("invalid date: %v", err)
	}
	p.Born = born
	p.Size, err = ParseShirtSize(aux.Size)
	return err
}


func main() {
    var p Person
    dec := json.NewDecoder(strings.NewReader(input))
    if err := dec.Decode(&p); err != nil {
        log.Fatalf("parse person: %v", err)
    }
    fmt.Println(p)
}

UnmarshalJSON for enums

Substitute ParseShirtSize: 

func ParseShirtSize(s string) (ShirtSize, error) {

with UnmarshalJSON: 

func (ss *ShirtSize) UnmarshalJSON(data []byte) error {
    // Extract the string from data.
    var s string
    if err := json.Unmarshal(data, &s); err != nil {
        return fmt.Errorf("shirt-size should be a string, got %s", data)
    }

    // The rest is equivalen to ParseShirtSize.
    got, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s]
    if !ok {
        return fmt.Errorf("invalid ShirtSize %q", s)
    }
    *ss = got
    return nil
}

UnmarshalJSON for enums (cont.)

Now use ShirtSize in the aux struct: 

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"log"
	"strings"
	"time"
)

const input = `{
    "name":"Gopher",
    "birthdate": "2009/11/10",
    "shirt-size": "XS"
}`

type Person struct {
	Name string
	Born time.Time
	Size ShirtSize
}

type ShirtSize byte

const (
	NA ShirtSize = iota
	XS
	S
	M
	L
	XL
)

func (ss ShirtSize) String() string {
	s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss]
	if !ok {
		return "invalid ShirtSize"
	}
	return s
}

func (ss *ShirtSize) UnmarshalJSON(data []byte) error {
	// Extract the string from data.
	var s string
	if err := json.Unmarshal(data, &s); err != nil { // HL
		return fmt.Errorf("shirt-size should be a string, got %s", data)
	}

	// The rest is equivalen to ParseShirtSize.
	got, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s]
	if !ok {
		return fmt.Errorf("invalid ShirtSize %q", s)
	}
	*ss = got // HL
	return nil
}


func (p *Person) UnmarshalJSON(data []byte) error {
    var aux struct {
        Name string
        Born string    `json:"birthdate"`
        Size ShirtSize `json:"shirt-size"`
    }

    dec := json.NewDecoder(bytes.NewReader(data))
    if err := dec.Decode(&aux); err != nil {
        return fmt.Errorf("decode person: %v", err)
    }
    p.Name = aux.Name
    p.Size = aux.Size
    // ... rest of function omitted ...

	born, err := time.Parse("2006/01/02", aux.Born)
	if err != nil {
		return fmt.Errorf("invalid date: %v", err)
	}
	p.Born = born
	return nil
}

func main() {
	var p Person
	dec := json.NewDecoder(strings.NewReader(input))
	if err := dec.Decode(&p); err != nil {
		log.Fatalf("parse person: %v", err)
	}
	fmt.Println(p)
}

Use the same trick to parse the birthdate.

Unmarshaling differently formatted dates

Create a new type Date: 

type Date struct{ time.Time }

And make it a json.Unmarshaler: 

func (d *Date) UnmarshalJSON(data []byte) error {
    var s string
    if err := json.Unmarshal(data, &s); err != nil {
        return fmt.Errorf("birthdate should be a string, got %s", data)
    }
    t, err := time.Parse("2006/01/02", s)
    if err != nil {
        return fmt.Errorf("invalid date: %v", err)
    }
    d.Time = t
    return nil
}

Unmarshaling differently formatted dates (cont.)

Now use Date in the aux struct: 

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"log"
	"strings"
	"time"
)

const input = `{
    "name":"Gopher",
    "birthdate": "2009/11/10",
    "shirt-size": "XS"
}`

type Person struct {
	Name string
	Born Date
	Size ShirtSize
}

type ShirtSize byte

const (
	NA ShirtSize = iota
	XS
	S
	M
	L
	XL
)

func (ss ShirtSize) String() string {
	s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss]
	if !ok {
		return "invalid ShirtSize"
	}
	return s
}

func (ss *ShirtSize) UnmarshalJSON(data []byte) error {
	var s string
	if err := json.Unmarshal(data, &s); err != nil {
		return fmt.Errorf("shirt-size should be a string, got %s", data)
	}
	got, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s]
	if !ok {
		return fmt.Errorf("invalid ShirtSize %q", s)
	}
	*ss = got
	return nil
}

type Date struct{ time.Time }

func (d Date) String() string { return d.Format("2006/01/02") }

func (d *Date) UnmarshalJSON(data []byte) error {
	var s string
	if err := json.Unmarshal(data, &s); err != nil {
		return fmt.Errorf("birthdate should be a string, got %s", data)
	}
	t, err := time.Parse("2006/01/02", s) // HL
	if err != nil {
		return fmt.Errorf("invalid date: %v", err)
	}
	d.Time = t
	return nil
}


func (p *Person) UnmarshalJSON(data []byte) error {
    r := bytes.NewReader(data)
    var aux struct {
        Name string
        Born Date      `json:"birthdate"`
        Size ShirtSize `json:"shirt-size"`
    }
    if err := json.NewDecoder(r).Decode(&aux); err != nil {
        return fmt.Errorf("decode person: %v", err)
    }
    p.Name = aux.Name
    p.Size = aux.Size
    p.Born = aux.Born
    return nil
}


func main() {
	var p Person
	dec := json.NewDecoder(strings.NewReader(input))
	if err := dec.Decode(&p); err != nil {
		log.Fatalf("parse person: %v", err)
	}
	fmt.Println(p)
}

Can this code be shorter?

Yes!

By making the Born field in Person of type Date.

Person.UnmarshalJSON is then equivalent to the default behavior!

It can be safely removed. 

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
	"time"
)

const input = `{
    "name":"Gopher",
    "birthdate": "2009/11/10",
    "shirt-size": "XS"
}`

type Person struct {
	Name string    `json:"name"`
	Born Date      `json:"birthdate"`
	Size ShirtSize `json:"shirt-size"`
}

type ShirtSize byte

const (
	NA ShirtSize = iota
	XS
	S
	M
	L
	XL
)

func (ss ShirtSize) String() string {
	s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss]
	if !ok {
		return "invalid ShirtSize"
	}
	return s
}

func (ss *ShirtSize) UnmarshalJSON(data []byte) error {
	var s string
	if err := json.Unmarshal(data, &s); err != nil {
		return fmt.Errorf("shirt-size should be a string, got %s", data)
	}
	got, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s]
	if !ok {
		return fmt.Errorf("invalid ShirtSize %q", s)
	}
	*ss = got
	return nil
}

type Date struct{ time.Time }

func (d Date) String() string { return d.Format("2006/01/02") }

func (d *Date) UnmarshalJSON(data []byte) error {
	var s string
	if err := json.Unmarshal(data, &s); err != nil {
		return fmt.Errorf("birthdate should be a string, got %s", data)
	}
	t, err := time.Parse("2006/01/02", s)
	if err != nil {
		return fmt.Errorf("invalid date: %v", err)
	}
	d.Time = t
	return nil
}


func main() {
    var p Person
    dec := json.NewDecoder(strings.NewReader(input))
    if err := dec.Decode(&p); err != nil {
        log.Fatalf("parse person: %v", err)
    }
    fmt.Println(p)
}

Was this really better?

Other ideas

Roman numerals

Roman numerals

Because why not? 

type romanNumeral int

And because Roman numerals are classier 

type Movie struct {
    Title string
    Year  romanNumeral
}

Roman numerals (cont.)

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
)

type romanNumeral int

var numerals = []struct {
	s string
	v int
}{
	{"M", 1000}, {"CM", 900},
	{"D", 500}, {"CD", 400},
	{"C", 100}, {"XC", 90},
	{"L", 50}, {"XL", 40},
	{"X", 10}, {"IX", 9},
	{"V", 5}, {"IV", 4},
	{"I", 1},
}

func (n romanNumeral) String() string {
	res := ""
	v := int(n)
	for _, num := range numerals {
		res += strings.Repeat(num.s, v/num.v)
		v %= num.v
	}
	return res
}

func parseRomanNumeral(s string) (romanNumeral, error) {
	res := 0
	for _, num := range numerals {
		for strings.HasPrefix(s, num.s) {
			res += num.v
			s = s[len(num.s):]
		}
	}
	return romanNumeral(res), nil
}

func (n romanNumeral) MarshalJSON() ([]byte, error) {
	if n <= 0 {
		return nil, fmt.Errorf("Romans had only natural (=>1) numbers")
	}
	return json.Marshal(n.String())
}

func (n *romanNumeral) UnmarshalJSON(data []byte) error {
	var s string
	if err := json.Unmarshal(data, &s); err != nil {
		return err
	}
	p, err := parseRomanNumeral(s)
	if err == nil {
		*n = p
	}
	return err
}

type Movie struct {
	Title string
	Year  romanNumeral
}


func main() {
    // Encoding
    movies := []Movie{{"E.T.", 1982}, {"The Matrix", 1999}, {"Casablanca", 1942}}
    res, err := json.MarshalIndent(movies, "", "\t")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Movies: %s\n", res)

    // Decoding
    var m Movie
    inputText := `{"Title": "Alien", "Year":"MCMLXXIX"}`
    if err := json.NewDecoder(strings.NewReader(inputText)).Decode(&m); err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%s was released in %d\n", m.Title, m.Year)
}

Secret data

Secret data

Some data is never to be encoded in clear text. 

type Person struct {
    Name string `json:"name"`
    SSN  secret `json:"ssn"`
}

type secret string

Use cryptography to make sure this is safe: 

func (s secret) MarshalJSON() ([]byte, error) {
    m, err := rsa.EncryptOAEP(crypto.SHA512.New(), rand.Reader, key.Public().(*rsa.PublicKey), []byte(s), nil)
    if err != nil {
        return nil, err
    }
    return json.Marshal(base64.StdEncoding.EncodeToString(m))
}

Note: This solution is just a toy; don't use it for real systems.

Secret data (cont.)

And use the same key to decode it when it comes back: 

func (s *secret) UnmarshalJSON(data []byte) error {
    var text string
    if err := json.Unmarshal(data, &text); err != nil {
        return fmt.Errorf("deocde secret string: %v", err)
    }
    cypher, err := base64.StdEncoding.DecodeString(text)
    if err != nil {
        return err
    }
    raw, err := rsa.DecryptOAEP(crypto.SHA512.New(), rand.Reader, key, cypher, nil)
    if err == nil {
        *s = secret(raw)
    }
    return err
}

Secret data (cont.)

Let's try it: 

package main

import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	_ "crypto/sha512"
	"encoding/base64"
	"encoding/json"
	"fmt"
	"log"
)

var key *rsa.PrivateKey

func init() {
	k, err := rsa.GenerateKey(rand.Reader, 2048)
	if err != nil {
		log.Fatalf("generate key: %v", err)
	}
	key = k
}

type Person struct {
	Name string `json:"name"`
	SSN  secret `json:"ssn"`
}

type secret string

func (s secret) MarshalJSON() ([]byte, error) {
	m, err := rsa.EncryptOAEP(crypto.SHA512.New(), rand.Reader, key.Public().(*rsa.PublicKey), []byte(s), nil)
	if err != nil {
		return nil, err
	}
	return json.Marshal(base64.StdEncoding.EncodeToString(m))
}

func (s *secret) UnmarshalJSON(data []byte) error {
	var text string
	if err := json.Unmarshal(data, &text); err != nil { // HL
		return fmt.Errorf("deocde secret string: %v", err)
	}
	cypher, err := base64.StdEncoding.DecodeString(text) // HL
	if err != nil {
		return err
	}
	raw, err := rsa.DecryptOAEP(crypto.SHA512.New(), rand.Reader, key, cypher, nil) // HL
	if err == nil {
		*s = secret(raw)
	}
	return err
}


func main() {
    p := Person{
        Name: "Francesc",
        SSN:  "123456789",
    }

    b, err := json.MarshalIndent(p, "", "\t")
    if err != nil {
        log.Fatalf("Encode person: %v", err)
    }
    fmt.Printf("%s\n", b)

    var d Person
    if err := json.Unmarshal(b, &d); err != nil {
        log.Fatalf("Decode person: %v", err)
    }
    fmt.Println(d)
}

But most JSON enums are boring

go generate to the rescue!

go generate:

You will see it as comments in the code like: 

//go:generate go tool yacc -o gopher.go -p parser gopher.y

More information in the blog post.

code generation tools: stringer

stringer generates String methods for enum types. 

package painkiller

//go:generate stringer -type=Pill

type Pill int

const (
    Placebo Pill = iota
    Aspirin
    Ibuprofen
    Paracetamol
)

Call go generate: 

$ go generate $GOPATH/src/path_to_painkiller

which will create a new file containing the String definition for Pill.

jsonenums

Around 200 lines of code.

Parses and analyses a package using:

And generates the code using:

And it's on github: github.com/campoy/jsonenums

Demo

Thank you

Francesc Campoy

Developer, Advocate, and Gopher

@francesc

campoy@golang.org