(cache)Go で書かれた API Gateway "KrakenD"

この記事は Go3 Advent Calendar 2017 の 10 日目の記事です。

小さな役割を持つサービス群でアプリケーションを構成する、いわゆる Microservices が流行して久しいですね。サービスが細かく分割されるとき、それらを取りまとめるサービスが必要になることがあります。一般に API Gateway や API Aggregator と呼ばれるものです。

そんな API Gateway のひとつに Go で書かれた KrakenD というものがあります。

f:id:tchssk:20171206184251p:plain KrakenD: API Gateway and Manager

今回はこれを試してみます。

仕様

複数のバックエンド API を集約 (aggregate) する動きを見るため、以下のような仕様でサービスを構築してみます。

バックエンドに GitHub と Qiita の Web API を使用する
KrakenD に /users/:id というエンドポイントを定義する
そこにリクエストが来たらバックエンドの GitHub と Qiita の API にリクエストする
- GitHub は GET https://api.github.com/users/:id
- Qiita は GET https://qiita.com/api/v2/users/:id
それぞれからのレスポンスを "github"、 "qiita" というフィールドに入れてクライアントにレスポンスする

シーケンス図にすると以下のような感じですね。

client     krakend    github      qiita
   |          |          |          |
   +--------->|          |          |    GET /users/:id
   | request  |          |          |
   |          |          |          |
   |          +--------->|          |    GET https://api.github.com/users/:id
   |          | request  |          |
   |          |          |          |
   |          +----------^--------->|    GET https://qiita.com/api/v2/users/:id
   |          |          | request  |
   |          |          |          |
   |          |<---------+          |
   |          | response |          |
   |          |          |          |
   |          |<---------^----------+
   |          |          | response |
   |          |          |          |
   |<---------+          |          |
   | response |          |          |
   |          |          |          |

KrakenD の設定ファイル

まず上記の仕様を KrakenD の設定ファイル (JSON) として用意する必要があります。なんと設定ファイルを作るための Web エディタが用意されているのでそれを使わせてもらいましょう。

KrakenDesigner - KrakenD: API Gateway and Manager

見た目はこんな感じ。

f:id:tchssk:20171206174522p:plain

UI もなかなか悪くなく、サクッと設定ファイルを作ることができました。

{
  "version": 1,
  "default_max_rate": 0,
  "client_max_rate": 0,
  "throttling_header": "",
  "timeout": "3000ms",
  "cache_ttl": "300s",
  "name": "My service",
  "host": [
    "https://api.github.com",
    "https://qiita.com"
  ],
  "endpoints": [
    {
      "endpoint": "/users/{id}",
      "method": "GET",
      "concurrent_calls": 1,
      "client_max_rate": 0,
      "querystring_params": [],
      "backend": [
        {
          "url_pattern": "/api/v2/users/{id}",
          "encoding": "json",
          "host": [
            "https://qiita.com"
          ],
          "group": "qiita"
        },
        {
          "url_pattern": "/users/{id}",
          "encoding": "json",
          "host": [
            "https://api.github.com"
          ],
          "group": "github"
        }
      ]
    }
  ]
}

KrakenD サービスの立ち上げ

最近のソフトウェアらしく Docker Hub でイメージが提供されているので、それを試してみましょう。

$ docker pull devopsfaith/krakend
$ docker run -p 8080:8080 -v $PWD:/etc/krakend/ devopsfaith/krakend

簡単ですね。ではリクエストを送ってみます。 HTTP クライアントは HTTPie です。

$ http localhost:8000/users/tchssk

{
    "github": {
        "avatar_url": "https://avatars3.githubusercontent.com/u/12257128?v=4",
        "bio": null,
        "blog": "https://tchssk.github.io",
        "company": null,
        "created_at": "2015-05-05T15:43:31Z",
        "email": null,
        "events_url": "https://api.github.com/users/tchssk/events{/privacy}",
        "followers": 6,
        "followers_url": "https://api.github.com/users/tchssk/followers",
        "following": 3,
        "following_url": "https://api.github.com/users/tchssk/following{/other_user}",
        "gists_url": "https://api.github.com/users/tchssk/gists{/gist_id}",
        "gravatar_id": "",
        "hireable": null,
        "html_url": "https://github.com/tchssk",
        "id": 12257128,
        "location": "Tokyo, Japan",
        "login": "tchssk",
        "name": "Taichi Sasaki",
        "organizations_url": "https://api.github.com/users/tchssk/orgs",
        "public_gists": 0,
        "public_repos": 33,
        "received_events_url": "https://api.github.com/users/tchssk/received_events",
        "repos_url": "https://api.github.com/users/tchssk/repos",
        "site_admin": false,
        "starred_url": "https://api.github.com/users/tchssk/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/tchssk/subscriptions",
        "type": "User",
        "updated_at": "2017-12-05T10:08:00Z",
        "url": "https://api.github.com/users/tchssk"
    },
    "qiita": {
        "description": "A web developer who loves Ginger Ale !!",
        "facebook_id": "",
        "followees_count": 0,
        "followers_count": 2,
        "github_login_name": "tchssk",
        "id": "tchssk",
        "items_count": 11,
        "linkedin_id": "",
        "location": "Tokyo, Japan",
        "name": "Taichi Sasaki",
        "organization": "",
        "permanent_id": 83093,
        "profile_image_url": "https://qiita-image-store.s3.amazonaws.com/0/83093/profile-images/1473702659",
        "twitter_screen_name": "tchssk",
        "website_url": "https://tchssk.github.io"
    }
}

ちゃんと "github" と "qiita" にそれぞれのレスポンスが入った形で返ってきました。

レスポンスの操作

KrakenD は、バックエンドから受け取った情報をクライアントにレスポンスする前に以下の方法で操作することができます。

Merging
Filtering
Grouping
Mapping
Capturing

では順番に試してみましょう。

Merging

複数のバックエンドからのレスポンスをマージすることができます。

  "endpoints": [
    {
      "endpoint": "/users/{id}",
      "method": "GET",
      "concurrent_calls": 1,
      "client_max_rate": 0,
      "querystring_params": [],
      "backend": [
        {
          "url_pattern": "/api/v2/users/{id}",
          "encoding": "json",
          "host": [
            "https://qiita.com"
          ]
        },
        {
          "url_pattern": "/users/{id}",
          "encoding": "json",
          "host": [
            "https://api.github.com"
          ]
        }
      ]
    }
  ]

上記のように設定すると GitHub と Qiita からのレスポンスがマージされます。

{
  "avatar_url": "https://avatars3.githubusercontent.com/u/12257128?v=4",
  "bio": null,
  "blog": "https://tchssk.github.io",
  "company": null,
  "created_at": "2015-05-05T15:43:31Z",
  "description": "A web developer who loves Ginger Ale !!",
  "email": null,
  "events_url": "https://api.github.com/users/tchssk/events{/privacy}",
  "facebook_id": "",
  "followees_count": 0,
  "followers": 6,
  "followers_count": 2,
  "followers_url": "https://api.github.com/users/tchssk/followers",
  "following": 3,
  "following_url": "https://api.github.com/users/tchssk/following{/other_user}",
  "gists_url": "https://api.github.com/users/tchssk/gists{/gist_id}",
  "github_login_name": "tchssk",
  "gravatar_id": "",
  "hireable": null,
  "html_url": "https://github.com/tchssk",
  "id": 12257128,
  "items_count": 11,
  "linkedin_id": "",
  "location": "Tokyo, Japan",
  "login": "tchssk",
  "name": "Taichi Sasaki",
  "organization": "",
  "organizations_url": "https://api.github.com/users/tchssk/orgs",
  "permanent_id": 83093,
  "profile_image_url": "https://qiita-image-store.s3.amazonaws.com/0/83093/profile-images/1473702659",
  "public_gists": 0,
  "public_repos": 33,
  "received_events_url": "https://api.github.com/users/tchssk/received_events",
  "repos_url": "https://api.github.com/users/tchssk/repos",
  "site_admin": false,
  "starred_url": "https://api.github.com/users/tchssk/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/tchssk/subscriptions",
  "twitter_screen_name": "tchssk",
  "type": "User",
  "updated_at": "2017-12-05T10:08:00Z",
  "url": "https://api.github.com/users/tchssk",
  "website_url": "https://tchssk.github.io"
}

フィールド名が重複している場合はどちらか片方のレスポンスが出力されるようです。

Filtering

2 種類のフィルタリングが利用できます。

Blacklisting

blacklist に指定されたフィールドを除外することができます。

  "endpoints": [
    {
      "endpoint": "/users/{id}",
      "method": "GET",
      "concurrent_calls": 1,
      "client_max_rate": 0,
      "querystring_params": [],
      "backend": [
        {
          "url_pattern": "/users/{id}",
          "encoding": "json",
          "host": [
            "https://api.github.com"
          ],
          "blacklist": [
            "avatar_url",
            "bio",
            "blog",
            "company"
          ]
        }
      ]
    }
  ]

上記のように設定すると avatar_url、 bio、 blog、 company が除外されます。

{
  "created_at": "2015-05-05T15:43:31Z",
  "email": null,
  "events_url": "https://api.github.com/users/tchssk/events{/privacy}",
  "followers": 6,
  "followers_url": "https://api.github.com/users/tchssk/followers",
  "following": 3,
  "following_url": "https://api.github.com/users/tchssk/following{/other_user}",
  "gists_url": "https://api.github.com/users/tchssk/gists{/gist_id}",
  "gravatar_id": "",
  "hireable": null,
  "html_url": "https://github.com/tchssk",
  "id": 12257128,
  "location": "Tokyo, Japan",
  "login": "tchssk",
  "name": "Taichi Sasaki",
  "organizations_url": "https://api.github.com/users/tchssk/orgs",
  "public_gists": 0,
  "public_repos": 33,
  "received_events_url": "https://api.github.com/users/tchssk/received_events",
  "repos_url": "https://api.github.com/users/tchssk/repos",
  "site_admin": false,
  "starred_url": "https://api.github.com/users/tchssk/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/tchssk/subscriptions",
  "type": "User",
  "updated_at": "2017-12-05T10:08:00Z",
  "url": "https://api.github.com/users/tchssk"
}

Whitelisting

whitelist に指定されたフィールドだけを出力することができます。

  "endpoints": [
    {
      "endpoint": "/users/{id}",
      "method": "GET",
      "concurrent_calls": 1,
      "client_max_rate": 0,
      "querystring_params": [],
      "backend": [
        {
          "url_pattern": "/users/{id}",
          "encoding": "json",
          "host": [
            "https://api.github.com"
          ],
          "whitelist": [
            "subscriptions_url",
            "type",
            "updated_at",
            "url"
          ]
        }
      ]
    }
  ]

上記のように設定すると subscriptions_url、 type、 updated_at、 url だけが出力されます。

{
  "subscriptions_url": "https://api.github.com/users/tchssk/subscriptions",
  "type": "User",
  "updated_at": "2017-12-05T10:08:00Z",
  "url": "https://api.github.com/users/tchssk"
}

Grouping

グルーピングを行うことができます。冒頭で扱った GitHub と Qiita の例が正にこれですね。

Mapping

バックエンドからのレスポンスを別名のフィールドにマッピングすることができます。

  "endpoints": [
    {
      "endpoint": "/users/{id}",
      "method": "GET",
      "concurrent_calls": 1,
      "client_max_rate": 0,
      "querystring_params": [],
      "backend": [
        {
          "url_pattern": "/users/{id}",
          "encoding": "json",
          "host": [
            "https://api.github.com"
          ],
          "mapping": {
            "blog": "personal_blog"
          }
        }
      ]
    }
  ]

上記のように設定すると blog が personal_blog にマッピングされます。

{
  "avatar_url": "https://avatars3.githubusercontent.com/u/12257128?v=4",
  "bio": null,
  "company": null,
  "created_at": "2015-05-05T15:43:31Z",
  "email": null,
  "events_url": "https://api.github.com/users/tchssk/events{/privacy}",
  "followers": 6,
  "followers_url": "https://api.github.com/users/tchssk/followers",
  "following": 3,
  "following_url": "https://api.github.com/users/tchssk/following{/other_user}",
  "gists_url": "https://api.github.com/users/tchssk/gists{/gist_id}",
  "gravatar_id": "",
  "hireable": null,
  "html_url": "https://github.com/tchssk",
  "id": 12257128,
  "location": "Tokyo, Japan",
  "login": "tchssk",
  "name": "Taichi Sasaki",
  "organizations_url": "https://api.github.com/users/tchssk/orgs",
  "personal_blog": "https://tchssk.github.io",
  "public_gists": 0,
  "public_repos": 33,
  "received_events_url": "https://api.github.com/users/tchssk/received_events",
  "repos_url": "https://api.github.com/users/tchssk/repos",
  "site_admin": false,
  "starred_url": "https://api.github.com/users/tchssk/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/tchssk/subscriptions",
  "type": "User",
  "updated_at": "2017-12-05T10:08:00Z",
  "url": "https://api.github.com/users/tchssk"
}

Caputuring

特定フィールドのオブジェクトだけを抽出することができます。ここでは GitHub API のリポジトリを取得するエンドポイントで試してみます。

$ http https://api.github.com/repos/golang/go

{
    "id": 23096959,
    "name": "go",
    "full_name": "golang/go",
    "owner": {
        "login": "golang",
        "id": 4314092,
        "avatar_url": "https://avatars3.githubusercontent.com/u/4314092?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/golang",
        "html_url": "https://github.com/golang",
        "followers_url": "https://api.github.com/users/golang/followers",
        "following_url": "https://api.github.com/users/golang/following{/other_user}",
        "gists_url": "https://api.github.com/users/golang/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/golang/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/golang/subscriptions",
        "organizations_url": "https://api.github.com/users/golang/orgs",
        "repos_url": "https://api.github.com/users/golang/repos",
        "events_url": "https://api.github.com/users/golang/events{/privacy}",
        "received_events_url": "https://api.github.com/users/golang/received_events",
        "type": "Organization",
        "site_admin": false
    },
    "private": false,
    "html_url": "https://github.com/golang/go",
    "description": "The Go programming language",
    "fork": false,
    "url": "https://api.github.com/repos/golang/go",
    "forks_url": "https://api.github.com/repos/golang/go/forks",
    "keys_url": "https://api.github.com/repos/golang/go/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/golang/go/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/golang/go/teams",
    "hooks_url": "https://api.github.com/repos/golang/go/hooks",
    "issue_events_url": "https://api.github.com/repos/golang/go/issues/events{/number}",
    "events_url": "https://api.github.com/repos/golang/go/events",
    "assignees_url": "https://api.github.com/repos/golang/go/assignees{/user}",
    "branches_url": "https://api.github.com/repos/golang/go/branches{/branch}",
    "tags_url": "https://api.github.com/repos/golang/go/tags",
    "blobs_url": "https://api.github.com/repos/golang/go/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/golang/go/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/golang/go/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/golang/go/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/golang/go/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/golang/go/languages",
    "stargazers_url": "https://api.github.com/repos/golang/go/stargazers",
    "contributors_url": "https://api.github.com/repos/golang/go/contributors",
    "subscribers_url": "https://api.github.com/repos/golang/go/subscribers",
    "subscription_url": "https://api.github.com/repos/golang/go/subscription",
    "commits_url": "https://api.github.com/repos/golang/go/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/golang/go/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/golang/go/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/golang/go/issues/comments{/number}",
    "contents_url": "https://api.github.com/repos/golang/go/contents/{+path}",
    "compare_url": "https://api.github.com/repos/golang/go/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/golang/go/merges",
    "archive_url": "https://api.github.com/repos/golang/go/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/golang/go/downloads",
    "issues_url": "https://api.github.com/repos/golang/go/issues{/number}",
    "pulls_url": "https://api.github.com/repos/golang/go/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/golang/go/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/golang/go/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/golang/go/labels{/name}",
    "releases_url": "https://api.github.com/repos/golang/go/releases{/id}",
    "deployments_url": "https://api.github.com/repos/golang/go/deployments",
    "created_at": "2014-08-19T04:33:40Z",
    "updated_at": "2017-12-07T12:12:01Z",
    "pushed_at": "2017-12-07T05:10:23Z",
    "git_url": "git://github.com/golang/go.git",
    "ssh_url": "git@github.com:golang/go.git",
    "clone_url": "https://github.com/golang/go.git",
    "svn_url": "https://github.com/golang/go",
    "homepage": "https://golang.org",
    "size": 166440,
    "stargazers_count": 35170,
    "watchers_count": 35170,
    "language": "Go",
    "has_issues": true,
    "has_projects": false,
    "has_downloads": true,
    "has_wiki": true,
    "has_pages": false,
    "forks_count": 4781,
    "mirror_url": null,
    "archived": false,
    "open_issues_count": 3158,
    "license": {
        "key": "bsd-3-clause",
        "name": "BSD 3-clause \"New\" or \"Revised\" License",
        "spdx_id": "BSD-3-Clause",
        "url": "https://api.github.com/licenses/bsd-3-clause"
    },
    "forks": 4781,
    "open_issues": 3158,
    "watchers": 35170,
    "default_branch": "master",
    "organization": {
        "login": "golang",
        "id": 4314092,
        "avatar_url": "https://avatars3.githubusercontent.com/u/4314092?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/golang",
        "html_url": "https://github.com/golang",
        "followers_url": "https://api.github.com/users/golang/followers",
        "following_url": "https://api.github.com/users/golang/following{/other_user}",
        "gists_url": "https://api.github.com/users/golang/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/golang/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/golang/subscriptions",
        "organizations_url": "https://api.github.com/users/golang/orgs",
        "repos_url": "https://api.github.com/users/golang/repos",
        "events_url": "https://api.github.com/users/golang/events{/privacy}",
        "received_events_url": "https://api.github.com/users/golang/received_events",
        "type": "Organization",
        "site_admin": false
    },
    "network_count": 4781,
    "subscribers_count": 2583
}

Go のリポジトリですね。これを KrakenD を通して取得します。

  "endpoints": [
    {
      "endpoint": "/repos/{owner}/{repo}",
      "method": "GET",
      "concurrent_calls": 1,
      "client_max_rate": 0,
      "querystring_params": [],
      "backend": [
        {
          "url_pattern": "/repos/{owner}/{repo}",
          "encoding": "json",
          "host": [
            "https://api.github.com"
          ],
          "target": "owner"
        }
      ]
    }
  ]

上記のように設定すると owner のオブジェクトだけが抽出されます。

{
  "avatar_url": "https://avatars3.githubusercontent.com/u/4314092?v=4",
  "events_url": "https://api.github.com/users/golang/events{/privacy}",
  "followers_url": "https://api.github.com/users/golang/followers",
  "following_url": "https://api.github.com/users/golang/following{/other_user}",
  "gists_url": "https://api.github.com/users/golang/gists{/gist_id}",
  "gravatar_id": "",
  "html_url": "https://github.com/golang",
  "id": 4314092,
  "login": "golang",
  "organizations_url": "https://api.github.com/users/golang/orgs",
  "received_events_url": "https://api.github.com/users/golang/received_events",
  "repos_url": "https://api.github.com/users/golang/repos",
  "site_admin": false,
  "starred_url": "https://api.github.com/users/golang/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/golang/subscriptions",
  "type": "Organization",
  "url": "https://api.github.com/users/golang"
}

その他の機能

今回は試しませんでしたが、一般的な API Gateway にあるような以下の機能も備えているようです。

複数のバックエンドへリクエストを分散する Load balancing
一定時間あたりのリクエスト数を制限する Rate limits
障害時などにバックエンドへのリクエストを遮断する Circuit breaker

性能

ベンチマークによると秒間処理リクエスト数は 3500 程度で、 OSS の API Gateway として有名な Kong の約 2 倍だそうです。

f:id:tchssk:20171208122904p:plain

利用プラン

公式ページに書かれている以下の 3 種類のプランがあるようです。

Open Source
Free
Enterprise

KrakenD のコア機能は OSS として GitHub で公開されていて、好きな Web framework と組み合わせて使えるようになっています。それがおそらく Open Source プラン。 DockerHub のイメージや RPM などを介してビルド済みのものを利用するのが Free プランでしょう。 Enterprise には mailto: のボタンが置かれているだけなので、まだ本格的なマネタイズは行なっていないのかもしれません。

今回は少し触ってみた程度ですが、 API Gateway が必要になったときはもう少し詳しく検証して利用を検討してみたいと思いました。