Prometheus meets Consul  Satoshi SUZUKI, @studio3104 Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 1
Who? Satoshi SUZUKI • Twitter: @studio3104 • GitHub: studio3104 • Works for: LINE Corporation Consul Casual Talks at LINE ...
Agenda • Consul ACL System • What's Prometheus? • Integrating with Prometheus Consul Casual Talks at LINE Corp. 27F Cafe. ...
Consul ACL System Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 4
Consul ACL System • Required to restrict running "consul exec" and "consul event" • To avoid unexpected accidents • We use...
Consul ACL System • If it's not restricted... $ consul exec 'rm -f /tmp/somethingimportant' ==> hostname1: finished with e...
Consul ACL System • If it's not restricted... $ consul exec 'rm -f /tmp/somethingimportant' ==> hostname1: finished with e...
Consul ACL System • So we need to restrict it { "server": true, "datacenter": "dc1", "start_join": ["consul-server-1"], "d...
Cosul ACL System • And then we get... $ consul exec 'rm -f /tmp/somethingimportant' Failed to create job file: Unexpected ...
Cosul ACL System • And then we get... $ consul exec 'rm -f /tmp/somethingimportant' Failed to create job file: Unexpected ...
Consul ACL System • acl_datacenter • Designates the datacenter which is authoritative for ACL information • acl_default_po...
Anonymous Token Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 12
Anonymous Token • Consul agent usually operate by the "Anonymous" permission implicitly • When the ACL is enabled • When t...
Anonymous Token • When acl_default_policy is set as "deny" • anonymous can't do anything • Of course exec and event • Cann...
Anonymous Token • For example, it's a setting of one of node { "services": [ { "name": "nginx", "address": "127.0.0.1", "p...
Anonymous Token • But it's not discovered by catalog API $ curl localhost:8500/v1/catalog/services | jq . { "consul": [] }...
Anonymous Token • So it's better to set the settings for anonymous token $ cat anonymous.json { "ID": "anonymous", "Type":...
Anonymous Token • Apply the json to acl API using master token $ curl -X PUT http://localhost:8500/v1/acl/update > ?token=...
Anonymous Token • or apply an ACL setting via WEB UI Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 19
Anonymous Token • Finally service discovery has enabled $ curl localhost:8500/v1/catalog/services| jq . { "consul": [], "n...
Anonymous Token • Finally service discovery has enabled $ curl localhost:8500/v1/catalog/service/nginx | jq . [ { "Node": ...
Enables "consul exec" Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 22
Enables "consul exec" • Apply write permissions to the token you'd like to use • _rexec/ key prefix • _rexec event Consul C...
Enables "consul exec" • For "consul exec 'command'" { "key": { "_rexec/": {"policy": "write"} }, "event": { "_rexec": {"po...
Enables "consul exec" • For "consul exec -prefix='_rexec/hoge' 'command'" { "key": { "_rexec/hoge/": {"policy": "write"} }...
Enables "consul exec" • Documentation mismatches     https://github.com/hashicorp/consul/issues/2187 Consul Casual Talks a...
What's Prometheus? Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 27
What's Prometheus?   • One of Monitoring system • Pull type architecture • Multi dimentional data model • Powerful query l...
What's Prometheus? Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 29
Integrating with Prometheus Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 30
Integrating with Prometheus • Prometheus has the feature to integrate with Consul natively    • consul_sd_config • Consul S...
Integrating with Prometheus • Example settings scrape_configs: - job_name: 'consul_sd' consul_sd_configs: - server: 'local...
Integrating with Prometheus • It's better to create a token dedicated to Prometheus $ curl localhost:8500/v1/acl/info/ > 6...
Integrating with Prometheus • It can retrieve only the services which are prefixed "prometheus-" "Rules": "{n "service": {n...
Integrating with Prometheus • But unfortunately it can't suppress "consul" itself even if targets are restricted by the to...
Integrating with Prometheus • So we need to handle them by "relabel_configs" relabel_configs: - source_labels: ['__meta_con...
Integrating with Prometheus • When are targets (exporters) removed from Prometheus? • When the consul agent changes state ...
Integrating with Prometheus • When aren't targets (exporters) left in Prometheus • When the Consul agent disconnects abnor...
Questions?  Satoshi SUZUKI, @studio3104 Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 39
Upcoming SlideShare
Loading in …5
×

Prometheus meets Consul -- Consul Casual Talks

87 views
 82 views

Published on

How to integrate Consul with Prometheus

Published in: Technology
0 Comments
1 Like
Statistics
Notes
no profile picture user

  • Be the first to comment

No Downloads
Views
Total views
87
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Prometheus meets Consul -- Consul Casual Talks

  1. 1. Prometheus meets Consul  Satoshi SUZUKI, @studio3104 Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 1
  2. 2. Who? Satoshi SUZUKI • Twitter: @studio3104 • GitHub: studio3104 • Works for: LINE Corporation Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 2
  3. 3. Agenda • Consul ACL System • What's Prometheus? • Integrating with Prometheus Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 3
  4. 4. Consul ACL System Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 4
  5. 5. Consul ACL System • Required to restrict running "consul exec" and "consul event" • To avoid unexpected accidents • We use consul straddling between various services Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 5
  6. 6. Consul ACL System • If it's not restricted... $ consul exec 'rm -f /tmp/somethingimportant' ==> hostname1: finished with exit code 0 ==> hostname2: finished with exit code 0 2 / 2 node(s) completed / acknowledged Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 6
  7. 7. Consul ACL System • If it's not restricted... $ consul exec 'rm -f /tmp/somethingimportant' ==> hostname1: finished with exit code 0 ==> hostname2: finished with exit code 0 2 / 2 node(s) completed / acknowledged Dangerous!! ! Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 7
  8. 8. Consul ACL System • So we need to restrict it { "server": true, "datacenter": "dc1", "start_join": ["consul-server-1"], "data_dir": "/usr/local/consul/data", "ui_dir": "/usr/local/consul/ui", "acl_datacenter": "dc1", "acl_default_policy": "deny", "acl_master_token": "7e2bea9c-87af-457d-b8e0-ff254819c234" } Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 8
  9. 9. Cosul ACL System • And then we get... $ consul exec 'rm -f /tmp/somethingimportant' Failed to create job file: Unexpected response code: 403 (Permission denied) Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 9
  10. 10. Cosul ACL System • And then we get... $ consul exec 'rm -f /tmp/somethingimportant' Failed to create job file: Unexpected response code: 403 (Permission denied) Safety!! ! Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 10
  11. 11. Consul ACL System • acl_datacenter • Designates the datacenter which is authoritative for ACL information • acl_default_policy • ACLs are a whitelist if it's set deny • acl_master_token • Token to allow operators to bootstrap the ACL system Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 11
  12. 12. Anonymous Token Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 12
  13. 13. Anonymous Token • Consul agent usually operate by the "Anonymous" permission implicitly • When the ACL is enabled • When the "acl_token" isn't described explicitly Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 13
  14. 14. Anonymous Token • When acl_default_policy is set as "deny" • anonymous can't do anything • Of course exec and event • Cannot register any services • Cannot discover any services • And everything... Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 14
  15. 15. Anonymous Token • For example, it's a setting of one of node { "services": [ { "name": "nginx", "address": "127.0.0.1", "port": 80 } ] } Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 15
  16. 16. Anonymous Token • But it's not discovered by catalog API $ curl localhost:8500/v1/catalog/services | jq . { "consul": [] } Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 16
  17. 17. Anonymous Token • So it's better to set the settings for anonymous token $ cat anonymous.json { "ID": "anonymous", "Type": "client", "Rules": "{ "service": { "": {"policy": "write"} } }" } Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 17
  18. 18. Anonymous Token • Apply the json to acl API using master token $ curl -X PUT http://localhost:8500/v1/acl/update > ?token=8f246b77-f3e1-ff88-5b48-8ec93abf3e05 > -d @./anonymous.json | jq . { "ID": "anonymous" } Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 18
  19. 19. Anonymous Token • or apply an ACL setting via WEB UI Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 19
  20. 20. Anonymous Token • Finally service discovery has enabled $ curl localhost:8500/v1/catalog/services| jq . { "consul": [], "nginx": [] } Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 20
  21. 21. Anonymous Token • Finally service discovery has enabled $ curl localhost:8500/v1/catalog/service/nginx | jq . [ { "Node": "hostname2", "Address": "172.31.0.1", "ServiceID": "nginx", "ServiceName": "nginx", "ServiceTags": [], "ServiceAddress": "127.0.0.1", "ServicePort": 80, "ServiceEnableTagOverride": false, "CreateIndex": 271, "ModifyIndex": 271 } ] Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 21
  22. 22. Enables "consul exec" Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 22
  23. 23. Enables "consul exec" • Apply write permissions to the token you'd like to use • _rexec/ key prefix • _rexec event Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 23
  24. 24. Enables "consul exec" • For "consul exec 'command'" { "key": { "_rexec/": {"policy": "write"} }, "event": { "_rexec": {"policy": "write"} } } Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 24
  25. 25. Enables "consul exec" • For "consul exec -prefix='_rexec/hoge' 'command'" { "key": { "_rexec/hoge/": {"policy": "write"} }, "event": { "_rexec": {"policy": "write"} } } Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 25
  26. 26. Enables "consul exec" • Documentation mismatches     https://github.com/hashicorp/consul/issues/2187 Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 26
  27. 27. What's Prometheus? Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 27
  28. 28. What's Prometheus?   • One of Monitoring system • Pull type architecture • Multi dimentional data model • Powerful query language • Alerting http://www.slideshare.net/wyukawa/prometheus-casual-talk1-63042281 Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 28
  29. 29. What's Prometheus? Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 29
  30. 30. Integrating with Prometheus Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 30
  31. 31. Integrating with Prometheus • Prometheus has the feature to integrate with Consul natively    • consul_sd_config • Consul SD configurations allow retrieving scrape targets from Consul's Catalog API https://prometheus.io/docs/operating/configuration/#<consul_sd_config> Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 31
  32. 32. Integrating with Prometheus • Example settings scrape_configs: - job_name: 'consul_sd' consul_sd_configs: - server: 'localhost:8500' token: '63c7ce81-1e3b-9081-dff5-d91ede002ceb' relabel_configs: - source_labels: ['__meta_consul_service'] regex: prometheus-(.+) replacement: '${1}' target_label: 'job' - source_labels: ['__meta_consul_service'] regex: consul action: drop - source_labels: ['__meta_consul_node'] target_label: 'node' Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 32
  33. 33. Integrating with Prometheus • It's better to create a token dedicated to Prometheus $ curl localhost:8500/v1/acl/info/ > 63c7ce81-1e3b-9081-dff5-d91ede002ceb > ?token=8f246b77-f3e1-ff88-5b48-8ec93abf3e05 | jq . [ { "ID": "63c7ce81-1e3b-9081-dff5-d91ede002ceb", "Name": "Prometheus", "Type": "client", "Rules": "{n "service": {n "prometheus-": {"policy": "read"}n }n}", "CreateIndex": 6257, "ModifyIndex": 6257 } ] Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 33
  34. 34. Integrating with Prometheus • It can retrieve only the services which are prefixed "prometheus-" "Rules": "{n "service": {n "prometheus-": {"policy": "read"}n }n}", • Normally Prometheus tries to take in every services which is discovered by Consul service discovery • Of course it can be specified statically but it'll lose most of the advantages to integrating with Consul Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 34
  35. 35. Integrating with Prometheus • But unfortunately it can't suppress "consul" itself even if targets are restricted by the token... $ curl localhost:8500/v1/catalog/services > ?token=63c7ce81-1e3b-9081-dff5-d91ede002ceb | jq . { "consul": [], "prometheus-node": [] } Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 35
  36. 36. Integrating with Prometheus • So we need to handle them by "relabel_configs" relabel_configs: - source_labels: ['__meta_consul_service'] regex: prometheus-(.+) replacement: '${1}' target_label: 'job' - source_labels: ['__meta_consul_service'] regex: consul action: drop Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 36
  37. 37. Integrating with Prometheus • When are targets (exporters) removed from Prometheus? • When the consul agent changes state to "left" • When the Prometheus exporter settings are removed from Consul SD Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 37
  38. 38. Integrating with Prometheus • When aren't targets (exporters) left in Prometheus • When the Consul agent disconnects abnormally ("failed" state) Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 38
  39. 39. Questions?  Satoshi SUZUKI, @studio3104 Consul Casual Talks at LINE Corp. 27F Cafe. Aug 1, 2016 39

×