Preparing for distributed system failures using akka #ScalaMatsuri

2. Copyright © 2017 TIS Inc. All rights reserved. 2 Who am I? TIS Inc. provides “Reactive Systems Consulting Service” @yugolf https://twitter.com/okapies/status/781439220330164225 - support PoC projects - review designs - review codes etc リアクティブシステムのコンサルティングサービスをやっています

7. Copyright © 2017 TIS Inc. All rights reserved. 7 - increasing the number of server means increasing failure points - face new enemies "network" サーバが増えれば障害点も増えるネットワークという新たな敵の出現 building distributed system is not easy

12. Copyright © 2017 TIS Inc. All rights reserved. Akka Actor 12 participant Actor processes messages in order of arrival $30 host アクターはメッセージを到達順に処理シンプルに⾮同期処理を実装可能 $10 $10 $10 status $10 $10 $10 mailbox

13. Copyright © 2017 TIS Inc. All rights reserved. Supervisor Hierarchy 13 let it crash スーパーバイザーが⼦アクターを監視し障害制御などを⾏う supervisor child actorchild actor supervise signal failure - restart - resume - stop - escalate

17. Copyright © 2017 TIS Inc. All rights reserved. timeout configuration 17 import akka.pattern.ask  import akka.util.Timeout  import system.dispatcher    implicit val timeout = Timeout(5 seconds)   val response = kitchen ? KitchenActor.DripCoffee(count)    response.mapTo[OrderCompleted] onComplete {  case Success(result) =>  log.info(s"success: ${result.message}")  case Failure(e: AskTimeoutException) =>  log.info(s"failure: ${e.getMessage}")  case Failure(t) =>  log.info(s"failure: ${t.getMessage}")  } askのタイムアウト設定

21. Copyright © 2017 TIS Inc. All rights reserved. implements of ask pattern 1/2 21 def ?(message: Any)(implicit timeout: Timeout, sender: ActorRef = Actor.noSender): Future[Any] = internalAsk(message, timeout, sender) private[pattern] def internalAsk(message: Any, timeout: Timeout, sender: ActorRef): Future[Any] = actorSel.anchor match {  case ref: InternalActorRef ⇒  if (timeout.duration.length <= 0)  Future.failed[Any](  new IllegalArgumentException(s"""Timeout length must not be negative, question not sent to [$actorSel]. Sender[$sender] sent the message of type "$ {message.getClass.getName}"."""))  else {  val a = PromiseActorRef(ref.provider, timeout, targetName = actorSel, message.getClass.getName, sender)  actorSel.tell(message, a)  a.result.future  }  case _ ⇒ Future.failed[Any](new IllegalArgumentException(s"""Unsupported recipient ActorRef type, question not sent to [$actorSel]. Sender[$sender] sent the message of type "${message.getClass.getName}"."""))  } ? internalAsk

22. Copyright © 2017 TIS Inc. All rights reserved. 22 akka.pattern.PromiseActorRef def apply(provider: ActorRefProvider, timeout: Timeout, targetName: Any, messageClassName: String, sender: ActorRef = Actor.noSender): PromiseActorRef = {  val result = Promise[Any]()  val scheduler = provider.guardian.underlying.system.scheduler  val a = new PromiseActorRef(provider, result, messageClassName)  implicit val ec = a.internalCallingThreadExecutionContext  val f = scheduler.scheduleOnce(timeout.duration) {  result tryComplete Failure(  new AskTimeoutException(s"""Ask timed out on [$targetName] after [${timeout.duration.toMillis} ms]. Sender[$sender] sent message of type "$ {a.messageClassName}"."""))  }  result.future onComplete { _ ⇒ try a.stop() finally f.cancel() }  a  } スケジューラを設定し時間がくれば AskTimeoutException送信 implements of ask pattern 2/2

27. Copyright © 2017 TIS Inc. All rights reserved. what is circuit breaker 27 https://martinfowler.com/bliki/CircuitBreaker.html ⼀定回数の失敗を繰り返すと接続を抑⽌ Once the failures reach a certain threshold, the circuit breaker trips

30. Copyright © 2017 TIS Inc. All rights reserved. apply circuit breaker: implement 30 val breaker =  new CircuitBreaker(  context.system.scheduler,  maxFailures = 5,  callTimeout = 10.seconds,  resetTimeout = 1.minute).onOpen(notifyMeOnOpen()) http://doc.akka.io/docs/akka/current/common/circuitbreaker.html def receive = {  case "dangerousCall" =>  breaker.withCircuitBreaker(Future(dangerousCall)) pipeTo sender()  } ５回失敗するとOpenになり、１分間はメッセージを送信させない

33. Copyright © 2017 TIS Inc. All rights reserved. CAP trade-off 33 return old information vs don't return anything just do my work vs need synchronize with others cache push - read - write 古い情報を返してもよいか？他者との同期なしで問題ないか？

38. Copyright © 2017 TIS Inc. All rights reserved. isolating the blocking calls to actors 38 val blockingActor = context.actorOf(Props[BlockingActor].  withDispatcher(“blocking-actor-dispatcher”),  "blocking-actor")    class BlockingActor extends Actor {  def receive = {  case GetCustomer(id) =>  // calling database  …  }  } ブロッキングコードはアクターごと分離してリソースを共有しない

45. Copyright © 2017 TIS Inc. All rights reserved. hardware will fail 45 If there are 365 machines failing once a year, one machine will fail a day Wouldn't a machine break even when it's hosted on the cloud? 1年に1回故障するマシンが365台あれば平均毎⽇1台故障する

51. Copyright © 2017 TIS Inc. All rights reserved. BackoffSupervisor 51 http://doc.akka.io/docs/akka/current/general/supervision.html#Delayed_restarts_with_the_BackoffSupervisor_pattern 3秒後、6秒後、12秒後、…の間隔でスタートを試みる   val childProps = Props(classOf[EchoActor])    val supervisor = BackoffSupervisor.props(  Backoff.onStop(  childProps,  childName = "myEcho",  minBackoff = 3.seconds,  maxBackoff = 30.seconds,  randomFactor = 0.2 // adds 20% "noise" to vary the intervals slightly  ))    system.actorOf(supervisor, name = "echoSupervisor") increasing intervals of 3, 6, 12, ...

55. Copyright © 2017 TIS Inc. All rights reserved. strategy 1/4: Static Quorum 55 quorum-size = 3 クラスタ内のノード数が⼀定数以上の場合⽣存 node2 node1 node4 node3 node5 Which can survive? - If the number of nodes is quorum-size or more

58. Copyright © 2017 TIS Inc. All rights reserved. strategy 4/4: Keep Referee 58 特定のノードが含まれている場合に⽣存 node2 node4 node3 node5 node1 Which can survive? - If contain the given referee node address = "akka.tcp://system@node1:port"

59. Copyright © 2017 TIS Inc. All rights reserved. 59 SBR is included in Lightbend Reactive Platform https://github.com/TanUkkii007/akka-cluster-custom-downing http://doc.akka.io/docs/akka/rp-current/scala/split-brain-resolver.html Lightbend Reactive Platform akka-cluster-custom-downing SBRはLightbend Reactive Platformで提供されています

61. Copyright © 2017 TIS Inc. All rights reserved. Failed to receive ack message 61 Order(coffee,1) Order(coffee,1) ackを受信できずメッセージを再送すると２重注⽂してしまう coffee please! becomes a duplicate order by resending the message

62. Copyright © 2017 TIS Inc. All rights reserved. idempotence 62 メッセージを複数回受信しても問題ないように冪等な設計で⼀貫性を維持 Order(id1, coffee, 1) Order(id1, coffee, 1) coffee, please! applying it multiple times is not harmful

68. THANK YOU