Microservices in the Apache Kafka Ecosystem

Microservices in the Apache Kafka Ecosystem

1. 1. Ben Stopford, Engineer, Confluent
2. 2. • Event Driven Microservices • The toolset: Kafka, KStreams, Connect • 10 Principals for Streaming Services What we’ll cover
3. 3. Microservices in the Kafka Ecosystem
4. 4. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Microservices
5. 5. So why Services?
6. 6. The Monolith Can we do reuse, encapsulation?
7. 7. The Monolith What happens when we grow?
8. 8. Companies are inevitably a collection of applications They must work together to some degree
9. 9. Inverse Conway Maneuver The 'Inverse Conway Maneuver' recommends evolving your team and organizational structure to promote your desired architecture. Org Structure Software Architecture
10. 10. Eschew shared, mutable state
11. 11. Service based approaches separate state
12. 12. But state must inevitably be shared between services
13. 13. Use a toolkit that embraces decentralisation
14. 14. Some simple patterns of distributed systems
15. 15. Request / Response
16. 16. When do we need Request Response? Looking things up What is in my shopping basket?
17. 17. Event Driven Async / Fire and Forget / Brokered
18. 18. Businesses are often modeled as a sequence events. Add to cart Buy Make Payment Update Stock Email Conf Package Item Dispatch Fraud
19. 19. When do we need Event Driven?
20. 20. SOA / Microservices Message Broker Event Based Request/Response
21. 21. Hybrids Event- Based Request/ Response
22. 22. Request Response Buy Items Order Service Stock Service Ful- filment Service Fraud Det- ection Mmm… New IPad REST etc UI Service
23. 23. Event Driven Buy Items Order Service Stock Service Ful- filment Service Fraud Det- ection Message Broker Mmm… New IPad UI Service
24. 24. Hybrid Buy Items Order Service Stock Service Ful- filment Service Fraud Det- ection Message Broker Mmm… New IPad REST UI Service
25. 25. As software engineers we are inevitably affected by the tools we surround ourselves with. Languages, frameworks, even processes all act to shape the software we build.
26. 26. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service The tools we choose have a big effect on our architecture GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Event Driven Request Response
27. 27. Kafka is well suited to Event Driven Architectures It will lead you down that path
28. 28. The Tool Set
29. 29. a Distributed Log
30. 30. What is a Distributed Log?
31. 31. Shard on the way in Producing Services Kafka Consuming Services
32. 32. Each shard is a queue Producing Services Kafka Consuming Services
33. 33. Consumers share load Producing Services Kafka Consuming Services
34. 34. Reduces to a globally ordered queue
35. 35. Load Balanced Services
36. 36. Fault Tolerant Services
37. 37. Build ‘Always On’ Services Rely on Fault Tolerant Broker
38. 38. Services can “Rewind & Replay” the log Rewind & Replay
39. 39. Compacted Log (retains only latest version) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
40. 40. A database engine for data-in-flight
41. 41. Max(price) From orders where ccy=‘GBP’ over 1 day window emitting every second Continuously Running Queries
42. 42. What is stream processing engine? Data Index Query Engine Query Engine vs Database Finite source Stream Processor Infinite source
43. 43. Windowing For unordered or unpredictable streams Sliding Fixed (tumbling)
44. 44. Features: similar to database query engine JoinFilter Aggr- egate View Window
45. 45. Stateful Stream Processing stream Compacted stream Join Stream Data Stream-Tabular Data Windowed Stream Locally Cached Table (disk resident) KafkaKafka Streams
46. 46. Useful for Enrichment stream Compacted stream Join Orders Customers KafkaKafka Streams
47. 47. Scales Out
48. 48. Embeddable Orders Service Kafka Orders Service Orders Service
49. 49. Kafka Connect View Replication
50. 50. Sometimes you need to physically move data
51. 51. Replicate it, so both copies are identical
52. 52. Iterate via regeneration
53. 53. Kafka Connect Kafka Connect Kafka Connect Kafka
54. 54. So…
55. 55. Service Backbone Scalable, Fault Tolerant, Concurrent, Strongly Ordered, Stateful
56. 56. Embeddable tool for data manipulation Adapt data streams (transformation, stream maintenance, analytic function)
57. 57. Replicate Data Sources Exactly Create Regenerable, Streaming Views
58. 58. How do we actually do this? 10 (opinionated) principals for Streaming Services
59. 59. 1. Don’t use Kafka for shopping carts! (OK, you can, but use sparingly) Shopping Cart UI Service Broker/durability/broadcast add little to request response
60. 60. Do use Kafka for your core business processing. Think “business events”. An order was created, a payment was received, a trade was booked etc. Pub/Sub. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service
61. 61. 2. Pick Topics with Business Significance Orders Payments Returns Invoices
62. 62. Give your messages meaningful IDs and version them OrdersService1-Order-1234-v2 Should relate to the real world Should be Versioned (if mutable) Include the service name Note the key used for sharding in Kafka may not be this key
63. 63. 3. Decouple publishers from subscribers GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service
64. 64. Add Request/Response only where needed GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service REST
65. 65. 4. Use the log to regenerate state Avoid journaling incoming events
66. 66. Event Source side effects • Use offsets to tie these back to the stream • Store in: • Kafka • Kstreams state store • Other DB
67. 67. 5. Apply the Single Writer Principal • Change at source (by calling that service) • Let the change propagate back • Keep local copies read only. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service (1) Change Orders at source (2) Let the change propagate through
68. 68. 6. Leverage keeping datasets inside the broker customer country supplier ex- rates
69. 69. Leverage keeping only the latest version (table view) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
70. 70. Join & Process on the fly stream Compacted stream Join Orders Customers KafkaKafka Streams
71. 71. 7. Prefer stream processing over maintaining historic views UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Orders Historic “copies” diverge
72. 72. Join & Process on the fly stream Compacted stream Join Orders Customers KafkaKafka Streams
73. 73. 8. Sometimes you need historic views. => Replicate & Keep Read Only
74. 74. Replicate Kafka Connect Kafka Connect Kafka Keep me Read Only
75. 75. Iterate
76. 76. Polyglotic Persistence
77. 77. 9. Use Schemas (especially if data is retained)
78. 78. Schemaless data doesn’t age well
79. 79. Messages
80. 80. Confluent Schema Registry can help Orders Service Schema Registry Email Service Avro
81. 81. 10. Consider “Stream Management” Services Stream Management Kafka • Retaining data => Admin tasks • Similar to the role of a DBA • Data Migration • Repartitioning • Latest/versioned • Environment Management • CQRS
82. 82. KStreams is a good toolset for this
83. 83. Stream Management KStreams Latest Stream Versioned Stream
84. 84. So… 1. Don’t use Kafka for shopping carts! 2. Pick Topics with Business Significance 3. Decouple publishers from subscribers 4. Use the log to regenerate state 5. Apply the Single Writer Principal 6. Leverage keeping datasets inside the broker 7. Prefer stream processing over maintaining historic views 8. Sometimes you need historic views. => Replicate Read Only 9. Use Schemas 10. Consider “Stream Management” Services
85. 85. Microservices push us away from shared, mutable state
86. 86. But state needs to be communicated
87. 87. In an increasingly data-heavy world we need tools to do this efficiently
88. 88. …and in real time.
89. 89. We need a data-centric toolset to do this All Your Data Immutable Streams Event Driven Services Stream Management Services Legacy CDC View Replication Polyglotic Persistence tables streams Streaming Services
90. 90. Keep it simple, Keep it moving
91. 91. @benstopford Thanks!