Tale of ISUCON and Its Bench Tools

Tale of ISUCON and Its Bench Tools

1. 1. Tales of ISUCON5 and Its Bench Tools BizReach internal meetup (Nov 27, 2015) Satoshi "Moris" Tagomori (@tagomoris)
2. 2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, Norikra, Hadoop, MessagePack-Ruby, Woothee, ... Treasure Data, Inc.
3. 3. http://www.treasuredata.com/
4. 4. HQ Branch
5. 5. ISUCON5 Main Topics • Qualify: "ISUxi" • Good old SNS • Friend relations, Footprint, Many N+1 queries, ... • Final: "AirISU" • API aggregate server • Parallel requests, Application processes/threads, Cache based on data/protocol, HTTP/2, ...
6. 6. How to Get High Score in ISUCON5 • Qualify • Add index, Cache master data, Remove N+1, ... • Final • Massive threads, Cache invariable data, Async/ Parallel requests to APIs, If-Modified-Since, HTTP/2, ...
7. 7. BUG
8. 8. Bugs in Organizer Side • Qualify • Nothing serious • Last bug was fixed at 1st day 11:30am • Final • Some serious bugs in scenario to make effects for top N players
9. 9. Benchmark Tool
10. 10. Why ISUCON Bench Tools Should Be Written Newly? • Two inconsistent requirements: • high performance • integrity check • 1 request pattern for 2 requirements • players can cheat w/ different requests for purposes
11. 11. Requirements in detail • Performance • throughput, concurrency, low latency • Content check • HTML parser, JSON parser, CSS/JS check, Image, other binaries, ... • Complex scenario coding • tools should simulate user behavior • Protocol handling in detail • HTTP protocols, HTTP headers, keepalive, cache control, timeouts, ... • Variable source data • disable "cache all requests/response" strategy
12. 12. Features • Sending GET/POST requests • w/ various query params a/o content body • w/ various HTTP headers • Sending request series for a session • Sending request series for several sessions • Checking response integrity/consistency • Skipping response check for performance if needed
13. 13. Rough Sketch • http_load + custom ruby script • http_load: requests for performance • ruby script: requests for checks
14. 14. Sessions! (http_load cannot handle sessions)
15. 15. Overview • Java: jetty-client + Java8 Lambda • jetty-client for performance • lambda for content check • jackson to parse input data • jsoup to parse response html (CSS selector) • json-path to parse response json (JsonPath)
16. 16. { getAndCheck(session, "/", "GET INDEX BEFORE SIGNUP", (check) -> { check.isRedirect("/login"); }); getAndCheck(session, "/login", "GET LOGIN BEFORE SIGNUP", (check) -> { check.isStatus(200); check.isContentType("text/html"); if (! check.hasViolations()) { check.exist("form.form-signin[action=/login]"); check.exist("input[name=email]"); check.exist("input[name=password]"); check.exist("button[type=submit]"); check.hasStyleSheet("/css/bootstrap.min.css"); check.hasStyleSheet("/css/signin.css"); } }); getAndCheck(session, "/css/bootstrap.min.css", "BOOTSTRAP CSS", (check) -> { check.isStatus(200); if (! check.hasViolations()) { check.isContentBodyChecksum("08df9a96752852f2cbd310c30facd934e348c2c5"); } }); getAndCheck(session, "/css/signin.css", "SIGNIN CSS", (check) -> { check.isStatus(200); if (! check.hasViolations()) { check.isContentBodyChecksum("702783cc5eff3d8d3532e339ddd15c57f7a08776"); } }); } Bootstrap.java
17. 17. while (true) { if (LocalDateTime.now().isAfter(stopAt)) break; Session s = sessions.get(random.nextInt((int) sessions.size())); get(s, "/login"); get(s, "/css/bootstrap.min.css"); get(s, "/css/signin.css"); post(s, "/login", formLogin(s)); if (LocalDateTime.now().isAfter(stopAt)) break; get(s, "/"); get(s, "/css/bootstrap.min.css"); get(s, "/css/jumbotron-narrow.css"); get(s, "/js/jquery-1.11.3.js"); get(s, "/js/bootstrap.js"); get(s, "/js/airisu.js"); get(s, "/user.js"); if (LocalDateTime.now().isAfter(stopAt)) break; for (int i = 0 ; i < 10 ; i++) { get(s, "/data"); if (LocalDateTime.now().isAfter(stopAt)) break; } if (LocalDateTime.now().isAfter(stopAt)) break; } Load.java
18. 18. @Override public Result finishHook(Result result) { long requests = result.requests; if (result.responses.exception * 100.0 / (requests * 1.0) >= 1.0) { result.addViolation("Too many exceptions", "通信エラー等の失敗が多過ぎます(1%以上)"); result.fail(); } if (result.responses.error * 100.0 / (requests * 1.0) >= 1.0) { result.addViolation("Too many errors", "ステータス 5xx のレスポンスが多過ぎます(1%以上)"); result.fail(); } if (result.responses.failure * 100.0 / (requests * 1.0) >= 5.0) { result.addViolation("Too many failures", "ステータス 4xx のレスポンスが多過ぎます(5%以上)"); result.fail(); } return result; } @Override public Step[] steps() { Step[] steps = new Step[3]; steps[0] = new Step(35000L, Init.class); steps[1] = new Step(60000L, Bootstrap.class); steps[2] = new Step( 70000L, Checker.class, ModifyLoader.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class ); return steps; } Full.java
19. 19. Code Example • Simple scenario • https://github.com/isucon/isucon5-final/blob/master/bench/src/main/java/net/isucon/ isucon5f/bench/Load.java • https://github.com/isucon/isucon5-final/blob/master/bench/src/main/java/net/isucon/ isucon5f/bench/Checker.java • https://github.com/isucon/isucon5-final/blob/master/bench/src/main/java/net/isucon/ isucon5f/bench/Bootstrap.java • https://github.com/isucon/isucon5-final/blob/master/bench/src/main/java/net/isucon/ isucon5f/bench/Init.java • Complex scenario • https://github.com/isucon/isucon5-final/blob/master/bench/src/main/java/net/isucon/ isucon5f/bench/Full.java
20. 20. Distributed Benchmarking • N nodes for 1 benchmark • Can a node perform fast enough? (CPU bounded) • Y -> 1 vs 1, N -> 2 vs 1 • "GET /": 5000req/thread on localhost -> enough • N nodes for many benchmarks • Scale out strategy • Queue/worker system
21. 21. Scaling out nodes ISUCON Portal (RoR) Queue (MySQL) Daemon script (ruby) Bench (Java) Daemon script (ruby) Daemon script (ruby) Daemon script (ruby) Bench (Java) ack req/set result Daemon script (ruby) Bench (Java)
22. 22. Recorded Performance • about 194,000 requests / 60 sec • OK: 185,000 • Redirects: 9,300 • Peak 30 nodes (Qualify) • 11,500 benchmarks in 2days
23. 23. Far more: Scenario • Checking for critical issues / non-critical issues • Critical/non-critical mode of Checker class • Checks w/ dependencies • If a check fails, following checks throws NPE :(
24. 24. Some More Topics • Async • Gigantic parallel requests • 2 or more simultaneous requests under control
25. 25. Java8: What and How I feel about it
26. 26. OSS: net.isucon.bench.*