jcmd #javacasual

6. •Introduction to Troubleshooting in JDK 9 •JDK 9 では jcmd / jhsdb を使おうという話 •スライド：https://goo.gl/4jaujJ JavaOne 2016 % jstack # JDK 8 : Options: -F to force a thread dump. Use when jstack <pid> does not respond (process is hung) -m to print both java and native frames (mixed mode) -l long listing. Prints additional information about locks -h or -help to print this help message % ./jstack # JDK 9 : Options: -l long listing. Prints additional information about locks -h or -help to print this help message # -F was removed by JDK-8155091: Remove SA related functions from tmtools % ./jstack -m Error: -m option used Cannot connect to core dump or remote debug server. Use jhsdb jstack instead % ./jhsdb jstack --mixed You have to set --pid or --exe. --locks to print java.util.concurrent locks --mixed to print both java and native frames (mixed mode) --exe executable image name --core path to coredump --pid pid of process to attach Copyright©2016 NTT corporation

7. •CLI の Local JVM process 診断ツール •jcmd [-l | -h | process command] •-h : help, -l | no argument: jps •process: <pid | main-class> • pid に 0 を指定すると全部 • main-class は一致したプロセス全部 •command: <command[ options] | -f file> • command は Domain.suffix 形式 jcmd Copyright©2016 NTT corporation

8. •jcmd process help •実行可能なコマンド一覧 (対象プロセス依存) •jcmd process help command •コマンドのヘルプ(使い方、負荷、オプション等) Help Copyright©2016 NTT corporation % jcmd 999 help GC.run 999: GC.run Call java.lang.System.gc(). Impact: Medium: Depends on Java heap size and content. Syntax: GC.run

9. Command: Domain.suffix (Java 8) Domain Num. suffix Abstract PerfCounter 1 各種 JVM 統計情報 (※) ManagementAgent 3 JMX エージェント操作 Thread 1 スレッドダンプ GC 7 GC 機構 (Heap / GC / Finalization) VM 7 ランタイム操作、情報取得 Copyright©2016 NTT corporation ※: 本来はコマンドとは別枠だが、見易さのため並べて記載している

10. Command: Domain.suffix (Java 9) Domain Num. suffix Abstract PerfCounter 1 - 各種 JVM 統計情報 (※) ManagementAgent 4 +1 JMX エージェント操作 Thread 1 - スレッドダンプ GC 7 +2 -1 GC 機構 (Heap / GC / Finalization) VM 15 +8 ランタイム操作、情報取得 Compiler 7 - コンパイラ操作、情報取得 JVMTI 2 - JVMTI エージェント操作 GC.rotate_log が VM ドメインへ移動 Copyright©2016 NTT corporation ※: 本来はコマンドとは別枠だが、見易さのため並べて記載している

11. •JVM 統計情報 •jstat や jinfo はこれを基に加工して出力してる • jdk/src/jdk.jcmd/share/classes/sun/tools/jstat/resources/jstat_options •無加工は辛いので加工スクリプトに渡して出力すると便利 PerfCounter Copyright©2016 NTT corporation suffix Description print JVM 統計情報を出力

12. •Management Agent 起動したり終わらせたり •JMX (Management Extensions) と JDP(Discovery Protocol) • JMXリモートアクセスを忘れてても後から設定可能 ManagementAgent suffix Description status 状態表示 start リモートエージェントを有効化 start_local ローカルエージェントを有効化 stop リモートエージェントを無効化 Copyright©2016 NTT corporation JDK 9 から

13. $ jcmd 999 ManagementAgent.start jmxremote.port=777 jmxremote.ssl=true 999: Commandexecuted successfully $ jcmd 999 ManagementAgent.status Agent: enabled Connection Type: remote Protocol : rmi Host : fedora24.novalocal URL : service:jmx:rmi://XXX…XXX Properties : com.sun.management.jmxremote.authenticate = true [default] com.sun.management.jmxremote.ssl.need.client.auth = false [default] com.sun.management.jmxremote.ssl = true [default] : com.sun.management.jmxremote.port = 8070 Copyright©2016 NTT corporation [config.file | jmxremote.* | jdp.*]

15. •GC機構 (Heap/GC/Finalization) GC suffix Description class_stats* クラスメタデータ一覧 (-helpで各項目説明) class_histogram クラスヒストグラム (jmap -histo:live) heap_info ヒープ・メタスペース使用量 heap_dump ヒープダンプ (jmap -dump, -allで参照切れも) finalizer_info Finalize 待ちキュー表示 run_finalization Finalization 実行 (System#runFinalization) run GC 実行 (System#gc) Copyright©2016 NTT corporation *:要 -XX:+UnlockDiagnosticVMOptions 参考：hotspot/src/share/vm/runtime/globals.hpp

16. •ランタイム周り。情報出力系 VM suffix Description info Fatal Error Log (クラッシュ時等で出すログ) uptime 起動時間 command_line VM 引数 system_properties システムプロパティ version バージョン flags 設定した(された)フラグ (-all で全フラグ) dynlibs メモリマップ付き Dynamic library リスト Copyright©2016 NTT corporation

17. •ランタイム周り。情報出力・動作変更系 VM suffix Description print_touched_methods* JVM 実行中に触ったメソッドを表示 classloader_stats メタデータ状況等の全クラスローダ情報 class_hierarchy 読込クラスをクラスローダ付き階層表示 native_memory** Native Memory Tracking (NMT) stringtable StringTable (-verboseで全Stringダンプ) symboltable SymbolTable (-verboseで全シンボル) set_flag flgname [value] JVM フラグの設定 (一部のみ) Copyright©2016 NTT corporation *:要 -XX:+UnlockDiagnosticVMOptions **: 要 -XX:NativeMemoryTracking 設定

18. • Native メモリの状態を記録する 1. Run Java with -XX:NativeMemoryTracking=summary 2. jcmd 999 VM.native_memory baseline 3. jcmd 999 VM.native_memory summary.diff • 詳細に見たい場合は上の summary を details に変える • NativeMemoryTracking was available since Java 8 • https://docs.oracle.com/javase/8/docs/technotes/ guides/troubleshoot/tooldescr007.html Native Memory Tracking Copyright©2016 NTT corporation

19. $ jcmd 999 VM.native_memory baseline 999: Baseline succeeded # --- Run method which wants to check the memory consumption --- $ jcmd 999 VM.native_memory summary.diff 999: Native Memory Tracking: Total: reserved=5664359KB -36096KB, committed=371555KB -36096KB - Java Heap (reserved=4108288KB, committed=258048KB) (mmap: reserved=4108288KB, committed=258048KB) - Class (reserved=1056953KB +15KB, committed=5433KB +15KB) (classes #767 +3) (malloc=185KB +3KB #741 -4) (mmap: reserved=1056768KB, committed=5248KB) - Thread (reserved=36128KB -112KB, committed=36128KB -112KB) (thread #36 +1) (stack: reserved=35980KB +325KB, committed=35980KB+325KB) (malloc=107KB +2KB #197 +4) (arena=41KB -130KB #70 +2) : Native Memory Tracking ±n shows diff from baseline Copyright©2016 NTT corporation

20. •ランタイム周り。ログ。 • -XX:+PrintGCDetails -Xloggc:gc.log.`date +%Y- %m-%d_%H-%M-%S` -XX:+PrintGCTimeStamps • -Xlog:gc*=debug:gc.log.%t:time,level • -Xlog:[<selection>]:[<output>]:[<decorators>] VM suffix Description log JVM Unified Logging (JEP158) 制御 Copyright©2016 NTT corporation

21. •jcmd 999 VM.log [options] •-Xlog:[<selection>]:[<output>]:[<decorators>] GC ログを設定するケース Copyright©2016 NTT corporation Options Description output output (出力先) output_options output (その他の設定) what selection (ロギング対象とそのレベル) decorators decorators (出力内容調整) disable Clears all configurations and turns off all loggings list Lists current log configurations rotate Rotate all logs (instead of GC.log_rotate in Java 8)

22. •Confirms current loggings GC ログを設定するケース Copyright©2016 NTT corporation $ jcmd 999 VM.log list 999: Available log levels: off, trace, debug, info, warning, error Available log decorators: time (t), uptime (u), timemillis (tm), … Available log tags: add, age, alloc, arguments, annotation, barrier, … : Described tag combinations: logging: Logging for the log framework itself Log output configuration: #0: stdout all=warning uptime,level,tags #1: stderr all=off uptime,level,tags default configurations

23. •Turns on a new logging: GC.log GC ログを設定するケース Copyright©2016 NTT corporation $ jcmd 999 VM.log output="file=gc.log" output_options="filecount=5,filesize=10m" what="gc=debug" decorators="time,level" 999: Command executed successfully $ jcmd 999 VM.log list 999: : Log output configuration: #0: stdout all=warning uptime,level,tags #1: stderr all=off uptime,level,tags #2: gc.log gc=debug time,level added a new GC logging gc=log_level name or path options (log rotation)

24. •Turns off all existing loggings •If you want to turn off #2 only •jcmd 999 VM.log output="#2" what="all=off" GC ログを設定するケース Copyright©2016 NTT corporation $ jcmd 999 VM.log disable 999: Command executed successfully $ jcmd 999 VM.log list 999: : Log output configuration: #0: stdout all=off uptime,level,tags #1: stderr all=off uptime,level,tags set all=off at stdout / stderr, and removed the other configuration(s) such as #2 output nothing

25. •jcmd 999 VM.log rotate •Rotates logs satisfying the following 1. Configured ‘output’ as file 2. Configured ‘filecount’ (by ‘output_option’) 手動ログローテート $ jcmd 999 VM.log list : Log output configuration: #0: stdout all=warning uptime,level,tags #1: stderr all=off uptime,level,tags #2: gc.log gc=debug filecount=5,filesize=20M level,tags #2 will be rotated, not #0 and #1 Obsoleted GC.rotate_log Copyright©2016 NTT corporation

26. •JIT Compiler •JEP 165: Compiler Control (JDK 9 Feature) Compiler Copyright©2016 NTT corporation suffix Description directive_add Adds directives of JIT by JSON format same as -XX:CompilerDirectivesFile directive_remove Removes a latest directive directive_clear Clears all directives directive_print Prints current directives codecache Prints summary of code cache codelist Lists all code cache queue Lists methods in waiting queue for JIT

27. % cat disable_c2_at_string-hashCode.json [ { "match":[ "*String.hashCode" ], "c2":{ "Exclude":true } } ] % jcmd 999 Compiler.directives_add disable_c2_at_string-hashCode.json 999: 1 compiler directives added % jcmd 999 Compiler.directives_print 999: Directive: matching: *String.hashCode c1 directives: inline: - Enable:false Exclude:false BreakAtExecute:false Log:false PrintAssembly:false … c2 directives: inline: - Enable:true Exclude:true BreakAtExecute:false Log:false PrintAssembly:false … Switch false to true Copyright©2016 NTT corporation

28. •jcmd 999 Compiler.codelist •Prints JIT compiled methods (in code cache) How JIT Compiler works? % jcmd 999 Compiler.codelist 999: : 18 3 java.lang.String.hashCode()I [0x00007ff301594090, 0x00007ff301594280 - 0x00007ff3015946e8] : ID Compile level of -XX:+TieredCompilation 0: Interpreter 1-3: C1 1: C1 without profiling (full optimization) 2: C1 with basic profiling (invocation and back-edge) 3: C1 with full profiling 4: C2 Method name nmethod start address code start address code end address Copyright©2016 NTT corporation

29. •jcmd 999 Compiler.codelist •Prints JIT compiled methods (in code cache) How JIT Compiler works? % jcmd 999 Compiler.codelist 999: : 18 3 java.lang.String.hashCode()I [0x00007ff301594090, 0x00007ff301594280 - 0x00007ff3015946e8] : ID Compile level of -XX:+TieredCompilation 0: Interpreter 1-3: C1 1: C1 without profiling (full optimization) 2: C1 with basic profiling (invocation and back-edge) 3: C1 with full profiling 4: C2 Method name nmethod start address code start address code end address Copyright©2016 NTT corporation

30. How JIT Compiler works? •JVM JIT compilation overview •http://www.slideshare.net/ZeroTurnaround/vlad imir-ivanovjvmjitcompilationoverview- 24613146 •Intrinsic Methods in HotSpot VM •http://www.slideshare.net/RednaxelaFX/green- teajug-hotspotintrinsics02232013 Copyright©2016 NTT corporation

31. •JVMTI Agent 制御 JVMTI suffix Description agent_load library_path [option] Attach JVMTI agent of library path data_dump Fire DataDumpRequest JVMTI event Copyright©2016 NTT corporation % jcmd 999 JVMTI.agent_load /path/to/libheapstats.so % jcmd 999 JVMTI.agent_load ¥ /path/to/libheapstats.so=heapstats.conf

34. •Gathers runtime information continuously, then draws up time-series graphs •Lightweight agent to run on the production systems •New solution to troubleshooting for OpenJDK users. HeapStats Copyright©2016 NTT corporation

35. Overhead of HeapStats 467.59 461.43 0 50 100 150 200 250 300 350 400 450 500 Without HeapStats With HeapStats (ops/m) 1.32 % Overhead Rate SPECjvm2008 Composite Result Copyright©2016 NTT corporation

36. $ jcmd 999 JVMTI.agent_load libheapstats.so 999: Commandexecuted successfully or java.lang.OutOfMemoryError: Java heap space $ java –jar heapstats-analyzer.jar $ rpm -ivh heapstats-2.0.1-0.*.rpm $ java -agentpath:heapstats … Your_Application then How to use HeapStats Copyright©2016 NTT corporation

jcmd #javacasual

Yuji Kubota

jcmd #javacasual