MySQL5.7で遊んでみよう

1. MySQL 5.7で遊んでみよう 2015/10/24 yoku0825 OSC 2015 Tokyo/Fall

2. MySQL 5.7.9 GA has been released at 2015/10/19!! 1/103

3. なんですが 2/103

4. MySQL 5.7を本番で使う話はしません 3/103

5. まずは新しい機能で遊んでみましょうという話をします 4/103

6. ＼こんにちは／ yoku0825＠とある企業のDBA オラクれない- ポスグれない- マイエスキューエる- 家に帰ると妻の夫- せがれの⽗- ムスメの⽗- Twitter: @yoku0825 Blog: ⽇々の覚書 MyNA ML: ⽇本MySQLユーザ会 5/103

7. 遊ぶ準備ホストがあると便利 VirtualBox- どこかでVPS借りてもいいけどTerminateを忘れずに InnoDBバッファプールサイズの変更を試すには、メモリー⼤盛りのが楽しいかも - 6/103

8. インストール CentOS 6.5です。。ちょっと遊ぶくらいならyumリポジトリーでいい新しいyumリポジトリーは既にデフォルトでインストールするのがMySQL 5.7になってる # rpm -i http://dev.mysql.com/get/mysql57-community-release-el6- 7.noarch.rpm # yum install mysql-community-server MySQL :: Download MySQL Yum Repository 7/103

9. 起動 service mysqld startのタイミングでmysqld --initialize (mysql_install_dbの後継)が⾛る # service mysqld start Initializing MySQL database: 2015-10-19T04:47:16.696030Z 0 [Warn ing] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see document ation for more details). 8/103

10. 接続…︖ # mysql ERROR 1045 (28000): Access denied for user 'root'@'localhost' (us ing password: NO) # ls -l ~/.mysql_secret ls: cannot access /root/.mysql_secret: No such file or directory 9/103

11. 初期パスワード初期パスワードは設定されている mysql_install_dbは~/.mysql_secretを作ったけど、mysqld --initializeは標準エラー出⼒にパスワードを出⼒する log_errorが設定されていればそのファイルに、設定されていなければコンソールに。 - 10/103

12. パスワードの在り処 # grep log-error /etc/my.cnf log-error=/var/log/mysqld.log # grep password /var/log/mysqld.log 2015-10-19T04:47:17.752156Z 1 [Warning] A temporary password is g enerated for root@localhost: jM*C.2dAc6dh 2015-10-19T04:47:22.244470Z 2 [Note] Access denied for user 'UNKN OWN_MYSQL_USER'@'localhost' (using password: NO) 2015-10-19T04:47:36.374844Z 3 [Note] Access denied for user 'root '@'localhost' (using password: NO) 11/103

13. ログインはできたけれど # mysql -p Enter password: mysql> status ERROR 1820 (HY000): You must reset your password using ALTER USE R statement before executing this statement. 12/103

14. パスワードを変更しようとするけど。。 mysql> SET PASSWORD = ''; ERROR 1819 (HY000): Your password does not satisfy the current po licy requirements 13/103

15. MySQL 5.7のrpmではvalidate̲passwordがデフォルトでインストールされているというよりは、今までが「デフォルトで⼊っているつもりが⼊っていなかった」ので、そのバグをFIX、ということらしい。 The validate̲password plugin was not installed by RPM packages for platorms using systemd or SysV- style initialization scripts. (Bug #18438833) MySQL :: MySQL 5.7 Release Notes :: Changes in MySQL 5.7.8 (2015-08-03, Release Candidate) 14/103

16. validate_password= OFFを/etc/my.cnfに書くやり⽅ # vim /etc/my.cnf [mysqld] validate_password= OFF # service mysqld restart # mysql -p Enter password: mysql> SET PASSWORD= ''; Query OK, 0 rows affected (0.00 sec) 15/103

17. ⼀度強⼒なパスワードで変更してから無効にするやり⽅ mysql> SET PASSWORD= 'Do_you_love_MySQL57?'; Query OK, 0 rows affected (0.00 sec) mysql> UNINSTALL PLUGIN validate_password; Query OK, 0 rows affected (0.01 sec) mysql> SET PASSWORD= ''; Query OK, 0 rows affected (0.00 sec) 16/103

20. sysスキーマ mysql> use sys Database changed mysql> SHOW TABLES; .. mysql> SELECT * FROM statement_analysis; .. 19/103

21. sysスキーマ performance̲schemaとinformation̲schemaのテーブルをベースにしたビュー performance̲schemaが無効になってるとほぼ空っぽ。- 5.7ではオーバーヘッドかなり⼩さくなってるし、有効にしておいていいと思う。 - CPUバウンドな場合はまだオーバーヘッド気になるかも。I/Oバウンドするなら誤差範囲。 - x$で始まる名前のビューは単位変換(G, M, K, m, u, n)なしデフォルトの時間単位はps- 20/103

22. はい次 21/103

23. MySQL 5.7の全⽂検索「全ツイート履歴をダウンロード」してきた。 mysql> CREATE TABLE tweet ( tweet_id BIGINT UNSIGNED PRIMARY KE Y, timestamp DATETIME NOT NULL, text TEXT); mysql> LOAD DATA INFILE '/tmp/tweets.csv' INTO TABLE tweet FIELD S TERMINATED BY ',' ENCLOSED BY '"' IGNORE 1 LINES (tweet_id, @d ummy, @dummy, @timestamp, @dummy, text, @dummy) SET timestamp= DA TE_ADD(@timestamp, INTERVAL 9 HOUR); ERROR 1290 (HY000): The MySQL server is running with the --secure -file-priv option so it cannot execute this statement 22/103

24. secure_file_privの暗黙のデフォルト値 rpm, dpkgのみ暗黙のデフォルト値がある。。 mysql> SELECT @@secure_file_priv; +-----------------------+ | @@secure_file_priv | +-----------------------+ | /var/lib/mysql-files/ | +-----------------------+ 1 row in set (0.00 sec) 23/103

25. sql_modeが強化ツイート履歴はʼyyyy/mm/dd hh:nn:ss +TZʼな形式のタイムスタンプを持っているので、MySQLがそのままは⾷えない。 ʻ+TZʼの部分を無理⽮理読み⾶ばして9時間⾜してやることで5.6では何とかなったが mysql> LOAD DATA INFILE '/var/lib/mysql-files/tweets.csv' INTO TABLE twe et FIELDS TERMINATED BY ',' ENCLOSED BY '"' IGNORE 1 LINES (tweet_id, @ dummy, @dummy, @timestamp, @dummy, text, @dummy) SET timestamp= DATE_ADD (@timestamp, INTERVAL 9 HOUR); ERROR 1292 (22007): Truncated incorrect datetime value: '2015-10-19 05:4 6:24 +0000' mysql> SELECT @@sql_modeG *************************** 1. row *************************** @@sql_mode: ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZE RO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTIT UTION 1 row in set (0.00 sec) 24/103

26. 申し訳ないが取り敢えず非strictにしようホントはなんかスクリプト書けばそれで済むんだけど⾯倒。。 mysql> SET sql_mode= ''; Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> LOAD DATA INFILE '/var/lib/mysql-files/tweets.csv' INTO TA BLE tweet FIELDS TERMINATED BY ',' ENCLOSED BY '"' IGNORE 1 LINE S (tweet_id, @dummy, @dummy, @timestamp, @dummy, text, @dummy) SE T timestamp= DATE_ADD(@timestamp, INTERVAL 9 HOUR); Query OK, 28051 rows affected, 56102 warnings (0.52 sec) Records: 28051 Deleted: 0 Skipped: 0 Warnings: 56102 mysql> SET sql_mode= default; Query OK, 0 rows affected (0.00 sec) 25/103

27. ngramパーサーは最初から有効になってる mysql> ALTER TABLE tweet ADD FULLTEXT KEY idx_ngram(text) WITH PA RSER ngram; Query OK, 0 rows affected, 1 warning (4.22 sec) Records: 0 Duplicates: 0 Warnings: 1 26/103

28. LIKEより遅くて草。。 mysql> SELECT COUNT(*) FROM tweet WHERE text LIKE '%mysql%'; +----------+ | COUNT(*) | +----------+ | 6265 | +----------+ 1 row in set (0.05 sec) mysql> SELECT COUNT(*) FROM tweet WHERE match(text) against ('mys ql' IN BOOLEAN MODE); +----------+ | COUNT(*) | +----------+ | 6265 | +----------+ 1 row in set (0.18 sec) 27/103

29. 転置索引の中⾝をのぞいてみる mysql> SET GLOBAL innodb_ft_aux_table= 'd1/tweet'; Query OK, 0 rows affected (0.00 sec) mysql> SELECT word, count(*) AS c FROM information_schema.INNODB_FT_INDE X_TABLE GROUP BY word ORDER BY c DESC LIMIT 10; +--------+------+ | word | c | +--------+------+ | my | 9409 | | sq | 9353 | | ql | 9346 | | ys | 8866 | | co | 8024 | | って | 7957 | | er | 6614 | | 。。 | 5997 | | tt | 5874 | | ht | 5715 | +--------+------+ 10 rows in set (8.84 sec) 28/103

30. トークンのサイズを変更するオンラインでは無理だった。。 mysql> SELECT @@ngram_token_size; +--------------------+ | @@ngram_token_size | +--------------------+ | 2 | +--------------------+ 1 row in set (0.00 sec) mysql> SET GLOBAL ngram_token_size= 3; ERROR 1238 (HY000): Variable 'ngram_token_size' is a read only va riable 29/103

31. ngram_token_sizeの指定 # vim /etc/my.cnf [mysqld] ngram_token_size= 3 # service mysqld restart 30/103

32. innodb_ft_aux_tableは毎回指定しないといけない mysql> SET GLOBAL innodb_ft_aux_table= 'd1/tweet'; Query OK, 0 rows affected (0.00 sec) mysql> SELECT word, count(*) AS c FROM information_schema.INNODB_FT_INDE X_TABLE GROUP BY word ORDER BY c DESC LIMIT 10; +--------+------+ | word | c | +--------+------+ | my | 9409 | | sq | 9353 | | ql | 9346 | | ys | 8866 | | co | 8024 | | って | 7957 | | er | 6614 | | 。。 | 5997 | | tt | 5874 | | ht | 5715 | +--------+------+ 10 rows in set (8.61 sec) 31/103

33. OPTIMIZE TABLEで転置索引の再構築 mysql> OPTIMIZE TABLE d1.tweet; -----------------------------+ | -----------------------------+ g recreate + analyze instead | | -----------------------------+ 2 rows in set (10.70 sec) DEX_TABLE GROUP BY word ORDER BY c DES C LIMIT 10; +------+------+ | word | c | +------+------+ | sql | 9094 | | ysq | 8328 | | mys | 8311 | | htt | 5389 | | ttp | 5388 | | :// | 5286 | | //t | 5222 | | .co | 5201 | | /t. | 5182 | | t.c | 5172 | +------+------+ 10 rows in set (13.08 sec) 32/103

34. MeCabプラグインはバンドルされているけど有効化されていない mysql> ALTER TABLE tweet ADD FULLTEXT KEY idx_mecab(text) WITH PA RSER mecab; ERROR 1128 (HY000): Function 'mecab' is not defined mysql> SHOW PLUGINS; .. 33/103

35. MeCabプラグインの有効化 # vim /etc/my.cnf [mysqld] loose-mecab-rc-file= /usr/lib64/mysql/mecab/etc/mecabrc # vim /usr/lib64/mysql/mecab/etc/mecabrc dicdir = /usr/lib64/mysql/mecab/dic/ipadic_utf-8 # service mysqld restart 34/103

36. MeCabを使った転置索引 mysql> INSTALL PLUGIN mecab SONAME 'libpluginmecab.so'; Query OK, 0 rows affected (0.02 sec) mysql> ALTER TABLE tweet ADD FULLTEXT KEY idx_mecab(text) WITH PA RSER mecab; Query OK, 0 rows affected (1.59 sec) Records: 0 Duplicates: 0 Warnings: 0 35/103

37. 取り敢えずCOUNT mysql> SELECT COUNT(*) FROM tweet WHERE text LIKE '%mysql%'; +----------+ | COUNT(*) | +----------+ | 6265 | +----------+ 1 row in set (0.04 sec) mysql> SELECT COUNT(*) FROM tweet FORCE INDEX(idx_mecab) WHERE ma tch(text) against ('mysql' IN BOOLEAN MODE); +----------+ | COUNT(*) | +----------+ | 5585 | +----------+ 1 row in set (0.00 sec) 36/103

38. 転置索引 mysql> SET GLOBAL innodb_ft_aux_table= 'd1/tweet'; Query OK, 0 rows affected (0.00 sec) mysql> ALTER TABLE tweet DROP KEY idx_ngram; -- このままだとngramとMeCabの転置索引が混じるのでドロップ Query OK, 0 rows affected (0.45 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> SELECT word, count(*) AS c FROM information_schema.INNODB_FT_INDEX_TABLE GROUP BY word ORDER BY c DESC LIMIT 10; +-----------+------+ | word | c | +-----------+------+ | mysql | 7374 | | :// | 5414 | | http | 4629 | | innodb | 1028 | | という | 1016 | | dbts | 883 | | https | 883 | | rkajiyama | 797 | | mariadb | 795 | | groonga | 681 | +-----------+------+ 10 rows in set (0.47 sec) 37/103

39. 以前やった全⽂検索のベンチマーク MySQLの全⽂検索に関するあれやこれや 3⾏で⾔うと Mroongaは速いし2gramでも1⽂字検索できて便利 1gramは死ぬからMeCab使う結果セットが⼤きくなってくるとMeCabでも死ぬ - 38/103

40. はい次 39/103

41. オフラインモードを試すためにSuperを持ってないユーザーを作る t'; Query OK, 0 rows affected, 1 warning (0.07 sec) mysql> SHOW WARNINGSG *************************** 1. row *************************** Level: Warning Code: 1287 oved in futu re release. Create new user with CREATE USER statement. 1 row in set (0.00 sec) mysql> SHOW GRANTS FOR yoku0825@localhost; +-----------------------------------------------------------------+ | Grants for yoku0825@localhost | +-----------------------------------------------------------------+ | GRANT USAGE ON *.* TO 'yoku0825'@'localhost' | | GRANT ALL PRIVILEGES ON `mysqlslap`.* TO 'yoku0825'@'localhost' | +-----------------------------------------------------------------+ 2 rows in set (0.00 sec) 40/103

42. MySQL 5.7(あるいはsql_mode= NO_AUTO_CREATE_USER)に怒られないユーザーの作成⽅法 mysql> CREATE USER yoku0826@localhost IDENTIFIED BY 'test'; Query OK, 0 rows affected (0.01 sec) mysql> GRANT ALL ON mysqlslap.* TO yoku0826@localhost; Query OK, 0 rows affected (0.00 sec) mysql> SHOW GRANTS FOR yoku0826@localhost; +-----------------------------------------------------------------+ | Grants for yoku0826@localhost | +-----------------------------------------------------------------+ | GRANT USAGE ON *.* TO 'yoku0826'@'localhost' | | GRANT ALL PRIVILEGES ON `mysqlslap`.* TO 'yoku0826'@'localhost' | +-----------------------------------------------------------------+ 2 rows in set (0.00 sec) 41/103

43. mysqlslapで負荷かけながらオフラインモードにしてみる $ mysqlslap --no-drop --auto-generate-sql --auto-generate-sql-exe cute-number=100000 -uyoku0826 -ptest mysqlslap: [Warning] Using a password on the command line interfa ce can be insecure. mysql> SET GLOBAL offline_mode= ON; Query OK, 0 rows affected (0.00 sec) mysqlslap: Error when storing result: 2013 Lost connection to MyS QL server during query mysqlslap: Cannot run query INSERT INTO t1 VALUES (95275444,'bNIr BDBl81tjzdvuOpQRCXgX37xGtzLKEXBIcE3k7xK7aFtqxC99jqYnpTviK83bf6lGD gsKd4R3KLmHPnI8TqnIKj1gjw7N2sXFZNS2Svyg8cpZN7atxL39w4igsp') ERRO R : MySQL server has gone away 42/103

44. オフラインモードでもう⼀度mysqlslapしてみる $ mysqlslap --no-drop --auto-generate-sql --auto-generate-sql-exe cute-number=100000 -uyoku0826 -ptest mysqlslap: [Warning] Using a password on the command line interfa ce can be insecure. mysqlslap: Error when connecting to server: The server is current ly in offline mode 43/103

45. はい次 44/103

46. InnoDBのバッファプールサイズオンライン変更(負荷かける⽅) $ mysqlslap --no-drop --auto-generate-sql --auto-generate-sql-exe cute-number=1000000 -c 32 -uyoku0826 -ptest & mysqladmin -uroot - r -i 1 ex | egrep "Com_(insert|select) " [1] 1256 45/103

47. InnoDBのバッファプールサイズオンライン変更(MySQL 側, 増やす) mysql> SET GLOBAL innodb_buffer_pool_size= 32 * 1024 * 1024 * 1024; Query OK, 0 rows affected (0.00 sec) mysql> SHOW STATUS LIKE '%resize%'; +----------------------------------+----------------------------------+ | Variable_name | Value | +----------------------------------+----------------------------------+ | Innodb_buffer_pool_resize_status | Resizing also other hash tables. | +----------------------------------+----------------------------------+ 1 row in set (0.00 sec) mysql> SHOW STATUS LIKE '%resize%'; +----------------------------------+----------------------------------------------------+ | Variable_name | Value | +----------------------------------+----------------------------------------------------+ | Innodb_buffer_pool_resize_status | Completed resizing buffer pool at 151019 10:00:03. | +----------------------------------+----------------------------------------------------+ 1 row in set (0.00 sec) 46/103

48. 増やしてる最中 | Com_insert | 144 | | Com_select | 119 | | Com_insert | 88 | | Com_select | 72 | | Com_insert | 0 | | Com_select | 0 | | Com_insert | 82 | | Com_select | 68 | | Com_insert | 150 | | Com_select | 125 | 47/103

49. SELECT-Only $ mysqlslap --no-drop --query="SELECT * FROM mysqlslap.t1 ORDER BY RAND() LIMIT 1" --number-of-queries=1000000 -c 32 -uyoku0826 -ptest & mysqladmin -r -i 1 -uroot ex | egrep "Com_select " .. | Com_select | 70 | | Com_select | 74 | | Com_select | 7 | | Com_select | 46 | | Com_select | 56 | .. | Com_select | 73 | | Com_select | 68 | | Com_select | 24 | | Com_select | 56 | | Com_select | 68 | 48/103

50. InnoDBのバッファプールサイズオンライン変更(MySQL 側, 減らす) mysql> SET GLOBAL innodb_buffer_pool_size= 128 * 1024 * 1024; Query OK, 0 rows affected (0.00 sec) mysql> SHOW STATUS LIKE '%resize%'; +----------------------------------+----------------------------- -----------------------+ | Variable_name | Valu e | +----------------------------------+----------------------------- -----------------------+ | Innodb_buffer_pool_resize_status | Completed resizing buffer po ol at 151019 10:01:06. | +----------------------------------+----------------------------- -----------------------+ 1 row in set (0.00 sec) 49/103

51. 減らしてる最中 | Com_insert | 123 | | Com_select | 100 | | Com_insert | 112 | | Com_select | 97 | | Com_insert | 118 | | Com_select | 96 | | Com_insert | 114 | | Com_select | 96 | 50/103

52. 次 51/103

53. JSONデータ型はカラムのデータ型なので、JSONとして有効でない値はエラーになる mysql> CREATE TABLE t1 (val JSON); Query OK, 0 rows affected (0.35 sec) mysql> INSERT INTO t1 VALUES ('"[{is_this_json?}]'); ERROR 3140 (22032): Invalid JSON text: "Missing a closing quotati on mark in string." at position 17 in value (or column) '"[{is_th is_json?}]'. 52/103

54. サンプルデータを突っ込んでみる # perl -MDBI -MConfig::Pit -MNet::Twitter::Lite::WithAPIv1_1 -MJSON -e 'my $conn = DBI->connect("dbi:mysql:d1", "root", "", {mysql_enable_utf8 => 1}); my $twitte r= Net::Twitter::Lite::WithAPIv1_1->new(%{pit_get("twitter")}, ssl => 1); $conn- >do("INSERT INTO t1 VALUES (?)", undef, JSON->new()->encode($twitter->search({q => "from:yoku0825"})));' mysql> SELECT val FROM t1 LIMIT 1G *************************** 1. row *************************** val: {"statuses": [{"id": 656334251193864192, "geo": null, "lang": "en", "text ": "RT @dimitrik_fr: A preview of the latest @MySQL 5.7 Performance results -- h ttps://t.co/0mAxN4abHI -- stay tuned for more details ;-)n#MySQ…", "user": {"i d": 184376567, "url": "http://t.co/iHeEe0ZUQj", "lang": "ja", "name": "yoku0825 ", "id_str": "184376567", "entities": {"url": {"urls": [{"url": "http://t.co/iHe Ee0ZUQj", "indices": [0, 22], "display_url": "yoku0825.blogspot.jp", "expanded_u rl": "http://yoku0825.blogspot.jp/"}]}, "description": {"urls": []}}, "location ": "Japan, Tokyo", "verified": false, "following": false, "protected": false, "t ime_zone": "Tokyo", "created_at": "Sun Aug 29 11:41:33 +0000 2010", "utc_offset ": 32400, "description": "妻とせがれと娘とふかふかのぬいぐるみとMySQLが好きです。本物のイルカが好きなことにも最近気づきました。", "geo_enabled": fals e, "screen_name": "yoku0 53/103

55. アクセスするにはJSON関数を使う mysql> SELECT json_keys(val) FROM t1; +---------------------------------+ | json_keys(val) | +---------------------------------+ | ["statuses", "search_metadata"] | +---------------------------------+ 1 row in set (0.02 sec) mysql> SELECT json_length(val, "$.statuses") FROM t1; +--------------------------------+ | json_length(val, "$.statuses") | +--------------------------------+ | 15 | +--------------------------------+ 1 row in set (0.00 sec) mysql> SELECT json_keys(val, "$.statuses[0]") FROM t1G *************************** 1. row *************************** json_keys(val, "$.statuses[0]"): ["id", "geo", "lang", "text", "user", "place", "id_str", "so urce", "entities", "metadata", "favorited", "retweeted", "truncated", "created_at", "coordina tes", "contributors", "retweet_count", "favorite_count", "is_quote_status", "retweeted_status ", "possibly_sensitive", "in_reply_to_user_id", "in_reply_to_status_id", "in_reply_to_screen_ name", "in_reply_to_user_id_str", "in_reply_to_status_id_str"] 1 row in set (0.00 sec) 54/103

56. アロー演算⼦はjson̲extractと等価 mysql> SELECT val->'$.statuses[*].id' FROM t1G *************************** 1. row *************************** val->'$.statuses[*].id': [656334251193864192, 65632248720307814 4, 656313502164914176, 656297985744044033, 656288740617273344, 65 6288011647238144, 656287478416994304, 656278339376680960, 6562771 97670318080, 656271731888287744, 656263672000483328, 656248886999 146497, 656134957589360640, 656130008205778944, 65612366700116377 7] 1 row in set (0.00 sec) 55/103

57. それMySQLでやる意味あるの︖ BLOB/TEXT型と⽐べて JSONとして有効でない⽂字列をエラーにできる- 容量の最適化が⾏われているらしい- JSONをクライアントサイドでゴニョるのに⽐べてクライアント/サーバー間の転送量が削減できる- サーバー内のフェッチ/更新サイズは削減できない- generated columnが使える 56/103

58. テーブルUDFが欲しい # perl -MDBI -MConfig::Pit -MNet::Twitter::Lite::WithAPIv1_1 -MJSON -e 'my $conn = DBI->connect("dbi:mysql:d1", "root", "", {mysql_enable_utf8 => 1}); my $twitte r= Net::Twitter::Lite::WithAPIv1_1->new(%{pit_get("twitter")}, ssl => 1); foreac h (@{$twitter->search({q => "from:yoku0825"})->{statuses}}) {$conn->do("INSERT I NTO t2 VALUES (?)", undef, JSON->new()->encode($_));}' mysql> SELECT val FROM t2 WHERE val->'$.lang' = 'ja' ORDER BY val->'$.created_at ' DESC LIMIT 1G *************************** 1. row *************************** val: {"id": 656322487203078144, "geo": null, "lang": "ja", "text": "RT @matsumot ory: あんちぽさんと喋るの楽しいから喋る系は良さそう / “エンジニア採用したさ過ぎて迷走している話 - Kentaro Kuribayashi's blog” https://t.co/YmNp7yNej4 ", "user": {"id": 184376567, "url": "http://t.co/iHeEe0ZUQj", "lang": "ja", "nam e": "yoku0825", "id_str": "184376567", "entities": {"url": {"urls": [{"url": "ht tp://t.co/iHeEe0ZUQj", "indices": [0, 22], "display_url": "yoku0825.blogspot.jp ", "expanded_url": "http://yoku0825.blogspot.jp/"}]}, "description": {"urls": []}}, "location": "Japan, Tokyo", "verified": false, "following": false, "protec ted": false, "time_zone": "Tokyo", "created_at": "Sun Aug 29 11:41:33 +0000 2010 ", "utc_offset": 32400, "description": "妻とせがれと娘とふかふかのぬいぐるみとMySQLが好きです。本物のイルカが好きなことにも最近気づきまし 57/103

59. ただの関数演算なのでJOINできる mysql> SELECT val->'$.text', reply.text FROM t2 JOIN (SELECT val->'$.in_reply_to _status_id' AS id, val->'$.text' AS text FROM t2 WHERE val->'$.in_reply_to_statu s_id' IS NOT NULL) AS reply ON t2.val->'$.id' = reply.id LIMIT 5G *************************** 1. row *************************** val->'$.text': "enforce_storage_engine" text: "興味を引いたのはざっとこんなもん。" *************************** 2. row *************************** val->'$.text': "InnoDBのデフラグ" text: "enforce_storage_engine" *************************** 3. row *************************** val->'$.text': "ROLEが拡張されてデフォルトロールが指定可能に" text: "InnoDBのデフラグ" *************************** 4. row *************************** val->'$.text': "RBRでもスレーブでトリガーが走る様になった" text: "ROLEが拡張されてデフォルトロールが指定可能に" *************************** 5. row *************************** val->'$.text': "ロスレス準同期" text: "RBRでもスレーブでトリガーが走る様になった" 5 rows in set (0.01 sec) 58/103

61. generated column #とは mysql> ALTER TABLE t2 ADD v_id BIGINT AS (val->'$.id') NOT NULL; Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> SELECT v_id FROM t2; +--------------------+ | v_id | +--------------------+ | 656334251193864192 | | 656322487203078144 | | 656313502164914176 | | 656297985744044033 | | 656288740617273344 | | 656288011647238144 | | 656287478416994304 | | 656278339376680960 | | 656277197670318080 | | 656271731888287744 | | 656263672000483328 | | 656248886999146497 | | 656134957589360640 | | 656130008205778944 | | 656123667001163777 | +--------------------+ 15 rows in set (0.00 sec) 60/103

62. UNIQUE制約をつけることもできる mysql> ALTER TABLE t2 ADD UNIQUE KEY (v_id); Query OK, 0 rows affected (0.12 sec) Records: 0 Duplicates: 0 Warnings: 0 61/103

63. カラムとして存在するので、カラム名を指定しない INSERTと相性が悪い # perl -MDBI -MConfig::Pit -MNet::Twitter::Lite::WithAPIv1_1 -MJS ON -e 'my $conn= DBI->connect("dbi:mysql:d1", "root", "", {mysql_ enable_utf8 => 1}); my $twitter= Net::Twitter::Lite::WithAPIv1_1- >new(%{pit_get("twitter")}, ssl => 1); foreach (@{$twitter->searc h({q => "from:yoku0825", count => 100})->{statuses}}) {$conn->do DBD::mysql::db do failed: Column count doesn't match value count at row 1 at -e line 1. .. 62/103

64. ちゃんとユニーク制約違反でエラーになる $ perl -MDBI -MConfig::Pit -MNet::Twitter::Lite::WithAPIv1_1 -MJS ON -e 'my $conn= DBI->connect("dbi:mysql:d1", "root", "", {mysql_ enable_utf8 => 1}); my $twitter= Net::Twitter::Lite::WithAPIv1_1- >new(%{pit_get("twitter")}, ssl => 1); foreach (@{$twitter->searc h({q => "from:yoku0825", count => 100})->{statuses}}) {$conn->do _ ));}' DBD::mysql::db do failed: Duplicate entry '656334251193864192' fo r key 'v_id' at -e line 1. .. mysql> SELECT COUNT(*) FROM t2; +----------+ | COUNT(*) | +----------+ | 100 | +----------+ 1 row in set (0.01 sec) 63/103

66. generated columnで全⽂検索 KEY (v_text) WITH PARSER MeCab; t supported for generated columns. ULLTEXT KEY (v_text) WITH PARSER MeCab; Query OK, 100 rows affected (0.37 sec) Records: 100 Duplicates: 0 Warnings: 0 ext) against ('MySQL') LIMIT 3G *************************** 1. row *************************** v_id: 654573993006010368 v_text: "RT @mysql_jp: MySQL 5.7 GAと同時にMySQL Fabricによる構成や各種レプリケーション構成へのアクセスをシンプルにするMySQL RouterもGAとなっています https://t.co/u3e1E2OSoV #mysql_jp" val->'$.created_at': "Mon Oct 19 14:17:58 +0000 2015" *************************** 2. row *************************** v_id: 654573993006010368 v_text: "RT @mysql_jp: MySQL 5.7 GAおよびMySQL Router GAなどの技術情報はMySQL 最新情報セミナー2015秋にてご紹介いたします。ぜひご参加下さい。なお東京は11月9日にも第2回を開催予定です http://t.c o/0Drp08uNbW #m…" val->'$.created_at': "Mon Oct 19 14:22:05 +0000 2015" *************************** 3. row *************************** v_id: 654573993006010368 v_text: "RT @mysql_jp: MySQL 5.7の機能概要はこちらです http://t.co/FZ1Qa7HTfon新機能の一覧はOracle ACE(MySQL)のyoku0825さんによるこちらの記事をご参照下さい https://t.co/WclCToU1Xrn#mysq…" val->'$.created_at': "Tue Oct 20 02:00:03 +0000 2015" 3 rows in set (0.00 sec) 65/103

67. generated columnの計算タイミング STORED or VIRTUAL INDEXあり INDEXなし VIRTUAL(デフォルト) ALTER TABLE時 SELECT時 STORED ALTER TABLE時 ALTER TABLE時 66/103

68. 逆パターン # wget http://downloads.mysql.com/docs/world.sql.gz # mysqladmin create world # zcat world.sql.gz | mysql world mysql> SELECT * FROM world.City LIMIT 3; +----+----------+-------------+----------+------------+ | ID | Name | CountryCode | District | Population | +----+----------+-------------+----------+------------+ | 1 | Kabul | AFG | Kabol | 1780000 | | 2 | Qandahar | AFG | Qandahar | 237500 | | 3 | Herat | AFG | Herat | 186800 | +----+----------+-------------+----------+------------+ 3 rows in set (0.00 sec) 67/103

69. generated columnでJSONにしてしまう mysql> ALTER TABLE City ADD v_json BLOB AS (json_object('name', N ame, 'countrycode', CountryCode, 'district', District, 'populatio n', Population)) STORED; Query OK, 0 rows affected (0.02 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> SELECT v_json FROM City LIMIT 3G *************************** 1. row *************************** v_json: {"name": "Kabul", "district": "Kabol", "population": 1780 000, "countrycode": "AFG"} *************************** 2. row *************************** v_json: {"name": "Qandahar", "district": "Qandahar", "population ": 237500, "countrycode": "AFG"} *************************** 3. row *************************** v_json: {"name": "Herat", "district": "Herat", "population": 1868 00, "countrycode": "AFG"} 3 rows in set (0.00 sec) 68/103

70. 挙句InnoDB Memcached(JSON型だと結果がおかしくなったのでBLOB型でSTORED) # mysql < /usr/share/mysql/innodb_memcached_config.sql mysql> INSTALL PLUGIN daemon_memcached SONAME "libmemcached.so"; mysql> ALTER TABLE world.City ADD c1 int DEFAULT 0, ADD c2 bigint DEFAUL T 0, ADD c3 int DEFAULT 0; Query OK, 0 rows affected (0.23 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> DELETE FROM innodb_memcache.containers; mysql> INSERT INTO innodb_memcache.containers VALUES ('world_json', 'wor ld', 'City', 'ID', 'v_json', 'c1', 'c2', 'c3', 'PRIMARY'); # service mysqld restart # memcat --servers=127.0.0.1:11211 1 {"name": "Kabul", "district": "Kabol", "population": 1780000, "countryco de": "AFG"} 69/103

75. MySQLにないCHECK制約もgenerated columnならこの通り mysql> ALTER TABLE Country ADD v_check tinyint AS (IF(GNPOld < 1, NULL, 1)) NOT NULL; Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> INSERT INTO Country SET Name= 'Dummy', GNPOld= 0.01; ERROR 1048 (23000): Column 'v_check' cannot be null mysql> INSERT INTO Country SET Name= 'Dummy', GNPOld= 1000.01; Query OK, 1 row affected (0.00 sec) 74/103

76. 夢は無限⼤ 75/103

77. はい次 76/103

78. レプリケーション関連のあれこれを遊ぶには MySQL::Sandboxがべんり # /usr/local/bin/cpanm MySQL::Sandbox # export PATH=$PATH:/usr/local/bin # wget http://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.8-r c-linux-glibc2.5-x86_64.tar.gz # SANDBOX_AS_ROOT=1 make_replication_sandbox mysql-5.7.8-rc-linux -glibc2.5-x86_64.tar.gz # cd ~/sandboxes/rsandbox_mysql-5_7_8/ 77/103

79. GTIDのオンライン有効化今まで gtid̲modeは起動時パラメーター(SET GLOBAL不可) masterslave gtid-mode= OFF gtid-mode= ON gtid-mode= OFF ○ × gtid-mode= ON × ○ 78/103

80. GTIDのオンライン有効化 5.7 gtid̲modeはダイナミックパラメーター(SET GLOBAL可) masterslave OFF OFF̲PERMISSIVE ON̲PERMISSIVE ON OFF ○ ○ ○ × OFF̲PERMIS SIVE ○ ○ ○ × ON̲PERMISS IVE × ○ ○ ○ ON × ○ ○ ○ 79/103

81. MySQL::Sandboxで起動したてはgtid_mode= OFF # ./use_all "SELECT @@gtid_mode" # master @@gtid_mode OFF # server: 1: @@gtid_mode OFF # server: 2: @@gtid_mode OFF 80/103

82. 負荷をかけながらgtid_mode= ONにもっていく # mysqlslap -S /tmp/mysql_sandbox13253.sock -umsandbox -pmsandbo x --auto-generate-sql --auto-generate-sql-execute-number=1000000 -c 2 & mysqladmin -uroot -pmsandbox -S /tmp/mysql_sandbox13253.so ck -r -i 1 ex | egrep "Com_(insert|select) " 81/103

83. gtid_mode= OFF_PERMISSIVE # ./m -e "SET GLOBAL gtid_mode= OFF_PERMISSIVE" # ./s1 -e "SET GLOBAL gtid_mode= OFF_PERMISSIVE" # ./s2 -e "SET GLOBAL gtid_mode= OFF_PERMISSIVE" # ./use_all "SELECT @@gtid_mode" # master @@gtid_mode OFF_PERMISSIVE # server: 1: @@gtid_mode OFF_PERMISSIVE # server: 2: @@gtid_mode OFF_PERMISSIVE 82/103

84. GTIDは振られない(が、GTIDが有効なマスターに接続しても⽂句は⾔わない) # at 2748010 #151021 11:06:35 server id 1 end_log_pos 2748075 CRC32 0x367e304 1 Anonymous_GTID last_committed=6992 sequence_number=69 93 SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/; # at 2748075 #151021 11:06:35 server id 1 end_log_pos 2748152 CRC32 0x916da7a 5 Query thread_id=16 exec_time=0 error_code=0 SET TIMESTAMP=1445393195/*!*/; BEGIN /*!*/; .. # at 2748372 #151021 11:06:35 server id 1 end_log_pos 2748403 CRC32 0x5fa8c12 b Xid = 51208 COMMIT/*!*/; 83/103

85. gtid_mode= ON_PERMISSIVE # ./m -e "SET GLOBAL gtid_mode= ON_PERMISSIVE" # ./s1 -e "SET GLOBAL gtid_mode= ON_PERMISSIVE" # ./use_all "SELECT @@gtid_mode" # master @@gtid_mode ON_PERMISSIVE # server: 1: @@gtid_mode ON_PERMISSIVE # server: 2: @@gtid_mode OFF_PERMISSIVE 84/103

86. GTIDを振るようになる(が、GTIDが有効でないスレーブがいても⽂句は⾔わない) # at 997588 #151021 11:09:16 server id 1 end_log_pos 997653 CRC32 0x985a4456 GTID last_committed=2538 sequence_number=2539 SET @@SESSION.GTID_NEXT= '00013253-1111-1111-1111-111111111111:2539'/*! */; # at 997653 #151021 11:09:16 server id 1 end_log_pos 997730 CRC32 0xed08a25b Query thread_id=15 exec_time=0 error_code=0 SET TIMESTAMP=1445393356/*!*/; BEGIN /*!*/; .. # at 997950 #151021 11:09:16 server id 1 end_log_pos 997981 CRC32 0xc3bfdb4c Xid = 68724 COMMIT/*!*/; 85/103

87. 1台だけgtid_mode= OFF_PERMISSIVEでも⽂句を⾔ってない # ./check_slaves master port: 13253 File: mysql-bin.000003 Position: 3236509 slave # 1 port: 13254 Master_Log_File: mysql-bin.000003 Read_Master_Log_Pos: 3236509 Slave_IO_Running: Yes Slave_SQL_Running: Yes Exec_Master_Log_Pos: 3236509 slave # 2 port: 13255 Master_Log_File: mysql-bin.000003 Read_Master_Log_Pos: 3236509 Slave_IO_Running: Yes Slave_SQL_Running: Yes Exec_Master_Log_Pos: 3236509 86/103

88. gtid_mode= ONの前に # ./m -e "SET GLOBAL gtid_mode= ON" ERROR 3111 (HY000) at line 1: SET @@GLOBAL.GTID_MODE = ON is no t allowed because ENFORCE_GTID_CONSISTENCY is not ON. # ./use_all "SELECT @@enforce_gtid_consistency" # master @@enforce_gtid_consistency OFF # server: 1: @@enforce_gtid_consistency OFF # server: 2: @@enforce_gtid_consistency OFF 87/103

89. enforce_gtid_consistency= ONの前にWARNで様⼦を⾒る mysql> SELECT @@enforce_gtid_consistency; +----------------------------+ | @@enforce_gtid_consistency | +----------------------------+ | WARN | +----------------------------+ 1 row in set (0.00 sec) mysql> CREATE TABLE t2 AS SELECT * FROM t1; Query OK, 1 row affected, 1 warning (0.04 sec) Records: 1 Duplicates: 0 Warnings: 1 mysql> SHOW WARNINGS; +---------+------+---------------------------------------------------------------+ | Level | Code | Message | +---------+------+---------------------------------------------------------------+ | Warning | 1786 | Statement violates GTID consistency: CREATE TABLE ... SELECT. | +---------+------+---------------------------------------------------------------+ 1 row in set (0.00 sec) 88/103

90. gtid_mode= ON # ./m -e "SET GLOBAL gtid_mode= ON" # ./use_all "SELECT @@gtid_mode" # master @@gtid_mode ON # server: 1: @@gtid_mode ON_PERMISSIVE # server: 2: @@gtid_mode OFF_PERMISSIVE # ./check_slaves 89/103

91. ここまでホントに無停⽌ gtid_modeを変更するたびにbinlogのローテートはされる CHANGE MASTER TO master_auto_position= 1にはSTOP SLAVE が必要 90/103

92. 次 91/103

93. シングルスレッドなレプリケーション 92/103

94. マルチスレッドレプリケーション(MTS) 93/103

95. マルチソースレプリケーション 94/103

96. マルチソースレプリケーション RESET MASTER, gtid_modeは⾯倒だからやってるだけで、必須ではないです # SANDBOX_AS_ROOT=1 make_multiple_sandbox 5.7.8 # cd ~/sandboxes/multi_msb_5_7_8/ # ./use_all "RESET MASTER" # ./use_all "SET GLOBAL enforce_gtid_consistency= ON" # ./use_all "SET GLOBAL gtid_mode= OFF_PERMISSIVE" # ./use_all "SET GLOBAL gtid_mode= ON_PERMISSIVE" # ./use_all "SET GLOBAL gtid_mode= ON" 95/103

97. チャンネルを指定してCHANGE MASTER TO # ./n1 -e "CHANGE MASTER TO master_host= '127.0.0.1', master_port= 838 0, master_user= 'msandbox', master_password= 'msandbox', master_auto_pos ition= 1 FOR CHANNEL 'node2'" ERROR 3077 (HY000) at line 1: To have multiple channels, repository cann ot be of type FILE; Please check the repository configuration and conver t them to TABLE. # ./n1 -e "SET GLOBAL master_info_repository= 'TABLE'" # ./n1 -e "SET GLOBAL relay_log_info_repository= 'TABLE'" # ./n1 -e "CHANGE MASTER TO master_host= '127.0.0.1', master_port= 838 0, master_user= 'msandbox', master_password= 'msandbox', master_auto_pos ition= 1 FOR CHANNEL 'node2'" # ./n1 -e "CHANGE MASTER TO master_host= '127.0.0.1', master_port= 838 1, master_user= 'msandbox', master_password= 'msandbox', master_auto_pos ition= 1 FOR CHANNEL 'node3'" # ./n1 -Ee "SHOW SLAVE STATUS" # ./n1 -Ee "SHOW SLAVE STATUS FOR CHANNEL 'node2'" # ./n1 -Ee "SHOW SLAVE STATUS FOR CHANNEL 'node3'" 96/103

98. レプリカされてる # ./n1 -e "START SLAVE" # mysqlslap --no-drop -S /tmp/mysql_sandbox8380.sock -umsandbox -pmsandbox --auto-generate-sql --create-schema=node2 # mysqlslap --no-drop -S /tmp/mysql_sandbox8381.sock -umsandbox -pmsandbox --auto-generate-sql --create-schema=node3 # ./use_all "SHOW DATABASES LIKE 'node%'" # server: 1: Database (node%) node2 node3 # server: 2: Database (node%) node2 # server: 3: Database (node%) node3 97/103

99. ⾯⽩そうなことはいっぱい 98/103

100. 罠もいっぱい 99/103

101. 参考URL MySQL :: MySQL 5.7 Reference Manual :: 1.4 What Is New in MySQL 5.7 MySQL :: MySQL 5.7 Release Notes Complete list of new features in MySQL 5.7 中の⼈が作った「新機能完全リスト」- MySQL 5.7の新機能完全リスト | Yakst ↑をがんばって⽇本語に訳したヤーツ- ⽇々の覚書: 5.7 yoku0825ʼs Presentations on SlideShare 100/103

102. その他の情報源 MySQL 5.7を本番で使いそうな話に興味のある⼈は MySQL 最新情報セミナー2015秋- MySQL 5.7関係なく、⽇本MySQLユーザ会は15周年です :) ⽇本MySQLユーザ会会15周年記念パーティー - connpass - MySQLで全⽂検索ができるMroongaのカンファレンスはこちら Groonga Meatup 2015 - Groonga | Doorkeeper- 101/103

103. その他の情報源⽇本MySQLユーザ会のブースは本⽇だけですがわたしはこの後懇親会に⾏きます。 MroongaとPGroongaの開発者の⼈もいるよ :)- 102/103

104. Questions and/or Suggestions? 103/103

MySQL5.7で遊んでみよう

yoku0825

MySQL5.7で遊んでみよう

A particular slide catching your eye?