IIJにおけるGlusterFS利用事例 GlusterFSの詳解と2年間の運用ノウハウ

IIJにおけるGlusterFS利用事例 GlusterFSの詳解と2年間の運用ノウハウ

1. 1. © 2016 Internet Initiative Japan Inc.
2. 2. © 2016 Internet Initiative Japan Inc. • • • • – • – – – •
3. 3. © 2016 Internet Initiative Japan Inc. • – – – – – – – – – • – – – –
4. 4. © 2016 Internet Initiative Japan Inc. • – – • – • – • – • – –
5. 5. © 2016 Internet Initiative Japan Inc.
6. 6. © 2016 Internet Initiative Japan Inc. Number of Bricks: 1 x (2 + 1) = 3 Redundancy
7. 7. © 2016 Internet Initiative Japan Inc. • – – • • • • • • • • • • • •
8. 8. © 2016 Internet Initiative Japan Inc. • –
9. 9. © 2016 Internet Initiative Japan Inc. •
10. 10. © 2016 Internet Initiative Japan Inc. • –
11. 11. © 2016 Internet Initiative Japan Inc. • – • • • – • • – • •
12. 12. © 2016 Internet Initiative Japan Inc. • – – – – • • – • –
13. 13. © 2016 Internet Initiative Japan Inc. • – • • • – – • – • – –
14. 14. © 2016 Internet Initiative Japan Inc. • – • 
 # gluster volume get <Volume名> all struct volume_options options[] = { ... { .key = {"page-size"}, .type = GF_OPTION_TYPE_SIZET, .min = 4096, .max = 1048576 * 64, .default_value = "131072", .description = "Page size with which read-ahead performs server I/O” }
15. 15. © 2016 Internet Initiative Japan Inc. • • .(mount point) foo/ sample.txt .(brick dir) .glusterfs 47/ 03/ 4703ed86-72b5-4573-997c-7e2339e2b1d7 foo/ sample.txt # getfattr -d -e hex -m . foo/sample.txt # file: foo/sample.txt trusted.bit-rot.version=0x020000000000000057a08889000abad6 trusted.ec.config=0x0000080301000200 trusted.ec.size=0x0000000000000007 trusted.ec.version=0x00000000000000010000000000000001 trusted.gfid=0x4703ed8672b54573997c7e2339e2b17d7
16. 16. © 2016 Internet Initiative Japan Inc. • – – • – • – – • – – –
17. 17. © 2016 Internet Initiative Japan Inc. • – • • • – – • – • – • – • – – • • • • • –
18. 18. © 2016 Internet Initiative Japan Inc.
19. 19. © 2016 Internet Initiative Japan Inc. 1. d2r2 をFUSEマウント GlusterFSのクラスタなら どこでも良い 2. 書き込み 3. DHT(分散ハッシュテーブル) から書き込み先を決定 4. Brickに file1 を書き込み
20. 20. © 2016 Internet Initiative Japan Inc. • – Brick1 Brick2 Brick3 /dir1 0 - 99 100 - 199 200 - 299 /dir2 100 - 199 400 – 499 300 - 399 /dir3 500 - 599 200 - 299 100 - 199
21. 21. © 2016 Internet Initiative Japan Inc. • – – • • – – – • –
22. 22. © 2016 Internet Initiative Japan Inc.
23. 23. © 2016 Internet Initiative Japan Inc.
24. 24. © 2016 Internet Initiative Japan Inc. • – • • – » – – • – • •
25. 25. © 2016 Internet Initiative Japan Inc. • echo "deadline" > /sys/block/<block name>/queue/scheduler echo "256" > /sys/block/<block name>/queue/nr_requests echo "16" > /proc/sys/vm/page-cluster sysctl -w vm.swappiness=10 sysctl -w vm.dirty_background_ratio=1 sysctl -w kernel.sched_wakeup_granularity_ns=15 sysctl –w vm.vfs_cache_pressure=50 • – PRUNEFS = “fuse.glusterfs” PRUNEPATHS = “<Brick Directory>”
26. 26. © 2016 Internet Initiative Japan Inc. • – gluster volume set <VOLNAME> cluster.entry-self-heal off • mount –t xfs –o inode64,noatime,nodiratime,attr2,delaylog,noquota
27. 27. © 2016 Internet Initiative Japan Inc. • – • • • – • – • • • • • – • – •
28. 28. © 2016 Internet Initiative Japan Inc. • – – – – – – – –
29. 29. © 2016 Internet Initiative Japan Inc. • • – • –
30. 30. © 2016 Internet Initiative Japan Inc. • – • – • – – • –
31. 31. © 2016 Internet Initiative Japan Inc. • – • – • – – –
32. 32. © 2016 Internet Initiative Japan Inc. • – • – • –