A Hands on Introduction to ZFS Pools; Part 1: Getting Started (Pool Creation, Export, Import)

The small hands on tutorial is the first one of three which shows how to manage disks through ZFS with very limited resources. It consists of the following parts:

1. Getting started, pool creation import and export of pools
2. Mirrors, managing disk failures and spare disks
3. RAIDZ2, resilvering and scrubbing 

The tutorial is based on the following extra components

• One 7 port USB 2.0 hub

• Six 1GB USB 2.0 memory sticks

The other requirement is a Solaris system. Any Solaris 10 system,
Solaris Developer Express or OpenSolaris 2008.05 will do. It works
even from the OpenSolaris Life CD boot. This means it can be tested
through any Windows PC without permanently changing the installation.

The total extra costs for this sand box should be around $60 USD. This minimal low cost kit will allow to simulate disk removal, disk transportation, mirroring, striping, RAIDZ, RAIDZ2, failing disks etc.

Getting USB disks with LEDs which flash when the USB stick is getting accessed are very helpful since they document what Solaris is doing.

Non Disclaimer: Please be cautious with your system. All commands apply significant changes to your system. Data corruption may easily occur. Have a good backup of your system. The system is not warranted to work since it depends on the USB sticks, the hub and how Solaris get's informed about status changes. ZFS hasn't been optimized for USB sticks and hick ups may occur at any time. Do not use such a configuration for important work. This is a low cost self learning sand box!

It may perform most tricks of a real storage system but it is not as reliable as a real one

The USB sticks have been initially formatted with FAT32 file systems. The FAT32 file systems will get deleted within the tutorial. The original data will be deleted through this exercise. The starting situation looks like below. A 7 port USB 2.0 hub which connected to your system. The USB sticks them self are still disconnected:

It may be useful to have a tool like

Become Super User

All operations need super user privilege since they apply severe changes to the systems

jack@opensolaris:~$

<font><font>         Password:</font></font> <br><font><font>      jack@opensolaris:~# id</font></font>

<font><font> uid=0(root) gid=0(root)<br>groups=0(root),1(other),2(bin),3(sys),4(adm),5(uucp),6(mail),7(tty),8(lp),9(nuucp),12(daemon)</font></font>

Stop the Volume Management

The USB Memory sticks are most likely formatted with FAT32. The colume manager would automatically mount them. This wouldn't be useful since they will be reformatted for ZFS.

jack@opensolaris:~# svcs -a | grep vol

online 2:40:18 svc:/system/filesystem/rmvolmgr:default

jack@opensolaris:~# svcadm disable svc:/system/filesystem/rmvolmgr:default

Remove unneeded Devices Entries in /dev

jack@opensolaris:~# devfsadm -C

Check for Devices connected

Figure out which devices are the once you plugin by using rmformat. The command below shows the initial status.

Warning: The assigned device entries will be most likely very different in every system. Do not rely on the controller ids used in this example. It's actually a good idea to label all USB sticks to avoid confusion later on.

jack@opensolaris:~# rmformat

Looking for devices...

1. Logical Node: /dev/rdsk/c6t0d0p0

Physical Node: /pci@0,0/pci108e,5347@2,1/storage@4/disk@0,0

Connected Device: Flash Drive SK_USB20 1.00

Device Type: Removable

Bus: USB

Size: 974.0 MB

Label: <None>

Access permissions: Medium is not write protected.

2. Logical Node: /dev/rdsk/c4t0d0p0

Physical Node: /pci@0,0/pci-ide@6/ide@0/sd@0,0

Connected Device: MATSHITA DVD-RAM SW-9585 B100

Device Type: DVD Reader/Writer

Bus: IDE

Size: 686.1 MB

Label: <None>

Access permissions: Medium is not write protected.

The list above shows the initial system with a DVD drive and a USB stick which is not part of the tutorial. The next step is to plugin the first USB stick:

The command below shows the discovery of the first “CnMemory” stick located in c0t0d0s0:

jack@opensolaris:~# rmformat

Looking for devices...

1. Logical Node: /dev/rdsk/ c0t0d0p0

Physical Node: /pci@0,0/pci108e 5347@2,1/hub@3/storage@4/disk@0,0

Connected Device: CnMemory 1.00

Device Type: Removable

Bus: USB

....

Now plug in the the second stick:

The command below shows the discovery of the second “CnMemory” stick located in c1t0d0s0

jack@opensolaris:~# rmformat

Looking for devices...

1. Logical Node: /dev/rdsk/ c1t0d0p0

Physical Node: /pci@0,0/pci108e5347@2,1/hub@3/storage@3/disk@0,0

Connected Device: CnMemory 1.00

Device Type: Removable

Bus: USB

Size: 991.8 MB

...

Repeat this step until all USB sticks are recognized and identified. See below

jack@opensolaris:~# rmformat

Looking for devices...

1. Logical Node: /dev/rdsk/c8t0d0p0

Physical Node: /pci@0,0/pci108e,5347@2,1/hub@3/hub@1/storage@2/disk@0,0

Connected Device: CnMemory 1.00

...

2. Logical Node: /dev/rdsk/c7t0d0p0

Physical Node: /pci@0,0/pci108e,5347@2,1/hub@3/hub@1/storage@3/disk@0,0

Connected Device: CnMemory 1.00

...

3. Logical Node: /dev/rdsk/c3t0d0p0

Physical Node: /pci@0,0/pci108e 5347@2,1/hub@3/hub@1/storage@4/disk@0,0

Connected Device: CnMemory 1.00

...

4. Logical Node: /dev/rdsk/c2t0d0p0

Physical Node: /pci@0,0/pci108e,5347@2,1/hub@3/storage@2/disk@0,0

Connected Device: CnMemory 1.00

...

5. Logical Node: /dev/rdsk/c1t0d0p0

Physical Node: /pci@0,0/pci108e,5347@2,1/hub@3/storage@3/disk@0,0

Connected Device: CnMemory 1.00

...

6. Logical Node: /dev/rdsk/c0t0d0p0

Physical Node: /pci@0,0/pci108e,5347@2,1/hub@3/storage@4/disk@0,0

Connected Device: CnMemory 1.00

...

7. Logical Node: /dev/rdsk/c6t0d0p0

Physical Node: /pci@0,0/pci108e,5347@2,1/storage@4/disk@0,0

Connected Device: Flash Drive SK_USB20 1.00

...

Label: <None>

Access permissions: Medium is not write protected.

8. Logical Node: /dev/rdsk/c4t0d0p0

Physical Node: /pci@0,0/pci-ide@6/ide@0/sd@0,0

Connected Device: MATSHITA DVD-RAM SW-9585 B100

...

Check your existing Zpools

jack@opensolaris:~# zpool list

no pools available

Create a striped Pool

Create a striped pool named stripedpool consisting initially of two devices:

jack@opensolaris:~# zpool create stripedpool c1t0d0p0 c2t0d0p0

Check the status

jack@opensolaris:~# zpool status stripedpool

pool: stripedpool

state: ONLINE

scrub: none requested

config:

NAME STATE READ WRITE CKSUM

stripedpool ONLINE 0 0 0

c1t0d0p0 ONLINE 0 0 0

c2t0d0p0 ONLINE 0 0 0

errors: No known data errors

Test the pool. Change the directory to it and create a tar file from directory /jack

jack@opensolaris:~# cd /stripedpool/

jack@opensolaris:/stripedpool# tar -cf jack.tar /jack

tar: Removing leading `/' from member names

jack@opensolaris:/stripedpool# ls -al

total 517

drwxr-xr-x 2 root root 3 2008-07-02 03:02 .

drwxr-xr-x 25 root root 512 2008-07-02 03:01 ..

-rw-r--r-- 1 root root 460800 2008-07-02 03:02 jack.tar

Dynamically expanding the striped Pool

Add the remaining 4 sticks to the stripe:

jack@opensolaris:/stripedpool# zpool add stripedpool c0t0d0p0 c3t0d0p0 c7t0d0p0 c8t0d0p0

New files will be striped over all 6 sticks. The write performance will grow by factor six. The availability will shrink as well by factor 6! This cinfiguration has no redundancy being build in.

Check the status of your pool.

jack@opensolaris:/stripedpool# zpool status -v stripedpool

pool: stripedpool

state: ONLINE

scrub: none requested

config:

NAME STATE READ WRITE CKSUM

stripedpool ONLINE 0 0 0

c1t0d0p0 ONLINE 0 0 0

c2t0d0p0 ONLINE 0 0 0

c0t0d0p0 ONLINE 0 0 0

c3t0d0p0 ONLINE 0 0 0

c7t0d0p0 ONLINE 0 0 0

c8t0d0p0 ONLINE 0 0 0

errors: No known data errors

Reenable the Volume Manager

All devices are now formatted. The volume manager may be reenabled: 

jack@opensolaris:~# svcadm enable svc:/system/filesystem/rmvolmgr:default

Testing the File System

A nice tool to have is iobar. It graphically displays the IO for all devices. Get this tool from the ToolsCD for free.

Create a large tar file based on /usr we now have a 6GB stripe

jack@opensolaris:/stripedpool# tar cf usr.tar /usr

tar: Removing leading `/' from member names

tar: Removing leading `/' from hard link targets

Use a different window to monitor the IO performance with iostat

jack@opensolaris:/stripedpool# iostat -xnc 2

cpu

us sy wt id

1 61 0 38

extended device statistics

r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device

838.8 0.0 3164.5 0.0 0.4 1.0 0.4 1.2 38 95 lofi1

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 lofi2

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 ramdisk1

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c5t0d0

41.2 0.0 1285.8 0.0 0.0 0.0 0.0 1.1 0 5 c4t0d0

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c6t0d0

0.0 3.0 0.0 385.7 0.7 0.6 236.6 200.7 20 20 c0t0d0

0.0 3.5 0.0 450.0 0.8 0.6 228.1 172.2 20 20 c1t0d0

0.0 3.5 0.0 450.0 0.8 0.6 227.0 172.3 20 20 c2t0d0

0.0 5.0 0.0 642.9 0.8 0.6 149.8 120.4 20 20 c3t0d0

0.0 5.5 0.0 652.7 0.6 0.6 109.9 109.5 20 20 c7t0d0

0.0 3.0 0.0 385.7 0.8 0.6 253.8 201.2 20 20 c8t0d0

Disconnecting the USB sticks

ZFS pools can be detached from systems after they have been exported.

Make sure that no shell is anymore in the directory /stripedpool by typing

jack@opensolaris:/# cd /

Export the pool to allow to disconnect the USB sticks physically

jack@opensolaris:/# zpool export stripedpool

jack@opensolaris:/# zpool list

no pools available

The USB sticks may be pulled out now:

A user wouldn't pull disks out in real live. What normally happens is that the real disks are multi attached and they can be seen from other Solaris systems. A second Solaris system would be able to import them no matter they are connected to it. The only requirement is to have the disks be visible in the /dev/dsk tree.

We demonstrate this capability by changing the order of the sticks randomly.

The remaining job is to import the file system. We skip the check for visible devices (rmformat) and assume that Solaris remapped the sticks no matter of the order. Use the command zpool import to get your filesystems back.

jack@opensolaris:/# zpool import stripedpool

jack@opensolaris:/# zpool status stripedpool

pool: stripedpool

state: ONLINE

scrub: none requested

config:

NAME STATE READ WRITE CKSUM

stripedpool ONLINE 0 0 0

c1t0d0p0 ONLINE 0 0 0

c2t0d0p0 ONLINE 0 0 0

c0t0d0p0 ONLINE 0 0 0

c3t0d0p0 ONLINE 0 0 0

c7t0d0p0 ONLINE 0 0 0

c8t0d0p0 ONLINE 0 0 0

Check your data:

jack@opensolaris:/# cd stripedpool

jack@opensolaris:/stripedpool# ls -al

total 517

drwxr-xr-x 2 root root 3 2008-07-02 03:02 .

drwxr-xr-x 25 root root 512 2008-07-02 03:01 ..

-rw-r--r-- 1 root root 460800 2008-07-02 03:02 jack.tar

Summary

This exercise showed how to

• find your devices in the device tree

• disable the volume management

• create a simple striped zpool with maximum performance

• export and reimport file systems with ZFS

The other chapters are dealing with

1. Getting started, pool creation import and export of pools
2. Mirrors, managing disk failures and spare disks
3. RAIDZ2, resilvering and scrubbing 

This tutorial has been inspired by the video “ A Thumper for the ones without budget (in German)”.

The  ZFS Administration Manual is what you will want to read to learn more about ZFS.