When a customer has a faulty disk, we will be following this procedure for SERVERware 3. However, this document will add a bit more details that can help a new engineer do the procedure without errors.
The first step is to determine which disk is faulted and on which server and this is the most common issue engineers make when providing the serial number to the customer of a faulty disk.
For example, let's presume this is the zpool status output:
NAME STATE READ WRITE CKSUM NETSTOR DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 SW3-NETSTOR-SRV1-1 FAULTED 3 0 0 too many errors SW3-NETSTOR-SRV2-1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 SW3-NETSTOR-SRV1-2 ONLINE 0 0 0 SW3-NETSTOR-SRV2-2 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 SW3-NETSTOR-SRV1-3 ONLINE 0 0 0 SW3-NETSTOR-SRV2-3 ONLINE 0 0 0
We can see that the faulted disk is SRV1-1, which means first disk on Server 1. This does not mean the first disk on the Primary server, but SRV1. SRV1 does not need to be the primary server necessarily if there was a failover in the past.
Please make sure to run sysmonit -S on the first server to see if it is in the primary state.
In the example below, the failover was already done so SRV1 was the secondary server. If this was not the case, the failover should be done because all maintenance actions must be performed on the secondary server.
Now, we run to see the list of disks with the corresponding partitions.
Garland-1 ~ # ls -lah /dev/disk/by-id/ total 0 drwxr-xr-x 2 root root 580 Maj 16 2022 . drwxr-xr-x 8 root root 160 Maj 16 2022 .. lrwxrwxrwx 1 root root 9 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV717303Z5150MGN -> ../../sdd lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV717303Z5150MGN-part1 -> ../../sdd1 lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV717303Z5150MGN-part2 -> ../../sdd2 lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV717303Z5150MGN-part9 -> ../../sdd9 lrwxrwxrwx 1 root root 9 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV71730EBD150MGN -> ../../sde lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV71730EBD150MGN-part1 -> ../../sde1 lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV71730EBD150MGN-part9 -> ../../sde9 lrwxrwxrwx 1 root root 9 Maj 16 2022 ata-KINGSTON_SEDC500M3840G_50026B7282A3CA9A -> ../../sdc lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-KINGSTON_SEDC500M3840G_50026B7282A3CA9A-part1 -> ../../sdc1 lrwxrwxrwx 1 root root 9 Maj 16 2022 ata-KINGSTON_SEDC500M3840G_50026B7282A3CAA9 -> ../../sdb lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-KINGSTON_SEDC500M3840G_50026B7282A3CAA9-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 9 Maj 16 2022 ata-KINGSTON_SEDC500M3840G_50026B7282A3CAC4 -> ../../sda lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-KINGSTON_SEDC500M3840G_50026B7282A3CAC4-part1 -> ../../sda1 lrwxrwxrwx 1 root root 9 Maj 16 2022 usb-SanDisk_Cruzer_Blade_4C530201401226108285-0:0 -> ../../sdf lrwxrwxrwx 1 root root 9 Maj 16 2022 wwn-0x50026b7282a3ca9a -> ../../sdc lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x50026b7282a3ca9a-part1 -> ../../sdc1 lrwxrwxrwx 1 root root 9 Maj 16 2022 wwn-0x50026b7282a3caa9 -> ../../sdb lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x50026b7282a3caa9-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 9 Maj 16 2022 wwn-0x50026b7282a3cac4 -> ../../sda lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x50026b7282a3cac4-part1 -> ../../sda1 lrwxrwxrwx 1 root root 9 Maj 16 2022 wwn-0x55cd2e414d7d5888 -> ../../sdd lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x55cd2e414d7d5888-part1 -> ../../sdd1 lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x55cd2e414d7d5888-part2 -> ../../sdd2 lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x55cd2e414d7d5888-part9 -> ../../sdd9 lrwxrwxrwx 1 root root 9 Maj 16 2022 wwn-0x55cd2e414d7dacf4 -> ../../sde lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x55cd2e414d7dacf4-part1 -> ../../sde1 lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x55cd2e414d7dacf4-part9 -> ../../sde9
Usually, the faulted disk would not have the partition (missing the part1), but that does not need to be necessary the case. Here, we have a good example where the following line is the only one without the partition:
lrwxrwxrwx 1 root root 9 Maj 16 2022 usb-SanDisk_Cruzer_Blade_4C530201401226108285-0:0 -> ../../sdf
However, we can see that the name of this drive is actually usb-xx which means that this is not a faulted drive, but the USB drive that was still plugged in the server, so we can ignore this part.
How to find the right drive and partition?
Since we know the faulted drive is SRV1-1, we can check the file /etc/tgt/mirror/SW3-NETSTOR-SRV1.conf. If it was SRV2-1, we would need to check /etc/tgt/mirror/SW3-NETSTOR-SRV2.conf.
Here, in this file, we see output like this:
<target SW3-NETSTOR-SRV1-1> <direct-store /dev/disk/by-id/ata-KINGSTON_SEDC500M3840G_50026B7282A3CAC4> write-cache off bs-type rdwr </direct-store> initiator-address 192.168.0.21 </target>
Now, we can locate the /dev/disk/by-id/ata-KINGSTON_SEDC500M3840G_50026B7282A3CAC4 within the initial list. We can see it is:
lrwxrwxrwx 1 root root 9 Maj 16 2022 ata-KINGSTON_SEDC500M3840G_50026B7282A3CAC4 -> ../../sda lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-KINGSTON_SEDC500M3840G_50026B7282A3CAC4-part1 -> ../../sda1
This tells us that it is /sda partition in question, which will help us determine the serial number.
To get the serial number, we run the following command:
udevadm info --query=all --name=/dev/X | grep ID_SERIAL
where we replace X with /sda.
This will give us output like this:
Garland-1 ~ # udevadm info --query=all --name=/dev/sda | grep ID_SERIAL E: ID_SERIAL=KINGSTON_SEDC500M3840G_50026B7282A3CAC4 E: ID_SERIAL_SHORT=50026B7282A3CAC4
Here, we got the serial number which can be provided to the customer to ensure the right disk is taken out.
Once the old disk is taken out and the new one put there, we list the drives again:
Garland-1 ~ # ls -lah /dev/disk/by-id/
total 0
drwxr-xr-x 2 root root 580 Jun 4 21:13 .
drwxr-xr-x 8 root root 160 Maj 16 2022 ..
lrwxrwxrwx 1 root root 9 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV717303Z5150MGN -> ../../sdd
lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV717303Z5150MGN-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV717303Z5150MGN-part2 -> ../../sdd2
lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV717303Z5150MGN-part9 -> ../../sdd9
lrwxrwxrwx 1 root root 9 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV71730EBD150MGN -> ../../sde
lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV71730EBD150MGN-part1 -> ../../sde1
lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-INTEL_SSDSC2BB150G7_PHDV71730EBD150MGN-part9 -> ../../sde9
lrwxrwxrwx 1 root root 9 Maj 16 2022 ata-KINGSTON_SEDC500M3840G_50026B7282A3CA9A -> ../../sdc
lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-KINGSTON_SEDC500M3840G_50026B7282A3CA9A-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 Maj 16 2022 ata-KINGSTON_SEDC500M3840G_50026B7282A3CAA9 -> ../../sdb
lrwxrwxrwx 1 root root 10 Maj 16 2022 ata-KINGSTON_SEDC500M3840G_50026B7282A3CAA9-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 Jun 4 21:13 ata-KINGSTON_SEDC600M3840G_50026B7686BB23F0 -> ../../sdg
lrwxrwxrwx 1 root root 10 Jun 4 21:13 ata-KINGSTON_SEDC600M3840G_50026B7686BB23F0-part1 -> ../../sdg1
lrwxrwxrwx 1 root root 9 Maj 16 2022 usb-SanDisk_Cruzer_Blade_4C530201401226108285-0:0 -> ../../sdf
lrwxrwxrwx 1 root root 9 Maj 16 2022 wwn-0x50026b7282a3ca9a -> ../../sdc
lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x50026b7282a3ca9a-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 Maj 16 2022 wwn-0x50026b7282a3caa9 -> ../../sdb
lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x50026b7282a3caa9-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 Jun 4 21:13 wwn-0x50026b7686bb23f0 -> ../../sdg
lrwxrwxrwx 1 root root 10 Jun 4 21:13 wwn-0x50026b7686bb23f0-part1 -> ../../sdg1
lrwxrwxrwx 1 root root 9 Maj 16 2022 wwn-0x55cd2e414d7d5888 -> ../../sdd
lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x55cd2e414d7d5888-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x55cd2e414d7d5888-part2 -> ../../sdd2
lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x55cd2e414d7d5888-part9 -> ../../sdd9
lrwxrwxrwx 1 root root 9 Maj 16 2022 wwn-0x55cd2e414d7dacf4 -> ../../sde
lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x55cd2e414d7dacf4-part1 -> ../../sde1
lrwxrwxrwx 1 root root 10 Maj 16 2022 wwn-0x55cd2e414d7dacf4-part9 -> ../../sde9The easiest way of recognizing the new disk is the date where it would be shown with the current date (when replaced).
Now, we proceed with the standard procedure.
- Create partition:
parted /dev/sdg --script -- mktable gpt
Note: /sdg is replaced depending on the partition for the new disk.
- Create label:
parted /dev/sdg --script -- mkpart "SW3-NETSTOR-SRV1-1" 1 -1
Note: 1-1 in SW3-NETSTOR-SRV1-1 is replaced depending on the faulty disk.
- Logout targets on primary server:
iscsiadm -m node -T SW3-NETSTOR-SRV1-1 --logout
Note: 1-1 in SW3-NETSTOR-SRV1-1 is replaced depending on the faulty disk.
- Edit /etc/tgt/mirror/SW3-NETSTOR-SRV2.conf on secondary server:
We see that /dev/disk/by-id/ata-KINGSTON_SEDC500M3840G_50026B7282A3CAC4 is there as the old one. We need to add the new one: /dev/disk/by-id/ata-KINGSTON_SEDC600M3840G_50026B7686BB23F0
- Run
tgt-admin -C 2 --update ALL -c /etc/tgt/mirror.conf -v
- Login targets on the primary:
iscsiadm -m node -T SW3-NETSTOR-SRV1-1 --login
Note: 1-1 in SW3-NETSTOR-SRV1-1 is replaced depending on the faulty disk.
- Run zdb to get the new uuid:
guid: 12957038959826436651
path: '/dev/disk/by-partlabel/SW3-NETSTOR-SRV1-1'
- Run zpool replace to replace the new uuid
zpool replace NETSTOR 12957038959826436651 /dev/disk/by-partlabel/SW3-NETSTOR-SRV1-1 -f
Note: 12957038959826436651 is the variable that is changed according to zdb result and 1-1 in SW3-NETSTOR-SRV1-1 is replaced depending on the faulty disk.