HOWTO :: SRV3 Rebuilding a single node after drive failure (3.1 and older)

If the OS disk fails to import the system ZFS pool, but its not completely dead, the following procedure can be done to pull the data from it. This is valid for older versions of serverware (3.1 or older) as versions 3.2 have backup configuration stored on NETSTOR that can be easily imported during reinstall. 

Boot from live CD and exit to shell

Import the system pool in read-only by using

zpool import

**To get the name of the OS pool**

echo 1 > /sys/module/zfs/parameters/zfs_recover

**To disable automatic zfs_recovery attempts**

zpool import -o readonly=on SYSTEM-XXXX -R /mnt/gentoo 

**To import and mount the pool into /mnt/gentoo**

Copy the following files to a USB drive, or offload them to an external server.

/etc/conf.d/net.* #except net.lo

/etc/conf.d/prerq.net.*

/etc/init.d/net.*

/etc/tgt/mirror/

/etc/passwd

/etc/group

/etc/shadow

/etc/conf.d/hostname

/etc/serverware/id

On the primary, create a script with the following commands and execute it. If it fails, check the messages and remove the specific lines from the script on which it fails. It may fail on one or more lines depending on the version of serverware being restored

#!/bin/bash



SETUPBAKDIR="/NETSTOR/.setup.bak"

mounted=$(cat /proc/mounts  | grep -P "^NETSTOR\s+/NETSTOR")



if [ -n "$mounted" ]; then

    if [ ! -d "$SETUPBAKDIR" ]; then

        mkdir -p $SETUPBAKDIR

    fi

    tar --exclude="/etc/tgt/mirror" \

        -jcf $SETUPBAKDIR/config.bak \

        /opt/sw-connector /opt/sipprot \

        /home/lxc /etc/lxc \

        /etc/sysmonit/mirror.cfg \

        /etc/sysmonit/sysmonit.cfg \

        /etc/serverware \

        /etc/tgt /etc/runlevels \

        /etc/init.d/controller \

        /etc/init.d/sw-connector \

        /etc/local.d/controller.start \

        /etc/hosts \

        /etc/resolv.conf \

        /etc/localtime

    

    if [ "$?" != "0" ]; then

        echo "Setup backup failed!" >&2

    fi

    sha1sum $SETUPBAKDIR/config.bak > $SETUPBAKDIR/config.bak.sha1

fi

Replace the OS drive with a new one, and install the same serverware version (from /etc/serverware.config) on it.

Restore all the config files from the USB to their appropriate places.

Bring up the networks (LAN/SAN/RAN) via /etc/init.d/ scripts. If there is an issue with dependencies, you can run 'rc-update -u' to clear cached dependencies.

Sync the backup files that you created from the primary (/NETSTOR/.setup.bak/config.bak) to the newly rebuilt server.

Extract the config.bak tar and its content into the apropriate directories. (tar xvf config.bak -C /)

Reboot the newly rebuilt server and confirm it joins the mirror as secondary.