Entries tagged mdadm

Today I rebooted my desktop for the first time in a few months. This did not go well. Probably as a result of this issue with lvm/dmsetup/cryptsetup conflicting my system didn't boot, and the error message was non-helpful.

The error shown just after grub2 had started to load a system was :

Cannot find LVM volume group gold-vol

The actual cause was that I was missing the mdadm package. D'oh. My desktop has 2x500Gb drives setup as:

 sda1 + sdb1 = md1  = /boot [1Gb]
 sda2 + sdb2 = md0  = LVM storage [460Gb]

(It's only as I write this that I'm surprised that md1 + md0 are opposite to the fashion I'd have expected them to be. I guess I just created them in the "wrong" order at install time. Oops)

So without mdadm the LVM volume group on /dev/md0 couldn't be found, and that in turn meant my root filesystem couldn't be accessed at /dev/gold-vol/root.

Fixing this was a real pain. Because the system is the PXE network host on my LAN I couldn't boot it that way, and the machine has no CD-ROM drive connected.

My solution was to download and install System Rescue CD, which I placed upon a USB stick. This worked beautifully once I realised I had to boot with rescue64 to get a 64-bit kernel capable of letting me run chroot.

Oddly enough I had problems booting from USB. If I powered down my system and hit the "on" switch the system just ignored the USB stick. I noticed that my USB mouse and card reader didn't show any power lights at all - not until after grub had failed to boot the system.

So the process of booting from USB was eventually determined to be:

• Poweroff system.
• Power on system - wait for grub to fail to boot kernel.
• At this point the USB mouse and card reader would be initialised in some fashion and would show their LED lights.
• Press Ctrl-alt-delete - at which point the BIOS would allow the USB booting to occur.

Very very odd. I guess its a question of what does the "USB enabling". I'd previously assumed the BIOS would do this setup - but looking over at another system I notice that the USB mouse doesn't "come alive" until mid-way through the Linux boot process even though I know that BIOS has options for "Enabling USB mouse & keyboard". Maybe I'm missing something obvious ..?

In conclusion .. I restarted GDM for the first time in weeks and rebooted, and this was a bad idea.

ObFilm: Never Been Kissed

Tags: computers:gold, mdadm, recovery, usb, usb booting | 3 comments

So we've recently spent our first week together in Helsinki, Finland.

Mostly this has been stress-free, but there are always oddities about living in new places, and moving to Europe didn't minimize them.

For the moment I'll gloss over the differences and instead document the computer problem I had. Our previous shared-desktop system had a pair of drives configured using software RAID. I pulled one of the drives to use in a smaller-cased system (smaller so it was easier to ship).

Only one drive of a pair being present make mdadm scream, via email, once per day, with reports of failure.

The output of cat /proc/mdstat looked like this:

md2 : active raid1 sdb6[0] [LVM-storage-area]
      1903576896 blocks super 1.2 2 near-copies [2/1] [_U]
md1 : active raid10 sdb5[1] [/root]
      48794112 blocks super 1.2 2 near-copies [2/1] [_U]
md0 : active raid1 sdb1[0]  [/boot]
      975296 blocks super 1.2 2 near-copies [2/1] [_U]

See the "_" there? That's the missing drive. I couldn't remove the drive as it wasn't present on-disk, so this failed:

mdadm --fail   /dev/md0 /dev/sda1
mdadm --remove /dev/md0 /dev/sda1
# repeat for md1, md2.

Similarly removing all "detached" drives failed, so the only thing to do was to mess around re-creating the arrays with a single drive:

lvchange -a n shelob-vol
mdadm --stop /dev/md2
mdadm --create /dev/md2 --level=1 --raid-devices=1 /dev/sdb6 --force
..

I did that on the LVM-storage area, and the /boot partition, but "/" is still to be updated. I'll use knoppix/similar to do it next week. That'll give me a "RAID" system which won't alert every day.

Thanks to the joys of re-creation the UUIDs of the devices changed, so /etc/mdadm/mdadm.conf needed updating. I realized that too late, when grub failed to show the menu, because it didn't find it's own UUID. Handy recipe for the future:

set prefix=(md/0)/grub/
insmod linux
linux (md/0)/vmlinuz-3.16.0-0.bpo.4-amd64 root=/dev/md1
initrd (md/0)//boot/initrd.img-3.16.0-0.bpo.4-amd64
boot

Tags: life, mdadm | 2 comments