CentOS boot problem

GargGarg Purveyor of Lincoln Nightmares Icrontian
edited October 2013 in Science & Tech
So, I became a server admin when I started my new job and found a server on my desk. We had to power it down to move it, and now on boot, it's complaining about a file system check (see attached).

/dev/sde was the 3TB external USB hard drive, but now it seems to be /dev/sdc. Here's what the devices looked like before powering down:
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_cuuats-lv_root
                       50G   18G   32G  36% /
tmpfs                  12G  460K   12G   1% /dev/shm
/dev/sda3             485M  135M  325M  30% /boot
/dev/mapper/vg_cuuats-lv_home
                       58G  966M   54G   2% /home
/dev/sdb2             1.8T   19G  1.6T   2% /media/cuuats.data2
/dev/sdb1             1.9T  195M  1.8T   1% /media/cuuats.data
/dev/sdc               50G   18G   32G  36% /
/dev/sdd               50G   18G   32G  36% /
/dev/sde               50G   18G   32G  36% /
/dev/sde              2.7T  2.7T     0 100% /media/extBU
Running e2fsk on /dev/sde just repeats the no such file or directory error.

Hitting Ctrl-D to boot results in a boot loop during the CentOS loading bar.

Halp?

Comments

  • mertesnmertesn I am Bobby Miller Yukon, OK Icrontian
    What boot loader are you using (grub, etc)?

    Might check its configuration and see if anything is relying on /dev/sde for boot files. If so, update the path to /dev/sdc if that's where the 3TB is being mounted now.
    Garg
  • Somehow this got doubleposted... and the other thread which I replied to didn't have the image.

    First problem, you shouldn't be mounting 4 separate partitions to the same mount point (/).

    As for dealing with the fsck error, if that drive doesn't contain anything critical to the actual boot process, you can boot into single mode and comment out the entry for the drive in /etc/fstab so that it won't attempt to mount on boot. That should at least get you into the system so you can troubleshoot further.
    Gargprimesuspect
  • AlexDeGruvenAlexDeGruven Wut? Meechigan Icrontian
    The prompt you're seeing is to get you to the single-user mode ardi mentioned.

    If /dev/sde (or whatever it should or ends up being) is an external drive AND it contains essential boot files, whoever set that up needs to be shot.

    Failing that, as Ardi said - Get into single-user mode, zap the drive's entry in /etc/fstab, and reboot. It should then boot cleanly and you should be able to mess around with the external drive manually.

    I would also recommend NOT having external drives set to auto in /etc/fstab for this very reason. If the drive drops out or another device somehow steals that drive's assignment (like a USB key that got plugged in while the drive was disconnected, etc), then you're going to run into a similar situation every time you need to boot.

    If it's essential data that must be available the entire time the server is running, then the drive needs to be plugged directly into the disk controller and live inside the case.
    Garg
  • GargGarg Purveyor of Lincoln Nightmares Icrontian
    Thanks everyone for the quick responses!
    ardichoke said:

    First problem, you shouldn't be mounting 4 separate partitions to the same mount point (/).

    Yeah, I have no idea what's up with that. I don't know what those partitions even are. There are no 50GB devices in the box. The other listed devices add up to the 5 SAS drives + 1 USB drive that I know about.

    If I run df -h at the root console after the e2fsck error, the only partitions mounted are those three and the root partition.
    $ df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/vg_cuuats-lv_root
                           50G   18G   32G  36% /
    /dev/sdc               50G   18G   32G  36% /
    /dev/sdd               50G   18G   32G  36% /
    /dev/sde               50G   18G   32G  36% /


    If /dev/sde (or whatever it should or ends up being) is an external drive AND it contains essential boot files, whoever set that up needs to be shot.

    Absolutely. But that's the weird thing - it's only for backup, doesn't have any boot files, and the SOP is to only have it mounted during a backup. I'm not sure why it would even be in fstab, except I read something saying that non-root users can only mount if the drive is in fstab.

    Anyway, for now, I removed /dev/sde from fstab. Since the root login (single user mode?) mounted a readonly file system, I ran mount --options remount,rw /, then was able to edit fstab and boot.
  • AlexDeGruvenAlexDeGruven Wut? Meechigan Icrontian
    edited October 2013
    Those three aren't mounted at the same point. They're a 3-disk software RAID-5 group presented as a single 50GB (probably 60GB disks) LVM volume mounted at /.

    Looking deeper, I'm not sure how you were ever able to mount the external drive since it appears to have the same device name as one of the disks in the RAID group, which SHOULD be impossible.

    I'm going to guess that the server hasn't gone down in a LONG time, or had the external drive disconnected/unmounted before shutting down/starting up every time.
    primesuspect
  • If those drives are part of a 3-disk software RAID5, why are they showing up in df? I've never seen that before. Usually when a disk is part of a software RAID array, it doesn't show up as mounted... only the resulting logical RAID volume shows up as mounted. Case in point, my own colo which is running on a software RAID1 which has the following output from df:

    [ardichoke@natasha.stealinurmegahurtz.com] [~/Downloads] [16:16]
    [557]$ sudo df -h
    Filesystem Size Used Avail Use% Mounted on
    rootfs 4.6G 1.5G 2.9G 35% /
    udev 10M 0 10M 0% /dev
    tmpfs 723M 400K 722M 1% /run
    /dev/disk/by-uuid/3a11c4ce-80be-4e5d-9384-a4c0159208e2 4.6G 1.5G 2.9G 35% /
    tmpfs 5.0M 4.0K 5.0M 1% /run/lock
    tmpfs 2.9G 0 2.9G 0% /run/shm
    tmpfs 3.6G 228K 3.6G 1% /tmp
    /dev/md0 471M 68M 379M 16% /boot
    /dev/md4 258G 65G 180G 27% /home
    /dev/md3 14G 3.3G 9.8G 26% /usr
    /dev/md2 14G 2.5G 11G 20% /var
    primesuspect
  • GargGarg Purveyor of Lincoln Nightmares Icrontian
    edited October 2013
    I believe my server's two 2TB drives are in RAID 0 or spanning. They show up as a single 4TB device /dev/sdb in fdisk, but then they are partitioned into two 2TB partitions (if it's set up for spanning, then it is the silliest thing I have ever seen).

    Everything seems to be good over here, now. Thanks guys!

    No trace of those three 50GB drives now, after rebooting. :screwy:
Sign In or Register to comment.