TC1 downtime for disk upgrade [now up]

Red Squirrel · Post by **Red Squirrel** » Thu Jun 19, 2008 4:15 pm

TC1 will be down for an unknown period of time starting in about an hour (wenever the file backup is done). I will be turning the single drive into a two disk raid1 for redundancy, which will affect all running virtual machines which includes TC1. This is being done as to avoid coding downtime in case a disk fails. There is not really any redundancy in my main server - only backups, so now there will be redundancy to avoid having to rebuild all the VMs should a disk fail. My secondary 1TB hard drive arrived today.

I will post once the downtime starts, as right now all files on the current drive are being copied as a precaution in case something goes wrong in the nonraid -> raid transition.

Archived topic from AOV, old topic ID:3131, old post ID:19918

Red Squirrel · Post by **Red Squirrel** » Thu Jun 19, 2008 4:41 pm

Oh right... should probably turn off the VMs BEFORE I start the transfer. Good thing /data3/vms comes after /data3/p2p so had chance to think before it gets to that spot!

So TC1 is currently down. I give this at least a few hours. I will most likely be able to bring them back online before the raid is done building though. I will just start it as a degraded state with 1 drive then add the other drive afterwards once data copied over.

Archived topic from AOV, old topic ID:3131, old post ID:19922

Red Squirrel · Post by **Red Squirrel** » Thu Jun 19, 2008 7:09 pm

Data backup is done, the new drive is installed, I'm just in the process of starting to build the raid now with the new and "old" (still rather new) drive. No data should be lost so I should not even need to touch the backup, this was just a precaution.

Archived topic from AOV, old topic ID:3131, old post ID:19934

Red Squirrel · Post by **Red Squirrel** » Thu Jun 19, 2008 9:05 pm

Getting there

Code: Select all

[root@borg ~]# mdadm --misc --detail /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Thu Jun 19 19:19:06 2008
     Raid Level : raid1
     Array Size : 976759936 (931.51 GiB 1000.20 GB)
    Device Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Jun 19 21:03:29 2008
          State : active, degraded, recovering
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 3% complete

           UUID : d3fba9ac:8a56e5a4:3824655f:41cbe66f
         Events : 0.2632

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       2       8       17        1      spare rebuilding   /dev/sdb1

Data is copying right now from backup, so the rebuild will probably take a very long time, my guess is the more disk activity the longer it takes as it also has to track changes. But once this is done it will be a redundant 1TB logical drive. Even got it to email me if a disk fails. (I'll have to test that by rebooting and unplugging one)

Archived topic from AOV, old topic ID:3131, old post ID:19943

Red Squirrel · Post by **Red Squirrel** » Thu Jun 19, 2008 9:33 pm

Data still transferring, should be done shortly.

Archived topic from AOV, old topic ID:3131, old post ID:19946

Red Squirrel · Post by **Red Squirrel** » Thu Jun 19, 2008 10:51 pm

TC1 now back up. Raid is still rebuilding, will probably take at least all night, but right now everything is working (except for disk redundancy).

Archived topic from AOV, old topic ID:3131, old post ID:19949

Red Squirrel · Post by **Red Squirrel** » Thu Jun 19, 2008 11:02 pm

Just random FYI on build status for the curious minded.

Code: Select all

[root@borg ~]# mdadm --misc --detail /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Thu Jun 19 19:19:06 2008
     Raid Level : raid1
     Array Size : 976759936 (931.51 GiB 1000.20 GB)
    Device Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Jun 19 23:01:11 2008
          State : clean, degraded, recovering
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 14% complete

           UUID : d3fba9ac:8a56e5a4:3824655f:41cbe66f
         Events : 0.7601

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       2       8       17        1      spare rebuilding   /dev/sdb1
[root@borg ~]#

Bed time for me now. (I start work at 7:00)

Archived topic from AOV, old topic ID:3131, old post ID:19950