Hardware RAID vs Software RAID.

[modem]

I ran into a not-so unique issue that had never crossed my mind before. In the process of building the new DA webserver, I decided on using hardware ATA raid since I plan on having a decent amount of customers on this server all of whom are really good sized businesses in this area. So the system I designed was a dual P3 that had the Promise FastTrak100 (PDC20267) onboard ATA RAID adapter. I connected 2 - 80GB drives to that and created a mirror boot/var/usr drive. Then I installed a FastTrak100 PCI adapter and connected 2 - 250GB drives to create a 250gb mirror for the /home directory only.


The issue I have ran into is this. I am using CentOS 3.3 which as you may know is a recompile of the RedHat Enterprise Linux source code so basically it's 99% RedHat. Well when I went to install CentOS I saw both mirrored arrays during the POST. When CentOS reached the disk druid partitioning area it showed 4 drives: 80gb, 80gb, 250gb, 250gb; rather than just an 80gb and 250gb.

I did a ton of reading on this issue this evening and found out that the FastTrak100 has little or no support in OS's newer than RedHat 8. I have found some sourcecode for drivers of which I'm going to attempt to compile in to the kernel tomorrow. Regardless, my question is if this doesn't work and I can't get the hardware raid array's recongized by CentOS, I am figuring on simply creating a Software RAID of these drives.

I do know that Software RAID is slower than hardware RAID, but how much slower is it. Is this a minor difference, or such a major difference that say 50 websites would slow teh system to a crawl??

Thoughts?? Advice??

 

[rldev]

Speed is merely one concern of software raid 1. If not properly configured it can pose other problems. You are under most cirmcumstances better off with hardware RAID 1 if possible.

 

[fusionictnl]

I would recommend hardware raid too if you want Raid

I'm not that big off a fan of Raid on Linux. But offcourse there a plenty of sites explaining the ++ and -- of raid

 

[nobaloney]

I'm going to go against the two previous posts.

I'm going to go further than that and point out why I feel the way I do, and perhaps give you some links to others who feel the same way...

Hardware RAID can be quite good if it's really hardware RAID.

But it's not really hardware RAID if you need support for it in the kernel.

If it were true hardware RAID you wouldn't need drivers for it. Go ahead and buy one of those true hardware RAID cards (it's easy to figure out which ones those are; they cost over us$400).

You'll find you plug multiple drives into them, you log into their own BIOS on system start, you set up the drives, and your OS, any OS, will see only the virtual drives you've set up... and it will see them as SCSI drives. Without any kernel support.

If you need kernel support, it's NOT hardware RAID; it's software RAID implemented in a hardware chip.

And the big problem isn't that it's software RAID in the hardware chip, it's that it requires drivers which may not be available for the next version of your OS.

For example, one of my clients bought a great (on paper) system from TigerDirect. It came with drivers for Linux. Or so it said.

It did come with drivers for RHL 7.3. So that's what we set it up on. The first problem came when we had to do a kernel update. We couldn't, because we couldn't get drivers for the new kernel. And we didn't get source code for the drivers, so we couldn't rebuild the kernel.

Even if we could, we'd have to shutdown the system and put it into single user mode each time we had to update the kernel, for as long as it took to do the rebuild... perhaps hours.

But to make it even worse, there was a but in RHL 7.3 that showed up as continuous kernel dumps when used with hardware RAID. Of course RH brought out a fix. But we could never get a driver for the fixed kernel.

So we now have a beautiful, fast, elegant, $2400 doorstop.

Really. It's sitting on the table behind me. It's been there for four or five months now. Neither the board manufacturer nor the chipset manufacturer has ever brought out upgraded kernel support. It was an expensive lesson.

Software RAID support in the Linux kernel is quite good, and many people (includiing us) get excellent speed out of it; many people who've run tests say it's much faster than linux without RAID.

From a HowTo at linux.com, explaining how to install Promise RAID on RHL 7.2 (while the version is old the Promise premise is current):

Type df -k and you should see your hard disks as /dev/sdaX instead of /dev/hdaX. This is because the Promise Driver is actually a special type of Software Emulation RAID, not exactly Hardware RAID. (Promise RAID works through a BIOS Hack).

We're very happy with software RAID.

Jeff

 

[modem]

Thanks for the information. Actually if software raid at the OS level (CentOS 3.3) is not of such a degraded or poor performance level then I may have to go with it. I do realize and always have that the Promise, Highpoint, other controller cards are SCSI emulation based; but the two promise fasttrak100's I'm using DO have their own BIOS to go inside of and to create the actual ATA array. I believe it's the CTRL-F key combo.

Anyway my issue was that when I intially started with this server I had the onboard Promise FastTrak100 with two 80's to create a mirror. Then I installed an additional PCI Promise FastTrak100 that has the two 250's connected for a single 250gb mirror. When I made it to Disk Druid instead of showing a single 80GB drive and a single 250GB drive it was showing the four drives individually. On my old server I had a 3Ware 5xxxx controller card which worked perfectly at DiskDruid by only showing the array's and not the individual drives.

I'm playing with CentOS to see if it's an OS thing or if I can go back to RH9 to see if that has better luck with it. From what I was reading online is that on possible way I might be able to get around this is to install CentOS onto one of the single 80gb drives. Configure it, recompile the kernel to turn on Promise 2026xx support, reboot, and create the array with the 80gb drive. The promise cards have a way to allow me to copy data from one drive to the second when creating an inital array so we'll see if that works.

Otherwise in the end if none of these hardware options work I will have to result to using software RAID. Just a side note, I do know that when using SCSI emulated arrays/devices that the devices appear as /mnt/sd0,1,2, etc. Supposedly after recompiling the kernel, this will then allow the drives to be seen as /mnt/ataraid.

 

[nobaloney]

Anyway my issue was that when I intially started with this server I had the onboard Promise FastTrak100 with two 80's to create a mirror. Then I installed an additional PCI Promise FastTrak100 that has the two 250's connected for a single 250gb mirror.

It appears from your response that you think you're getting true hardware RAID with those Promise Controllers. Your'e not.

You're getting software RAID either way.

Configure it, recompile the kernel to turn on Promise 2026xx support, reboot, and create the array with the 80gb drive. The promise cards have a way to allow me to copy data from one drive to the second when creating an inital array so we'll see if that works.

That's a lot of work to go through to get software RAID. And if you ever have to upgrade your kernel you'll have to do it again, which means your server will be down while you do it.

And if any updated kernel doesn't have support (that's what happened to us) you won't be able to update your kernel without completely rebuilding your drive system.

For example, this post by Linus Torvolds (for any who don't know, he created Linux) and the rest of the thread, explain the problems with using the Promise controller with the newest 2.6 kernels.

Jeff

 

[modem]

Jeff - I will expand more on your previous message here a bit later. I can say that doing software RAID via the OS as well as using the controllers (they may be scsi emulation but they do offload the processing work from the P3's onto their own minature chip) results in a non-bootable system. Mainly because linux is trying to use default ide drivers for the promise controllers. I've found a link that I'm trying out to see if this works.

http://lumpi.the-lions.org/Fasttrak100.htm

If worst comes to worst, I'll just put the drives on the standard mobo controller and software raid them that way. While I could have done that yesterday, I'm someone that likes to track down a problem and solve it or at least reach the point where I can say it would require this or that to be solved.

 

[nobaloney]

If worst comes to worst, I'll just put the drives on the standard mobo controller and software raid them that way.

Which is exactly what we've done with systems that offer both Promise and standard controllers, after our own testing.

Jeff

 

[modem]

What is the speed of the software raid? I know I asked that before but I'm wondering if it's the equivelant of like an ATA66 controller, ATA33, etc?

 

[nobaloney]

Software RAID actually speeds up a lot of disk access (as does well-implemented Hardware RAID) because it allows reading and writing to happen at the same time.

The speed is of course dependent on the underlying controller.

An ATA66 controller is still an ATA66 controller, and an ATA33 controller is still an ATA33 controller.

Or am I misunderstanding something in your question?

Jeff

 

[modem]

Nope, you full understood it. The reason I asked was that back in the old college days when P2 350 was the best cpu out there, we learned that NT4's software RAID sucked up too much of the cpu time, system resources, and was overally 50% slower than that of SCSI. Obviously that WAS back in the day of 350mhz cpu's, 66mhz pci buses, ATA33 devices compared to... SCSI. Now that we have dual p3's, 133mhz busses, ATA100/133 controllers and 7200rpm/8mbcache/ATA133 drives I was wondering if that would take too harsh of a toll on system resources or response time. Although I think I've pretty much answered my own question.

This does lead into another question. I've already situated myself as to using software RAID since CentOS/RedHat has issues fully recognizing the Promise controller in RAID mode. I tried a sample installation of CentOS in preparing for what I would do for the final install, update/package wise.

Should I create a single "/" partition that is mirrored on both 2x80's? Or should I create multiple partitions "/", "/usr", "/var" and have them mirrored on both drives? Does performance suffer at either of these?

The reason I ask is that last night during my test install I created Linux RAID partitions equal on both drives. For example I created a 100mb linux raid partition on both drives, then went back and created a RAID1 md0 device by mirroring the two 100mb partitions. I did this for "/", "/boot", "swap", "/tmp", "/usr", "/var".

Did that hurt anything, did I loose any performance that way? Or is it absolutely no different than just having a single "/" mirrored?

Thanks for all your help, i GREATLY appreciate it.

 

[josetann]

Partitioning is up to you. Personally, I prefer a 100MB /boot, then around 250-500MB of swap, then the rest dedicated to /. I prefer the swap to be near the beginning, since you get the highest speed there. Also, it's nearest the beginning of the / partition, so the drive heads won't ever have to move too far (unless you completely fill up the / partition). Either way, just partition it the way you want. Raid will work just as well with many partitions as it will with just a few. Good idea on mirroring swap, it may slow things down just a tad whenever it's used, but this way someone could actually physically remove one of the drives while the system's running, and it'd never skip a beat. Had the swap partition only been on one drive, and that's the one that went out, you'd have a dead server (until you rebooted).

The main reason I choose to use linux's built-in raid, is because it's compatible with virtually any system. Does your server have ide ports? It's compatible! I had a server fail once, just had the datacenter guys pop one of the drives into a new server so I could pull the data off of it. Had the drives been setup on a promise controller in raid, they would have had to take the raid card out, plus one or both drives. Then install the drivers in linux, and hope it works. And if the promise controller was on the motherboard, forget about it. Nope, all they had to do was put the drive in a new server, and I was able to access everything just fine. Had I wanted to, I probably could have just replaced the drives in the new server with the old ones, and it would have booted up fine.

 

[nobaloney]

Nope, you full understood it. The reason I asked was that back in the old college days when P2 350 was the best cpu out there, we learned that NT4's software RAID sucked up too much of the cpu time, system resources, and was overally 50% slower than that of SCSI. Obviously that WAS back in the day of 350mhz cpu's, 66mhz pci buses, ATA33 devices compared to... SCSI.

The problem wasn't the P2 350, the 66Mhz bus or the ATA33 drives. IT was NT4's implementation of RAID.

The Cobalt RaQ3 had a 300Mhz AMD processor and ATA33 drives (I'm not sure of the bus speed). I still manage a few of them, and they run Linux software RAID very well and quite fast.

Should I create a single "/" partition that is mirrored on both 2x80's? Or should I create multiple partitions "/", "/usr", "/var" and have them mirrored on both drives? Does performance suffer at either of these?

There have been a lot of arguments on both sides.

My stand is that since your server (hopefully) runs 24/7 without your constant intervention, you want to protect it from itself as much as possible. Therefore you don't wants logs (from a DOS attack, for example) to fill up the space your /etc directory uses to write passwd and shadow; otherwise any reseller adding a new user will crash your system because the rewriting of those files by the useradd function will fail.

You don't want a user with unlimited space to use so much of it that you can't write logs any more.

You don't want /tmp files on a partition where hackers can execute files.

And so forth.

I've made two replies in this thread where I explain what I do, and why. Perhaps they'll be helpful to you; perhaps not.

Originally posted byjosetann
I prefer the swap to be near the beginning, since you get the highest speed there. Also, it's nearest the beginning of the / partition, so the drive heads won't ever have to move too far (unless you completely fill up the / partition).

This was probably very true under Windows, and especially under Windows installations which involve power saving, because when a drive isn't busy, it parks its heads at the "beginning" of the drive.

It's not true on a Linux server because the heads on a Linux server are almost always moving.

And since Linux software RAID enables reading from one (physical) drive while writing to another, you should get faster results from software RAID then without it.

A year or two ago I had dinner with Hans Reiser (of Reiser File System fame); I thought I knew a bit about drives then, but it's amazing how much you can learn even during one dinner, from someone who really knows.

Either way, just partition it the way you want. Raid will work just as well with many partitions as it will with just a few.

Under normal conditions yes, but under failure conditions, perhaps not. We had a partition failure once. Not a drive failure, but rather a partition failure. But it was limited to a small partition, and the other partitions still had full RAID functionality. I prefer that to losing all RAID functionality.

Good idea on mirroring swap, it may slow things down just a tad whenever it's used, but this way someone could actually physically remove one of the drives while the system's running, and it'd never skip a beat. Had the swap partition only been on one drive, and that's the one that went out, you'd have a dead server (until you rebooted).

Great idea!

The main reason I choose to use linux's built-in raid, is because it's compatible with virtually any system.

You make a very good point. Linux software ATA RAID drives will run on any system and will mount as non-RAID drives. I can't speak for SCSI because I don't know. Do you know if SCSI-based Linux software RAID drives will mount singly in a non-RAID system?

I probably could have just replaced the drives in the new server with the old ones, and it would have booted up fine.

We've done it when we had a system with a motherboard problem. Just moved the two drives to another server and everything worked.

But to be fair, in the same scenario with two systems with Promise controllers, it would have worked as well.

Thanks for some very good points.

Jeff