RAID 5 in Windows on Mac Pro

[pipomolo42]

Hello,

I've posted this to InsanelyMac a few days ago, I'm reposting it here, because It's more about "running things on a mac" than "running OS X on things" ... I hope you don't mind.

I've been looking for a solution to setup a raid system in my mac pro that would work equally in Linux, Windows and maybe OS X.

the ESB2 southbridge supports 3 different modes:
- Legacy IDE (as PCI vendor id 2680)
- AHCI (as PCI vendor id 2681)
- RAID (as PCI vendor id 2682)

It seems that when the system is booted, AHCI mode is used (I checked the status in rEFIt's efi console), and Mac OS X also reports that it's using AHCI mode.

Unfortunately, when booting Linux or Windows through the apple CSM, the ESB2 is switched back to Legacy IDE mode, so you cannot use the RAID features of the board.

While reading through the ESB2 specification (http://www.intel.com/design/chipsets/datashts/313082.htm) it seems that AHCI mode supports RAID 0 and 1, and RAID mode supports RAID 0, 1 and 5. but RAID mode also requires an "Option ROM" to be on the mainboard. Specification also states that it is possible to switch the chipset's mode, by just writting to offset 0x90 of the ide controller PCI configuration space (the PCI device at address 00:00:1f.2).

I confirmed this mode switching under Linux, using the setpci tool to write to the config space and to check the vendor id afterwards.

Eventually, the goal would be to be able to set up a RAID 5 array that would be accessible from both Windows and Linux (don't know if OS X would support it, as it is not provided as a standard feature, but I only use OS X for firmware updates anyways)

The question I still have is : what about the "Option ROM" ? is it only used for array configuration ? In which case Linux and Windows can achieve the same, or is it really necessary for the array's "sanity" ?

Then, my guess would be that Apple's CSM is responsible for setting this controller mode (They probably decided to use the legacy IDE mode as AHCI and RAID mode require a specific driver to be loaded from a floppy at windows installation, and the Mac Pro does not have a floppy drive).

Actually the driver could also be slipstreamed into the installation CD. And Vista supports "F6 loading" from a USB key or CD as well as floppy. And the driver is included in the linux kenel for some versions now. So, it wouldn't have been that difficult to offer "bootcampers" a better option than legacy IDE.

The options I see to restore the full fearured behaviour would be:

a) patch a bootloader (like grub or lilo) to write to the PCI config space, and use this bootloader to boot all operating systems that require BIOS emulation. The problem is that these bootloader generally use the BIOS to access the hard drives, so I don't know how safe it is to switch the controller mode outside of the bios.

b) patch the apple CSM so that it does not set the controller to Legacy IDE mode. I first thought this would require reading/writing to the firmware directly, but actually, I think it could be achieved more easily from rEFIt, just writting to RAM between the LoadImage and StartImage calls.

Anyone has ideas or hints about this ?

[pipomolo42]

Hello,

For your information, I successfully managed to install and boot Linux and WinXP with the ESB2 in RAID mode. I patched grub's stage1 so that it writes to the correct pci configuration space address. patch and resulting stage1 are here: http://boeglin.org/static/macpro/ .

For Linux to get it to work, it's as simple as:
- make sure your kernel has the ahci module
- reboot

For windows it's like:
- add an entry to grub to chainload to your windows partition
- add an entry to grub following this trick: http://www.lrz-muenchen.de/~bernhard/grub-chain-cd.html (http://www.lrz-muenchen.de/%7Ebernhard/grub-chain-cd.html)

I then simply used this last entry to chainload to the windows installation CD (in which I previously slipstreamed the chipset drivers (infinst) and AHCI drivers available here: http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&Inst=Yes&ProductID=2308&DwnldID=11207&strOSs=All&OSFullName=All%20Operating%20Systems&lang=eng , using nlite)


After next reboot, I simply used "refit -> grub -> windows entry" instead "refit ->windows"
to get to a winXP capable of RAID.

Unfortunately, it doesn't completely work: the Intel RAID configuration utility appears to be confused as no Option ROM was loaded by Apple's CSM, and it only offers RAID 0 (maybe as a "default" first entry, as it could not get the capabilities from the Option ROM INT13h functions) ...

From there, there are several possible options:

- simply write 0x40 instead of 0x80 to the PCI config space to benefit at least from AHCI instead of RAID. (it can simply be achieved by replacing the 0x80 by a 0x40 at offset 0x5d of the stage1 image I provided, or directly in the first disk sector in which you installed grub). According to Intel's spec, AHCI mode would allow at least the use of RAID 0 and 1 in Windows (I didn't have time to test yet).

- "if" the Option ROM is present in Apple's CMS and "if" the CSM respects the BIOS specs and "if" I manage to set the ESB2 in RAID mode before the CSM tries to load the Optopn ROM, it should be executed, and RAID should then be working 100% (maybe writing a EFI driver that will patch the CSM in memory before it is loaded).

- other possible way (?): the EFI API has a method named LoadOptionRom, I haven't checked yet what it does exactly.

- yet other possible way (?): there are many mainboards or systems by TYAN, Intel, HP, Dell, IBM using the same ESB2 southbridge and with downloadable bios file. We could just extract the Option ROM from one of these, and see how we can load it to memory.

- create an online petition and whine at apple so that they publish and document a new firmware for the mac pro including the Option ROM and the ability to select in which mode we want the ESB2 to boot (using a NVRAM variable, or known memory address).

Edit: 0x40 has to be written at offset 0x5d, not 0x5b as I previously wrote, sorry for the mistake (but anyway, there is no 0x80 at 0x5b in the file, so that was easy to guess, I hope ;) )

[pipomolo42]

(Could one of the admins or moderators change the topic to something like "RAID 5 in Windows on Mac Pro", maybe it will attract more people with knowledge or ideas about this. Thanks!)

[zarmanto]

No problem; 'tis edited as requested.

[pipomolo42]

Ok, so I finally tried AHCI mode: it does not provide any option for handling RAID arrays, so it seems that hooking an Option ROM to int13h is the only way to go for this ...

On the other hand, AHCI should still be a much better option than the legacy IDE selected by Apple: it supports NCQ, and it also allows DMA transfers (whereas Legacy IDE mode uses only PIO, which consumes CPU), and the Intel spec also claims that AHCI offers a better power managment ...

Now I need to think a bit more about the possibilities to get an option rom and have it "executed" at bootup.

( and thanks for the edit :) )

[pipomolo42]

Hi,

I managed to get the option rom from a bios image of a motherboard using the samle chipset.

I also started to look more closely at the Mac Pro EFI firmware:

The compatibility BIOS is included in it, as GUID 1547B4F3-3E8A-4FEF-81C8-328ED647AB1A. It starts with the strings "Copyright (C) 1998-2004 Insyde Software Corp.", "SYS CORE", "5.31.DT", "07/29/05".

This GUID is referenced in the DXE driver BC6D08DC-865D-4FFE-8B7A-FB5FB04F12F1, which is in turn referenced in the EFI application 2B0585EB-D8B8-49A9-8B8C-E21B01AEF2B7, which is the legacy OS loader

The next step will be to find where in these files is the controller put in legacy IDE mode, and put it in RAID mode instead, and then load the option rom in the option rom space (0xc0000-0xf0000).

[Beliyaal]

This is great news. I was struggling with this when I first installed my Mac Pro and gave up. I only need AHCI as I use Windows software raid 0 only.

Hopefully it's possible to change the driver without reinstalling Windows.

When I did my testing I noticed that the throughput was shared by two SATA ports. This meant if I put two drives in the same raid on port 0 and 1, or port 2 and 3 it wouldn't perform as well. Because AHCI wasn't supported (port 4 and 5 isn't recognized by windows) I ended up buying an PCIE sata card so I could get 3 channels.

Would this throughput limit be fixed by putting the controller in AHCI mode or is it a hardware limitation?

[pipomolo42]

Hi,

As far as I know, it's not possible to switch from IDE to AHCI or RAID mode without reinstalling, as winXP seems to have issues finding the disk to boot on ...

Edit: actually, it *could* work if you install the AHCI driver in IDE mode and then reboot to AHCI mode, but I'll have to try first to confirm

Edit2: I just tested, and it works:
- update your IDE controller (the one whose device ID is 2680) with the AHCI driver in http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&Inst=Yes&ProductID=2308&DwnldID=11207&strOSs=All&OSFullName=All%20Operating%20Systems&lang=eng
- reboot windows, using the patched grub as a bootstrap, that will set the controller to AHCI mode

But I think it'll not work for RAID, as winXP registers it as a different controller class (SCSI/RAID)

If you have good disks benchmarks to point me to, I could run them in all modes, and if they show a big difference in performances, maybe try to use them as an argument to ask Apple to provide a firmware that fully supports its hardware capabilities ... Would be easier than me trying to fix it on my own ;)

[pipomolo42]

Here is also one link you'ld want to read before switching to AHCI (although I didn't need to apply it, forcing the AHCI driver installation was enough):
http://support.microsoft.com/kb/922976

Edit: Ah, sorry, this one's only for Vista ... Well, Wikipedia also contains good advices:
http://en.wikipedia.org/wiki/AHCI

[pipomolo42]

So, I finally made some benchmarks, using Intel Iometer (http://www.iometer.org/) (http://www.iometer.org/%29).

I plugged in two hard disks, in the first two Sata slots, and setup 4 workers on each disk, and tested using 512B, 2kB and 64kB blocks.

As you can see in the results I posted on http://www.flickr.com/photos/10695072@N04/sets/72157601084232467/, the AHCI mode completely beats the legacy IDE mode.


with 512B blocks: Total I/O per second: +24%

with 2kB blocks: Total I/O per second: +22%

with 64kB blocks: Total I/O per second: +19%

So, the more the controller is stressed, the better it performs in AHCI ...

Also, with 64kB blocks, it's interresting to note that the 19% increase also means that the Total MB per second goes from 17.09 to 20.32!

[Beliyaal]

I used ATTO disk benchmark. This benchmark measures raw throughput rather than accesstime. I tried it with two 150g raptors and it was obvious that the performance was worse for some ports together. I don't remember if it was port 1 and 2 or port 1 and 3 that was the problem though.

https://www.attotech.com/software/files/drivers/win_app_hbautil_340.exe

[scaife]

I know this is definitely off topic, but I have been trying all night to get this solution working.

All I really want is to get my controller in AHCI in Windows. I don't use Linux so I was trying to build a single 100mb ext3 filesystem to boot grub from. I have 3 hard drives, #s 1 and 2 are HFS+ only and #3 is for Windows.

I booted to a Ubuntu 7.04 cd I have lying around and installed grub to that 100mb partition I talked about earlier. I was going to just boot the windows install CD from grub, so I created menu.lst and added the requisite entries. I copied the stage1 file from the posted link to /mnt/boot/grub/ (where I had mounted it).

Rebooted, select "boot linux" in refit... and get a message saying that no bootable disks were found.

I see that most people have a linux distro already installed for this to work, but I am trying to avoid that. Is there a good way to get grub running on this drive so I can boot Windows?

[pipomolo42]

Hi, You also have to run grub for it to be installed on the partition. You'll need my patched stage1, a normal stage2 and the grub binary.

run "grub --device-map=/dev/null"
(--device-map=/dev/null is to bypass grub's disk detection)
then issue the following commands within the grub interpreter:

- device (hd2) /dev/sdc
(to tell grub that /dev/sdc is your third hard disk (grub counts from 0))

- root (hd2,0)
(to tell grub that stage2 is on the first partition of the third hard disk, you might need to replace the 0 by your actual partition number)

- setup (hd2)
(to install the stage1 to the disk's mbr, and make it point to the stage2, it should display wether it manages to do so)

- quit
(to quit grub)

Then, build your menu.lst and it should work.

[scaife]

Ther part I was missing on the grub install was the MBR -- I mounted the boot partition I created:

mount -t ext2 /dev/sdc1 /mnt

then:

grub-install --root-directory=/mnt /dev/sdc

Which worked fine... I copied stage1 to /mnt/boot/grub and created a menu.lst that matched the entries at the cd-booting-with-grub site listed in your post. I dropped in memdisk and the sbm disk image... rebooted, voila! Up comes SBM -- unfortunately it doesn't list a cd-rom drive and the only two options I have are Boot to BIOS and Reboot. Interestingly enough, if I muck around too much in SBM it just locks up hard.

I got rid of the SBM and memdisk images and got the newest memdisk image and tried not one, but two other SBM images I found, including the one in /install on the Ubuntu disc. One locked up immediately, the other (the one from the disc) booted fine -- it now listed all of my hard drives as well as floppy fd0 and "removable disk". I dropped down and hit removable disk and it locked up promptly again.

Honestly I think the removable disk it's referring to is my USB flash drive, but maybe not. I still don't think SBM is detecting my CD drive and I am sure as hell not able to boot the Windows CD via grub/memdisk/sbm.

Anyway, I really appreciate the help you've been so far! I have been trying to get this to work since I got home from work and it's kind of disheartening... I want to play some Half-Life but I don't have Windows installed! :P Any ideas?

[pipomolo42]

Ah, sorry, I forgot to mention: I use the SBM image I linked to, and when it proposes the two options Boot to BIOS and Reboot, I just select Boot to BIOS, and for some reason, it boots the cdrom. Anyway, if it doesn't work like this for you, SBM is not mandatory: you can just install windows in IDE mode, and then "convert" it to AHCI mode, by installing the AHCI driver and then start using grub to boot windows.

Also:
- if you have a USB drive connected, SBM might try to boot on it, instead of the CD
- you need to make sure that grub installs "my" stage1, just copying it on the partition will not be effective

I don't exactly know why SBM works although no option shows up, and from where it decides the boot order, so there are changes it does not work, and you might want to use "the install IDE windows + convert to AHCI" way instead, which does not have such randomness issue ;)

[pipomolo42]

I used ATTO disk benchmark. This benchmark measures raw throughput rather than accesstime. I tried it with two 150g raptors and it was obvious that the performance was worse for some ports together. I don't remember if it was port 1 and 2 or port 1 and 3 that was the problem though.

https://www.attotech.com/software/files/drivers/win_app_hbautil_340.exe
I don't know how noticeable it was for you, but I tried your benchmark with windows stripped volumes on ports (2, 3) and then ports (3, 4) in AHCI mode, and the difference is less than 5%.

[scaife]

Ha! Fantastic that when I put the old stuff back and hit quit to BIOS it worked fine....... except now it is still days later and I still can't get it to work. Windows refuses to install on anything that's not the first drive and removing the other drives takes away my refit install.

I guess I am going to try the IDE switcheroo on it -- I will let you know how it goes. I will try to install Windows via the Boot Camp method, then resize the partition and drop grub on the new partitioned space...

[pipomolo42]

Hi,

there is another trick you can use in grub to virtually swap your disks 1 and 3. Just add the following commands:

map (hd0) (hd2)
map (hd2) (hd0)

[Beliyaal]

After twelve torturous hours I managed to get Vista to boot with AHCI. I should have read your post more carefully and I would have noticed that your stage1 file puts the controller in RAID rather than AHCI mode :) Btw the offset is 0x5d not 0x5b to patch the file. I recompiled grub stage1 and have put it up on my website for convenience.

To get it to install in Vista:
* Run regedit and change "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servic es\msahci" key "Start" to 0
* Install ubuntu on a partition somewhere.
* From terminal:

wget http://www.olofsson.info/grub/stage1
sudo cp stage1 /boot/grub/stage1
sudo grub --device-map=/dev/null

Then issue the commands as described in the previous post, but with the correct drive and partition numbers. Add your Vista partition to /boot/grub/menu.lst


The next time you start Vista from the grub menu it will automatically install drivers for the AHCI controller. They are horribly slow though and you need to install Intel Matrix Storage Manager (http://downloadcenter.intel.com/Detail_Desc.aspx?agr=N&ProductID=2532&DwnldID=13806&strOSs=150&OSFullName=Windows%20Vista*%2064&lang=eng) to get full performance.

With the controller in AHCI mode all channels perform at full speed at the same time for me, and I'm able to use port 5 and 6 on the motherboard so I could throw out my PCIE controller.

Now all I need for my Mac Pro to be perfect would be Apple to release a Geforce 8800 equipped Mac Pro so I could flash my bios and be able to run OS X :)

[scaife]

After twelve torturous hours I managed to get Vista to boot with AHCI. I should have read your post more carefully and I would have noticed that your stage1 file puts the controller in RAID rather than AHCI mode :) Btw the offset is 0x5d not 0x5b to patch the file. I recompiled grub stage1 and have put it up on my website for convenience.


Beliyaal -- a mini update here. I have got Windows to boot normally... and it will also boot normally if I boot it from grub. Once I get it working, I will be happy to document my steps ----- If I install the AHCI drivers, then reboot via grub, it blue screens immediately. I can tell it to install from last known good config and it will reboot with the old drivers, but obviously that doesn't help.

A little further digging and it appears that even when booting from grub with the patched ACHI (not RAID) stage1, the PCI_DEV that is coming across is the 2680. I downloaded the stage1 again and diffed it against the one I currently have installed, and they are the same.

That said, I think I am on the downhill stretch with just one hurdle left. I haven't been able to roll my own stage1 from the Ubuntu live CD, so I have been working with (until today) the RAID stage1. Can you verify that the stage1 at your site is working properly, Beliyaal?

Or, as a secondary question, does the RAID stage1 also enable AHCI as well as RAID?

OR, as a tertiary question, am I just installing the damn stage1 wrong? Or not using it at all? It's installed in /boot/grub, I overwrote the stock one.

[Beliyaal]

Can you verify that the stage1 at your site is working properly, Beliyaal?

Or, as a secondary question, does the RAID stage1 also enable AHCI as well as RAID?

OR, as a tertiary question, am I just installing the damn stage1 wrong? Or not using it at all? It's installed in /boot/grub, I overwrote the stock one.

Yes I have verified that the file is configured properly (0x40 instead of 0x80).

No, AHCI will not work if the controller is configured as RAID (windows will not recognize any disk).

When you have copied the stage1 file to the grub directory you have to run grub again to apply the stage1 (stage1 is the code that goes in the mbr on the harddrive). If your controller comes up as 2680 it's in compatibility mode not AHCI or RAID.

[pipomolo42]

After twelve torturous hours I managed to get Vista to boot with AHCI. I should have read your post more carefully and I would have noticed that your stage1 file puts the controller in RAID rather than AHCI mode :) Btw the offset is 0x5d not 0x5b to patch the file.


Ah, you're right, sorry for the mistake. I edited my previous post.

The next time you start Vista from the grub menu it will automatically install drivers for the AHCI controller. They are horribly slow though and you need to install Intel Matrix Storage Manager (http://downloadcenter.intel.com/Detail_Desc.aspx?agr=N&ProductID=2532&DwnldID=13806&strOSs=150&OSFullName=Windows%20Vista*%2064&lang=eng) to get full performance.

With the controller in AHCI mode all channels perform at full speed at the same time for me, and I'm able to use port 5 and 6 on the motherboard so I could throw out my PCIE controller.

Nice, I'm glad to hear it works for others as well :)

[scaife]

FYI -- I was able to get everything working, I really appreciate everyone's help! My biggest problem was not putting the Windows partition at the end of the drive. Actually, it may have been not actually installing the new stage1 file :P

Just wanted to give a heads-up that this actually works and I am so happy someone figured it out!

[scaife]

Anyone else that has done this -- have you seen issues with XP only showing 1GB of RAM instead of 2? I've got 3GB and am used to seeing it show 2gb, but after this install I'm showing 1....

[downset04]

i have been looking for a way to get my sata dvd drives to work in windows XP under bootcamp, after searching for more than a year and many failed attempts with drivers and refit i stumbled on this page

i installed unbuntu and windows xp
changed the driver in xp to the intel ahci one
booted in ubunted and loaded the stage1 grub file as posted in this thread

and my xp finaly sees my sata drives

you cannot believe how happy this makes me

thanks you for figuring this out

[kameleon2004]

Been having a hell of a time getting this to work... I've got a MackBook Pro (Santa Rosa). I have 4 partitions on a 160GB sata disk: EFI System (200mb), HSF+ (35GB), NTFS (110GB), and a 10GB boot [fat32]. I have rEFIt on the EFI partition, working great. I've been trying like mad to install grub to the last partition. I copied grub into /boot/grub/, then load up GParted LiveCD (since it's built with grub), get to a terminal, and use:

grub --device-map=/dev/null
- device (hd0) /dev/sda
- root (hd0,3)
- setup (hd0,3)
- quit

per the previous post (I just want it installed on the partition, not the entire disk). It returns:

searching for stage 1... successful
searching for stage 2... successful
embed fat 1.5... fail
embed fat 1.5... fail
install grub blah blah.... successful

I then reboot, rEFIt shows Linux on sda4 (hd0,3), and I choose it. It then shows:

GRUB Loading Stage 2...
_

with a blinking cursor. Forever. And Ever.

Any ideas? I haven't even tried the patched stage 1 yet, just trying to get grub functional.

As a side note, I burned an iso of SGD (super grub cd), and it did not boot either, with the same result - hanging on "loading stage 2".

[kameleon2004]

Problem fixed. Apparently, I needed a patched version of stage 2 to work on my mac book pro. Found what I needed here (http://www.scl.ameslab.gov/Projects/mini-xen/index.html).

After that, I booted into vista w/ grub, and ahci was recognized. I installed the Intel driver, and everything works great. Thanks for the grub hack!

Also, it seems a rEFIt patch would be a bit more elegant... any work in that direction?

[sirris101]

I'm offering $100 to anyone who can help me do this on my Mac Pro!

http://forums.macrumors.com/showthread.php?t=457725