Oracle 10g Data warehouse ORION benchmark, size 20TB, 1200MB per second

During the last days, I have had the opportunity to test and benchmark a data warehousing hardware, that’s really fast for its money. It’s not suitable for real/available production, since it depends on disk striping over a bunch of components, but I considered it as a good way to push the limits a bit.

Result: A throughput of about 1200MB/s and nearly 1200 iops. Period. You may quickly want to see detailed results on the bottom of the page, but for fully understanding of the results look up the setup as well.

• Test hardware and its setup:
  • HP ProLiant DL585 G5
    • 4x Quad-Core AMD Opteron(tm) Processor 8356
      (=16 Cores)
    • 48GB RAM
    • 2x Adaptec 5085 SAS RAID-Controller
      • Controller firmware version: 5.2-0 (Build 16116)
      • Driver version: 1.1-5 (Build 2459)
      • 512MB Cache
      • BBU
      • writecache enabled
      • plugged into two PCI-e x8 ports serviced by different PCIe-controllers
    • 2x SAS JBOD manufactured by AIC, type XJ-SA24-316R-B with 16 HDDs each
      • HDDs: Seagate ST31000340NS, 3.5″, 1TB, 32MB Cache, 7200rpm, SATA2
      • HDD firmware version: AN05
      • each JBOD has its own RAID-Controller
    • per JBOD :
      • 2 RAID5-Arrays over 7 disks each
      • 1 hot spare
    • => 28 active disks
    • => 4 arrays
    • => no intersection in disks, no double striping
    • 4 raw devices, one raw device for each disk
• Operating system
  • SUSE Linux Enterprise Server 10 SP2
  • Kernel 2.6.16.60-0.21-smp
• Orion-Syntax
  • sequential IO

    ./orion10.2_linux -run advanced
    -testname dwhprofile_seq_fullmatrix
    -num_disks 28
    -cache_size 1024
    -size_small 8
    -size_large 16
    -type seq
    -simulate raid0
    -write 50
    -duration 30
    -matrix detailed

  • random IO

    ./orion10.2_linux -run advanced
    -testname dwhprofile_rand_fullmatrix
    -num_disks 28
    -cache_size 1024
    -size_small 8
    -size_large 16
    -type rand
    -simulate raid0
    -write 50
    -duration 30
    -matrix detailed

• Personal annotations and facts not proven hereby
  • All this was benchmarking the optimized combination possible with these hardware components. All other layouts (PCIe slots, array sizing, controller-side caching…) have been slower.
  • These Adaptec controllers surprisingly are delivering nearly the same throughput for RAID5/6/10, maybe due to their dual-core RAID CPU layout.
  • In practice, the sequential throughput of 600MB/s for reading or writing (with enabled cache, mind your BBU!) per controller is absolutely realistic. If the adapters are plugged into seperate PCIe busses, you can just add up the two values to 1200MB/s. The benchmark discussed here was tested this way. Otherwise, maybe if you are using a DL385G2 with one single PCIe bus controller, the concurrency in the PCIe will reduce your rate to 850-1000MB/s.
  • The random IO values are about 18MB/s, and seem to be rather slow in absolute numbers. It was tested with 8k up to 16k block size, where 16k was the desired database block size. The slow positioning of the 7200rpm disks slows down the whole thing with cumulative latency. This proofes the setup being a financial trade-off for a data warehousing environment with a focus on bulk loading.
    Increasing the block size to values above 256k makes IO numbers rocketing immediately (values not in the test results). The high bandwitdh of the Adaptec 5085 RAID controllers brings its profit in this case.
  • Unfortunately, the Adaptec 5085 SAS RAID controller seem not to be rock solid and comprehensible in taking automa(g/t)ic actions. One time it acted as if one disk shelf (JBOD) wasn’t available for a while, resulting in total damage of the overlaying ASM “external redundancy” diskgroup.
    Another highlight was a total lock-up of the IO subsystem after a disk failure. Rebooting fixed it without persistent damage.
    Using the firmware versions mentioned above (mind the relevance of the HDD firmware!) seems to help a bit.

Please see the detailed benchmark results and my graphs from this file (1,5MB). Within next days, I will provide the charts as pictures, but for now you will have to look into the .ods files by yourself.

Regards
Usn