SGI: Hardware

High-def s.w. JBOD RAID results on Octane2 (500MB/sec)...

mapesdhs
From Edinburgh, Scotland
Who joined Nov. 10, 2003, 4:17 p.m.

Wrote the following at March 29, 2007, 7:36 p.m...

Dear All,

I finally had a chance to finish off the Octane SCSI software RAID tests I was doing
a couple of months back, so here are the results.

Btw, is this a record for Octane? Not seen numbers this high before...

1920 x 1080 HD, 4:4:4:4 (4 bytes per pixel):

Code: Select all


   # req_size  fwd_wt  fwd_rd  bwd_wt  bwd_rd  rnd_wt  rnd_rd
   

   #  (bytes)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)
   

   #---------------------------------------------------------
   

   8404992  453.61  502.87  438.06  316.56  438.17  309.13
   

   16809984  455.71  509.40  451.16  392.44  450.71  383.15
   

   33619968  455.03  511.44  452.22  444.92  453.11  442.38
   

   67239936  451.82  511.88  457.21  475.78  457.68  477.13
   

   134479872  446.51  511.30  456.71  493.18  457.22  498.42
   

   268959744  433.79  507.24  454.86  503.18  456.14  502.94

That's about 2X faster than real-time for HD. 8) And I broke past 500MB/sec! Woohoo!! :D

The results for HD 4:2:2 (2 bytes per pixel) are:

Code: Select all


   # req_size  fwd_wt  fwd_rd  bwd_wt  bwd_rd  rnd_wt  rnd_rd
   

   #  (bytes)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)
   

   #---------------------------------------------------------
   

   4276224  436.45  493.96  418.19  244.55  376.18  235.73
   

   8552448  441.24  502.54  431.44  360.59  430.71  344.31
   

   17104896  444.07  507.07  440.83  426.15  440.62  409.81
   

   34209792  443.23  508.76  443.35  461.72  443.36  455.78
   

   68419584  441.08  509.27  445.51  483.95  444.81  484.57
   

   136839168  435.90  508.20  445.48  494.18  444.77  496.57
   

   273678336  423.67  504.80  443.51  498.36  443.54  499.06

Here's the system spec (note that one of the QLA12160s in the PCI
cage was not used for any of these tests - no point as the XIO connection
is already maxed out with just one card; using both would actually slow
things down):

Dual-R14K/600 Octane2, 3.5GB RAM, 73GB disk, V12 graphics
PCI cage with 2 x QLA12160 cards + Gbit Enet
2 x XTalk Adapter, each with QLA12160
3 x dual-channel JBOD RAID unit, 12 x 73GB disks per unit, 2.45TB total space.

Here's the hinv (controllers 4 and 5 not used for these tests):

Code: Select all


   2 600 MHZ IP30 Processors
   

   CPU: MIPS R14000 Processor Chip Revision: 2.4
   

   FPU: MIPS R14010 Floating Point Chip Revision: 0.0
   

   Main memory size: 3584 Mbytes
   

   Xbow ASIC: Revision 1.4
   

   Instruction cache size: 32 Kbytes
   

   Data cache size: 32 Kbytes
   

   Secondary unified instruction/data cache size: 2 Mbytes
   

   Integral SCSI controller 0: Version QL1040B (rev. 2), single ended
   

   Disk drive: unit 1 on SCSI controller 0
   

   Integral SCSI controller 1: Version QL1040B (rev. 2), single ended
   

   Integral SCSI controller 4: Version QL12160, low voltage differential
   

   Integral SCSI controller 5: Version QL12160, low voltage differential
   

   Integral SCSI controller 2: Version QL12160, low voltage differential
   

   Disk drive: unit 8 on SCSI controller 2
   

   Disk drive: unit 9 on SCSI controller 2
   

   Disk drive: unit 10 on SCSI controller 2
   

   Disk drive: unit 11 on SCSI controller 2
   

   Disk drive: unit 12 on SCSI controller 2
   

   Disk drive: unit 13 on SCSI controller 2
   

   Integral SCSI controller 3: Version QL12160, low voltage differential
   

   Disk drive: unit 8 on SCSI controller 3
   

   Disk drive: unit 9 on SCSI controller 3
   

   Disk drive: unit 10 on SCSI controller 3
   

   Disk drive: unit 11 on SCSI controller 3
   

   Disk drive: unit 12 on SCSI controller 3
   

   Disk drive: unit 13 on SCSI controller 3
   

   Integral SCSI controller 10: Version QL12160, low voltage differential
   

   Disk drive: unit 8 on SCSI controller 10
   

   Disk drive: unit 9 on SCSI controller 10
   

   Disk drive: unit 10 on SCSI controller 10
   

   Disk drive: unit 11 on SCSI controller 10
   

   Disk drive: unit 12 on SCSI controller 10
   

   Disk drive: unit 13 on SCSI controller 10
   

   Integral SCSI controller 11: Version QL12160, low voltage differential
   

   Disk drive: unit 8 on SCSI controller 11
   

   Disk drive: unit 9 on SCSI controller 11
   

   Disk drive: unit 10 on SCSI controller 11
   

   Disk drive: unit 11 on SCSI controller 11
   

   Disk drive: unit 12 on SCSI controller 11
   

   Disk drive: unit 13 on SCSI controller 11
   

   Integral SCSI controller 8: Version QL12160, low voltage differential
   

   Disk drive: unit 8 on SCSI controller 8
   

   Disk drive: unit 9 on SCSI controller 8
   

   Disk drive: unit 10 on SCSI controller 8
   

   Disk drive: unit 11 on SCSI controller 8
   

   Disk drive: unit 12 on SCSI controller 8
   

   Disk drive: unit 13 on SCSI controller 8
   

   Integral SCSI controller 9: Version QL12160, low voltage differential
   

   Disk drive: unit 8 on SCSI controller 9
   

   Disk drive: unit 9 on SCSI controller 9
   

   Disk drive: unit 10 on SCSI controller 9
   

   Disk drive: unit 11 on SCSI controller 9
   

   Disk drive: unit 12 on SCSI controller 9
   

   Disk drive: unit 13 on SCSI controller 9
   

   IOC3/IOC4 serial port: tty1
   

   IOC3/IOC4 serial port: tty2
   

   IOC3 parallel port: plp1
   

   Graphics board: V12
   

   Integral Fast Ethernet: ef0, version 1, pci 2
   

   Gigabit Ethernet: eg0, PCI slot 1, firmware version 12.4.10
   

   Iris Audio Processor: version RAD revision 12.0, number 1

In case anyone's interested, here are the results for using the same array for
uncompressed PAL and NTSC in both 4:2:2 and 4:4:4:4. Overall the results
are always at least 10X faster than real-time:

Summary (MB/sec):

Code: Select all


   PAL:
   

   4:2:2     4:4:4:4
   

   

   Write Speed:     354        394
   

   

   Read Speed:     432        470
   

   

   

   NTSC:
   

   

   4:2:2     4:4:4:4
   

   

   Write Speed:     321        375
   

   

   Read Speed:     412        453

Now for some other interesting info...

If using just 6 x 73GB disks on 2 controllers (3 disks per controller), with one
QLA12160 card, it gives 161/169 write/read speed. These are older model drives,
max speed about 35MB/sec each.

If using 6 x 146GB disks in the same way, but newer drives (U320 Seagate,
more like 75MB/sec in theory), the results are identical (161/169 write/read sped).

Next, if one uses 6 x older 73GB on 2 controllers but this time with 2 different
QLA12160 cards in different XIO slots (eg. 2 x XTalk, or PCI cage = XTalk), using
just 1 contoller from each card, then the speed does go up to 181/214 write/read,
ie. faster, but only by 12%/26% respectively. Are the disks maxing out? Well...

Now use 6 x fast 146GB disks, 2 controllers, 2 cards, different XIO slots, etc.,
one gets 248/273 write/read speed - a huge increase!

This proves the typical maximum XIO-to-PCI bandwidth is about 166MB/sec
per XIO channel. It varies a bit, and can peak at 175MB/sec, but 166 seems to be
about average for a sustained rate.

Thus, if going for mega speed, it makes more sense to use as many different XIO
channels as possible, with a smaller number of disks on each channel. The same
applies to anyone using FC, eg. to go beyond a PCI cage with a QLA2342
installed, get an XTalk adapter with another FC card, rather than put an extra
FC card in the PCI cage.

As a matter of interest, the above means that the maximum storage bandwidth
Octane could ever provide is probably about 650MB/sec (actually 665 to be precise),
ie. by using a 1-slot MGRAS gfx board and installing another XTalk+QLA12160 in
the top-right XIO slot. This speed is not far off the maximum speed possible through
any of the crossbar links (800MB/sec) so if it could do it that would be pretty
impressive. Might try it some time just for the hell of it...

Right, that's all for now, time for some zzzz...

Cheers!

Ian.

I'm working on a charitable PC build for the Learn Engineering YouTube channel. Please PM/email/call if you'd like to contribute!
Donations of any kind of item I can sell to provide funds are also most welcome.
[email protected]
+44 (0)7434 635 121

ipaddict
From ::1
Who joined March 14, 2006, 2:14 a.m.

Wrote the following at March 30, 2007, 10:36 a.m...

Nice numbers! Let's see 600 now, eh? :wink:

mapesdhs
From Edinburgh, Scotland
Who joined Nov. 10, 2003, 4:17 p.m.

Wrote the following at March 30, 2007, 1:19 p.m...

ipaddict wrote: :shock: Nice numbers! Let's see 600 now, eh?

I'll have to change the gfx to 1-slot MGRAS, but will see what I can
do.

Might be a few days before I can look into it though.

I ran a couple more tests, some interesting extra info. Reveals more
about how the speed limits of older drives affect overall
performance, and suggests spending more to get a smaller number of
newer faster disks is a better initial approach.

Thus, compare the following...

18 x 73GB, 3 SCSI cards on 3 XIO channels, using just 1 controller from each
card, 6 disks per controller, HD 4 bytes per pixel:

Code: Select all


   # req_size  fwd_wt  fwd_rd  bwd_wt  bwd_rd  rnd_wt  rnd_rd
   

   #  (bytes)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)
   

   #---------------------------------------------------------
   

   8404992  266.99  413.06  266.94  223.98  267.29  219.00
   

   16809984  269.64  373.90  268.43  296.10  267.19  291.56
   

   33619968  268.00  393.36  264.25  340.70  263.71  334.96
   

   67239936  269.65  386.56  264.49  367.93  265.25  369.20
   

   134479872  262.61  400.28  266.38  393.89  266.51  392.67
   

   268959744  255.11  410.89  266.68  405.38  267.32  407.61

Next, just 6 x 73GB, 3 SCSI cards on 3 XIO channels, 1 disk per controller,
HD 4 bytes per pixel:

Code: Select all


   # req_size  fwd_wt  fwd_rd  bwd_wt  bwd_rd  rnd_wt  rnd_rd
   

   #  (bytes)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)
   

   #---------------------------------------------------------
   

   8306688  169.10  214.41  158.19  169.03  161.23  167.29
   

   16613376  166.89  213.91  162.45  187.64  162.56  187.63
   

   33226752  167.24  213.99  166.55  198.79  165.56  198.66
   

   66453504  165.00  210.73  167.29  203.61  166.44  203.55
   

   132907008  163.12  210.41  165.57  206.60  167.26  206.53

The write speed is maxing out the drives perfectly, but I don't know why the
read speed isn't better. Probably something to do with the drive parameters.

Anyway, the most interesting result is this last one...

6 x modern 146GB, 3 SCSI cards on 3 XIO channels, 1 disk per
controller, PAL 3 bytes per pixel (my goof, will rerun and edit
later for HD 4 bytes/pixel):

Code: Select all


   # req_size  fwd_wt  fwd_rd  bwd_wt  bwd_rd  rnd_wt  rnd_rd
   

   #  (bytes)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)
   

   #---------------------------------------------------------
   

   1327104  399.03  416.81  158.09  139.51  157.39  104.13
   

   2654208  402.43  416.47  218.73  153.62  209.20  138.68
   

   5308416  398.58  417.02  289.01  160.54  272.20  156.14
   

   10616832  398.00  416.86  342.88  241.34  319.41  232.39
   

   21233664  394.54  415.03  351.04  304.66  349.66  307.63
   

   42467328  392.38  412.61  376.23  366.38  371.90  362.66
   

   84934656  383.10  404.65  386.13  384.81  385.04  382.42
   

   169869312  372.81  400.45  382.40  389.57  385.54  390.46

This shows it's better if possible to start off with a smaller number
of faster drives, despite the higher cost, as it means better expandability,
in this case up to 5TB. On the other hand, fewer drives means less
flexibility for including parity drives, multiple file systems, etc.

Ian.