SGI: Hardware

QLA2212 works on O2, gives you 2 1Gb FC-AL interfaces

So, I tried the AUA-3020 on a Fuel, no dice - but it apparently works on an O2. So of course that got me to thinking...

I had a QLA2212 here (which also does not work on a Fuel), the reason being that the AUA3020 and the QLA2212 (basically 2 QLA2200's-on-a-card) both use PCI bridge chips to map two devices into a single PCI slot, and apparently PCI bridge chips do not play well with XIO-based systems . So that got me to thinking some more...

So I put the QLA2212 (should be the same with a QLA2202, they're basically the same cards) into the PCI slot on this O2, and voila - 2 FC-AL interfaces! This also explains Ian's recent observation about one of the Adaptec 4-port SCSI cards working, I'd bet dollars to donuts that it's basically a version of the card which has two separate SCSI interfaces with a bridge chip (3 chips) as per his warning that it is " this version from the picture " that works.

Anyways, the dual-port Qlogic card shows up as two discrete interfaces, and should be able to pull some fairly good bandwidth - though I don't have a scratch-array handy to do some testing right at the moment, but will try to do the test in the next couple of weeks. Should be able to easily max out whatever bandwidth the PCI slot can provide in an O2, as 64-bit PCI-33 theoretically can push 266 MB/sec, though I don't think that you'd get nearly that much with an O2. It also doesn't require weird drivers, nor case hacking - basically stuff it in, Irix automatically "reconfigures the kernel" and on your next boot-up, you have two interfaces! Woot! :lol: :twisted: :lol:

Here's the hinv after the card was installed and the system rebooted:

Code: Select all

LilDude 1% hinv -vvm
CPU: MIPS R5000 Processor Chip Revision: 2.1
FPU: MIPS R5000 Floating Point Coprocessor Revision: 1.0
1 200 MHZ IP32 Processor
Main memory size: 256 Mbytes
Secondary unified instruction/data cache size: 1 Mbyte on Processor 0
Instruction cache size: 32 Kbytes
Data cache size: 32 Kbytes
FLASH PROM version 4.18
Integral SCSI controller 0: Version ADAPTEC 7880
Disk drive: unit 1 on SCSI controller 0 (unit 1)
CDROM: unit 4 on SCSI controller 0
Integral SCSI controller 1: Version ADAPTEC 7880
Integral SCSI controller 3: Version Fibre Channel QL2200A, 33 MHz PCI
Integral SCSI controller 4: Version Fibre Channel QL2200A, 33 MHz PCI
On-board serial ports: tty1
On-board serial ports: tty2
On-board EPP/ECP parallel port
CRM graphics installed
Integral Ethernet: ec0, version 1
Iris Audio Processor: version A3 revision 0
PCI Adapter ID (vendor 0x9004, device 0x8078) PCI slot 1
PCI Adapter ID (vendor 0x9004, device 0x8078) PCI slot 2
PCI Adapter ID (vendor 0x1011, device 0x0026) PCI slot 3
PCI Adapter ID (vendor 0x1077, device 0x2200) PCI slot 4
PCI Adapter ID (vendor 0x1077, device 0x2200) PCI slot 5
Video: MVP unit 0 version 1.4
AV: AV1 Card version 1, Camera not connected.
Vice: TRE


Cheers!
:O3000: <> :O3000: :O2000: :Tezro: :Fuel: x2+ :Octane2: :Octane: x3 :1600SW: x2 :O2: x2+ :Indigo2IMP: :Indigo2: x2 :Indigo: x3 :Indy: x2+

Once you step up to the big iron, you learn all about physics, electrical standards, and first aid - usually all in the same day
Very cool, thanks for the report!

Strange as it may seem, I've never had a QLogic 1-Gbit dual-port card in-house. I've had Emulex 1-Gbit duals, and plenty of 2-Gbit and 4-Gbit duals, but no QLA2212's, so I've never tried.

I guess this means a QLA2312 would work in an O2 as well... It doesn't play nice in Octanes, and the bridge chip was suspected of being the primary culprit. This pretty much confirms it.

Chris
:O2000R: (<-EMXI/IO6G) :O200: :O200: :O200: (<- quad R12k O200 w/GIGAchannel and ESI+Tex) plus a bunch of assorted standalone workstations...
I got this card off of eBay a few weeks back to try to figure out the best way to stuff two fibre interfaces in one slot on the IP35 architecture, which of course failed because of the bridge issue, so I'm kinda happy that it wasn't a total waste of investment. The exact card was a "Qlogic SANBlade QLA2212F/66"

I ended up getting a couple of QLA2342's, which seem to work fine, and are fully supported. Glad that the IRIX Qlogic drivers aren't picky in the slightest... :D
:O3000: <> :O3000: :O2000: :Tezro: :Fuel: x2+ :Octane2: :Octane: x3 :1600SW: x2 :O2: x2+ :Indigo2IMP: :Indigo2: x2 :Indigo: x3 :Indy: x2+

Once you step up to the big iron, you learn all about physics, electrical standards, and first aid - usually all in the same day
Yes, as a matter of fact, the current price of QLA2342's is noteworthy... And I agree with you on the convenience of a generic IRIX driver -- the dual-channel 2-Gbit FC card that I have in my Origin 2000 isn't technically a QLA2342 (basically the same card, but the PCI info is for a different vendor), but IRIX treats it like a QLA2342 anyway.

Glad the 2212 didn't go to waste!

(Shameless plug -- if anyone wants to get into some crazy fast data rates with FC using these 2-Gbit interfaces, let me know... Any system newer than an O2/Octane/Origin2 can make good use of 2-Gbit FC...)

Chris
:O2000R: (<-EMXI/IO6G) :O200: :O200: :O200: (<- quad R12k O200 w/GIGAchannel and ESI+Tex) plus a bunch of assorted standalone workstations...
So I hooked up an array of 10 Seagate 9GB 10k fibre channel drives in a Clariion rack and ran some tests:

Test 1 - QLA2212, two channels:

Code: Select all

LilDude 3# diskperf -D -W -n "Speedy (10x9GB 10k)" -c512m /speedy/testfile
#---------------------------------------------------------
# Disk Performance Test Results Generated By Diskperf V1.2
#
# Test name     : Speedy (10x9GB 10k)
# Test date     : Sat Jun 14 01:05:04 2008
# Test machine  : IRIX LilDude 6.5 10070056 IP32
# Test type     : XFS data subvolume
# Test path     : /speedy/testfile
# Request sizes : min=4096 max=4194304
# Parameters    : direct=1 time=10 scale=1.000 delay=0.000
# XFS file size : 536870912 bytes
#---------------------------------------------------------
# req_size  fwd_wt  fwd_rd  bwd_wt  bwd_rd  rnd_wt  rnd_rd
#  (bytes)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)
#---------------------------------------------------------
4096    4.44    5.19    0.70    0.67    1.93    0.71
8192    6.07    7.08    1.62    1.43    3.50    1.34
16384    9.10   11.43    3.63    2.98    5.82    2.36
32768   12.18   17.13    7.75    4.54    9.80    4.31
65536   15.74   23.70   15.28    7.11   13.82    6.56
131072   20.62   35.54   20.28   11.69   18.78   11.05
262144   24.16   43.79   23.90   19.07   23.60   18.23
524288   26.28   49.42   25.97   29.25   26.02   28.14
1048576   29.50   79.52   29.27   52.25   29.24   51.11
2097152   30.94   91.64   30.80   71.88   30.80   70.82
4194304   31.48   98.50   31.46   85.74   31.37   85.56


Test 2 - yank out one of the cables, yielding just 1 channel (with failover goodness) :

Code: Select all

LilDude 4# diskperf -D -W -n "Speedy (1 channel)" -c512m /speedy/testfile
#---------------------------------------------------------
# Disk Performance Test Results Generated By Diskperf V1.2
#
# Test name     : Speedy (1 channel)
# Test date     : Sat Jun 14 01:31:10 2008
# Test machine  : IRIX LilDude 6.5 10070056 IP32
# Test type     : XFS data subvolume
# Test path     : /speedy/testfile
# Request sizes : min=4096 max=4194304
# Parameters    : direct=1 time=10 scale=1.000 delay=0.000
# XFS file size : 536870912 bytes
#---------------------------------------------------------
# req_size  fwd_wt  fwd_rd  bwd_wt  bwd_rd  rnd_wt  rnd_rd
#  (bytes)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)
#---------------------------------------------------------
4096    4.61    5.37    0.73    0.70    1.98    0.75
8192    6.36    7.33    1.71    1.50    3.65    1.38
16384    9.50   12.01    4.13    3.18    6.09    2.47
32768   12.70   17.66    8.14    4.80    9.73    4.32
65536   15.72   23.72   15.23    7.28   13.70    6.61
131072   20.13   31.22   19.66   12.65   18.61   11.06
262144   23.27   37.16   22.97   19.55   22.81   18.41
524288   25.12   41.26   24.98   29.52   24.98   28.30
1048576   27.71   61.65   27.55   47.87   27.60   46.73
2097152   29.01   68.00   28.89   59.22   28.90   58.19
4194304   29.54   72.01   29.49   66.48   29.53   66.11


Test 3 - 10k Fujitsu system disk (for a baseline):

Code: Select all

LilDude 5# diskperf -D -W -n "System Disk (Fujitsu 10k)" -c512m /testfile
#---------------------------------------------------------
# Disk Performance Test Results Generated By Diskperf V1.2
#
# Test name     : System Disk (Fujitsu 10k)
# Test date     : Sat Jun 14 01:43:05 2008
# Test machine  : IRIX LilDude 6.5 10070056 IP32
# Test type     : XFS data subvolume
# Test path     : /testfile
# Request sizes : min=4096 max=4194304
# Parameters    : direct=1 time=10 scale=1.000 delay=0.000
# XFS file size : 536870912 bytes
#---------------------------------------------------------
# req_size  fwd_wt  fwd_rd  bwd_wt  bwd_rd  rnd_wt  rnd_rd
#  (bytes)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)
#---------------------------------------------------------
4096    8.28    7.89    2.94    0.69    1.68    0.78
8192    9.32    9.02    5.96    1.37    3.00    1.41
16384   13.53   10.06    9.60    2.86    5.24    2.76
32768   17.71   15.59   15.07    6.04    8.71    4.95
65536   21.88   20.52   20.22   13.27   13.45    8.20
131072   25.53   22.92   13.64   13.57   16.61   12.47
262144   27.70   24.52   21.18   19.47   23.26   16.94
524288   28.95   25.31   26.78   19.82   27.20   20.60
1048576   31.72   31.09   30.27   29.57   30.72   27.21
2097152   32.86   32.48   32.54   29.99   32.16   30.28
4194304   33.37   33.23   33.26   32.63   33.13   31.98


Haven't got a clue whats up with the write speeds (note: write caching is enabled - I checked) - could be many things including the fact that this O2 is only an R5k-200. But 100 MB/sec isn't anything to sneeze at, and it likely exceeds the combined bandwidth of the two native SCSI channels, at least for reads. Someone needs to try a high-bandwidth SCSI card in the PCI slot in one of these (*cough*Ian*cough*) , and perhaps run the test with an R5k-200 to see if some of these numbers are somehow processor-related, or related to the PCI implementation on an O2, or even a bandwidth issue through the PCI-PCI bridge chip.

In any regard, they're some numbers to crunch. This is the exact same array I could get ~170 MB/sec read-write on an Octane, so it's not the array.
:O3000: <> :O3000: :O2000: :Tezro: :Fuel: x2+ :Octane2: :Octane: x3 :1600SW: x2 :O2: x2+ :Indigo2IMP: :Indigo2: x2 :Indigo: x3 :Indy: x2+

Once you step up to the big iron, you learn all about physics, electrical standards, and first aid - usually all in the same day
Dr. Dave wrote: So, I tried the AUA-3020 on a Fuel, no dice - but it apparently works on an O2. So of course that got me to thinking...

I had a QLA2212 here (which also does not work on a Fuel), the reason being that the AUA3020 and the QLA2212 (basically 2 QLA2200's-on-a-card) both use PCI bridge chips to map two devices into a single PCI slot, and apparently PCI bridge chips do not play well with XIO-based systems .


XIO machines may very well have problems, but it seems like SGI has gone to quite a bit of trouble to write a PCI-PCI bridge driver for XIO/Bridge PCI busses. A relevant note from /var/sysgen/mtune/kernel:

Code: Select all

*
* As of 6.5.19, general support for PCI-PCI bridges was added with the
* pciio_ppb driver.  This can expose the well known RRB thrashing issue that
* Bridge and its derivatives (Xbridge and PIC-in-PCI-mode) exhibit.
*
* By default, pciio_ppb will disallow any PPB hierarchy hung off of a Bridge
* slot which contains more than one function advertising DMA master ability,
* generating a console message.
*
* When set to non-zero, pciio_multimaster_override will override this
* behavior and allow an arbitrary PPB hierarchy to be built up on Bridge.
*
pciio_multimaster_override      0       0               1

Looking through the 6.5.19 release notes , I couldn't find any specific hardware fixes in 6.5.19 that would have warranted this driver. The only big change was adding support for the O3900, which maybe has some PCI-PCI bridged hardware.

There wasn't much mention of pciio_ppb on Google, but one hit was extremely interesting: an LKML post from SGI with a bunch of Linux IA64 src to be put into 2.6. The reply is scathing:
Guy you drive me nuts with your silly hack it up on irix and
glue it into linux strategy!

And indeed, the code is straight out of Irix, and grafted onto Linux in a less-than-pretty way. For example, this massive 1MB patch contains the entire source for the pciio_ppb driver. The code still uses IRIX's vertex_hdl_t type, which maps perfectly onto IRIX's hwgraph, but I have no idea they made it map onto Linux. For posterity, I'll post the pciio_ppb header here:
+ * Fairly generic PCI-PCI Bridge (PPB) driver module.
+ *
+ * This driver implements provider-independent PPB support for Irix.
+ *
+ * Goals
+ * =====
+ * Transparantly provide pciio services to PCI devices downstream of a PPB
+ * hierarchy.
+ *
+ * Implement as a provider-independent module.
+ *
+ * Design
+ * ======
+ *
+ *
+ * For the most part, devices behind a PPB are used in the same way as devices
+ * directly connected to a host bus. Drivers are written using the guidelines
+ * set forth in the device drivers programming guide. The most notable
+ * exception is that some providers might not allow config space pio maps to
+ * devices residing on secondary busses because of shared address space when
+ * accessing these devices.
+ *
+ * Other restrictions might come into play if devices behind a PPB request
+ * resources or mappings which are not compatable with each other when you
+ * consider that all subordinate devices originate at a common host slot. An
+ * example might be interrupt routing using DEVICE_ADMIN statements.
+ *
+ * All subordinate bus numbers in the system come from a shared pool, at start
+ * at 16. Bus numbers < 16 are left to the providers. A future modification to
+ * this code might move bus number allocation to be a per-provider function.
+ *
+ * While this code should be provider-independent, no attempt was made to
+ * retrofit this to O2. The O2 already had PPB support, and it did not seem
+ * worth the risk to break customer code that might relay on how O2 works.
+ *
+ * No attempt was made to restrict the configuration for the well-known
+ * bridge/xbridge issue of RRB thrashing when there are multiple PCI masters
+ * doing reads from a single bridge. The reasoning is that there is no easy way
+ * to tell if downstream PCI masters intend to do host DMA or not. It is
+ * perfectly ok to allow multiple masters doing device-to-device traffic.
+ *
+ * Provider implementations must make the following accomodations for this
+ * driver to work correctly:
+ *
+ * When accessing config space (pciio_config_get()/pciio_config_set())
+ * the provider should examine the pciio_info_t of the passed vhdl
+ * to determine if TYPE1 addressing should be used. This can be checked
+ * by pciio_info_type1_get() returning non-zero. This means that
+ * the bus/slot/function to be used are encoded in the address
+ * to be read/written, and the macros PCI_TYPE1_BUS,
+ * PCI_TYPE1_SLOT, and PCI_TYPE1_FUNC should be used to extract
+ * the correct values.
+ *
+ * In addition, if the provider stores non-zero bus numbers for the host
+ * PCI bus in a device's pciio_info_t->c_bus (as bridge/xbridge do), the
+ * provider code must determine if the register to be read resides on the
+ * host bus (actually PCI bus 0), or a subordinate bus (PCI bus != 0).
+ * pcibr accomplishes this by checking the pciio_info c_vertex against
+ * c_hostvertex. If they are the same, the device is on the host bus, and
+ * this is PCI bus 0. If not, the device is on a subordinate bus, and the
+ * bus number can be taken from c_bus.
+ *
+ * Providers must decide if it is appropriate for pio maps to be
+ * constructed for config space of devices on subordinate busses. For
+ * bridge/xbridge, this should not be allowed because all devices on
+ * subordinate busses share the same address space (the bus/slot to use is
+ * programmed in a bridge/xbridge register).
+ *
+ */

It doesn't appear that this code ever made it into the Linux kernel tree.
That FTP site has many other patches--not sure how interesting they are from an Irix POV.
:Octane: R12K/300x2, MXE
:1600SW: :ChallengeL:
smokin' work Dr.!

just added 2 the list

quick question: a quick search in ebay (com/de/uk), returned 0. is this one some really exotic, hard to find model or just "old"?

(i read that chris, our fc man, has never tested one, so this model is probably exotic(?))

edit: my bad, looks like the "f" after the model makes a difference when searching ebay
It's old hardware, but at the time showed up fairly easily on eBay.

This link will probably die pretty quickly, however but there's one up now:

eBay linky
:O3000: <> :O3000: :O2000: :Tezro: :Fuel: x2+ :Octane2: :Octane: x3 :1600SW: x2 :O2: x2+ :Indigo2IMP: :Indigo2: x2 :Indigo: x3 :Indy: x2+

Once you step up to the big iron, you learn all about physics, electrical standards, and first aid - usually all in the same day
Sorry for bumping an old thread, but thought you'd all like to know the QLA2342 works ok in O2:

Code: Select all

CPU: MIPS R12000 Processor Chip Revision: 2.3
FPU: MIPS R12010 Floating Point Chip Revision: 0.0
1 300 MHZ IP32 Processor
Main memory size: 1024 Mbytes
Secondary unified instruction/data cache size: 1 Mbyte on Processor 0
Instruction cache size: 32 Kbytes
Data cache size: 32 Kbytes
FLASH PROM version 4.18
Integral SCSI controller 0: Version ADAPTEC 7880
Disk drive: unit 2 on SCSI controller 0
CDROM: unit 4 on SCSI controller 0
Integral SCSI controller 1: Version ADAPTEC 7880
Disk drive: unit 5 on SCSI controller 1
Integral SCSI controller 2: Version Fibre Channel QL2342 Port 1
Disk drive: unit 111 on SCSI controller 2
Disk drive: unit 114 on SCSI controller 2
Disk drive: unit 116 on SCSI controller 2
Disk drive: unit 118 on SCSI controller 2
Disk drive: unit 120 on SCSI controller 2
Disk drive: unit 122 on SCSI controller 2
Disk drive: unit 124 on SCSI controller 2
Integral SCSI controller 3: Version Fibre Channel QL2342 Port 2
Disk drive: unit 113 on SCSI controller 3
Disk drive: unit 115 on SCSI controller 3
Disk drive: unit 117 on SCSI controller 3
Disk drive: unit 119 on SCSI controller 3
Disk drive: unit 121 on SCSI controller 3
Disk drive: unit 123 on SCSI controller 3
Disk drive: unit 125 on SCSI controller 3
On-board serial ports: tty1
On-board serial ports: tty2
On-board EPP/ECP parallel port
CRM graphics installed
Integral Ethernet: ec0, version 1
Iris Audio Processor: version A3 revision 0
Video: MVP unit 0 version 1.4
AV: AV2 Card version 0, Camera not connected.
Vice: TRE


This is an original Discreet Effect system. The array is 14 x 73GB Seagate 10K FC,
all with orig Discreet firmware.

Before posting full results though, I'll bump it to an R12K/400, and then try the same tests later
with an R7K/600 (setting up the disks as an XLV rather than a stonefs). Once done, I'll do
the tests with an R5K/200, see if there's any CPU bottleneck re Dr. Dave's comments earlier.
Meanwhile, here's a simple XLV test with eight 73GB 10K FCs:

Code: Select all

# req_size   fwd_wt  fwd_rd  bwd_wt  bwd_rd  rnd_wt  rnd_rd
#  (bytes)   (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)
#----------------------------------------------------------
1769472   46.50   60.85   46.16   56.38   49.54   56.05
3538944   50.76   68.85   46.34   64.12   50.83   63.14
7077888   41.96   70.84   39.50   68.34   48.48   68.97
14155776   42.59   71.15   37.96   69.39   51.15   69.32
28311552   49.49   73.80   41.97   73.80   42.08   69.94
56623104   27.24   70.02   42.48   71.31   33.84   75.05


The first line is the important one as that's the optimised request size for PAL. It's sufficient
for real-time SD.

I notice the QLA2342's LEDs show the links are only 1Gbit, which is a bit odd since I'm
sure the array modules can run at 2Gbit. Not that it would matter, looks like O2 couldn't
push it that fast anyway.

One point worthy of note: the peak numbers in the above test are very similar to the best
STREAM result I've obtained for O2, so maybe this really is just a question of limited
main-CPU/RAM/disk bandwidth. Here's a STREAM run for an R7K/600:

Code: Select all

Function      Rate (MB/s)   RMS time     Min time     Max time
Copy:          66.4418       0.4865       0.4816       0.4922
Scale:         67.0130       0.4804       0.4775       0.4842
Add:           72.5161       0.6638       0.6619       0.6678
Triad:         72.4564       0.6653       0.6625       0.6680



Either way, hope this brightens your day. 8)

Ian.
(14/Jan/2016) FREE! (collection preferred) 16x Sagitta 12-bay dual-channel U160 SCSI JBOD units.
Email, phone or PM for details, or see my forum post .
[email protected]
+44 (0)131 476 0796