SGI: hinv

Once again, thanks to Voralyan's howto, here's the second Altix box on Debian 6. This one is a tad larger; there are 13 compute nodes, two of which have PCI slots and IOC10's (SATA drives instead of SCSI), and it's running with a single-plane router. The hwinfo and dmesg follow.

And pics.....

Update:

After working for a while on the additional two Altix 350 compute nodes from this system that had issues, I've now populated this system to 15 compute nodes, making 30 CPUs, and 54GB of RAM.

I had updated the PROMs on the 13 working nodes, but not yet on the two non-working ones. A PROM version mix, as has been noted elsewhere, will cause the system to enter POD, before ever loading EFI. And on compute nodes without disks or an IO9 or IO10 (there are three in this system with IO10's, but neither of the repaired nodes have any IO or disks) booting a node outside the system and updating it individually is not an option.

However, POD has the tools to fix this. So, if the system drops you to POD due to a PROM version mismatch, there is hope. In POD, you can flash the downrev nodes by nasid; you can get the nasid in POD by (note that I entered POD from the EFI shell for this example):

Suppose that brick 001c26 was at PROM 4.43, however. In my output above, I find that the nasid is 22 for 001c26; I can program its PROM from POD by:


Then I can exit POD and reboot. As a note, if anyone knows where a POD reference document can be found, that would be cool.....

Also as a note: the Altix 350 router brick has the same embedded L2 capability as the 3700BX2 does; all that's required is to plug in a 'USB to Ethernet' adapter into the L1 USB port on the router. Finding what kind of 'USB to Ethernet' adapter was difficult, but I finally found and just tried a 3Com 3C460B that worked right away. Simply plugging in the proper USB to Ethernet adapter into the L1 USB causes the L1 on the r-brick to load the L2 emulator, access to which is Ethernet only. You can set the IP address from the L1; the instructions for doing this I found, ironically, in the 3700BX2 documentation and not the Altix 350 documentation. It appears that the same NL4 router is used for Altix 330 systems, and maybe even for the same vintage Origin systems; if your NL4 router has a host type A USB port labeled 'L1 USB' it may have this capability as well. Altix 330 systems appear to have this capability on the compute node, too, so the separate L2 controller isn't required for these systems to get L2 functionality.

A reboot_l1 seems to be a good thing to do after plugging the adapter in; the L1 serial port stays an L1 serial port and can't access either the L2 prompt or the system console (have to use one of the other compute node's serial ports for console, but have to use the router's L1 to power up the whole system, since the router is the only brick with full L1 connectivity in a routed, non-ring, configuration. In a ring config, the L1 at any node can see all nodes, but not in routed mode, where node visibilities from the compute bricks' L1's are, to say it bluntly, weird).

But the emulated L2 works great once a supported USB to Ethernet adapter is found. Again, I found that a 3Com 3C460B works fine; I haven't tried any other USB to Ethernet adapters. Would be interesting to see what others work.

The box is still running Debian 6, but as soon as I can get CentOS 5 to rebuild it's gong CentOS, since the ProPack and Foundation tools will work there, and I can rebuild system updates from source (can't do that with SLES) can keep the OS updated (with ia32el, and i386 binary capability) at least until 2017, the EOS year for CentOS 5 (and all RHEL5 rebuilds).

It's gorgeous! Absolutely incredible. I want one!

While you've got it on Debian 6, were you looking to run any benchmarks?

Just for fun I'd like to see how this thing would do with UnixBench ( http://code.google.com/p/byte-unixbench/ ) and compare it to my weak little dual-socket web server.

Hmm, the BYTE UnixBench is still around and updated? Cool; I remember running a much older version on my Tandy 6000 running Xenix System III back in the '80's (that box was a 68000 CPU at 8MHz and 1MB of RAM with two 70MB MFM hard drives and a pair of 8 inch floppies; but it was real Unix!). I most recently ran the 4.10 version a while back; didn't know it was a google code project now.

Right now I am waiting for a second 64 process run of the HPCC Challenge benchmark to finish; the hpcc package is distributed in the Debian repos, so it's an easy install and a pretty simple configure. It is fun to watch all 30 CPUs hit and stay at 100% for half an hour. (Wall time: it is really strange to see something like:

as output of 'time' though!)

The HPCC benchmarks make a good stress test; I know this box isn't going to break any records, but since I'm going to be using it for planetarium graphics rendering I need to stress it pretty hard before putting it in production. It is also somewhat disquieting to see the load average parked at 64 and the system still in a responsive and usable state.... A really large Altix (say a 512 processor 3700) would be really interesting to see, but I wouldn't want the power bill. This Altix 350 is one compute node (and one router) shy of a full dual plane max config, so is a good indication of how powerful such a maxxed out box would be.

In any case, UnixBench doesn't seem to like >16 CPU's..... I'll see what I can do to fix, perhaps. I'm also running a comparison on an 8 CPU Xeon system; two quad core 2.0GHz 64-bit Xeons, running CentOS 6.3 with 16GB of RAM (Dell Precision 690 workstation).

I will report back.

Ok, some results. First, the 30 CPU Altix 350. Since UnixBench is a system benchmark, the slow SATA drives do somewhat lower the total score (and I've snipped some repetitive sections out):


Now, the Dell Precision 690, two quad core Xeons (E5335's) at 2.0GHz:



There are some areas where the Altix shines, especially considering the age difference and the much faster drives loaded in the 690. But that gives an idea of system speed differences.

I'm working on getting the 20 CPU Altix 3700 running the benchmark, but a script is tossing an error, so it hasn't happened yet.

And, for grins and giggles, a dual-processor 2.4GHz 32-bit Xeon (Dell PowerEdge 1600SC):



The Altix fares quite well against the NetBurst core Xeon.

The 32-bit Xeon's score is pretty much what I expected compared to my own. And of course the 8-core Xeon box just blows everything out of the water.

Compare to my dual 3.06Ghz Xeon, 2GB RAM, 32-bit webserver:


So from this I can gather that the UnixBench workload obviously wasn't designed with distributed MPI-friendly architectures in mind, but nonetheless it's fun to compare. I'll have to give UnixBench a go when I get my new 8-core UltraSparc T1/16GB RAM box up and running.

How responsive does the Altix system feel compared to working on the newer Core Xeon and the old Netburst?

The Altix systems are both at least as responsive as the Dell 690 8 core, even for remote ssh-tunneled X programs. The 690 is going to be used for virtualization, but was available for a quick benchmarking before getting KVM guests going....

I'm overall quite pleased at the responsiveness, even under heavy load, of both Altix boxes. The old NetBurst box is definitely less responsive under load.

And the Altix boxen are destined to do MPI-friendly rendering tasks, so they will do quite well indeed.

[edit] Also, I have a Sun Enterprise 6500 here that I might can run the benchmarks on at some point, but it will be a while. The E6500 has 18 CPUs and 22GB of RAM. Will be an interesting test.