Nekonomicon - Origin is sick

pentium
Who joined Aug. 28, 2006, 5:29 p.m.
and authored 4256 notes

Wrote the following at May 27, 2013, 6:16 p.m...

Over the years my O2K was rolled around a bit, saw an install, got shoved into storage, had parts added while in storage and then finally it was refilled with 250mhz nodeboards. Finally I have it in a permanent location however I'm running into problems.
First time applying power from the perspective of reading the LEDs on the nodeboards everything came up no problem. Unfortunately at the time I didn't have a terminal attached and I forgot about pulling one out for another week.
Fast forward to today Finally I have the VT100 plugged in but now it's behaving weird.
The two boards in N1 and N3 look OK from their respective LED's however in N2 and N4 the boards are locked at:

Code:

After some board swapping and shifting ram around I found that this was present on three of my nodeboards (one was a spare) and the MSC says that the CPU tests are not responding regardless of what slot you put the nodeboard in. Another nodeboard seems to be failing and disabling any ram (even good ram) that is in the first bank. Only two of my boards are behaving right now. :/

_________________
:Crimson:

<-------- A very happy forum member.

PymbleSoftware
Who joined July 15, 2008, 4:48 p.m.
and authored 1877 notes

Wrote the following at May 27, 2013, 7:58 p.m...

Not sure which LEDs you're talking about...is it one of these...?

http://www.nekochan.net/wiki/Origin2000 ... outer_LEDs

http://www.nekochan.net/wiki/Origin2000 ... plane_LEDs

http://www.nekochan.net/wiki/Origin2000 ... Board_LEDs

R.

_________________
死の神はりんごだけ食べる

開いた括弧は必ず閉じる -- あるプログラマー

:Tezro:

J5600, 2 x Mac, 3 x SUN, Alpha DS20E, Alpha 800 5/550, 3 x RS/6000, Amiga 4000 VideoToaster, Amiga4000 -030, 733MHz Sam440 AmigaOS 4.1 update 1. Tandem Himalaya S-Series Nonstop S72000 ServerNet.

Sold: :Indy:

Cortex ---> http://www.facebook.com/pages/Cortex-th ... 11?sk=info
Minnie ---> http://www.facebook.com/pages/Minnie-th ... 02?sk=info
Book ----> http://pymblesoftware.com/book/
Github ---> https://github.com/pymblesoftware
Visit http://www.pymblesoftware.com
Search for "Pymble", "InstaElf", "CryWhy" or "Cricket Score Sheet" in the iPad App store or search for "Pymble" or "CryWhy" in the iPhone App store.

pentium
Who joined Aug. 28, 2006, 5:29 p.m.
and authored 4256 notes

Wrote the following at May 27, 2013, 8:53 p.m...

Nodeboard LED's.

I managed to find the diagram on the wiki (the search seems oddly broken) and it means the CPU's are disabled. Enableall only enables the failing ram banks so I use enable -n -s n3 -cpu ab and a reboot to get them back talking to the system but then things get ugly.
The nodeboards then start screaming about no ram (it's there) and the system goes flying into POD in a panic. It almost feels like I really own two pairs of actually good ram and the other half dozen pairs are all bad but it's hard to say as I have boards declaring banks being faulty when there's nothing in them. I can't even try to pinpoint and filter out bad sticks.

Edited: Yeah, even if I enable the cpu's and put four known good sticks in a questionable nodeboard the result is:

Code:

Or...

Code:

...which translate to...

Quote:

98 Node has no Local memory No RAM or Disabled RAM

....followed by the PROM dumping to POD.
I can understand one nodeboard going south from a memory controller failure, but two nodes PLUS my two spares??

_________________
:Crimson:

<-------- A very happy forum member.

foetz
Who joined April 14, 2003, 3:34 a.m.
and authored 4732 notes

Wrote the following at May 28, 2013, 1:19 p.m...

i'd try one board at a time. eventually the same with the ram sets. it's nasty but the only proper way to track down errors

SAQ
Who joined July 19, 2006, 7:37 a.m.
and authored 5632 notes

Wrote the following at May 28, 2013, 4:22 p.m...

Have you tried reseating the MSC? I don't have Origin diagrams showing how the MSC communicates with the nodeboards, but it could be an error there.

N2 and N4 share XBOW but not router, so the router board doesn't stand out.

_________________
Damn the torpedoes, full speed ahead!

There are those who say I'm a bit of a curmudgeon. To them I reply: "GET OFF MY LAWN!"

:Indigo:

(single-CM)

recondas
Moderator
Who joined June 6, 2004, 4:55 p.m.
and authored 4968 notes

Wrote the following at May 28, 2013, 5:58 p.m...

viewtopic.php?f=3&t=17194&p=134279&#p134279

_________________
***********************************************************************
Welcome to ARMLand - 0/0x0d00
running...(sherwood-root 0607201829)
* InfiniteReality/Reality Software, IRIX 6.5 Release *
***********************************************************************

pentium
Who joined Aug. 28, 2006, 5:29 p.m.
and authored 4256 notes

Wrote the following at May 29, 2013, 7:19 p.m...

Okay, so boy following by Reconda's word I stripped down to the base system (MSC, one nodebaord and an IO6G, because my IO6 appears to be PROM passworded and otherwise useless right now)

Using one of the good nodes with good ram I went into POD, nuked everything and then restarted to let everything get rediscovered. On restart I see:

Code:

  WARNING: Board with freq 100 mhz at module 1 slot 1 is attached to a back-plane of 360 mhz frequency
  
  WARNING: Hub at 100 mhz and router at 90 mhz is an unsupported configuration

...otherwise the system runs normally.

Then powered down, removed the node and transferred the known good memory from banks 0 and 1 to one of the questionable nodes. Put it all back together and powered up.

Code:

  1B 000: ***No useable ram installed. Need working and enabled memory in bank 0 or 1
  
  1B 000: ***Add working and enabled memory present in bank 0 or 1 and reset the syst

MMSC also displays

Code:

5C0 M 1

Oh yeah, plus the system has locked up and had to be powered down from the breaker.

Reinstall the ram in the good node and confirm that it is good. It is and I get to the PROM.
Okay. So can confirm two good banks worth of ram and one healthy nodeboard. Onwards.

Perform the same POD nuke and reset as before, again the same two warnings. Can now confirm two good nodeboards and four healthy banks worth of ram.

Now repeat putting healthy ram into another questionable nodeboard.

Code:

  1B 000: ***No useable ram installed. Need working and enabled memory in bank 0 or 1
  
  1B 000: ***Add working and enabled memory present in bank 0 or 1 and reset the syst

...and again, it locks solid.

Repeat putting known pairs of good ram into remaining two spare nodeboards. Exact same result.

Okay. So now we put one healthy node + ram combo into slot 1 and stuff a questionable node with good ram in the second slot.

on top of our previous memory errors we also now have

Code:

  ***Warning: Found a new IP27 board in module 1, slot n2, serial KAH055
  
  Please use the 'update' command from the PROM Monitor to update the inventory

and...

Code:

  IP27 Node Board, Module 1, Slot n2
  
  ASIC HUB Rev 5, 100 MHz, (nasid 1)
  
  Processor A: **Disabled, Reason = Unuseable bank 0.
  
  Processor A: 250 MHz R10000 Rev 0.0
  
  Secondary Cache 4MB 250MHz Tap 0x9 , (cpu 2)
  
  R10010FPC Rev 0.0
  
  Processor B: **Disabled, Reason = Unuseable bank 0.
  
  Processor B: 250 MHz R10000 Rev 0.0
  
  Secondary Cache 4MB 250MHz Tap 0x9 , (cpu 3)
  
  R10010FPC Rev 0.0
  
  Memory on board, 0 MBytes (Standard), (256 Mbytes - Bank(s) 0 1 disabled)
  
  Bank 0, 128 MBytes Disabled, Reason: Some DIMMs failed mem test.
  
  Bank 1, 128 MBytes Disabled, Reason: Some DIMMs failed mem test.

Running UPDATE and a reset completely disables the node.

Code:

  ***Warning: Board in module 1, slot n2 is missing or disabled It previously contained a node-board, barcode KAH055 laser 3f3404
 

Same applies with the other three.

The one thing I have not yet done is put any of the questionable sticks in one of the good nodes in the event I am dealing with a bad stick that killed every node I put it into so far. I have switched around my two routers in case they were acting funny.

On the other hand if you installed both healthy nodes with good ram...

Code:

  *** Barrier sync warning: local=4418, NIC 0x3408b0=4400 (promrev mismatch?)
  
  *** Barrier sync warning: local=4446, NIC 0x3408b0=4428 (promrev mismatch?)
  
  *** Barrier sync warning: local=4908, NIC 0x3408b0=4890 (promrev mismatch?)
  
  *** Barrier sync warning: local=4981, NIC 0x3408b0=4963 (promrev mismatch?)

....it appears they need to be flashed. I don't know immediately how to do that without booting into irix.

Edited: Also, if you have in both healthy nodes and then add in one of the questionable nodes with questionable ram you get:

Code:

  2A 001:Testing/Initializing all memory ...........              DONE
  
  2A 001: waiting for node with nic 3472e2 at module 1 slot 1 at global barrier...
  
  ....
  
  2A 001: ........
  
  2A 001: *** Barrier sync warning: local=5090, NIC 0x3472e2=5108 (promrev mismatch?)
  
  2A 001:Checking partitioning information .........              DONE
  
  2A 001: > Global barrier failed at line 5337 node 0
  
  2B 001: Local slave entering slave loop
  
  2A 001: Local master entering slave loop
  
  1A 000:
  
  1A 000: *** General Exception on node 0
  
  1A 000: *** EPC: 0xc00000001fc3a2e8 (0xc00000001fc3a2e8)
  
  1A 000: *** Press ENTER to continue.

_________________
:Crimson:

<-------- A very happy forum member.

jan-jaap
Who joined June 17, 2004, 10:35 a.m.
and authored 3659 notes

Wrote the following at May 29, 2013, 11:36 p.m...

pentium wrote:

Using one of the good nodes with good ram I went into POD, nuked everything and then restarted to let everything get rediscovered. On restart I see:

Code:

   WARNING: Board with freq 100 mhz at module 1 slot 1 is attached to a back-plane of 360 mhz frequency
   
   WARNING: Hub at 100 mhz and router at 90 mhz is an unsupported configuration

...otherwise the system runs normally.

The midplane is configured for 180MHz nodeboards.

You have to toggle one or two jumpers, see: http://www.vdheijden-messerli.net/sgist ... .02-onyx2/

_________________
Now this is a deep dark secret, so everybody keep it quiet

It turns out that when reset, the WD33C93 defaults to a SCSI ID of 0, and it was simpler to leave it that way... -- Dave Olson, in comp.sys.sgi
Currently in commercial service:

(2x)

In the museum : almost every MIPS/IRIX system.
Wanted : GM1 board for Professional Series GT graphics (030-0076-003, 030-0076-004)

jan-jaap
Who joined June 17, 2004, 10:35 a.m.
and authored 3659 notes

Wrote the following at May 29, 2013, 11:59 p.m...

A couple of random observations & tips:

* Try to keep a healthy nodeboard and one bank of RAM in slot N0 in the system and put the hardware under test in slot N1. With a working nodeboard you should eventually reach POD whatever is in the other slots.

* With a RAM-less nodeboard in N1 you will eventually reach POD (if you have a working node in N0). But it takes ages (20 - 30 minutes?). This allows you to recover disabled nodes, flash the PROM of the secondary nodeboard, or flash CPU speed parameters in it. It will still try to test all RAM you put in, so put in the least you can.

* 256MB RAM banks were supported in 250MHz and faster nodes. But IIRC there's a minimum PROM rev. for them. You've got downrev PROM warnings, and 256MB banks?

* PROM version mismatches are an endless source of misery.

If I were you, I would:

1. Fix that 90/100MHz midplane speed selector.

2. Install the newest IRIX you have with a good nodeboard and one bank of RAM. This gives you a working config for flashing firmware. Flash the good node and the IO6G to the last rev. (6.156 I think). Congrats, now you have a functional system to work from.

3. Add a secondary nodeboard. With or without RAM. It may take a while and it may disable bit left and right, but you still have a working system capable of booting IRIX and flashing the secondary board if needed.

4. If a 250MHz node with the last firmware still won't recognize the 256MB bank, the first bank may be bad (bad contact, or otherwise broken). If you add another bank of RAM, it can swap banks 0 and 1 and use bank1 as the first bank.

5. Origins can accumulate loads of fine black dust. You could try to clean the memory slots with compressed air, clean the DIMMs with alcohol, and install/remove DIMMs still 'wet' from the alcohol in the slot in an attempt to clean them.

_________________
Now this is a deep dark secret, so everybody keep it quiet

It turns out that when reset, the WD33C93 defaults to a SCSI ID of 0, and it was simpler to leave it that way... -- Dave Olson, in comp.sys.sgi
Currently in commercial service:

(2x)

In the museum : almost every MIPS/IRIX system.
Wanted : GM1 board for Professional Series GT graphics (030-0076-003, 030-0076-004)

jan-jaap
Who joined June 17, 2004, 10:35 a.m.
and authored 3659 notes

Wrote the following at May 30, 2013, 12:15 a.m...

Forget what I said about 256MB banks, looks like you have 128MB banks.

Need more coffee

_________________
Now this is a deep dark secret, so everybody keep it quiet

It turns out that when reset, the WD33C93 defaults to a SCSI ID of 0, and it was simpler to leave it that way... -- Dave Olson, in comp.sys.sgi
Currently in commercial service:

(2x)

In the museum : almost every MIPS/IRIX system.
Wanted : GM1 board for Professional Series GT graphics (030-0076-003, 030-0076-004)

pentium
Who joined Aug. 28, 2006, 5:29 p.m.
and authored 4256 notes

Wrote the following at May 30, 2013, 8:45 a.m...

I got a mix of everything. I have several sets of bright green banded ram (none seem to work), a few sets of dark green banded ram (two of them are in use and the rest apparently do not work) and eight sticks of white banded stuff (also in use).

_________________
:Crimson:

<-------- A very happy forum member.

SAQ
Who joined July 19, 2006, 7:37 a.m.
and authored 5632 notes

Wrote the following at May 30, 2013, 7:04 p.m...

pentium wrote:

I got a mix of everything. I have several sets of bright green banded ram (none seem to work), a few sets of dark green banded ram (two of them are in use and the rest apparently do not work) and eight sticks of white banded stuff (also in use).

White/silver = 128MB/stick
Green (light? haven't seen the other type so I think it might be light) = 32MB/stick
Yellow=16MB/stick

_________________
Damn the torpedoes, full speed ahead!

There are those who say I'm a bit of a curmudgeon. To them I reply: "GET OFF MY LAWN!"

:Indigo:

(single-CM)

pentium
Who joined Aug. 28, 2006, 5:29 p.m.
and authored 4256 notes

Wrote the following at May 30, 2013, 9:08 p.m...

Question: how do you flash nodes that the system marks as disabled? flash -m 1 -s 2, flash -n n2, or even flash -c either say...

Code:

  NOTE: m1 n1: prom file version 6.156 is the same as current version 6.156: prom
  
  not flashed

...or say nothing is there to be flashed.
Also, I see this at startup:

Code:

WARNING: downrev router board in module 1 slot r1

I wasn't aware you could flash those too.

_________________
:Crimson:

<-------- A very happy forum member.

SAQ
Who joined July 19, 2006, 7:37 a.m.
and authored 5632 notes

Wrote the following at June 1, 2013, 2:12 p.m...

pentium wrote:

Question: how do you flash nodes that the system marks as disabled? flash -m 1 -s 2, flash -n n2, or even flash -c either say...

Code:

   NOTE: m1 n1: prom file version 6.156 is the same as current version 6.156: prom
   
   not flashed

...or say nothing is there to be flashed.
Also, I see this at startup:

Code:

WARNING: downrev router board in module 1 slot r1

I wasn't aware you could flash those too.

You can't. It sounds like your Origin is a first-gen model, probably originally 180MHz. The first generation of SPIDER chips on the routers was different from the ones that were qualified/shipped with the later IP31 machines. I'm not sure if that's a "won't work" (V12 on XBOW 1.3) or a "not guaranteed to work" (big SIMMS in early IP20s) issue, but that's what it's referring to.

_________________
Damn the torpedoes, full speed ahead!

There are those who say I'm a bit of a curmudgeon. To them I reply: "GET OFF MY LAWN!"

:Indigo:

(single-CM)

pentium
Who joined Aug. 28, 2006, 5:29 p.m.
and authored 4256 notes

Wrote the following at June 5, 2013, 7:47 a.m...

Thought as much. Still however can't persuade the other nodes to flash.

_________________
:Crimson:

<-------- A very happy forum member.

pentium
Who joined Aug. 28, 2006, 5:29 p.m.
and authored 4256 notes

Wrote the following at June 22, 2013, 1:49 p.m...

Lets try and get back on this.

After one false start where the MSC kicked out the PSU with a fault It came up the second time with a console spamming:

Code:

  1A 000: =====> bridge_sanity diag  took an exception. <=====
  
  1A 000:  EPC    : 0xc00000001fc11530
  
  1A 000:  BadVA  : 0x0000000000000000
  
  1A 000:  Cause  : 0x000000008000c01c

Another power cycle stopped that. Configuration is still sitting at one flashed and working node, a healthy Irix install and a bunch of other nodes with disabled CPU's disabled ram and PROM versions that I cannot seem to get irix to flash.

Edited: Woah.

Code:

  IP27 PROM SGI Version 6.156  built 11:27:56 AM Nov 18, 2003
  
  Testing/Initializing memory ...............             DONE
  
  Copying PROM code to memory ...............             DONE
  
  Discovering local IO ......................
  
  Found an io Board with nic 030-1275-002.
  
  Found an io Board with nic 030-1275-004.
  
  DONE
  
  Discovering NUMAlink connectivity .........             DONE
  
  Found 5 objects (3 hubs, 2 routers) in 55798 usec
  
  Waiting for peers to complete discovery....             DONE
  
  Recognized 390 MHz midplane
  
  Global master is /hw/module/1/slot/n1
  
  *** Barrier sync warning: local=4418, NIC 0x329e00=4400 (promrev mismatch?)
  
  *** Barrier sync warning: local=4446, NIC 0x329e00=4428 (promrev mismatch?)
  
  *** Barrier sync warning: local=4908, NIC 0x329e00=4890 (promrev mismatch?)
  
  *** Barrier sync warning: local=4981, NIC 0x329e00=4963 (promrev mismatch?)
  
  Testing/Initializing all memory ...........             DONE
  
  waiting for node with nic 329e00 at module 1 slot 2 at global barrier.....
  
  *** Barrier sync warning: local=5108, NIC 0x329e00=5090 (promrev mismatch?)
  
  waiting for node with nic 3472e2 at module 1 slot 3 at global barrier...........
  
  Checking partitioning information .........             DONE
  
  *** Barrier sync warning: local=5355, NIC 0x329e00=5337 (promrev mismatch?)
  
  Loading BASEIO prom .......................             DONE
  
  BASEIO PROM Monitor SGI Version 6.156  built 11:26:28 AM Nov 18, 2003 (BE64)
  
  6 CPUs on 3 nodes found.
  
  Installing PROM Device drivers ............
  
  Base I/O Ethernet set to /dev/ethernet/ef0
  
  Walking SCSI Adapter 0 (/hw/module/1/slot/io1), (pci id 0)
  
  1+ 2- 3- 4- 5- 6+ 7- 8- 9- 10- 11- 12- 13- 14- 15- = 2 device(s)
  
  Walking SCSI Adapter 1 (/hw/module/1/slot/io1), (pci id 1)
  
  1- 2- 3- 4- 5- 6- 7- 8- 9- 10- 11- 12- 13- 14- 15- = 0 device(s)
  
  Initializing PROM Device drivers ..........             DONE
  
  Checking hardware inventory ...............                      Found new or re
  
  -enabled component MEM BANK 0 1 2 3
  
  ***Warning: Found a new MSCSI board in module 1, slot io7, serial EAX394
  
  Please use the 'update' command from the PROM Monitor to update the inventory
  
  ***Warning: Found a new IP27 board in module 1, slot n3, serial HXC897
  
  Please use the 'update' command from the PROM Monitor to update the inventory
  
  DONE
  
  **** System Configuration and Diagnostics Summary ****
  
  CONFIG:
  
  No. of NODEs enabled    = 3
  
  No. of NODEs disabled   = 0
  
  No. of CPUs enabled     = 6
  
  No. of CPUs disabled    = 0
  
  Mem enabled             = 1792 MB
  
  Mem disabled            = 0 MB
  
  No. of RTRs enabled     = 2
  
  No. of RTRs disabled    = 0
  
  DIAG RESULTS:
  
  ALL DIAGS PASSED.
  
  **** End System Configuration and Diagnostics Summary ****
  
  System Maintenance Menu
  
  1) Start System
  
  2) Install System Software
  
  3) Run Diagnostics
  
  4) Recover System
  
  5) Enter Command Monitor
  
  Option? 5
  
  Command Monitor.  Type "exit" to return to the menu.
  
  >> hinv -v
  
  IP27 Node Board, Module 1, Slot n1
  
  ASIC HUB Rev 5, 100 MHz, (nasid 0)
  
  Processor A: 250 MHz R10000 Rev 3.4
  
  Secondary Cache 4MB 250MHz Tap 0x9 , (cpu 0)
  
  R10010FPC Rev 3.4
  
  Processor B: 250 MHz R10000 Rev 3.4
  
  Secondary Cache 4MB 250MHz Tap 0x9 , (cpu 1)
  
  R10010FPC Rev 3.4
  
  Memory on board, 256 MBytes (Standard)
  
  Bank 0, 128 MBytes (Standard)  <-- (Software Bank 0)
  
  Bank 1, 128 MBytes (Standard)
  
  IP27 Node Board, Module 1, Slot n2
  
  ASIC HUB Rev 5, 100 MHz, (nasid 1)
  
  Processor A: 250 MHz R10000 Rev 3.4
  
  Secondary Cache 4MB 250MHz Tap 0x19 , (cpu 2)
  
  R10010FPC Rev 3.4
  
  Processor B: 250 MHz R10000 Rev 3.4
  
  Secondary Cache 4MB 250MHz Tap 0x19 , (cpu 3)
  
  R10010FPC Rev 3.4
  
  Memory on board, 512 MBytes (Standard)
  
  Bank 0, 128 MBytes (Standard)  <-- (Software Bank 0)
  
  Bank 1, 128 MBytes (Standard)
  
  Bank 2, 128 MBytes (Standard)
  
  Bank 3, 128 MBytes (Standard)
  
  IP27 Node Board, Module 1, Slot n3
  
  ASIC HUB Rev 5, 100 MHz, (nasid 2)
  
  Processor A: 250 MHz R10000 Rev 3.4
  
  Secondary Cache 4MB 250MHz Tap 0x9 , (cpu 4)
  
  R10010FPC Rev 3.4
  
  Processor B: 250 MHz R10000 Rev 3.4
  
  Secondary Cache 4MB 250MHz Tap 0x9 , (cpu 5)
  
  R10010FPC Rev 3.4
  
  Memory on board, 1024 MBytes (Standard)
  
  Bank 0, 256 MBytes (Standard)  <-- (Software Bank 0)
  
  Bank 1, 256 MBytes (Standard)
  
  Bank 2, 256 MBytes (Standard)
  
  Bank 3, 256 MBytes (Standard)
  
  BASEIO IO Board, Module 1, Slot io1
  
  ASIC BRIDGE Rev 4, (widget 8)
  
  adapter PCI-SCSI Rev 5
  
  (pci id 0)
  
  peripheral SCSI DISK, ID 1, IBM DXHS18Y
  
  peripheral SCSI CDROM, ID 6, TOSHIBA CD-ROM XM-5401TA
  
  adapter PCI-SCSI Rev 5
  
  (pci id 1)
  
  adapter IOC3 Rev 1
  
  (pci id 2)
  
  controller multi function SuperIO
  
  controller Ethernet Rev 1
  
  adapter IOC3 Rev 1
  
  (pci id 6)
  
  controller multi function SuperIO
  
  controller Keyboard/Mouse
  
  controller Parallel Port
  
  adapter RAD
  
  (pci id 7)
  
  PCI_XIO IO Board, Module 1, Slot io2
  
  ASIC BRIDGE Rev 3, (widget 12)
  
  adapter ID (Vendor 14e4 Device 1645 class 2 subclass 2)
  
  (pci id 0)
  
  adapter ID (Vendor 10df Device fc00 class c subclass c)
  
  (pci id 1)
  
  XTALK_PCI IO Board, Module 1, Slot io6
  
  ASIC BRIDGE Rev 4, (widget 13)
  
  adapter ID (Vendor 10a9 Device 9 class 2 subclass 2)
  
  (pci id 1)
  
  XTALK_PCI IO Board, Module 1, Slot io5
  
  ASIC BRIDGE Rev 4, (widget 14)
  
  adapter ID (Vendor 1077 Device 2200 class 1 subclass 1)
  
  (pci id 1)
  
  MSCSI IO Board, Module 1, Slot io7
  
  ASIC BRIDGE Rev 3, (widget 14)
  
  adapter PCI-SCSI Rev 5
  
  (pci id 0)
  
  adapter PCI-SCSI Rev 5
  
  (pci id 1)
  
  adapter PCI-SCSI Rev 5
  
  (pci id 2)
  
  adapter PCI-SCSI Rev 5
  
  (pci id 3)
  
  ASIC ROUTER , Module 1, Slot r1 (nasid 1)
  
  ASIC ROUTER , Module 1, Slot r2 (nasid 2)
  
  MIDPLANE, Module 1 Frequency 390 MHz
  
  ASIC XBOW Rev 4, on midplane of Module 1
  
  MIDPLANE, Module 1 Frequency 390 MHz
  
  ASIC XBOW Rev 4, on midplane of Module 1
  
  MARDIGRAS Graphics Board, Module 1, Slot io4
  
  >>

Suddenly two nodes started working fine. CPU's and ram are behaving as well. Lemme see if it boots.

Edited: Yeah. Aside from PROM rev mismatches the system is running on three nodeboards suddenly with not a peep about CPU or ram failures.

Code:

  >> auto
  
  Starting up the system...
  
  Loading dksc(0,1,8)/sash: 896+111764+16853+3848 entry: 0xa8000000012a6ee4
  
  4460239+1061168+992208 entry: 0xa80000000001a750
  
  WARNING: downrev router board in module 1 slot r1
  
  WARNING: downrev router board in module 1 slot r1
  
  WARNING: downrev router board in module 1 slot r2
  
  IRIX Release 6.5 IP27 Version 01090133 System V - 64 Bit
  
  Copyright 1987-2006 Silicon Graphics, Inc.
  
  All Rights Reserved.
  
  Setting rbaud to 19200
  
  /hw/module/1/slot/io5/xtalk_pci/pci/1/scsi_ctlr/0: Firmware version: 2.2.6: TP.
  
  /hw/module/1/slot/io5/xtalk_pci/pci/1/scsi_ctlr/0: Response in of 32768 is inval
  
  id, rereading.
  
  /hw/module/1/slot/io5/xtalk_pci/pci/1/scsi_ctlr/0: Response in of 32768 is inval
  
  id, ignoring.
  
  /hw/module/1/slot/io5/xtalk_pci/pci/1/scsi_ctlr/0: Response in of 32768 is inval
  
  id, rereading.
  
  /hw/module/1/slot/io5/xtalk_pci/pci/1/scsi_ctlr/0: Response in of 32768 is inval
  
  id, ignoring.
  
  /hw/module/1/slot/io5/xtalk_pci/pci/1/scsi_ctlr/0: Response in of 32768 is inval
  
  id, rereading.
  
  /hw/module/1/slot/io5/xtalk_pci/pci/1/scsi_ctlr/0: Response in of 32768 is inval
  
  id, ignoring.
  
  /hw/module/1/slot/io5/xtalk_pci/pci/1/scsi_ctlr/0: no cable detected.
  
  NOTICE: Starting failsoftd
  
  The system is coming up.
  
  network: WARNING: IRIS's Internet address is the default.
  
  Using standalone network mode.
  
  WARNING: ef0: link fail - check ethernet cable
  
  Warning:  Internet Gateway web server running as root.
  
  Use "chkconfig webface_apache off" to disable.
  
  Starting new eventmond...
  
  No System change detected.
  
  IRIS console login: root
  
  IRIX Release 6.5 IP27 IRIS
  
  Copyright 1987-2006 Silicon Graphics, Inc. All Rights Reserved.
  
  Last login: Thu May 30 22:00:28 PDT 2013 on ttyd1
  
  TERM = (vt100)
  
  IRIS 1# hinv -v
  
  6 250 MHZ IP27 Processors
  
  CPU: MIPS R10000 Processor Chip Revision: 3.4
  
  FPU: MIPS R10010 Floating Point Chip Revision: 3.4
  
  CPU 0 at Module 1/Slot 1/Slice A: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x9
  
  CPU 1 at Module 1/Slot 1/Slice B: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x9
  
  CPU 2 at Module 1/Slot 2/Slice A: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x19
  
  CPU 3 at Module 1/Slot 2/Slice B: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x19
  
  CPU 4 at Module 1/Slot 3/Slice A: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x9
  
  CPU 5 at Module 1/Slot 3/Slice B: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x9
  
  Main memory size: 1792 Mbytes
  
  Instruction cache size: 32 Kbytes
  
  Data cache size: 32 Kbytes
  
  Secondary unified instruction/data cache size: 4 Mbytes
  
  Memory at Module 1/Slot 1: 256 MB (enabled)
  
  Bank 0 contains 128 MB (Standard) DIMMS (enabled)
  
  Bank 1 contains 128 MB (Standard) DIMMS (enabled)
  
  Memory at Module 1/Slot 2: 512 MB (enabled)
  
  Bank 0 contains 128 MB (Standard) DIMMS (enabled)
  
  Bank 1 contains 128 MB (Standard) DIMMS (enabled)
  
  Bank 2 contains 128 MB (Standard) DIMMS (enabled)
  
  Bank 3 contains 128 MB (Standard) DIMMS (enabled)
  
  Memory at Module 1/Slot 3: 1024 MB (enabled)
  
  Bank 0 contains 256 MB (Standard) DIMMS (enabled)
  
  Bank 1 contains 256 MB (Standard) DIMMS (enabled)
  
  Bank 2 contains 256 MB (Standard) DIMMS (enabled)
  
  Bank 3 contains 256 MB (Standard) DIMMS (enabled)
  
  ROUTER in Module 1/Slot 2: Revision 2: Active Ports [4,5,6] (enabled)
  
  ROUTER in Module 1/Slot 3: Revision 2: Active Ports [5,6] (enabled)
  
  Integral SCSI controller 3: Version QL1040B (rev. 2), differential
  
  Integral SCSI controller 4: Version QL1040B (rev. 2), differential
  
  Integral SCSI controller 5: Version QL1040B (rev. 2), differential
  
  Integral SCSI controller 6: Version QL1040B (rev. 2), differential
  
  Integral SCSI controller 0: Version QL1040B (rev. 2), single ended
  
  Disk drive: unit 1 on SCSI controller 0 (unit 1)
  
  CDROM: unit 6 on SCSI controller 0
  
  Integral SCSI controller 1: Version QL1040B (rev. 2), single ended
  
  Integral SCSI controller 2: Version Fibre Channel QL2200A, 33 MHz PCI
  
  IOC3/IOC4 serial port: tty1
  
  IOC3/IOC4 serial port: tty2
  
  IOC3/IOC4 serial port: tty3
  
  IOC3/IOC4 serial port: tty4
  
  IOC3 parallel port: plp1
  
  Graphics board: EMXI
  
  Gigabit Ethernet: eg0, module 1, XIO slot io6, firmware version 0.0.0
  
  Integral Fast Ethernet: ef0, version 1, module 1, slot io1, pci 2
  
  Iris Audio Processor: version RAD revision 7.0, number 1
  
  Origin MSCSI board, module 1 slot 7: Revision 3
  
  Origin BASEIO board, module 1 slot 1: Revision 4
  
  Origin PCI XIO board, module 1 slot 2: Revision 3
  
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 0
  
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 1
  
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 2
  
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 3
  
  PCI Adapter ID (vendor 0x14e4, device 0x1645) PCI slot 0
  
  PCI Adapter ID (vendor 0x10df, device 0xfc00) PCI slot 1
  
  PCI Adapter ID (vendor 0x10a9, device 0x0003) PCI slot 6
  
  PCI Adapter ID (vendor 0x10a9, device 0x0003) PCI slot 2
  
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 0
  
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 1
  
  PCI Adapter ID (vendor 0x10a9, device 0x0005) PCI slot 7
  
  PCI Adapter ID (vendor 0x10a9, device 0x0009) PCI slot 1
  
  PCI Adapter ID (vendor 0x1077, device 0x2200) PCI slot 1
  
  IOC3/IOC4 external interrupts: 1
  
  HUB in Module 1/Slot 1: Revision 5 Speed 100.00 Mhz (enabled)
  
  HUB in Module 1/Slot 2: Revision 5 Speed 100.00 Mhz (enabled)
  
  HUB in Module 1/Slot 3: Revision 5 Speed 100.00 Mhz (enabled)
  
  IP27prom in Module 1/Slot n1: Revision 6.156
  
  IP27prom in Module 1/Slot n2: Revision 6.69
  
  IP27prom in Module 1/Slot n3: Revision 6.156
  
  IO6prom on Global Master Baseio in Module 1/Slot io1: Revision 6.156
  
  IRIS 2#

Edited: WTF? Yes we still have the rev mismatch but I'm now getting healthy boots with all CPU's enabled and no disabled ram and I didn't do anything but add nodes.

EDITED: Oh..

Code:

  WARNING: /hw/module/1/slot/n2/node/cpubus/0/a: Number of consecutive exceptions
  
  (physical  HARDWARE ERROR STATE:
  
  +  Errors on node Nasid 0x2 (2)
  
  +    IP27 in /hw/module/1/slot/n3 [serial number HXC897]
  
  +      HUB signalled following errors.
  
  +        ICRB4_A entry Register: 0xa0000190052040
  
  +          39<->02: SN0Net address 0x64014810
  
  +          56<->56: CRB has an error
  
  +        ICRB4_B entry Register: 0xa5104e000000
  
  +        ICRB6_A entry Register: 0xa0000190052040
  
  +          39<->02: SN0Net address 0x64014810
  
  +          56<->56: CRB has an error
  
  +        ICRB6_B entry Register: 0xa5104e000000
  
  +        ICRB9_A entry Register: 0xa0000190052040
  
  +          39<->02: SN0Net address 0x64014810
  
  +          56<->56: CRB has an error
  
  +        ICRB9_B entry Register: 0xa5104e000000
  
  +        ICRBb_A entry Register: 0xa0000190052040
  
  +          39<->02: SN0Net address 0x64014810
  
  +          56<->56: CRB has an error
  
  +        ICRBb_B entry Register: 0xa5104e000000
  
  +        ICRBc_A entry Register: 0xa0000190052040
  
  +          39<->02: SN0Net address 0x64014810
  
  +          56<->56: CRB has an error
  
  +        ICRBc_B entry Register: 0xa5104e000000
  
  End Hardware Error State
  
  ++FRU ANALYSIS BEGIN
  
  ++
  
  ++
  
  ++                      FRU Analysis Summary
  
  ++
  
  ++      Software                                        : 70%
  
  ++
  
  ++FRU ANALYSIS END
  
  PANIC: /hw/module/1/slot/n2/node/cpubus/0/a: Kernel Data Bus error in Cached spa
  
  ce at physical address 0x3200a4010 /hw/module/1/slot/n4/node/memory/dimm_bank/1
  
  (EPC 0xc000000000020440)
  
  NOTICE - cpu 4 didn't dump TLB, may be hung
  
  Dumping to /hw/module/1/slot/io1/baseio/pci/0/scsi_ctlr/0/target/1/lun/0/disk/pa
  
  rtition/1/block at block 0, space: 0x2000 pages
  
  Waiting 5 seconds for I/O processor.
  
  CPU 3 is the I/O processor.
  
  Dumping low memory...
  
  Dumping static kernel pages......
  
  Dumping pfdat pages....
  
  Dumping backtrace pages...
  
  Dumping dynamic kernel pages.........
  
  Dumping buffer pages...
  
  Dumping remaining in-use pages......WARNING: /hw/module/1/slot/n2/node/cpubus/0/
  
  a: Number of consecutive exceptions (physical address 0x3200a4000) exceeded limi
  
  t
  
  WARNING: /hw/module/1/slot/n2/node/cpubus/0/a: Unrecoverable bus error exception
  
  , node 0x3 paddr 0x3200a4000
  
  WARNING: /hw/module/1/slot/n2/node/cpubus/0/a: Mem info: standard dir entry hi 0
  
  x0 entry lo 0x33a3
  
  WARNING: /hw/module/1/slot/n2/node/cpubus/0/a: Number of consecutive exceptions
  
  (physical address 0x3202a4000) exceeded limit
  
  WARNING: /hw/module/1/slot/n2/node/cpubus/0/a: Unrecoverable bus error exception
  
  , node 0x3 paddr 0x3202a4000
  
  WARNING: /hw/module/1/slot/n2/node/cpubus/0/a: Mem info: standard dir entry hi 0
  
  x0 entry lo 0x33a3
  
  Dumping free pages............WARNING: /hw/module/1/slot/n2/node/cpubus/0/a: Num
  
  ber of consecutive exceptions (physical address 0x320000000) exceeded limit
  
  WARNING: /hw/module/1/slot/n2/node/cpubus/0/a: Unrecoverable bus error exception
  
  , node 0x3 paddr 0x320000000
  
  WARNING: /hw/module/1/slot/n2/node/cpubus/0/a: Mem info: standard dir entry hi 0
  
  x0 entry lo 0x312c
  
  WARNING: /hw/module/1/slot/n2/node/cpubus/0/a: Number of consecutive exceptions
  
  (physical address 0x320004000) exceeded limit
  
  WARNING: /hw/module/1/slot/n2/node/cpubus/0/a: Unrecoverable bus error exception
  
  , node 0x3 paddr 0x320004000
  
  (this continues on and on...)

A reboot found that the ram in the last node failed again. The system came up fine after it was disabled I'll work with that later but for now the whole system is now up and running.

Code:

  IRIS 2# hinv -v
  
  8 250 MHZ IP27 Processors
  
  CPU: MIPS R10000 Processor Chip Revision: 3.4
  
  FPU: MIPS R10010 Floating Point Chip Revision: 3.4
  
  CPU 0 at Module 1/Slot 1/Slice A: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x9
  
  CPU 1 at Module 1/Slot 1/Slice B: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x9
  
  CPU 2 at Module 1/Slot 2/Slice A: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x19
  
  CPU 3 at Module 1/Slot 2/Slice B: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x19
  
  CPU 4 at Module 1/Slot 3/Slice A: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x9
  
  CPU 5 at Module 1/Slot 3/Slice B: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x9
  
  CPU 6 at Module 1/Slot 4/Slice A: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x0
  
  CPU 7 at Module 1/Slot 4/Slice B: 250 Mhz MIPS R10000 Processor Chip (enabled)
  
  Processor revision: 3.4. Scache: Size 4 MB Speed 250 Mhz  Tap 0x0
  
  Main memory size: 1920 Mbytes
  
  Instruction cache size: 32 Kbytes
  
  Data cache size: 32 Kbytes
  
  Secondary unified instruction/data cache size: 4 Mbytes
  
  Memory at Module 1/Slot 1: 256 MB (enabled)
  
  Bank 0 contains 128 MB (Standard) DIMMS (enabled)
  
  Bank 1 contains 128 MB (Standard) DIMMS (enabled)
  
  Memory at Module 1/Slot 2: 512 MB (enabled)
  
  Bank 0 contains 128 MB (Standard) DIMMS (enabled)
  
  Bank 1 contains 128 MB (Standard) DIMMS (enabled)
  
  Bank 2 contains 128 MB (Standard) DIMMS (enabled)
  
  Bank 3 contains 128 MB (Standard) DIMMS (enabled)
  
  Memory at Module 1/Slot 3: 1024 MB (enabled)
  
  Bank 0 contains 256 MB (Standard) DIMMS (enabled)
  
  Bank 1 contains 256 MB (Standard) DIMMS (enabled)
  
  Bank 2 contains 256 MB (Standard) DIMMS (enabled)
  
  Bank 3 contains 256 MB (Standard) DIMMS (enabled)
  
  Memory at Module 1/Slot 4: 128 MB (enabled)
  
  Bank 0 contains 128 MB (Standard) DIMMS (enabled)
  
  Bank 1 contains 128 MB (Standard) DIMMS (disabled)
  
  ROUTER in Module 1/Slot 2: Revision 2: Active Ports [4,5,6] (enabled)
  
  ROUTER in Module 1/Slot 4: Revision 2: Active Ports [4,5,6] (enabled)
  
  Integral SCSI controller 3: Version QL1040B (rev. 2), differential
  
  Integral SCSI controller 4: Version QL1040B (rev. 2), differential
  
  Integral SCSI controller 5: Version QL1040B (rev. 2), differential
  
  Integral SCSI controller 6: Version QL1040B (rev. 2), differential
  
  Integral SCSI controller 1: Version QL1040B (rev. 2), single ended
  
  Integral SCSI controller 0: Version QL1040B (rev. 2), single ended
  
  Disk drive: unit 1 on SCSI controller 0 (unit 1)
  
  CDROM: unit 6 on SCSI controller 0
  
  Integral SCSI controller 2: Version Fibre Channel QL2200A, 33 MHz PCI
  
  IOC3/IOC4 serial port: tty1
  
  IOC3/IOC4 serial port: tty2
  
  IOC3/IOC4 serial port: tty3
  
  IOC3/IOC4 serial port: tty4
  
  IOC3 parallel port: plp1
  
  Graphics board: EMXI
  
  Gigabit Ethernet: eg0, module 1, XIO slot io6, firmware version 0.0.0
  
  Integral Fast Ethernet: ef0, version 1, module 1, slot io1, pci 2
  
  Iris Audio Processor: version RAD revision 7.0, number 1
  
  Origin PCI XIO board, module 1 slot 2: Revision 3
  
  Origin MSCSI board, module 1 slot 7: Revision 3
  
  Origin BASEIO board, module 1 slot 1: Revision 4
  
  PCI Adapter ID (vendor 0x14e4, device 0x1645) PCI slot 0
  
  PCI Adapter ID (vendor 0x10df, device 0xfc00) PCI slot 1
  
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 0
  
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 1
  
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 2
  
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 3
  
  PCI Adapter ID (vendor 0x10a9, device 0x0009) PCI slot 1
  
  PCI Adapter ID (vendor 0x10a9, device 0x0003) PCI slot 6
  
  PCI Adapter ID (vendor 0x10a9, device 0x0003) PCI slot 2
  
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 0
  
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 1
  
  PCI Adapter ID (vendor 0x10a9, device 0x0005) PCI slot 7
  
  PCI Adapter ID (vendor 0x1077, device 0x2200) PCI slot 1
  
  IOC3/IOC4 external interrupts: 1
  
  HUB in Module 1/Slot 1: Revision 5 Speed 100.00 Mhz (enabled)
  
  HUB in Module 1/Slot 2: Revision 5 Speed 100.00 Mhz (enabled)
  
  HUB in Module 1/Slot 3: Revision 5 Speed 100.00 Mhz (enabled)
  
  HUB in Module 1/Slot 4: Revision 5 Speed 100.00 Mhz (enabled)
  
  IP27prom in Module 1/Slot n1: Revision 6.156
  
  IP27prom in Module 1/Slot n2: Revision 6.69
  
  IP27prom in Module 1/Slot n3: Revision 6.156
  
  IP27prom in Module 1/Slot n4: Revision 6.57
  
  IO6prom on Global Master Baseio in Module 1/Slot io1: Revision 6.156

Edited. I hate computers.
Removed the node, clean the ram and the slots. Put it back in....

Code:

  IP27 PROM SGI Version 6.156  built 11:27:56 AM Nov 18, 2003
  
  Testing/Initializing memory ...............             DONE
  
  Copying PROM code to memory ...............             DONE
  
  Discovering local IO ......................
  
  Found an io Board with nic 030-1275-002.
  
  Found an io Board with nic 030-1275-004.
  
  DONE
  
  Discovering NUMAlink connectivity .........             DONE
  
  Found 6 objects (4 hubs, 2 routers) in 66460 usec
  
  Waiting for peers to complete discovery....             DONE
  
  Recognized 390 MHz midplane
  
  Global master is /hw/module/1/slot/n1
  
  *** Barrier sync warning: local=4418, NIC 0x329e00=4400 (promrev mismatch?)
  
  *** Barrier sync warning: local=4446, NIC 0x329e00=4428 (promrev mismatch?)
  
  *** Barrier sync warning: local=4908, NIC 0x329e00=4890 (promrev mismatch?)
  
  *** Barrier sync warning: local=4981, NIC 0x329e00=4963 (promrev mismatch?)
  
  Testing/Initializing all memory ...........             DONE
  
  Initializing headless node at nasid 3
  
  .----- MEMORY FAILURE (miscompare, node slot 1) -----
  
  No hub MD error registered
  
  Address      0xa800000300001000
  
  Actual       0x9e0de114090dde55
  
  Expected     0x9e0de51c090dde55
  
  Difference   0x0000040800000000
  
  ----- MEMORY FAILURE (miscompare, node slot 1) -----
  
  No hub MD error registered
  
  Address      0xa800000300001020
  
  Actual       0xa20d994439360ab9
  
  Expected     0xa20d9d4c39360ab9
  
  Difference   0x0000040800000000
  
  ----- MEMORY FAILURE (miscompare, node slot 1) -----
  
  No hub MD error registered
  
  Address      0xa800000300001040
  
  Actual       0xa60d5174695e371d
  
  Expected     0xa60d557c695e371d
  
  Difference   0x0000040800000000
  
  ----- MEMORY FAILURE (miscompare, node slot 1) -----
  
  No hub MD error registered
  
  Address      0xa800000300001060
  
  Actual       0xaa0d09a499866381
  
  Expected     0xaa0d0dac99866381
  
  Difference   0x0000040800000000
  
  Exceeded maximum error count (stopping)
  
  .......*** Swapping bank 0 with bank 1 on headless node nasid 3
  
  Discovering local IO ......................             DONE
  
  *** Barrier sync warning: local=5108, NIC 0x329e00=5090 (promrev mismatch?)
  
  waiting for node with nic 3472e2 at module 1 slot 3 at global barrier.......
  
  Checking partitioning information .........             DONE
  
  *** Barrier sync warning: local=5355, NIC 0x329e00=5337 (promrev mismatch?)
  
  Loading BASEIO prom .......................             DONE
  
  BASEIO PROM Monitor SGI Version 6.156  built 11:26:28 AM Nov 18, 2003 (BE64)
  
  6 CPUs on 4 nodes found.
  
  Installing PROM Device drivers ............
  
  Base I/O Ethernet set to /dev/ethernet/ef0
  
  Walking SCSI Adapter 0 (/hw/module/1/slot/io1), (pci id 0)
  
  1+ 2- 3- 4- 5- 6+ 7- 8- 9- 10- 11- 12- 13- 14- 15- = 2 device(s)
  
  Walking SCSI Adapter 1 (/hw/module/1/slot/io1), (pci id 1)
  
  1- 2- 3- 4- 5- 6- 7- 8- 9- 10- 11- 12- 13- 14- 15- = 0 device(s)
  
  Initializing PROM Device drivers ..........             DONE
  
  Checking hardware inventory ...............                      Found new or re
  
  -enabled component MEM BANK 0 1 2 3
  
  ***Warning: Found a new IP27 board in module 1, slot n3, serial HXC897
  
  Please use the 'update' command from the PROM Monitor to update the inventory
  
  ***Warning: Found a new IP27 board in module 1, slot n4, serial JHL636
  
  Please use the 'update' command from the PROM Monitor to update the inventory
  
  ***Warning: Found a new MSCSI board in module 1, slot io7, serial EAX394
  
  Please use the 'update' command from the PROM Monitor to update the inventory
  
  DONE
  
  **** System Configuration and Diagnostics Summary ****
  
  CONFIG:
  
  No. of NODEs enabled    = 4
  
  No. of NODEs disabled   = 0
  
  No. of CPUs enabled     = 6
  
  No. of CPUs disabled    = 2
  
  Mem enabled             = 1920 MB
  
  Mem disabled            = 128 MB
  
  No. of RTRs enabled     = 2
  
  No. of RTRs disabled    = 0
  
  DIAG RESULTS:
  
  /hw/module/1/slot/n4/node/cpu/0: CPU A disabled
  
  Reason: PROM copied to memory (bank 0) is bad.
  
  /hw/module/1/slot/n4/node/cpu/1: CPU B disabled
  
  Reason: PROM copied to memory (bank 0) is bad.
  
  /hw/module/1/slot/n4/node/mem: MEMBANK(S) 0  disabled
  
  Reason:
  
  Bank 0: Some DIMMs failed mem test.
  
  **** End System Configuration and Diagnostics Summary ****

Lovely. The node went back to being completely unusable again. Should of left it alone when it was at least leaving the CPU's enabled.

Last edit for now, I swear: I got the CPU's back by rotating the ram around (which still fail) and cleaning the compression connector, but now node 1 is silently failing at boot and locking the system up......I'm done for now.

_________________
:Crimson:

<-------- A very happy forum member.

SAQ
Who joined July 19, 2006, 7:37 a.m.
and authored 5632 notes

Wrote the following at June 23, 2013, 11:07 a.m...

My suspicion is that the downrev router is in fact a "won't work" configuration with your boards. Perhaps try pulling that router and seeing if you can get the two nodes connected to the other router up reliably. Rotate your other IP31s through those Nx slots to see if they work.

Is it only one router or are both downrev?

_________________
Damn the torpedoes, full speed ahead!

There are those who say I'm a bit of a curmudgeon. To them I reply: "GET OFF MY LAWN!"

:Indigo:

(single-CM)

pentium
Who joined Aug. 28, 2006, 5:29 p.m.
and authored 4256 notes

Wrote the following at June 23, 2013, 1:34 p.m...

I've never upgraded them. They were both in there when I bought the system and it's four original 180mhz nodeboards.
I'm kind of scared to start moving the nodes. The last time I rotated them around everything went to hell. Right now aside from 128mb or ram being offline they are all generally behaving. It could be that the many, many insertions and removals have finally caught up to the compression connectors.

_________________
:Crimson:

<-------- A very happy forum member.

SGI: Hardware

Origin is sick