SGI: Development

Are there any known details on programming the ASICs responsible for managing the CPUs in multiprocessor SGIs? I mean, details from an OS-development point of view (ie: ASIC commands needed to create a new process on a certain CPU, or for scheduling each CPU, etc...).

I don't mind if the details are for Challenge, for the Octane, or for NUMA or later MP boxes.

The purpose is that I feel curious about the protocol of commands used by SGI for controlling MP systems. I'm beginning a pet project which needs (simple) MP and maybe I could get some inspiration from older SGI designs...

good question, I do not have an answer, but I can say OpenBSD and Linux has SMP for IP30

The O2000 and Onyx2 (and by extension the Octane and Tezro) are based on an academic computer design from Stanford called DASH. There are many papers about it on citeseer. This design uses directory memories to produce coherency from a loosely-coupled mesh of processing nodes (each of which has several CPUs in a traditional SMP arrangement).

robespierre wrote: The O2000 and Onyx2 (and by extension the Octane and Tezro) are based on an academic computer design from Stanford called DASH. There are many papers about it on citeseer. This design uses directory memories to produce coherency from a loosely-coupled mesh of processing nodes (each of which has several CPUs in a traditional SMP arrangement).

Stanford DASH

Stanford DASH was a cache coherent multiprocessor developed in the late 1980s by a group led by Anoop Gupta, John L. Hennessy, Mark Horowitz, and Monica S. Lam at Stanford University. It was based on adding a pair of directory boards designed at Stanford to up to 16 SGI IRIS 4D Power Series machines and then cabling the systems in a mesh topology using a Stanford-modified version of the Torus Routing Chip. The boards designed at Stanford implemented a directory-based cache coherence protocol allowing Stanford DASH to support distributed shared memory for up to 64 processors. Stanford DASH was also notable for both supporting and helping to formalize weak memory consistency models, including release consistency. Because Stanford DASH was the first operational machine to include scalable cache coherence, it influenced subsequent computer science research as well as the commercially available SGI Origin 2000 . Stanford DASH is included in the 25th anniversary retrospective of selected papers from the International Symposium on Computer Architecture and several computer science books, has been simulated by the University of Edinburgh, and is used as a case study in contemporary computer science classes.

• dash paper
• lenoski '92
• The Directory-Based Cache Coherence Protocol
• merez, lecture17
• weems lecture 21
• performances and implementation
• Meng Zhang

any good book & paper to learn about that ?

I opened this topic, but …

This is the DASH book: Scalable Shared-Memory Multiprocessing - 30% (100 pages) of it is "Experience with DASH". I've got it some time ago for less than $10.

GL1zdA wrote: This is the DASH book: Scalable Shared-Memory Multiprocessing - 30% (100 pages) of it is "Experience with DASH". I've got it some time ago for less than $10.

bought in 20 milli seconds since I realized that baham_books sells (UK book seller) it for €9,31 euro including shipping

I think that the Octane and Desktop Tezro only have a single node, either with 2 CPUs (for Octane) or 4 CPUs (for Tezro). You would think that the Rackmount Tezro is really an O350 so it would support multiple nodes, but the second brick has graphics and I/O only, no CPUs.

robespierre wrote: I think that the Octane and Desktop Tezro only have a single node, either with 2 CPUs (for Octane) or 4 CPUs (for Tezro). You would think that the Rackmount Tezro is really an O350 so it would support multiple nodes, but the second brick has graphics and I/O only, no CPUs.

You're right with the Octane and Desktop Tezro, but the Rackmount Tezro was also available as 2 nodes 2 CPUs each using the Workstation Expansion Module (since you couldn't use a 4x 1 GHz CPU nodeboard with the V12). I've tried to compile the available options here , but I don't think anyone has experimented with extreme (unsupported) Tezro configs like the ones in my questions. But there were successful attempts to change the various personalities (Origin 350/Onyx 350/Tezro) of the Chimera.

ivelegacy wrote: good question, I do not have an answer, but I can say OpenBSD and Linux has SMP for IP30

Thanks, I forgot this. I'll take a look at the source. Thanks a lot too for the pointers towards DASH documentation to everybody who contributed.

Most of the Linux SMP code came out of SGI, if you're of a mind to look at any of that. Probably you're not...

and got my book

cesss wrote: Are there any known details on programming the ASICs responsible for managing the CPUs in multiprocessor SGIs? I mean, details from an OS-development point of view (ie: ASIC commands needed to create a new process on a certain CPU, or for scheduling each CPU, etc...).

I don't mind if the details are for Challenge, for the Octane, or for NUMA or later MP boxes.

The purpose is that I feel curious about the protocol of commands used by SGI for controlling MP systems. I'm beginning a pet project which needs (simple) MP and maybe I could get some inspiration from older SGI designs...

If you are interested in experimenting a bit with multiprocessors, inter-processor interrupts etc. then Stanford's SimOS/MIPS may also be of interest. Its related to the FLASH architecture ( http://mprc.pku.edu.cn/mentors/training ... kuskin.pdf ) which many Stanford DASH people (and future VMware people) were involved in. Its a partial implementation though and probably about 50% of its commands and registers are no-ops. Documentation isn't great, but you do however have the source code to see how the 'hardware' works, making it possible to understand enough to do CPU startup/shutdown, interrupt setup, interprocessor interrupts, timers etc, and it doesn't take a lot of code - maybe 10% of what an x86/Intel MPS multiprocessor from that same era would have taken. If learning basic SMP operations is your goal, its probably a good starting point. For example, it would be a good exercise to try to port an existing OS like Linux to it.

Linux SMP isn't that hard to understand (though admittedly I haven't kept up with it for many years now, and it tends to grow in complexity each year). Even though they were obsolete, I found it easier to look at simpler early SMP architectures first (e.g. SPARC32, Alpha) - PROM and hardware handled many of the really low level details and so you can concentrate on understanding how stuff like how tracking and managing CPUs, IPIs, cache operations work. I certainly think diving straight into something like a high end NUMA architecture is going to be a more difficult approach.

Also, I'd recommend you get a really good grounding in cache/TLB coherency, atomic locking, memory ordering, and interrupts on multiprocessor systems first. Otherwise you will probably have to redesign all your code when it doesn't work as you expect on real hardware

a very interesting Architecture: The Onyx

The Onyx is actually a very classic SMP design - a fast bus integrating all components. Onyx2 is where it becomes interesting, when they try to find the right balance between tightly coupled SMP and loosely coupled cluster.

Linus Torvald's on 13 November 2012:



Link to the original interview