Miscellaneous Operating Systems/Hardware

(A bit of background: I have an O350 netbooting, accessing its hard drive, keeping track of the time of day, and so on. Basically mostly working in single-processor mode, though serial is untested, the PS/2 keyboard/mouse ports are unsupported, the only use of the L1 is as a console device, and so on. I have an O300 that is not extremely far behind, but definitely needs some more work, and I'm still buying more hardware to work with.)

Kumba wrote:

nyef wrote: For that matter, I still don't have any "fully working" MIPS64 Linux systems quite yet (by which I mean with a working bootloader, and SMP if it's not a Fuel, and there's that IOC3 IRQ conflict, and if it is a Fuel or Tezro then the graphics card driver, and then there's the L1 controller stuff...).

Not to threadjack, but IOC3 IRQ conflict? IIRC, at least on Octane with the Metadriver, IOC3 should be requesting two IRQs, one for ethernet, and ethernet+2 for I/O by kb/mouse. Serial uses a UART polling hack built into 8250 core to run the serial ports for now (by telling it to use IRQ 0), but the end goal would be to make the Altix DMA-capable IOC3 driver work.

First off, I'm not (yet?) using the IOC3 Metadriver, just the IP27 drivers.

Second, that "just add $n$ to the IRQ number" idea is horrifically broken in a modern Linux kernel (dynamically-allocated IRQ numbers, after all). It probably works on IP30 because of the pcibr driver allocating numbers for all eight IRQ pins at once. Next thing we know, it'll turn out to vary depending on which type of system or expansion card the IOC3 is in.

Third, I misremembered. My notes say that it's a conflict between the SCSI controller and the USB host controller leading to lost interrupts, not the IOC3. Which isn't to say that there's not an IOC3 IRQ issue still, but that it's not at the level of "known IRQ conflict"... yet, if ever.

Fourth, none of the MIPS Bridge/XBridge/PIC drivers, even for IP30, support all of the song and dance involved with shared IRQs, or the mapping of interrupt lines other than INTA.

Fifth, while I have a plan of attack for sorting out my IRQ issues, currently it only affects my O300, so it's not a huge critical thing for me right now.

Sixth, Ralf was suggesting switching to IRQ domains...

Kumba wrote:

nyef wrote: Second, can we use strace against Xsgi on IRIX itself to any useful purpose?

Apparently so. I was digging into my chat logs from 2004, and Stan used a patched strace to monitor Xsgi calls to MGras, which helped him figure out some of the DMA bits needed to write the Impact driver. I don't think he ever released that patch, though. Would probably be useful for a lot of things outside of Linux porting.

Hrm... I wonder if we could do that to get better support with the VPro driver? Although first I'd need to get a working VPro machine...

nyef wrote: First off, I'm not (yet?) using the IOC3 Metadriver, just the IP27 drivers.

Yup, Metadriver isn't in the tree just yet. I hope to start sending it in after 4.2, but we'll see. Technically, it's already in the tree, just under drivers/sn, and it's Altix only, so my actual patches move it to drivers/misc (where ioc4.c is at) and fixes it up to work on MIPS. Only problem is, I don't know if those changes break ia64/Altix, since I don't have one of those systems...

nyef wrote: Second, that "just add $n$ to the IRQ number" idea is horrifically broken in a modern Linux kernel (dynamically-allocated IRQ numbers, after all). It probably works on IP30 because of the pcibr driver allocating numbers for all eight IRQ pins at once. Next thing we know, it'll turn out to vary depending on which type of system or expansion card the IOC3 is in.

Won't argue with you there, but it does call request_irq() to get the IRQ numbers, and this works fine on IP30 and IP27 so far. It'll be interesting to see if it JustWorks(TM) with IP35 as well, once I can combine your code with some of my out-of-tree patches. I think the IRQ thing would only be an issue for the IOC3 PCI card, also known as a CADDuo (which I have stashed in a storage bin somewhere...).

nyef wrote: Third, I misremembered. My notes say that it's a conflict between the SCSI controller and the USB host controller leading to lost interrupts, not the IOC3. Which isn't to say that there's not an IOC3 IRQ issue still, but that it's not at the level of "known IRQ conflict"... yet, if ever.

It might be IOC3. It's a weird little piece of hardware, and Cthulhu knows if the existing in-tree driver isn't messing things up somehow. In reality, the MIPS IOC3 bits are all in ioc3-eth.c. Only Altix uses the drivers/sn/ioc3.c copy (for now).

nyef wrote: Fourth, none of the MIPS Bridge/XBridge/PIC drivers, even for IP30, support all of the song and dance involved with shared IRQs, or the mapping of interrupt lines other than INTA.

There's a lot not currently supported, such as hard-wiring the RRB allocations, Octane's screwed-up DMA mapping (which I highly suspect is giving me grief with >2GB RAM), using the one-wire drivers to query the NIC chips to build a hardware inventory, etc...

nyef wrote: Sixth, Ralf was suggesting switching to IRQ domains...

That or this new irqchip thing. Or are both used? It seems right as I finally figure something out in Linux, some wise guy goes off and implements an entirely new subsystem to replace it. I'll loathe the day fbdev/CONFIG_VT become dependent on systemd...if those devs ever get their way, that is :/

nyef wrote: Hrm... I wonder if we could do that to get better support with the VPro driver? Although first I'd need to get a working VPro machine...

Entirely possible, but way beyond my skill level. Stan was the graphics god. Even though I understand things better now than I did 10 years ago, it's still boggling how easy hardware hacking came to him.

Kumba wrote:

nyef wrote: First off, I'm not (yet?) using the IOC3 Metadriver, just the IP27 drivers.

Yup, Metadriver isn't in the tree just yet. I hope to start sending it in after 4.2, but we'll see. Technically, it's already in the tree, just under drivers/sn, and it's Altix only, so my actual patches move it to drivers/misc (where ioc4.c is at) and fixes it up to work on MIPS. Only problem is, I don't know if those changes break ia64/Altix, since I don't have one of those systems...

Looking at the in-kernel driver, it looks like only the BaseIO IOC3s are supported for dual interrupts, everything else is a single-interrupt setup... And the MENET 1/2/3 and MENET 4 card classes aren't even detected, or they're lumped in with the "serial" class!

Kumba wrote:

nyef wrote: Second, that "just add $n$ to the IRQ number" idea is horrifically broken in a modern Linux kernel (dynamically-allocated IRQ numbers, after all). It probably works on IP30 because of the pcibr driver allocating numbers for all eight IRQ pins at once. Next thing we know, it'll turn out to vary depending on which type of system or expansion card the IOC3 is in.

Won't argue with you there, but it does call request_irq() to get the IRQ numbers, and this works fine on IP30 and IP27 so far. It'll be interesting to see if it JustWorks(TM) with IP35 as well, once I can combine your code with some of my out-of-tree patches. I think the IRQ thing would only be an issue for the IOC3 PCI card, also known as a CADDuo (which I have stashed in a storage bin somewhere...).

I have a CAD Duo myself, though I haven't tried installing it anywhere yet. But, as mentioned above, it looks like the IRQ thing is only an issue for BaseIO "cards".

Kumba wrote:

nyef wrote: Fourth, none of the MIPS Bridge/XBridge/PIC drivers, even for IP30, support all of the song and dance involved with shared IRQs, or the mapping of interrupt lines other than INTA.

There's a lot not currently supported, such as hard-wiring the RRB allocations, Octane's screwed-up DMA mapping (which I highly suspect is giving me grief with >2GB RAM), using the one-wire drivers to query the NIC chips to build a hardware inventory, etc...

"Proper" PCI enumeration / resource allocation. Yet another case where I'm fairly sure I know how to get the hardware to work, but have no idea about how to get the kernel side to work / not mess things up.

Kumba wrote:

nyef wrote: Sixth, Ralf was suggesting switching to IRQ domains...

That or this new irqchip thing. Or are both used? It seems right as I finally figure something out in Linux, some wise guy goes off and implements an entirely new subsystem to replace it. I'll loathe the day fbdev/CONFIG_VT become dependent on systemd...if those devs ever get their way, that is :/

That might be the day to sit down and figure out how to replicate whatever the kernel needs from... it ... and start our own linux distro. Or just switch to FreeBSD. We'd have to somewhat start over with porting the kernel, but we could crib far more directly from OpenBSD...

Kumba wrote:

nyef wrote: Hrm... I wonder if we could do that to get better support with the VPro driver? Although first I'd need to get a working VPro machine...

Entirely possible, but way beyond my skill level. Stan was the graphics god. Even though I understand things better now than I did 10 years ago, it's still boggling how easy hardware hacking came to him.

A good part of that is practice and having a system for going about it. I've figured things out from as little as a ROM dump, some basic idea of the system architecture, and a list of entry points before. Not even any actual hardware!

extremely interesting thread, unfortunately I can only contribute like one of your fans =)

nyef wrote:

Kumba wrote: I think the IRQ thing would only be an issue for the IOC3 PCI card, also known as a CADDuo (which I have stashed in a storage bin somewhere...).

Allow me to shed some light on the subject.

First, you should accept that the IOC3 chip is not a PCI chip. Even though it is connected to the PCI bus, it does not follow the PCI specs.

In the PCI world, a PCI device, by default, is allowed to use only one of the four PCI interrupt pins (A-D). The pin number is reported in the PCI Interrupt Register in the PCI Configuration Space. But if, and only if, it advertizes itself as a multifunction device, then the subfunctions, which will answer to different addresses in the PCI Configuration Space, are allowed to use different values for the PCI Interrupt Register, and thus use different interrupt pins. Note that the way pins A-D are routed to actual interrupt sources can (and will) vary between PCI slots, as configured by the PCI Controller (which might hardwire the logic), so that PCI devices defaulting to interrupt pin A (as recommended) will not necessarily share the same interrupt source on the host.

Now, IOC3 violates the PCI specifications in two ways:
- it will not answer to the first 0x40 bytes of configuration space access, with some addresses being shared (such as the first two BARs), and some not being answered, but without releasing the bus, so you can't even rely upon a bus timeout to recover in software.
- when it actually uses two interrupts, it does not advertize itself as a multi-function device.

Note that I write `when'. The reason is that IOC3 may not necessarily be fitted with all the subcomponents: some IOC3 flavours lack the Ethernet part, for example.

The IOC3 interrupt logic is to use pin A for the SuperIO devices (serial ports, parallel port, PS/2 ports) and pin B for the Ethernet, when it has both. An Ethernet-only IOC3 (such as the first three ones on the MENET board) will only use pin A, and a SuperIO-only IOC3 (such as those found on SIO UFC boards, or the second onboard IOC3 on Origin 2000) will only use pin A as well.

When the IOC3 device exists as a PCI card (such as the CADDuo), the interrupt assignment for pin A and pin B is controlled by the PCI Controller they are plugged into.

When the IOC3 device is onboard, and connected to a Bridge, XBridge, or PIC chip, routing of its interrupt pins is static and will vary accross systems; the rule of thumb being that it uses the lowest unused Bridge interrupt sources. For example, on an Octane, the on-board devices are the two SCSI controllers, Bridge interrupts 0 and 1, and RAD audio, Bridge interrupt 3. IOC3 will use Bridge interrupt 2 for SuperIO and Bridge interrupt 4 for Ethernet. But on a Fuel, the on-board devices are the SCSI controller, Bridge interrupt 1, the two upper PCI slots, Bridge interrupts 2 and 3, and the USB Controller, Bridge interrupt 5. IOC3 here will use Bridge interrupt 0 for SuperIO, and Bridge interrupt 4 for Ethernet.

Don't you love IOC3 now?

maybe (in the next life) I will find a way to home-build pretty (cheap) PCI fpga dev boards (they are already existing, but too expensive), in this case we could domesticate the SGI_PCI, and It should be nice to add new hw and sw features without all the troubles we have now because the SGI_PCI is not INTEL_PCI compliant.

let me say, having to deal with the PCI has a lot of troubles, e.g. I have a RS/PRO machine (it's a router, MIPS32, Atheros9) in where the PCI controller completely misses the PCI_IO mode, it's only able to perform PCI_MEM, and … the 95% of the mini_PCI boards on the markets are PCI_IO, so … no party.

I think Ubiquiti wanted to sell their custom minIPCI modules for RS/PRO (typically, wifi miniPCI modules).
I think SGI wanted to do the same with their products. I can't see any other valid reason.

the PCI, what a drama

ivelegacy wrote: maybe (in the next life) I will find a way to home-build pretty (cheap) PCI fpga dev boards (they are already existing, but too expensive), in this case we could domesticate the SGI_PCI, and It should be nice to add new hw and sw features without all the troubles we have now because the SGI_PCI is not INTEL_PCI compliant.

The "5I25 Superport FPGA based PCI Anything I/O card" from Mesa Electronics
is a Spartan-6 based PCI-card that is not expensive. It costs 89 US$.
See:
http://www.mesanet.com/fpgacardinfo.html
The manual is here:
http://www.mesanet.com/pdf/parallel/5i25man.pdf

Excellent, thank you! Never heard about Mesa Electronics before, I was shocked by all the expensive kits I have seen from Digilent and Xilinx resellers, they sell big Virtex boards with PCI and ePCI interface, but their price start form 500 USD.

Check the price of the Altium Nanoboard... lol

rwengerter wrote: The "5I25 Superport FPGA based PCI Anything I/O card" from Mesa Electronics
is a Spartan-6 based PCI-card that is not expensive. It costs 89 US$.
See:
http://www.mesanet.com/fpgacardinfo.html
The manual is here:
http://www.mesanet.com/pdf/parallel/5i25man.pdf

I have bought a pair of those boards. They are on their way.
plans: to develop something cool by the end of the next year.

ivelegacy wrote:

rwengerter wrote: The "5I25 Superport FPGA based PCI Anything I/O card" from Mesa Electronics
is a Spartan-6 based PCI-card that is not expensive. It costs 89 US$.
See:
http://www.mesanet.com/fpgacardinfo.html
The manual is here:
http://www.mesanet.com/pdf/parallel/5i25man.pdf

I have bought a pair of those boards. They are on their way.
plans: to develop something cool by the end of the next year.

If you are looking for a free processor core, I suggest the Microblaze MCS which
is free, programmable in C (with free C Compiler) and does not use much resources.
Here is a suitable tutorial for implementation in a Spartan-6:
http://ece.wpi.edu/~rjduck/Microblaze%2 ... l%20v5.pdf

rwengerter wrote: If you are looking for a free processor core

are you into soft core ? thank you, I am already fine

I have a pretty MIPS-R3K soft core, multi cycles pipelined version, being developed for three years, I have been carefully reading reports for years, the catalogue of defects is basically always the same, basically up to 2 (and no more than 2) bugs a semester been fixed without regression. It means stable progress. it's now working good within -O0..-O3 (gcc-mips), with no pipeline hazards and unexpected behaviors, everything is compliant with the ISA/MIPS-R3K so we (two friends and me) want to evolve it into MIPS32, adding more instructions and features. Actually we want to finalize the first stage, so there are no plans for the "upgrade", may be in the late future.

Btw I have other plans for the Octane2's PCI. I'd like to had a custom PCI board for I/O. I have already developed a deferred I/O usb driven LCD, it's EHCI High Speed (1) , It has a kernel driver and it's able to provide a text console plus X11 support. Unfortunately, due to the hw (I mean the PCB, the connecting cable, hardware troubles … and I am not so skilled) limitations I have with the bandwidth … I can't use a laptop LCD (signals on such device are 400..600Mhz and you need extremely care with the LVDS driver), so I am using a STN LCD, 800x640. It's an small 8" LCD used by the industry, it has a built-in video controller, it doesn't need LVDS driver, signals are below 100Mhz (slow refresh rated and just 4 bit of colors) so it's OK! Its driver has been developed (by my friend and I) for Atheros SoC with linux 2.6.39. I have back ported it to linux 2.6.17-RC4, the USB_LCD has been plugged to the USB bus using the NEC PCI board on my Octane2, and .. it works, but It doesn't work as expected, on Octane2 the USB bandwidth seems to be compromised, ugly reduced below 10Mbyte/sec (20Mbyte/sec with bulk-only removing the MPRO/V6 GFX) and the deferred I/O is not working at all (well, 2.6.17 is completely different from 2.6.39), so I had to reinvent the wheel, I changed the driver's code in order to force raw mode. It's now updating the whole LCD screen, so it generated a constant data traffic, from the XIO_PCI to the PCI_USB controller and the result is: still decent for text console, indeed too slow for X11.


(1) USB 2.0

• High Speed mode, maximum transfer rate of 480 Mbit/s (40x faster than USB 1.1)
• Full Speed mode, maximum transfer rate of 12 Mbit/s
• Low Speed mode, maximum transfer rate of 1.5 Mbit/s

So for reasons like these I want to develop something custom about the Octane2 PCI, and I want a damn frame buffer! Matrox Millennium, VoodooII and pretty PCI video cards like these are not working at all when plugged into the XIO_PCI.

Who has the time of course, but I have no deadlines.

Due to the fact I have no Octane2 like you can read in my signature, I am now working around a VDU (video display unit) for CGA, it will be used in the WoodyBox project to drive the CRT tube, the VDU is written in VHDL and it's a modular design and the driver is completely separated to the high level display unit, so if you want "to port" you have only to care about the driver-module, recycling the rest of the project code. In my plans I want to have it converted to VGA or (directly to LCD). So it will be cool to have it in the PCI board I want to plug in the XIO_PCI of Octane 2.

I think the most trouble will be around the PCI, customized by SGI, from the Intel specification I feel it's not a piece of cake, so I am expecting a true blood horror story with zombies, vampires werewolves, witches, black masses and a lot of black magic.

ivelegacy wrote: I think the most trouble will be around the PCI, customized by SGI, from the Intel specification I feel it's not a piece of cake, so I am expecting a true blood horror story with zombies, vampires werewolves, witches, black masses and a lot of black magic.

It's not the first you write this, and I think you are confused. The Bridge chip (and its offsprings XBridge and PIC) are PCI 2.1 compliant. It's only their IOC3 chip which is a fraud. If you don't deal with IOC3 in your PCI board experiments, you have nothing to worry about.

miod wrote: The Bridge chip (and its offsprings XBridge and PIC) are PCI 2.1 compliant. It's only their IOC3 chip which is a fraud. If you don't deal with IOC3 in your PCI board experiments, you have nothing to worry about.

The PIC ASIC is is also the only bridge I've encountered which enforces the PCI-X spec to an annoying and pedantic degree.

I had to troubleshoot a PCI-X device driver in an Onyx3900 once, which mysteriously failed to translate DMA buffer addresses to PCI bus addresses. The IRIX function for that is pciio_dmatrans_addr() and you can tell it to generate 64bit PCI bus addresses with the PCIIO_DMA_A64 flag. Which we didn't because this was a 32bit device.

Except this flag was ignored on the Onyx3900 IX-brick.

Finally an engineer at SGI pointed me to this fine print in the PCI-X Addendum to the PCI Local Bus Specification, sect. 1.8, "Bus Width":

... and therefore PCIIO_DMA_A64 was always enforced on the PIC chip, whether you wanted it or not.

In other words: please fix your non-compliant PCI-X hardware. Which we did...

miod wrote: The Bridge chip are PCI 2.1 compliant. It's only their IOC3 chip which is a fraud. If you don't deal with IOC3 in your PCI board experiments, you have nothing to worry about.

Good points, but is't a bit hard to avoid to have to deal with the IOC3, so from my point of view - " what I deal with " - it's not compliant.


miniPCI to PCI

I am also having a lot of troubles with Atheros9 SoC because they do not support PCI_IO , they are PCI_MEM only. I bought an extender board, it exports the miniPCI to PCI in where I can toy with the Mesanet FPGA . It's a bit expensive (~240 euro), I have found a cheap second hand (90 euro) on ebay dot com