Miscellaneous Operating Systems/Hardware

The first head-node in Cerberus, 24 June 2009
It's still running kernel 2.6.15 on a single pATA

Kumba wrote: Still need to get you off of 2.6.x entirely. I can only wonder about the security flaws that cumulatively exist in such an old release.

oh well, let me say a few words about:

Cerberus (click on the link in my signature to see it) is a NAS system, build on PowerPC technology, composed by 4 nodes, and loaded with different fs services (FUSE/SSHFS, NFS, SAMBA, …), and repo services (GIT, SVN, MERCURIAL, ...), ables to performs soft-mirroring (softRAID). A pretty old Kernels 2.6.15 and 2.6.24 are running on such a system since the beginning with no troubles in years of continuously working. Cerberus was born in single node configuration, then I added more nodes, all my source code archives are stored by that multi-heads-monster & in all of his heads I trust

All my routers (RS/PRO, TL703) are 2.6.* are 2.6.39.4 and no troubles with them, even if recent routers they are migrating to 3.3.* (e.g. in TL703 the RF wifi module needs a kernel workaround to fix an hardware bug, and this "patch" is available only for kernel 3.*, so I'd best move up)

Acme System is still supporting NetusG20 (arm5/embedded) with 2.6.39.4, no modern kernel are available from them without troubles, so, for the production I am using 2.6.39.4.

My PowerPC laptop is 2.6.26, my PDA C1K is 2.6.23 and it's the last kernel available really usable, so I have back ported an USB ethernet module from 2.6.26 (which sucks greatly about the battery charge and the suspend feature, which are the most important things in a portable device)

In short, everything I have developed for hobby & job is based on 2.6, this implies that my kernel modules (e.g. USB_LCD, used in shipbuilding, job purpose) are based on 2.6, and in order to "port them" to kernel 4.* I just need effort: time, a lot of time, in first place!

I am planning to attach my USB_LCD to ip30 through a NEC USB/OHCI chip installed inside the ShoeHorn, it's currently working fine with 2.6.17.4, but it's time to move , no doubt about that, and no doubt kernels 4.* are really supporting Btrfs, the new copy on write CoW filesystem for Linux aimed at implementing advanced features while focusing on fault tolerance, repair and easy administration.

It's time to move from ext3, I want the Btrfs filesystem

however, consider that moving kernel modules from 2.6 to 4.* requires a lot of effort in first place, and consider that I have other pending tasks in my queues (e.g. VHDL, soft core & TAP, planned to be completed the sooner )


OK, let's get rock & thank you for your help

(start--stop - who - where - what - result)

so, gcc took longer to compile

@kumba



is there any problem if the kernel image is 22Mbyte (kernel + ramrootfs) ?
just to know, I currently also have a smaller 9Mbyte file (kernel + uclibc ramrootfs)
is your testing machine a dual R12K ? or R14K ?

I have

• forced R10K (with R12K, R14K, R16K it hangs up)
• force early serial
• removed gfx kprintf (we do not have a gfx installed, so it's no sense)
• attached a compressed ramrootfs

&& it boots !!!!!!!!!!!!!!!!!



hApPyhApPy
Celebration








p.s.
the SMP is not enabled (cat /proc/cpuinfo says 1 CPU), the Menet has 2 NICs instead of 4

I have forced cpu_type=R10000 && Multi-Processing support && Preemption Model=No Forced Preemption
and … something bad is happening now

ivelegacy wrote:

Code: Select all

Instruction bus error, epc == 0000000000443b3c, ra == 0000000000444b88
/usr/bin/myNET-web-machine-info: line 51:   246 Bus error               myNET-networktraffic $netdev >>$page_name2


How old is your initramfs? The problems I ran into on the Octane were Data Bus Errors (DBE), not IBEs, and those were solved once I learned to use the correct handle_*_irq() function. I'm wondering if you've got that initramfs built using really old toolchains/kernel headers and it's just not playing nice.

Kumba wrote: How old is your initramfs?

built in 2009
multi arch[]={ ppc/G3, mips3/be, x86/i586 }
used everywhere with my routers, Cerberus, etc

no reason to update, since now

Kumba wrote: The problems I ran into on the Octane were Data Bus Errors (DBE), not IBEs

it works with SMP=disabled



it does not work properly with SMP=enabled

Kumba wrote: and those were solved once I learned to use the correct handle_*_irq() function

isn't kernel affair ? is the users pace also involved ?
can you tell me more ?

Kumba wrote: I'm wondering if you've got that initramfs built using really old toolchains/kernel headers and it's just not playing nice

she has



I used gcc- v4.1.2, uclibc profiled (cross compiler, mips3-be-glibc ---> mips3-be-uclibc)
and I can reuse my OpenWrt builder (kernel v4.2.*, gcc v4.6.*)

do you have a "demo" version ?
do you have a good ram-rootfs ?

is there a good & updated version of gentoo-stage3-mips3-be-uclibc ?
I haven't checked yet



my ram-rootfs is included within the kernel

ivelegacy wrote: built in 2008

Okay, about as old as the last one I built (2007).

ivelegacy wrote: it works with SMP=disabled

I figured as much. Although I have not had any severe problems with SMP once I fixed the IRQ setup, I can't rule out that all SMP-related problems have been solved, though.

ivelegacy wrote:

Kumba wrote: and those were solved once I learned to use the correct handle_*_irq() function

isn't kernel matter?

Not following what you're asking here?

ivelegacy wrote:

Code: Select all

kika # uname -r
2.6.39-flesh-eating-bats


she has

Code: Select all

kika # gcc-config -l
[1] mips-unknown-linux-gnu-4.1.2 *
[2] mips-unknown-linux-gnu-4.5.2
[3] mips-unknown-linux-gnu-4.6.3


I used gcc- v4.1.2, uclibc profiled (cross compiler, mips3-be-glibc ---> mips3-be-uclibc)
and I can reuse my OpenWrt builder (kernel v4.2.*, gcc v4.6.*)

do you have a "demo" version ?
do you have a good ram-rootfs ?

I don't have a current initramfs at the moment. This is the thing that has been irritating me to no end over the weekend, is trying to get a uclibc-based mips3 chroot working so that I can create a new initramfs, but there's something jacked up with this uclibc root and the ncurses library. If I recompile ncurses, anything linked to libncurses.so.5.9 starts segfaulting, and I am stumped as to why. I am trying one last nutty idea before I call it quits and just focus instead on making a glibc-based initramfs.

Although, using OpenWRT to kickstart a new uclibc root is an interesting idea. I tried this with buildroot a while back, but they sadly chose to drop support for the old MIPS ISAs prior to mips32r1/mips64r1. When I asked how to undo that, the answer basically involved inserting lots of #ifdef macros in their code, and they didn't seem inclined to specify where. So I abandoned that idea entirely.

As for a demo kernel, I haven't had time to spin you one yet, since I've got the Octane tied up trying to get this uclibc idea working.

My thinking at the moment is, you've just got an incompatibility between the really old code of your initramfs, likely built against 2.6.x kernel headers or such, and the 4.3.x nature of the current kernel. If you were getting the same problems I was, you'd be seeing Data Bus Errors, not Instruction Bus Errors. An IBE means that the CPU attempted to decode a CPU opcode that it doesn't understand, so do make sure your binaries are compiled for MIPS-III or MIPS-IV at a minimum, and that no mips32rX/mips64rX stuff snuck in by accident.

Kumba wrote: Not following what you're asking here?

I haven't understood if "handle_*_irq() function" is related only to the kernel
or if it has a relation with the userspace, and in case where/what?

is it related to "something" in libc? some kernel-helper?

Kumba wrote: make sure your binaries are compiled for MIPS-III or MIPS-IV at a minimum, and that no mips32rX/mips64rX stuff snuck in by accident.

they are mips3

ivelegacy wrote: I haven't understood if "handle_*_irq() function" is related only to the kernel
or if it has a relation with the userspace, and in case where/what?

is it related to "something" in libc? some kernel-helper?

Oh, sorry, it's a kernel function. In the IP30's IRQ initialization code, I was assigning the "handle_level_irq" to all IRQ numbers (HEART has 64 interrupt vectors available). handle_level_irq() isn't percpu-aware, and so it takes a longer code path, which adds latency, when handling a specific interrupt. This is fine for normal IRQs, but not SMP IPI's. So a quick-rejig to assign "handle_percpu_irq" for SMP IPI's was the final piece of the SMP puzzle, and that's how I got my Octane to boot in SMP mode and actually work well.

ivelegacy wrote: they are mips3

So in theory, it should work fine. What PAGE_SIZE is set in your kernel? 4KB should be universal and work with everything. If I get this uclibc-based netboot working, though, I'll find out if there are PAGE_SIZE incompatibilities there, as there were in the musl-based netboot I built. I run 64KB PAGE_SIZE on the Octane right now, as it helps speed compiles up by reducing TLB pressure. musl couldn't handle that well, though.

Kumba wrote: My thinking at the moment is, you've just got an incompatibility between the really old code of your initramfs, likely built against 2.6.x kernel headers or such, and the 4.3.x nature of the current kernel.

I can say "no!". It randomly works, and randomly fails, which sounds related to the kernel (more specifically to the SMP, with no SMP enabled I have no issue, it never fails), perhaps something wrong or missing with the SMP initialization.

Kumba wrote: I run 64KB PAGE_SIZE on the Octane right now, as it helps speed compiles up by reducing TLB pressure. musl couldn't handle that well, though

I am on 4Kbyte PAGE_SIZE, with 64KB it crashes soon



Currently, where is your rootfs ? Attached to an external scsi bus ? NFS imported ?

busybox has been compiled as "MIPS-I", can cause problems on a MIPS-IV machine ?
R12K claims to be



I am going to recompile it as MIPS-III


I have also modified my rootfs to work with an hard drive instead of running within ramfs

ivelegacy wrote: busybox has been compiled as "MIPS-I", can cause problems on a MIPS-IV machine ?

IIRC, MIPS-II is MIPS-I plus a handful of new instructions plus pipeline interlocks to eliminate some of the hazards. MIPS-IV is superscalar with speculative execution and whatnot, so the pipeline hazards are even less of an issue. On the other hand, it might still be an issue for some reason or another.

nyef wrote: MIPS-II is MIPS-I plus a handful of new instructions plus pipeline interlocks to eliminate some of the hazards

I wonder if these new instructions & pipeline interlocks introduced by MIPS-II && MIPS-III && MIPS-IV to eliminate some of the hazards, are exposing a "weakness" within the spin-lock in the kernel-SMP, as I still and randomly get Instruction Bus Error on CPU1.

There is also this problem about stages: no man power to build them!

deadline respected by the end of the year:
completed 20 minutes ago
we have a working kernel v4 with a working uclib-rootfs
Yahhhhooooooo

+H~a~P~p~Y~n~E~w~Y~e~A~r+

ivelegacy wrote:

nyef wrote: MIPS-II is MIPS-I plus a handful of new instructions plus pipeline interlocks to eliminate some of the hazards

I wonder if these new instructions & pipeline interlocks introduced by MIPS-II && MIPS-III && MIPS-IV to eliminate some of the hazards, are exposing a "weakness" within the spin-lock in the kernel-SMP, as I still and randomly get Instruction Bus Error on CPU1.

There is also this problem about stages: no man power to build them!

I'm making progress on the MIPS-II uClibc netboot thing. Had to rebuild my three stages all over again after running into some oddities with a few options I had in uclibc's config. Setting up a chroot now to built the netboot by hand in, while mapping out file/libs and looking into scripting the process for the future.

That said, your problem is not tied to spinlocks or anything. If it was a spinlock issue, you'd get a proper oops message and the kernel would likely die pretty fast due to a race condition, sleeping while atomic, etc. I still think it's something really weird about your hardware...maybe there's a hardware bug that simply isn't known about and thus, not being worked around? Hard to know right now. If I can get this netboot actually built and working without any more screwy issues popping up (uClibc apparently has quite a lot of them), then you'll be one of the first to test it to see if you're still having problems. I'll update the Gentoo bug you opened with further info when I have it. Only have time on the weekends to really get stuff done.

Additionally, the R10000 family handles *all* hazards in hardware, I think the only MIPS CPU to do so. Ralf's indicated that's one of the nicest features about this class of CPU.

All that said, stay away from the 4.4.x kernel on Octane for now. Something changed with (I think) Device DMA memory setups and the contiguous memory allocator. You'll get a Oops message when pci-ip30.c tries probing something, then updates devtmpfs (via kdevtmpfsd), and hits an unaligned access bug. So that's on my list to fix now. Cannot reproduce the problem on the Onyx2 and pci-ip27.c, so it's probably Octane-specific. The other possibility is you'll get ~27+ Oops messages in a row, if you have a graphics board installed and booting with the early printk hack enabled. Have to figure that one out, too...

Kumba wrote: I'm making progress on the MIPS-II uClibc net boot thing

instead of using gentoo.Catalyst I used Aboriginal (which looks similar to O.E.), it's Qemu based and works fine!
mips-3-uclibC-be was completed by the end of the year, hardly tested, it's working with kernel 4.*

Almost forgot to add, the MIPS-I, MIPS-II, etc, are Instruction Set Architecture (ISA) references. They don't really have much to do with the actual CPU hardware, in a way. Instead, they specify features a CPU should support, such as register size (32bits, 64bits), specific opcodes (MIPS-IV added square root), etc.

The way it works in MIPS is kinda like this:

• MIPS-I (32-bit)
  • MIPS-II (32-bit)
    • MIPS32R1
      • MIPS32R2 ... MIPS32R6
    • MIPS-III (64-bit)
      • MIPS-IV (64-bit)
        • MIPS64R1
          • MIPS64R2 ... MIPS64R6

You can see how ugly it looks when visualized in a tree-like layout, but that shows the compatibility chain. In other words, MIPS32r* branches off of MIPS-II, with some 32-bit backports of 64-bit opcodes added in various MIPS32R* releases (the latest being R6; there was no R4 due to some Asian cultural fear of the number 4). MIPS64R1 was spun off of what WAS going to be MIPS-V, but since MIPS-V never made it into actual silicon (it was supposed to be instituted by the aborted R18000 CPU), it technically doesn't exist. So MIPS64R* can instead be visualized as branching off of MIPS-IV.

ivelegacy wrote:

Kumba wrote: I'm making progress on the MIPS-II uClibc net boot thing

mips-3-uclibC-be was completed them by the end of the year, it's working with kernel 4.*

I can't use your netboot for Gentoo purposes, though. Stuff we support has to be built through our release tools (which is where all of the fun is at...). Glad your build is working for you (would still be curious to know if the problem really was old rootfs + new kernel or something else). But in the end, I still have to make this MIPS-II netboot work for (theoretically) anyone that has an Octane.

Kumba wrote: would still be curious to know if the problem really was old rootfs + new kernel or something else

I do not know, but I can say that the old rootfs is working fine with MIPS32r2, successfully used with kernel 2.6.39 and v3.* on my routers, so, I suspect the kernel v4.* is bugged somewhere as I can't run the last 2015 MIPS4- glibc -stage3 made by Catalyst with PAGESIZE={ 4K, 16K }, while it works fine with PAGESIZE=64K




do you think things like the one above will work with ip30 ?
planning to interface a micro drive