Miscellaneous Operating Systems/Hardware

I don't consider Wikipedia "newer literature" and it is certainly far from authoritative. I do consider IBM authoritative since they were there with multitasking and multiprogramming and SMP much earlier even than UNIX and two decades before Sinclar QDOS (if we believe the dates on Wikipedia). Even so, the page you cited does use the word multiprogramming as I used it and doesn't explain the difference between that and "Preemptive multitasking/time-sharing" very well, nor could it, since preemptive multitasking and time sharing are not necessarily related. These words do all have clear, well-known definitions for the last 40 years or so. Revisionism doesn't help clarify anything! The people who invent stuff generally get to say what it's called. Not always, but more often than not.

Would you consider the textbook used in MIT's Operating Systems I class a few years ago to be authoritative enough?

According to Operating System Concepts, 7th edition , multitasking and time sharing are one and the same thing [p. 16] (also, under the index entry for "multitasking" it says "see time sharing"). It defines it as an extension of multiprogramming, where switches between jobs occur frequently and not only when a job is waiting for I/O or another task, so that the users can interact with every running program. It later divides it into preemptive and cooperative multitasking (as is done on the Wikipedia page).

This is the accepted definition in use today.

35 years ago, the word "terminal" was primarily used to describe a physical machine, but now when someone says "I'll open a terminal", it means he'll start a terminal emulator program, not that he'll take apart a VT100.

I don't decide how people choose to name or rename concepts

No, but this does explain the confusion if you know the history on IBM, DEC, Bell Labs, and MIT.



This book is noted for being overly general. It's not a good reference. It's a very high level overview, very weak on actual details, and full of oversimplifications and overgeneralizations. It's not for people who actually go on to write operating systems or work on them.

Multitasking and time sharing are not one and the same thing at all. That's a broken analogy. It's like saying virtual storage and large address spaces are the same thing because most large address spaces are implemented through virtual storage.

Multitasking is a popular way timesharing systems are implemented, it's hardware and software. Timesharing is an application of technology to allow multiple users to share the same machine (hence the name timesharing). Nobody ever said that timesharing systems were designed to run one program, even with multiple tasks, contrary to what your quote seems to be suggesting by conflating the two terms. Timesharing systems let more than one person use the same box at the same time. That's all. Multitasking is the way timesharing is usually implemented, it is not timesharing itself. Multitasking is used for other things like making the best use of the processor or processors and getting more throughput, and to achieve a high level of concurrrency by scheduling the use of resources effectively. Multitasking has value independent of timesharing, timesharing is just one of the many possible benefits of multitasking. Timesharing is not multitasking. They're two totally separate things, although they are often related.



I don't have enough context. Are you saying the book defines multitasking as an extension of multiprogramming or timesharing as an extension of multiprogramming? Either way, it's the cart before the horse. Multitasking is one way multiprogramming and timesharing (again two different things) are implemented. Multiprogramming and timesharing can also be implemented through multiprocessing without multitasking. Conflating these things isn't correct or helpful in understanding anything. It's technically wrong, even if that's a popular or the most popular implementation. Reading the IBM doc helps since they invented and implemented most of this stuff originally and their hardware and software is still in use today, using the same techniques and principles as in the 1960s obviously with many enhancements etc. Like I said the guy who delivers something usually gets to call it whatever he wants. It's hard to argue on them.



That's also an oversimplification because one of the earliest, most pervasive and long-lived multitasking models is neither strictly preemptive nor cooperative but a hybrid. Doesn't the fact you're trying to bring proofs from wikipedia trouble you at all? Most of the people who actually wrote and work on this stuff don't hang out on wikipedia.



I'm sure it was in your operating systems class. But that doesn't really change anything. You have to actually know what you're talking about, you have to have been there, and written and used this stuff to know. That's how I know. Not from textbooks, not from people with an agenda, not from an MIT operating systems class, but by actually coding on those systems and seeing how they work, and reading the manuals written by the people who actually wrote the OS and designed and implemented all the technology. Not from some academic that never produced anything in his life except misinformation.



Renaming concepts is exactly what you're doing only you're trying to hang the responsibility on somebody else.

I actually agree with most of your points (and thanks for the detailed explanations)
The low-level OS stuff I've worked on was dominated by UNIX-like systems, so this has skewed my view of operating systems in general. To be frank, I didn't like that book either, is there a better one you could recommend?

The broad and general description I used was simply because that is what the vast majority of the systems used today implement, so this is what people reading this forum are most familiar with. As you say, it is not correct to lump everything together like that, but this is a public forum, not a scientific article
The Wikipedia link was never meant as a proper proof, but as an example of such usage (which is why I preceded it with "e.g.").


So now that we've cleared up the terminology and almost completely derailed the thread, perhaps we can return to the original topic of the number-of-CPUs detection script

jwp : on IRIX you can also use "sysconf NPROC_ONLN" to get the number of CPUs currently online. This way you don't have to parse the hinv output, so it's a little bit faster and will now also work on mixed-CPU Origins, where hinv reports processors as:

Thanks for your post. I'm sorry I got a little bent out of shape. Your good manners put an end to this hopefully and it now ends on a positive note because of that.



It's a good sign you weren't happy with the book. I have not seen any favorable reviews of it and when I skimmed it I found it very misleading. I do not know of a good general book, probably because to be general is to create exactly such a book that can't cover anything accurately enough or in sufficient detail. The other choices are really specific to a particular OS or family- for instance Bach or McKusick on UNIX, or Tannenbaum on Minix (but not on anything else!) etc. UNIX has portability as an overriding concern so most of the focus is on the software part of an OS. But other OS run on dedicated hardware designed for the OS (for example MVS and historically VMS etc.) and it is hard to find any information that addresses the specifics of those issues and without that you don't have both sides of the equation. Open source OS are much more approachable and lots of stuff has been written both good and bad. A lot of the proprietary stuff has virtually no 3rd party documentation or books on it so you only learn it from the manuals and (cough) reverse engineering (cough) and that was in a time when things were on a much simpler scale and they didn't hold things so close to the chest. Back then everything was in the manuals. Today things have become so complicated it's not possible for a person to even understand all of one compiler let alone a whole OS. And I'm not sure how useful it is to understand every operating system concept and implementation unless you're going to design a new OS. Most people (myself included) are happy just to find one OS they like working on and then drill down. The other stuff you just get in bits and pieces as you need it. A lot of people are learning OS concepts today by running vintage OS to see how things were done when a person could understand almost all of it.

I tend to focus on original sources and I view with suspicion anything from academia unless the guy has had some success at actually delivering something. Stroustup is a rare example. Wirth did some important work but in the end nothing he did seems to have ever been implemented well, despite the good concepts behind it. People picked up where he left off, fixed the shortcomings, and produced something usable. He himself never did AFAIK. You never go wrong by finding out the history behind something and understanding the way it was first designed and implemented. That's the best basis for building on stuff because of the "those who don't know history are condemned to repeat it" thing. I'm not talking about stuff that came out and died a week later or fringe concepts but stuff from the early days that passed the test of time and proved it was a good design and has turned out to be extensible for decades. That's what I love about what I work on. Everything from the old days still runs unchanged but today's version is more capable and smoother than ever before. OTOH I see a lot of thrashing and bad implementations today of stuff that was already done correctly and elegantly many years ago and it's frustrating to have to keep banging your head against it and that is mostly what bothered me about this discussion. Even though UNIX and Intel are the overwhelming popularity contest winners today they are not the best of breed and weren't the first in almost anything and really didn't blaze any technical trails. They did a lot to bring computing to the masses and get lots of people coding which is certainly important. But they're far from all there is. Let's give credit where credit is due. And you have to keep your eyes open for that.

It's OK
Writing makes it harder to convey the tone and English isn't my first language. I'm sure if we had this discussion in person, there would have been no friction.


I like that Maurice Bach's book, it's well written.


HP also made some pretty awesome patents for their hardware -- all information on how to build the machine was in there, including ROM contents with disassembly. I recently saved a fully working 9830A from being thrown away and am slowly going through its patent application to see how it works internally. It's similar to the early HP minicomputers, except it has a much slower CPU, simpler I/O, and a BASIC interpreter in its ROM (this dialect of BASIC is actually much nicer that what was available on home computers, it's more similar to Fortran and can do matrix multiplication etc.).


I like doing that, but some hardware is very hard to find nowadays or it's too large to ship. Emulators exist, but not for every system, so the less popular ones slowly fade away, which is sad.

Output from my Intel Core i5 based Hackintosh running OSX 10.8.4:

Just as an aside here on the subject, I put gr_osview on the desktop for the past few days. Irix does not handle multiple cpu's as well as OS/2. If you watch, cpu0 is consistently harder-worked than cpu1. I'd bet the case is even worse with more processors.

In OS/2 no cpu shows a tendency to work harder than the others. They tend to be equally loaded.

I don't remember what BeOS did, but don't remember a bias like Irix shows.

As Bluecode has pointed out, Unix does not do this well. I find it hard to believe that yet another horseshit gnu "library" applied to all kinds of situations where it doesn't apply is going to improve the situation. Those idiots can't even figure out autoconf.

Thank you, ShadeOfBlue, I have updated the script accordingly. I trust that this sysconf approach will work for older systems as well? I would guess so, since it's something from the System V heritage...


Ah, interesting. Would you be able to post output of the following script?

Yep



Forgot to say that the '4' on the second line did also appear on my previous output - I just didn't include it for some reason!

Your English is great, better than I could speak your language I'm sure! Yeah a pitcher of beer always seems to smooth things out. If that doesn't work there's always the waitresses who bring it



Yes, I wish I had certain pieces of vintage (and current!) hardware but if you don't live in America a lot of that is real hard to come by. Emulators have done such an amazing job of letting people try stuff they never had a chance to work on, but there's nothing like real hardware.

You named an Intel Mac octane ? Heresy, sir, heresy!

Oops.. good spot. I've used that as the hostname for my primary system for years, mostly because of my love for SGI boxes and the names they used.. I never thought about how that might go down amongst actual SGI enthusiasts though. I always name my secondary system (e.g. a laptop) 'FUEL' - and yet I own neither an Octane nor a Fuel. Go figure. Maybe when I finally get my O2 installed, I'll call it 'Dell' or something just to keep the convention going

Ah, great, just a copy-and-paste issue.

It seems that the heuristic approach has proved fruitful. I just tried the script in Minix 3, and it worked perfectly (another "/proc/cpuinfo" instance).

From my end, I've tested it on Linux, DragonFly BSD, FreeBSD, NetBSD, OpenBSD, Illumian, OpenIndiana, and a few others.

Adding RISC iX to your list: check for "RISC iX" or "RISCiX" in the output of uname, and return one core (which would be an ARM2 or ARM3), should do the trick - but would be pretty pointless!

That said, there are now some dual/quad core ARMs around (but I expect your Linux method would apply there), while the mid-90's RiscPC Hydra system could have five ARM610s or ARM710s on a daughterboard (and run NetBSD).

How complete would you like this list to be?!?

It should, but I'm not 100% sure, since SGI yanked the manpages for anything older than 6.5 from Techpubs

Wow, I didn't know about RISC iX before -- it's very interesting to see an mwm look and feel on a green screen! Fascinating how many variants Unix has spawned over the decades...

http://en.wikipedia.org/wiki/RISC_iX

As for how complete the list should be, I'm mostly interested in the Unix and Unix-like systems that have seen some degree of widespread use in the last 10-15 years. I think we currently have the bases covered with the big ones (but someone correct me if I'm wrong).

Finally had a minute ...

Been thinking about this a while and came to the conclusion that you have been misled, no, it is not different.

You're looking at this from the wrong point of view. I am only speaking of the situation as it pertains to workstations .

In the case of a workstation, the top priority is user responsiveness. . Not ultimate performance. Not least power consumption, not the best benchmarks on a single type of task; the overriding primary requirement is responsiveness to user input. I guess this point is somewhat lost on people who have only used Unix or Windows or OS X. But if you had ever used a system that truly prioritized the user, you would never be happy with the rest of this crap again.

Irix is not very good at this. I am quite disappointed in the performance of a dual processor O350. If you keep gr_osview open on the desktop you will see that one p is almost always more heavily loaded than the other. This is poor quality software. OS/2 does not do that. BeOS does not do that.

In practice, it does make a difference. With the Fuel, I would get situations when (for example) a swiped bit of text would not middle mouse down into nedit instantly. The processor was busy with something else. I was looking forward to going two-up ... but what do I find ? 2 p does the same thing. Irix is not well-designed (from a workstation point of view). This should never ever ever happen. I am god here and the damned computer better understand that.

Your example was also not so great for the following reason - a computer cannot spawn thirty threads simultaneously. If it spawns thirty threads, they will be spaced out at some time interval - let's say 3 milliseconds just for fun. During those three milliseconds a lot of other events will be occuring as well. And then the next thread, another three milliseconds. The rendering program does not own the entire computer, ever, and you can't spawn multiple threads simultaneously ever, either. So your example doesn't really hold true in practice.

But more to the point, it ignores the prime consideration for a workstation : PAY ATTENTION TO THE USER. If I am writing an email while some app is rendering a 3840x2400 graphic in the backround, I really don't give a shit if it takes .25 seconds longer or even 25 seconds longer. What I care about is that the edits in my email do not get hung up even for 1/10th of a second. What I am currently doing is what counts, not the stupid background render job.

This is an area where Unix, Linux, and Windows are not very good.

Here's another odd thing - with OS/2, for desktop use more than 2 processors makes no difference. I tried 4, it made no difference. One person could not load down four processors. (Except that a quad PPro Netfinity takes up a *lot* of space !) I was surprised at how zoomy a quad Origin 500 was. This is pure conjecture but it might be that the scheduler in Irix is so crappy that having an extra couple processors overcomes some of this complaint. As it is, I am not impressed in any way with a 2p O350. For a workstation , the system does not handle threading or processes well at all.

Back to the subject of this thread, the entire philosophy expressed by it is just wrong . This thing is a tricycle - putting a fatter tire on the front isn't going to solve the problem. Go fast around a corner and the thing is still going to fall over.

I like the concept of open source. I've bought software for (to me) big money, only to have the lovely vendor decide three months later that I needed to "migrate" and that the flaws in what I had bought were never going to be fixed. It's an odd thing but the money I handed over was never flawed So I have a soft spot for the concept of Open Source.

And Linux was once upon a time a cool idea with some talented people and good ideas. Back in the beginning they stressed all the things I admire. Remember "standards" ? Now it's "well, it works in gcc." Thanks. Once I made you rich enough, rich enough to forget my name. . List the dependencies ? Right again me boyo, "just apt-get the yummy debian !"

Fucking idiots.

I know there are still good people doing work in Linux or other open source projects. I have talked to a few who were happy to hear about problems (okay, not exactly happy but responsive and cooperative.) The quality of a few projects is great, really good. But too much of it is rancid vomit.

Gtk was okay. Gtk2 is not. Even when you get it to work, at 2.13.3 the entire file system handling goes to crap. It disappears and doesn't work. "Oh that's deprecated, man" ... go fuck yourself, you little shithead. The code is garbage, untested, stinking worthless shit. "We don't have a Solaris /SGI /AIX box to test with, man ..." Then go buy one, you little twats. Put up fifty bucks and actually test your goddamned worthless software. The average greaser motorcycle mechanic has twenty thousand dollars' worth of tools, these "hi tech", highly skilled brilliant degreed individuals can't bother to buy a few tools ? Many of these people are worthless little snotnoses who couldn't find their ass with both hands in a dark room. Pixman 0.30's configure told me "your system has broken thread support." Yeah, right. Irix was 64-bit running on 128 processors when the twit who wrote that configure script was in diapers but MIPSPro has "broken thread support." Then gtk3 ... let's not even go there, I don't want to barf up my lunch.

What I have come across playing with the stinking gtk series is that the newer it gets, the worse it gets. It seems that people understand less and less of the basic requirements for software - hell, less and less about anything , as time goes by. Basically, the more "modern" it is, the shittier. So when I hear that squeal about "modern, current, uptodate" or even worse, "deprecated" I want to scream. Modern is synonymous with pigshit. At least Motif works .

Shall we bring in Fireflop for an example ? My favorite turd ? (I have dozens of screenshots of failed fireflop, btw, so this is not an anomoly.)


What's this ? A jpeg .. a stinking simple-ass jpeg ... that Fireflop couldn't display. There is simply no excuse for this, ever. Those dickcheeses have all the time in the world to play with their awesumbars and their non-standard msql and change toolkits and their el-coolo urlbar color changing and the stupid little text popups "Did you want to go to a website ?' No thanks, I want you to go wash my socks. (And if that worthless goddamned Open Orifice asks me one more time "Are you sure you want to save this in a Microsoft format ?" I'm going to kill someone. The only reason that piece of shit is allowed within 100 yards of my computer is to "save this file in a Microsoft format."

But they don't have the time and manpower to make the damned thing actually work.

It's a real temptation to say, "Screw Open Source. The quality is garbage."

Which brings me back to this project ... I am all for people writing software. Really. Even if it's something I could care less about, it's still good. Without software computers are not useful. And this is kind of a fun little utility, it could be nifty to know hw many cores are in a box or how many hyper-threaded pseudo-processors or whatever. But as a means to make multi-threaded software work better, it's tits on a boar. Developing more poorly-thought-out, badly coded untested crappy gnu libraries is the wrong approach to getting the most out of multiple processors. The programs have to be written with the task in mind, not just have some cool modern 'parallel library' tacked on to a junky badly-structured application. Junky single-threaded, junky multi-threaded, who cares ? Junk is junk.

Hi Hamei, I'm not saying that knowing the number of processors on a system is useful for the average application. I'm saying that it's one piece of information that may be necessary if someone is writing a special type of library or special application that uses multiprocessing. It's not for the typical program, but rather for a very CPU-intensive task that takes a long time to complete. Most people don't have these types of problems. Their programs take a second to complete, and are typically I/O bound, or network-bound, or something like that.

The reason for using all available processors would not be because the program would complete in 1 second instead of 2 seconds -- the idea is that it's something very significant, for which one processor is insufficient to get the job done in a reasonable amount of time. For example, a job that might take 90 minutes with one process can potentially complete in 30 minutes with four processes. Of course, not every concurrency problem can be resolved with simple approaches like this, but for some it's an eloquent solution. It's important to use the right tool for the job, and to know when it's necessary and not necessary to do optimization at this level.

Without using multiprocessing, the machine would be idle, and my time would be wasted (to me, that's the truly bad user experience -- waiting around for an extra hour). As it is, even with the program pushing the CPU 4 ways to 100%, the system is still responsive. If there were a problem, I would just "nice" the program to a low scheduler priority. I've found, though, that the Linux scheduler seems to handle everything fine, and the only difference I see in my user experience is just the noise of the fans when they kick in. Basically, the processors are there to be used, not sit around idle, and the kernel's scheduler should decide how to divide CPU time between the processes.

In the case of IRIX, it sounds like the scheduler isn't being very fair about how much CPU time each process gets. In that case, the process scheduling algorithm is basically the problem. In the Linux 2.6 era, there was a scheduler rewrite to fix similar clashes between background processes and user applications: http://en.wikipedia.org/wiki/Completely_Fair_Scheduler . As an anecdote, it appears there was a big stink about it from some guy at Red Hat, to which Linus replied, "Numbers talk, bullshit walks. The numbers have been quoted. The clear interactive behavior has been seen."


As for the relative strengths and weaknesses of open source, there are good things and bad things. Certainly some development efforts like the Linux kernel, Perl, Python, Ruby, MySQL, Apache, etc. have been very successful. Some of it was famous and widely used before Linux was even around, like Perl, Emacs, the GNU toolchain, BSD Unix, etc. There's also a lot of crap code out there that is unprofessional, poorly conceived, and poorly tested. Taken as a whole, it's a big mixed bag, but that's the nature of having thousands of open development projects.

I do think it's much healthier than the era of 1990s commercial workstations, though, which were enjoyed by the few and privileged (while the vast majority were stuck on toy operating systems like DOS and Windows). For me, the earlier era of timesharing and community development at Bell Labs and UCB is like an ideal, and I think that open source projects are closer to that ideal than the approaches taken later at HP, Sun, IBM, etc.

I think we're mixing up the current situation on UNIX systems and elsewhere .

What I said before was that on a UNIX system , if a CPU-bound program knows how many CPUs the system has, it can run better than if it just spawns hundreds of threads willy-nilly. This is a measurable fact, not speculation. GUI programs are IO-bound (waiting for user input), so they won't benefit from knowing the number of CPUs.

The script made by jwp is for UNIX systems, so it makes sense to talk about those in this thread

I'll now address a few of the points you made...


1. Responsiveness while running CPU-bound background jobs
The responsiveness issues you describe can be fixed by running the CPU-intensive stuff at a lower priority (higher nice(1) value). If you run the rendering job at a nice value of 20, then every UI app will get priority over the rendering job and there shouldn't be any sluggishness. You could also try boosting the priority of UI apps (lowering their "nice" value with renice(1) ).

However, you must set the nice value manually, since the kernel has no clue which program does what. IIRC, Maya runs the rendering processes at a lower priority, so the user can still work normally on the rest of the scene at the same time. And this actually works, people use it this way.

If you're compiling something in the background and you don't use "nice" to start those jobs at a much lower priority, they will run at the same priority as nedit and firefox and the rest of the stuff you've started. As far as the kernel is concerned, these tasks are equally important, so it won't favour nedit and firefox.



2. CPU-bound programs spawning as many threads as they can
Now if a renderer starts 2160 threads (one for each line in a typical 4K image), it will burden the system needlessly, especially if you run it at the same priority as your GUI apps. It doesn't matter if the threads are created a few milliseconds apart, that's not the issue here (thread creation is cheap). Context switches are expensive, it simply makes no sense to spawn that many threads, because then the system will spend too much time switching between them (the threads will fight each other for CPU time!) instead of doing work, it is inefficient (I've explained this in more detail in a previous post). This is not speculation, I know this because I've written parallel renderers before and I measured that spawning threads for every line in the output image was much slower and used more memory. This is why no proper renderer does that. Look at Maya or any other good 3D suite, every single one will check how many cores the system has. There is much less overhead if the program spawns only as many threads as there are CPUs and then distributes work between them. Every scientific app using OpenMP works like this. The OpenMP library checks the number of cores available and then spawns the appropriate number of worker threads.

The only scenario where spawning all the jobs at once would make sense is if the kernel provided a batch job scheduler. So you could simply put your tasks into a global queue and the kernel would work on one or more tasks from that queue (depending on number of CPUs) whenever the system is idle.

In a time-sharing OS that doesn't provide this functionality, spawning lots of CPU-bound threads will result in lower overall performance (the threads will fight each other) and consume more resources (you need memory for every thread's stack at the very least) than simply checking how many CPUs are available and using an appropriate number of threads.



3. Using an Origin as a desktop machine & behaviour of IRIX's scheduler
Origins are not desktop machines. The behaviour you've seen (one CPU more loaded than the other) is normal. This is because the scheduler also takes cache locality into account, along with a lot of other metrics (on NUMA systems it is also aware of the system's topology which also influences the decisions it makes). Having one CPU loaded more that the others is not an indication of a crappy scheduler.

If you have lots of short-lived tasks, it makes no sense to spread them out to other CPUs and thrash their caches. IRIX will actually distribute threads to other CPUs once it figures out they will run for more than a few seconds (or if the main CPU is very busy). Also, if you re-run a single program many times, IRIX will try to run it on the same CPU, because that CPU already has the data that program needs in its cache, which is why it would be inefficient to run it on a different CPU. If the kernel didn't care about cache locality and just tried to keep every CPU busy, it would be similar to one of those parts boxes with multiple drawers and having one of each screw type in every single drawer, because it's a waste to keep drawers empty. Finding a specific screw would be very slow

What you want is something like BeOS, a single-user desktop OS. IRIX was made to run on everything from workstations with a single CPU and one user to servers with 1024 CPUs and thousands of users; it's difficult to have optimal performance in every scenario and for every workload. Adding a batch job scheduler to the kernel would help a bit with background jobs such as rendering and compiling (and then those programs wouldn't need to know how many CPUs the system has). But even as it is now, if you lower the priority of such tasks, the system should remain quite responsive. I'm not saying it's perfect, it could use improvements, but it's not a complete disaster




I hope it's clearer now why checking the number of available CPUs is a perfectly normal thing for a CPU-bound program to do on a UNIX system