The collected works of mia - Page 3

Not bad, my turn.



4xIP53@1Ghz, xvm raid 0+1, 4xST9300653SS.

mapesdhs wrote: What controller setup was this using?

mapesdhs wrote: And in which slots?

Let's take this thing up a notch, with raid0, same hardware configuration:

what model cards are they? 3442X-R? 3800X?

3800 rebranded, but probably identical to the LSI one.

Any tips on how you setup the RAID in XVM?

I'm going to try cxfs this weekend.

For completeness, I have run an identical diskperf benchmark on a single drive, to understand, perhaps to some extend, the xvm "overhead", those results are shared below:

Ian,

Impressive results.

I currently use those 4xSAS drives for backups (where streaming matters) with Amanda on IP35.

I'm satisfied with those results; although I might also include some large SATA drives for long term backups; while the SAS will be for the daily backup tasks, which are mostly incremental per nature.

I use this same startech in my tezro, in place of the dvd; with those 4xSAS I benchmarked above.
Cool setup; but I could use the fuel to get 8 like that.

mapesdhs wrote: True, the Fuel could hold 8. I don't like the idea though that they'd always be powered on, thus the external case.

If you want to have data "always" available to your Irix box (regardless what constitutes your dataset: movies, isos, documents, pictures, archives etc.) I suggest that you give DMF (Data Migration Facility) a try (documentation id #007-3681-019). It would allow you to use local or remote destinations to store files, which would be migrated on demand, in a transparent manner.

In a nutshell, under a xfs filesystem with dmf attributes:

mplayer ian.avi

The file ian.avi, if not locally present, will be automagically retrieved from either tape/ftp/nfs/xfs/cxfs/cifs... and, depending on your policy, sent back (if modified) to the remote location (which can be a windows, osx, openbsd or linux server if you want, it doesn't really matter); or even to multiple locations (great for online data mirroring). To you, it will be entirely transparent, if your network is slow, a disk cache (on a separate filesystem/directory) can be added to the mix. While this is not really a backup solution, it's a convenient way to unify "archipelagos of storage", in a transparent manner for the user, and requiring no administration overhead.

Policies can be set based on uid, file size, disk space left, etc. it's very versatile. It's meant to handle a lot of data, think NASA-large datasets, where you can't obviously store everything on a single disk, or even a few of these; if the data is modified, it will be pushed back to the remote store(S) (singular or plural) based on a weight factor (assuming you have rights to modify the data). Those "remote store(S)" don't have to run irix (but they can), so if you need cheap storage readily available on irix without the disadvantages of NFS, you should look into this.

It's relatively trivial to setup DMF to automatically fetch a file from tape (movie?) when one request access (mplayer ian.avi) to this file, the end-user will not even know the data is coming from tape (unless you use a DDS3 perhaps, heh).

At least, it's worth investigating; I intend to deploy it for data rarely accessed, which I don't want to keep on NFS; simply because NFS, despite its advantages in unix-land, doesn't offer a great way to replicate the data amongst remote stores (unless if you use EMC/Netapp/Hitachi asynchronous mirroring).

To store data (remotely) DMF supports: NFS, FTP, CXFS, XFS, SMB/CIFS; it's a transparent filesystem migrator. Data can be pushed (and pulled) to offline media as well, assuming it's used conjointly with openvault or tape management facility; but most commonly, the secondary storage can be any bulk-storage device accessible through NFS or FTP. Data is verified (checksumed) on the fly.

So rather than spending a lot of money to store a lot of data on very fast storage, you can automatically tier it, data will be automatically migrated between fast and slower drives (or stores) in a transparent way.

To sum things up, in your case, you could have a primary storage on SSD, secondary storage on SATA or SAS, to the end user, it's a single filesystem with absolutely no way to know it's automatically tiered. Cool no?

Ian,

Understandable; of course. I like to throw those things to give people new ideas or different viewpoints.
It's cool however that DMF is available for Irix, it used to be mostly on big Unicos systems (Cray C90 & J90 and such) with some relatively big tape transports (thousands of slots in silos). In other words, it's good to see software that was available on systems worth tens of millions of dollars come to "affordable" workstations.

Even for "standalone" systems, it makes sense to tier between SSD/SAS and SATA (no network connection necessary, all XFS); and it's well supported by xfsdump/xfsrestore.

In any case, I'm still curious to hear what you're doing for backups these days, I imagine one cannot talk storage without mentioning backup strategies.

mapesdhs wrote: For the Fuel, the Startech unit is basically it. The only other often-used SGI I have atm is my gateway O2, and that just has a 2nd disk for cloning every now & then (very little changes on it, just running ipfilter). All my other 'for keeps' SGIs are not
really setup properly yet (original Effect O2 with all manuals, general R7K/600 O2, quad-1gig Tezro, Octane2).

You seem to prefer using the Fuel as your primary workstation, while you own a 4x1Ghz-IP53, is it just a color preference?

Makes sense. I think we have a pretty similar setup; I use the O2 as a firewall, and shell server (~50W is well worth it); and IP27/30/53 for compute intensive tasks (compilation and optimization); where a lot of ram and IO helps.

I had one of those, but I ended up tossing it away, it was just crippling my system, so I suspect their proprietary drivers are not worth a dime; I think I still have a 4x100 one (which didn't work in IP32).
In the US, unlike in Europe and Asia, Internet is still pretty slow (for the masses), all things are relative, it's still much faster than 10 years ago.

http://www.netindex.com/download/allcountries/

Does your site goes through the O2, or is it hosted somewhere else? just curious.

Ian,

I'm sorry, this is a dumb question, but have you tried to disable ipfilter on your O2 when using the gigE card? By disabling I mean unloading the kernel module all together.

Another one for fun: (this one is work in progress)

I've replicated FC jedi's Chris Kalisiak's configuration from http://www.futuretech.blinkenlights.nl/fc.html (Ian's website):

Took a "relatively large Origin server", a netapp 14-disks (Seagate 300GB FC) lun array, and got those laughable results:



At the time of the tests the load was roughly ~40% on the netapp; so I'm trying to understand where is the bottleneck, plausible causes:

- qla2342 driver?
- single channel (2Gbps) FC, not appropriate? (This netapp, while using a 2Gbps GBic is pushing only 1Gbps at most). I should trunk more ports.
- gremlins

The array is created and managed by the netapp head; then exposed to the host/switch using either the fibre channel protocol or iscsi.

Netapp encapsulate lun blocks within those 4k WAFL (netapp's filesystem) blocks. WAFL itself is stripped between all the drives (with redundancy bits, here raid-DP). WAFL is the underlying filesystem of iscsi, nfs, cifs, fcp on netapp. Yes, this is not a typo, a block device is chunked on a filesystem.

There are advantages and (performance) drawbacks of this method. But this overhead probably explains why I can't reach 1Gbps line rate; because the cost of IO operations (including parity checks). Add to this that WAFL is ironed on disk blocks, which are checksumed (in the case of those FC drives, the disk blocks are 524 bytes, where 8 bytes are reserved for checksums, and 512 bytes are available.

So there's a lot of overhead, but also a lot of features (mirroring to another array, symmetrically or asymmetrically, snapshots etc.) it's a tradeoff of features vs. throughput; as it's often the case. Finally, this specific array doesn't have a lot of cache available for read-ahead, write-back, etc. But I do appreciate its features. Regardless, I think this is a good test anyway. Please note that this disk shelf is 10 years old; but again, so is this Origin.

JJ,

> This is NetApp a SAN filer and not a JBOD or an FC RAID shelf, right?

Yup.

> I have an SGI TP9100 as well. It's loaded with a grab bag of old disks, some even 1Gb/s models (Cheetah 73LP). But with some striping I got this: [...]

Neat! XVM stripe I assume?

> My experience with the QL2342 is quite good. It will deliver the expected performance. My experience with the LS driver (SAS/SATA, U320 SCSI and 4Gb FC) is more spotty: [...]

I couldn't agree more; there's something fishy with the LS driver; I found that under serious loads (500MB/s and more) the LS driver seems to drop data, likely a bug, either in the firmware or driver; I've had XFS corruptions with very fast 15k drives, regardless if I enable/disable read+write cache, CFQ, etc. Of course, maybe I did hit a XFS bug, that's a possibility as well; or simply some interaction between XFS and LS doesn't quite work right. Amazingly, I've looked into this, and those same errors are being seen with the linux implementation of XFS (and I'm referring to "current" (10-2012) issues; no patch yet, that said, only one person currently supports XFS, so I'm not sure he has all the time necessary to look into this.

seriously, no; you'd kill yourself.

My Altix 350 rack is in the basement, a direct shot from the street (through the garage), and it took 3 people to move it. It was as empty and dismantled as could be. Next time I'll just build the house around it.