Clusters and scalability

Hey,

I read this on a site a few minutes ago, and I'm not quite sure what it means:

"In September 2001 Hewlett-Packard clustered 225 PCs running the Open Source Linux operating system. The resulting system (called I-cluster) benchmarked itself right into the global top-500 of supercomputers, using nothing but unmodified, out-of-the-box hardware. (A significant number of entries in that top-500, by the way, runs Linux, and more and more Unix clusters are being used for supercomputing applications.) "

I'm not sure what clustering is. Could anyone give me a simple explanation of what they mean by clusters? I know a good deal about networking (mainly network programming), but I've never learnt about clustering (or scalability for that matter). So could someone give me a quick explanation of what clustering and scalable servers means? I would be most obliged, thanks :)

Sounds like all the computers are combined into one big computer.

You know, like Voltron.

lol, yeah, but how exactly does that work? Do they share their CPU's, with all the 225 computers and just one operating system, or do all the computers have their own OS and the jobs are just shared?

I wonder if I could turn 4 of my old 486's out in the shed into a cluster... could I get better performance from them do you think?

I dunno. With Voltron, it seemed like all the robots had their own operating systems (in this case, a pilot), but when they combined, they went to a central operator. My memory's fuzzy, since I'm no longer 6 years old, though. I can't remember if just one guy piloted or if they all did. Anyways, I know the Power Rangers all sat in the cockpit. I don't know who piloted that thing, though.

hth

I think they portrayed it as if they were all still piloting their parts. Granted, it is just a cartoon and they weren't giving a lot of attention to little details like that. Voltron was awesome. It needs a live action movie! ILM should do the effects.

edit: I heard they are doing a live action movie with Transformers. Anyone hear anything about that?

edit: edit: http://www.corporate-ir.net/ireye/ir...item_id=421174

Clustering is when you put a lot (or a few if you want) of computers in LAN and they share the tasks, using all of the processors. I think they need to have an OS designed for this, but I'm not sure.
Scalability is about the total processor power you get from all of them together. Linear Scalability is, for example, when you put two 2Ghz computers together and the total speed is 4Ghz. However, this is more of an idea, than an achievable goal since they need to comunicate, share memory and other stuff that consumes cycles.
A good scalable computer is a computer that wastes few cycles communicating. A bad scalable computer is one that wastes a lot of cycles.
Hope this is clear enough for you.

With that out of the way, does anyone have a definitive answer on my Voltron and Power Ranger questions?

with power rangers, isn't it some sort of magic ring thing? If not, they're all controlling it from the same "bridge". I would assume the controls are there.

WTF? I think you're confused. In power rangers, when they make Megazord, they're all on the bridge, but they never really control it, from the looks - they just kind thrust their fists and look dramatic. I don't know what this magic ring deal's all about - maybe you're thinking of Captain Planet?

no, it's a magic sword or something isn't it?

what do they use to summon the "ships"? isn't there some kind of ridiculous magic thing?

I thought they just kinda said like "Anuszord!" and it showed up.

Damn, I wish I was a little kid again ;)

I grew up with voltron and tranformers. This new fangled power rangers isn't for us old guys!

edit: wait, wait, wait!! I specifically remember some sort of knife that was played like a flute. That's power rangers isn't it? that's how they summoned the vehicles.

edit: ^^^^^
I thought that was just the green ranger. I'm so confused!

No kidding. I'll call my brother and ask - he's about the right age for that. He's at cross country practice right now, but I should be able to get a hold of him in a couple hours.

Didn't they use remote controls in the latter episodes/films?
Or they just went to that control room with the robot and that face suspended in the air and shouted something like "Lets metamorph!" (tranlated from spanish, and my memory's not that good either) and suddently they dressed up and the "vehicles" appeared.

>>I specifically remember some sort of knife that
>>was played like a flute. That's power rangers isn't it?
>>that's how they summoned the vehicles.
Doesn't Zelda do that? I'm very confused now:confused:

Quote:

---

Originally posted by -=SoKrA=-
Didn't they use remote controls in the latter episodes/films?
Or they just went to that control room with the robot and that face suspended in the air and shouted something like "Lets metamorph!" (tranlated from spanish, and my memory's not that good either) and suddently they dressed up and the "vehicles" appeared.

---

I'm not talking about the later series, though. I never watched those, but you may be right. Anyways, I'm about to leave for lunch - I'll call my brother and ask him.

Not for the first time, I haven't the faintest idea what you lot are talking about.

>> Not for the first time, I haven't the faintest idea what you lot are talking about.

Children's show from the early 90s. I think it's actually still on, in some form.

Alright, I got the skinny from my brother. According to him, their magic belt buckles or something lame like that turns them into Power Rangers, and when they're in the Power Ranger state, they can call their Zords just by asking for them, and usually jumping into the air. He also said the Green/White ranger was the only one that called by playing his flute of gayness.

Power rangers is the worst TV show I have ever seen.

You can use something like OpenMosix.

It load into the (Linux/Unix) kernel and lets you split programs into tasks. The best way to do it is to have a program with a lot of threads, theres also some special libraries you can use(normally c or fortran, could be anything).

Each computer is a node and you set up groups. Every group gets a switch and every node gets that many nics.

This is fun stuff to play with, a good reason to keep all those all pcs or get a cheap Lindows pc(like $200 bucks, you can build a nice node for alittle over $300).

The normally dont have monitors, video cards, disk drives(some times just a hard drive), keyboards or mice. You can have them boot off the network card(load teh os over the network, no hard drive/cdrom) if you set up a tftp server, i havent played with that(my school is starting soon, i will soon).

Its kool stuff to look into. But theres bunch of different kind of cuper computer. This is the "poor mans supercomputer".



lol, power ranges....

Download a copy of OSCAR. It is a collection of tools used to set up a Beowulf Cluster. I tried it at work a few years ago. We get better performance from our software using 10 PC (Pentiums 400 - 700 MHz, with two NIC per, ie we ran two seperate LANS, one for data passing, the other for control) than from the SGI machine that software was targetted for. The PC were old office Gateway machine, resale value of about $1000 total. The SGI was an 8 CPU Power Onyz costing $800,000.

To get the best performance, you will need to optimize your code to take advantage of multiple node. For your message passing, MPI is becoming the standard and should work on most versions of Unix/Linux. PVM is still used, but I have never tried it.

BTW: openMosix will work with RH9.0. The last time I had looked at the website, it had only been tested with RH7.3.

Thanks, I was looking at an article on intel and they said beowulf clusters are very powerful, but code must be specially written to take advantage of it. openMosix on the other hand doesn't require recompiling the code, so if I ever do try out clustering, I think thats the way I'll do it.

Wow, I can't believe 10 PC's outperformed the SGI, it must be pretty powerful stuff. Thanks for all the info guys!

Remember that SGI specializes graphics processing and it is hard to find something better. The software I was working on at that time did not take advantage of the SGI's strengths.

Ohh ok, but still, anything worth $800,000 should be able to out pace $1000 worth of old computers. I've been reading the openMosix documentation and as soon as I install linux again (in a few months) I'll hook up my other computers and use their power together.

Okay, onto another question. Lets say I have a web server, with about 100 gig of data. I want to set up a cluster of, say, 6 servers with 1.2ghz athlons.

Where would I keep the 100 gig of data that needs to be sent out across the net? On all of the servers, or just one? Or, on a seperate data storage server? But wouldn't that mean I'd have a bottle neck at the storage server? Or, would I have everything "mirrored" somehow on all the servers?

How do sites like microsoft keep all the data on their servers the same? If I had three storage servers with 100gig each, how could I make sure they all held the same files? Would it be some sort of CVS for normal files? This is interesting....

On the OpenMosix documentation, it says that if you're running a webserver or a database server, you can't take advantage of clustering (at least with OpenMosix).
I have no idea how MS keeps al their data. They probably use a super-secret server that can access data on any computer in a record time and they don't tell anyone obout;)
Seriously though, I've no idea.

For a website, the bottle neck is not in your machine, it is in the ethernet connection.

A cluster can be used for a datase.

The space should probally be in one machine shared withe the rest. Maybe a LVM is possible over ethernet, I have no idea.

The internet connection would be the problam, the slashdot effect.

>>For a website, the bottle neck is not in your machine, it is in the ethernet connection.
Not if you're using server-side scripting.

Quote:

---

Not if you're using server-side scripting.

---

Like PHP or a Database?

Yeah, a cluster would be nice there.

Now im wondering about a Etherner-LVM

Yeah thats what I was thinking, a site of mostly PHP and MySQL databases. The PHP on all computers would be easy, but keeping the data in the database the same on all the computers would be hard. Actually, how could you keep all the files for the site on the same computer? Would it be possible to use CVS? If you can't I'm going to invent a way of keeping them all mirrored :)

>>> Actually, how could you keep all the files for the site on the same computer?

You set one up a file server and mount it's disk as a shared drive. You can then put a decent RAID controller in your single file server for reliability, and backups are simpler because you only need to back up the data from 1 machine. A UPS is nice as well.

Ahh, such a simple answer, I can't believe I didn't think of it *blushes* thanks very much!

Quote:

---

>>For a website, the bottle neck is not in your machine, it is in the ethernet connection.
Not if you're using server-side scripting.

---

A server can serve out pages faster than your connection can get them out.

If you are going to have only one IP address, everything will be going through your gateway and out through one connection to the internet (ie bottleneck). If you want a mega site to works fast, then you will need multiple IP address and a seperate connection for each you. Now if we were in a perfect world every computer on the internet would be connected to each other with fiber optics, and then the server would be the bottle neck. Maybe in my lifetime, it will be a total fiber optic internet.