Thousands of hosts on a gigabit LAN, maybe not

[mfidelman at meetinghouse.net (Miles Fidelman)]

Forgot to mention - you might also want to check out Beowulf clusters - 
there's an email list at http://www.beowulf.org/ - probably some useful 
info in the list archives, maybe a good place to post your query.

Miles

Miles Fidelman wrote:
> John Levine wrote:
>> Some people I know (yes really) are building a system that will have
>> several thousand little computers in some racks.  Each of the
>> computers runs Linux and has a gigabit ethernet interface.  It occurs
>> to me that it is unlikely that I can buy an ethernet switch with
>> thousands of ports, and even if I could, would I want a Linux system
>> to have 10,000 entries or more in its ARP table.
>>
>> Most of the traffic will be from one node to another, with
>> considerably less to the outside.  Physical distance shouldn't be a
>> problem since everything's in the same room, maybe the same rack.
>>
>> What's the rule of thumb for number of hosts per switch, cascaded
>> switches vs. routers, and whatever else one needs to design a dense
>> network like this?  TIA
>>
>>
>
> It's become fairly commonplace to build supercomputers out of clusters 
> of 100s, or 1000s of commodity PCs, see, for example:
> www.rocksclusters.org
> http://www.rocksclusters.org/presentations/tutorial/tutorial-1.pdf
> or
> http://www.dodlive.mil/files/2010/12/CondorSupercomputerbrochure_101117_kb-3.pdf 
> (a cluster of 1760 playstations at AFRL Rome Labs)
>
> Interestingly, all the documentation I can find is heavy on the 
> software layers used to cluster resources - but there's little about 
> hardware configuration other than pretty pictures of racks with lots 
> of CPUs and lots of wires.
>
> If the people you know are trying to do something similar - it might 
> be worth some nosing around the Rocks community, or some phone calls.  
> I expect that interconnect architecture and latency might be a bit of 
> an issue for this sort of application.
>
> Miles Fidelman
>
>
>
>


-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra