Hadoop

Hadoop is a Java framework for dealing with huge data stores. Microsoft is bringing Hadoop to Windows Azure soon. Although Microsoft already has a map/reduce solution called Dryas, it is not ready for prime time production use. Hadoop will fill that gap now.

Map/reduce operates at a lower level than plain old SQL. It moves your regular business layer processing down to the data storage layer. And it does so in a parallel mode. That is how you get the good performance at a large scale.

There are other map/reduce solutions out there. Hadoopdotnet implements Hadoop for .NET. And MySpace has their Qizmt solution. However look to Microsoft for the new Hadoop port to the Microsoft cloud.