S A R A: H i g h P e r f o r m a n c e C o m p u t i n g i n t h e N e t h e r l a n d s
By Erik Kluit
SARA, the High Performance Computing (HPC) Center of the Dutch universities and research centers, recently acquired six POWER4 Regatta-eServer pSeries 690 supercomputers. The new machines will mainly be used by the institutes of the Netherlands Organization for Scientific Research (NWO), the Vrije Universiteit (VU) and individual researchers in the area of Computational Research (natural science and chemistry simulations). The new Regatta's will deliver around five times the computational power of the present IBM supercomputer at SARA.
Rijks (Systems Manager and Consultant HPC) and Jules Wolfrat (Systems Manager
and Programmer HPC), both specialised in parallelization and
optimalization of software) go a long way with SARA and AIX: before 1990
SARA hosted a 3090 mainframe with six processors, running AIX/370.
Early 90's SARA started using one of the first RS/6000 models: 7013-550
(41 MHz) and three 7013-590's
(66 MHz). Nowadays SARA hosts an SP configuration that consists of three
"towers" or "frames". Two of the frames, used for
batch processing, contain eight so-called "nighthawk-2"
high nodes, each of which contains 16 Power3-II processors (375 MHz), with
1 Gbyte of memory per processor. The third frame, used for interactive
access and job submission, contains 3 "winterhawk-2" wide nodes,
each consisting of 4 Power3-II (375 MHz) processors with 1 Gbyte of memory
of the Regatta's
The Regatta's (eServer pSeries 690) will be introduced in three phases. At the beginning of January 2002 the first two arrived. Each of these Regatta's contains one node with 32 Power4-processors, with a clock cycle of 1,1 GHz. This adds 280 GFlop/s in one blow to the existing peak performance of 210 GFlop/s, more than doubling the capacity of the system. Each node contains 32 Gbyte of shared memory. Each tower has a local scratch space of 135 Gbyte. The home file systems are mounted from an external file server, just like on the 'old' SP systems.
However, the new towers are not yet connected to the fast switch (SP Switch2), which means that the General Parallel File System (GPFS) is not available yet. GPFS is designed to provide high performance by "striping" I/O across multiple disks and high scalability (by utilizing multiple servers) through the SP Switch2.
Wim Rijks and Jules Wolfrat point out that the performance of the GPFS file system becomes more and more a bottleneck, it can't keep up with the speed of the other resources (processors and memory-access).
For connectivity to the outside world and to the Regatta servers Gigabit-Ethernet is available.
By the end of 2002 the next two Regatta's and the fast switch will be installed, replacing the "nighthawk-2" high nodes. With the installation of the last two machines in 2004 the configuration will consist of 204 processors (six Regatta's and the "winterhawk-2" wide nodes), 204 GB RAM and around 1 TFlop/s of computational power.
The Top 10 Supercomputer Sites of June 2002 (www.top500.org) for comparison:
|Rank||Manufacturer||Computer||# Proc||Rmax (TFlops)||Installation Site||Year|
|1||NEC||Earth-Simulator||5120||35.86||Earth Simulator Center Yokohama, Japan||2002|
|2||IBM||ASCI White, SP Power3 375 MHz||8192||7.22||Lawrence Livermore National Laboratory Livermore, USA||2000|
|AlphaServer SC ES45/1 GHz||3016||4.46||Pittsburgh Supercomputing Center Pittsburgh, USA||2001|
|AlphaServer SC ES45/1 GHz||2560||3.98||Commissariat a l'Energie Atomique (CEA) Bruyeres-le-Chatel, France||2001|
|5||IBM||SP Power3 375 MHz 16 way||3328||3.05||NERSC/LBNL Berkeley, USA||2001|
|AlphaServer SC ES45/1 GHz||2048||2.91||Los Alamos National Laboratory Los Alamos, USA||2002|
|7||Intel||ASCI Red||9632||2.37||Sandia National Laboratories Albuquerque, USA||1999|
|8||IBM||pSeries 690 Turbo 1.3GHz||864||2.31||Oak Ridge National Laboratory Oak Ridge, USA||2002|
|9||IBM||ASCI Blue-Pacific SST, IBM SP 604e||5808||2.14||Lawrence Livermore National Laboratory Livermore, USA||1999|
|10||IBM||pSeries 690 Turbo 1.3GHz||768||2.00||IBM/US Army Research Laboratory (ARL) Poughkeepsie, USA||2002|
Keep it dry !
I was also invited to take a look on the computer-floor. Computers don't use space these days, true, but a lot of computers do! I was impressed by the size of the Dutch national supercomputer: a 1024-CPU system consisting of two 512-CPU SGI Origin 3800 systems (picture). The machine is fitted with 500MHz R14000 CPUs organized in 256 4-CPU nodes, and has a peak performance of 1 TFlop/s.
Other systems SARA hosts are a CRAY Origin2000 (parallel system), an SGI Onyx2 RealityMonster (grafical system) and three Beowulf clusters (parallel systems).
Several systems at SARA are in the Top500 list of supercomputers.
While we walk down the stairs Wim Rijks explains why the computers are on the first floor: water! In case of a flood (remember we are in the Netherlands here, dikes can break) the computers keep it dry!
Parallelization of Programs
Parallel computer architectures are common in the world of supercomputers. SARA has the necessary skills to adapt or build computer programs to these parallel architectures: analyzing the opportunities for parallelization, choosing the parallelization method, optimizing the serial version of the program and parallelizing the program. On the website of SARA you can find a list of finished Parallel projects. Most of the projects done were funded by NCF (the Dutch National Foundation for Computerfacilities). NCF is responsible for the HPC infrastucture for scientific research in the Netherlands.
The Regatta's are best suited for programs that make use of parallelization
(using up-to 32 processors with shared memory) in
combination with high processing power needs.
The SP configuration uses OpenMP for running shared-memory parallel programs (C/C++ and Fortran) on one node. OpenMP is a portable, scalable model that gives shared-memory parallel programmers a simple and flexible interface for developing parallel applications.
On the SP it is possible to run parallel programs involving either up to eight 16-CPU nodes or two 32-CPU nodes. However, these programs have to be parallelized with MPI. It is also possible to combine MPI with OpenMP compiler directives. Technically it is possible to run a parallel job involving all of the SP's nodes but the current setup of the batch system does not support this.
MPI is an international accepted standard for a message passing library, for use in C en Fortran programs, and designed for high performance on both massively parallel machines and on workstation clusters.
One of the main users, the Department of Theoretical Chemistry (VU Research Group), will use the Regatta's to run chemistry simulations using the Amsterdam Density Functional software package (ADF) that uses MPI.