|
|
Before going on to the descriptions of the machines themselves, it is
important to consider some mechanisms that are or have been used to
increase the performance. The hardware structure or
architecture determines to a large extent what the possibilities
and impossibilities are in speeding up a computer system beyond the
performance of a single CPU. Another important factor that is
considered in combination with the hardware is the capability of
compilers to generate efficient code to be executed on the given
hardware platform. In many cases it is hard to distinguish between
hardware and software influences and one has to be careful in the
interpretation of results when ascribing certain effects to hardware or
software peculiarities or both. In this chapter we will give most
emphasis to the hardware architecture. For a description of machines
that can be considered to be classified as "high-performance" one is
referred to [9] and
[42].
Since many years the taxonomy of Flynn
[16] has proven to be useful for the
classification of high-performance computers. This classification is
based on the way of manipulating of instruction and data streams and
comprises four main architectural classes. We will first briefly sketch
these classes and afterwards fill in some details when each of the
classes is described separately.
- SISD machines: These are the conventional systems that
contain one CPU and hence can accommodate one instruction stream that
is executed serially. Nowadays many large mainframes may have more than
one CPU but each of these execute instruction streams that are
unrelated. Therefore, such systems still should be regarded as (a
couple of) SISD machines acting on different data spaces. Examples of
SISD machines are for instance most workstations like those of DEC,
Hewlett-Packard, IBM and SGI. The definition of SISD machines
is given here for completeness' sake. We will not discuss this type of
machines in this report.
- SIMD machines: Such systems often have a large number of processing
units, ranging from 1,024 to 16,384 that all may execute the same instruction on
different data in lock-step. So, a single instruction manipulates many data
items in parallel. Examples of SIMD machines in this class are the CPP DAP Gamma
II and the Quadrics Apemille which are not marketed anymore since about 2 years.
Nevertheless, the concept is still interesting and it may be expected that this
type of system will come up again or at least as a co-processor in large,
heterogeneous HPC systems.
- Another subclass of the SIMD systems are the vectorprocessors.
Vectorprocessors act on arrays of similar data rather than on single data items
using specially structured CPUs. When data can be manipulated by these vector
units, results can be delivered with a rate of one, two and — in special
cases — of three per clock cycle (a clock cycle being defined as the basic
internal unit of time for the system). So, vector processors execute on their
data in an almost parallel way but only when executing in vector mode. In this
case they are several times faster than when executing in conventional scalar
mode. For practical purposes vectorprocessors are therefore mostly regarded as
SIMD machines. An example of such a system is for instance the NEC SX-8B.
- MISD machines: Theoretically in these types of machines multiple
instructions should act on a single stream of data. As yet no
practical machine in this class has been constructed nor are such systems
easily to conceive. We will disregard them in the following discussions.
- MIMD machines: These machines execute several instruction
streams in parallel on different data. The difference with the
multi-processor SISD machines mentioned above lies in the fact that the
instructions and data are related because they represent different
parts of the same task to be executed. So, MIMD systems may run many
sub-tasks in parallel in order to shorten the time-to-solution for the
main task to be executed. There is a large variety of MIMD systems and
especially in this class the Flynn taxonomy proves to be not fully
adequate for the classification of systems. Systems that behave very
differently like a four-processor NEC SX-8 and a thousand
processor IBM p690 fall both in this class. In the following we will
make another important distinction between classes of systems and treat
them accordingly.
- Shared memory systems: Shared memory systems have multiple CPUs all
of which share the same address space. This means that the knowledge of
where data is stored is of no concern to the user as there is only one
memory accessed by all CPUs on an equal basis. Shared memory systems can be
both SIMD or MIMD. Single-CPU vector processors can be regarded as an
example of the former, while the multi-CPU models of these machines
are examples of the latter. We will sometimes use the abbreviations SM-SIMD and
SM-MIMD for the two subclasses.
- Distributed memory systems: In this case each CPU has its
own associated memory. The CPUs are connected by some network and may
exchange data between their respective memories when required. In
contrast to shared memory machines the user must be aware of the
location of the data in the local memories and will have to move or
distribute these data explicitly when needed. Again, distributed memory
systems may be either SIMD or MIMD. The first class of SIMD systems
mentioned which operate in lock step, all have distributed memories
associated to the processors. As we will see, distributed-memory MIMD
systems exhibit a large variety in the topology of their connecting
network. The details of this topology are largely hidden from the user
which is quite helpful with respect to portability of applications.
For the distributed-memory systems we will sometimes use DM-SIMD and
DM-MIMD to indicate the two subclasses.
As already alluded to, although the difference between shared- and distributed
memory machines seems clear cut, this is not always entirely the case from
user's point of view. For instance, the late Kendall Square Research systems
employed the idea of "virtual shared memory" on a hardware level. Virtual
shared memory can also be simulated at the programming level: A specification
of High Performance Fortran (HPF) was published in 1993 [23] which by means of compiler directives
distributes the data over the available processors. Therefore, the system on
which HPF is implemented in this case will look like a shared memory machine to
the user. Other vendors of Massively Parallel Processing systems (sometimes
called MPP systems), like HP and SGI, also are able to support proprietary
virtual shared-memory programming models due to the fact that these physically
distributed memory systems are able to address the whole collective address
space. So, for the user such systems have one global address space
spanning all of the memory in the system. We will say a little more about the
structure of such systems in the ccNUMA
section. In addition, packages like TreadMarks ([1]) provide a virtual shared memory
environment for networks of workstations. A good overview of such systems is
given at [12]
Distributed processing takes the DM-MIMD concept one step
further: instead of many integrated processors in one or several
boxes, workstations, mainframes, etc., are connected by (Gigabit)
Ethernet, FDDI, or otherwise and set to work concurrently on tasks in
the same program. Conceptually, this is not different from DM-MIMD
computing, but the communication between processors is often orders of
magnitude slower. Many packages to realise distributed computing are
available. Examples of these are PVM (standing for
Parallel Virtual Machine)
[17], and MPI
(Message Passing Interface,
[27]),
[28]).
This style of programming, called the "message passing" model has becomes so
much accepted that PVM and MPI have been adopted by virtually all major vendors
of distributed-memory MIMD systems and even on shared-memory MIMD systems for
compatibility reasons. In addition there is a tendency to cluster shared-memory
systems, for instance by HiPPI channels, to obtain systems with a very high
computational power. E.g., the NEC SX-8, and the Cray X1 have this
structure. So, within the clustered nodes a shared-memory programming style can
be used while between clusters message-passing should be used. It must be said
that PVM is not used very much anymore and that MPI has more or less become the
de facto standard.
For SM-MIMD systems we should mention OpenMP
([33],
[8]), that
can be used to parallelise Fortran and C(++) programs by inserting comment
directives (Fortran 77/90/95) or pragmas (C/C++) into the code. OpenMP
has quickly been adopted by the major vendors and has become a well
established standard for shared memory systems.
|