In parallel computation scenarios, the main objective is often to reduce the total wall clock execution time rather than simply reduce CPU time. Because so many different factors are present, you cannot expect a linear improvement in performance just by adding more and more nodes.

One of the most important factors is the inherent parallelism present in the code (i.e., how well the problem is broken into pieces for parallel execution). From an infrastructure point of view, many additional factors can also contribute to improved performance.

In most LAM/MPI cluster implementations, because client nodes have to communicate with each other through the MPI architecture, it is important to have a fast and dedicated network between nodes (e.g., gigabit Ethernet interfaces with bonding).

Also, it is a good idea to create a separate VLAN for a private communication network so that no other traffic can contribute to performance degradation.

If your application is performing any kind of data mining (which is often the case for commercial implementations of LAM/MPI), disk I/O from master and client nodes also has an effect on performance. However, because of the nature of parallel execution, it is important that source data for data mining (or the executables in simpler implementations) is available to all nodes for simultaneous read and write operations.

If you are using SAN-based external disks along with NFS, setting NFS parameters can be beneficial in terms of performance improvement. If you are using NAS storage subsystems and NFS/CIFS protocols to make shared data sources available to all nodes for simultaneous read/write, it is highly recommended that you use a separate VLAN and Ethernet interfaces on each node for disk I/O from the NAS subsystem, so that storage traffic is isolated from MPI traffic.

Finally, cluster filesystems (such as GFS, GPFS, and Veritas) can also help speed up disk I/O for large LAM/MPI implementations.