Communication API

The classes handling distribution over MPI processes.

The global mpi object provides basic MPI operations over all processors, such as broadcasts and sums, and is used to initialise MPI at the start of a program.

Each State holds a Communicator, created automatically from its potential, which handles the distribution of the state’s data over the processors it uses. It is rarely necessary to interact with the communicator directly, but it can be accessed via state.comm to make use of its data assignment and MPI reduction functions.

See Parallelisation for how the parallelisation works and how to use it.

Mpi minim::mpi

The global MPI helper instance.

This is the only instance of Mpi that should be used; it is shared by the whole library.

Mpi

class Mpi

A helper for MPI operations over all processors, without a communicator.

Programs should not create their own instances of this class; use the global minim::mpi object, which is created automatically when the library is included. It is initialised by calling mpi.init() (or mpiInit()) at the start of the program, after which the size and rank members give the number of processors and the rank of the current processor.

For operations specific to a state, use its Communicator instead, which is restricted to the processors used by that state.

Public Members

int size: The number of processors.

int rank: The rank of this processor.

Initialisation

void init(): Initialise MPI, if not already initialised.

void init(int *argc, char ***argv): Initialise MPI with command line arguments.

MPI reduction functions

Perform operations on local values before reducing the result over all processors.

Every processor receives the result.

double sum(double a) const: Sum a value over all processors.

double sum(const vector<double> &a) const: Sum the elements of a vector over all processors.

double dotProduct(const vector<double> &a, const vector<double> &b) const: Compute the dot product of two vectors over all processors.

Communication

void bcast(int &value, int root = 0) const: Broadcast an integer from the given root processor.

void bcast(double &value, int root = 0) const: Broadcast a double from the given root processor.

void bcast(vector<double> &data, int root = 0, int nData = 0) const: Broadcast a vector of doubles from the given root processor.

void barrier() const: Wait for all processors to reach this point.

Communicator

class Communicator

Handles the distribution of data over MPI processes.

An abstract base class. Derived classes define how the degrees of freedom are divided into blocks and assigned to processors: CommGrid for data on a structured grid, and CommUnstructured for element-based data.

The degrees of freedom are divided into blocks, one for each MPI rank used by the communicator. Each rank holds its block plus a halo of degrees of freedom, owned by neighbouring ranks, that are needed to compute the energy and gradient of the block. This processor data has size nproc. A communicator is created automatically when a State is created, and handles all communication between the ranks used by the state.

Subclassed by minim::CommGrid, minim::CommUnstructured

Public Functions

int rank() const: The MPI rank of this processor within the communicator (-1 if this processor is not used).

int size() const: The number of MPI ranks used by the communicator.

virtual std::unique_ptr<Communicator> clone() const = 0: Create a deep copy of the communicator.

virtual void setup(Potential &pot, size_t ndof, vector<int> ranks) = 0: Set up the distribution of the given potential over the given ranks.

Public Members

size_t ndof: Total number of degrees of freedom.

size_t nproc: Number of degrees of freedom held by this processor (block plus halo).

size_t nblock: Number of degrees of freedom assigned to this processor (excluding halo).

bool usesThisProc = true: Whether the current processor is used by the communicator.

vector<int> ranks = vector<int>(): The global MPI ranks used by the communicator.

Assign data

Convert global data to the local block or processor data.

virtual vector<int> assignBlock(const vector<int> &in) const = 0

Assign the local block from global or block data.

The layout of the returned data depends on the derived class: CommUnstructured returns just the block (size nblock), while CommGrid returns the processor data (size nproc), with the halo assigned if global data is given and zeroed otherwise.

Parameters:: in – The data to assign: either the full set of degrees of freedom, or the local block data.
Returns:: The local block data.

virtual vector<char> assignBlock(const vector<char> &in) const = 0

Assign the local block from global or block data.

Parameters:: in – The data to assign: either the full set of degrees of freedom, or the local block data.
Returns:: The local block data.

virtual vector<double> assignBlock(const vector<double> &in) const = 0

Assign the local block from global or block data.

Parameters:: in – The data to assign: either the full set of degrees of freedom, or the local block data.
Returns:: The local block data.

virtual vector<int> assignProc(const vector<int> &in) const = 0

Assign the local processor data (block plus halo) from global data.

Parameters:: in – The full set of degrees of freedom (size ndof).
Returns:: The local processor data (size nproc).

virtual vector<char> assignProc(const vector<char> &in) const = 0

Assign the local processor data (block plus halo) from global data.

Parameters:: in – The full set of degrees of freedom (size ndof).
Returns:: The local processor data (size nproc).

virtual vector<double> assignProc(const vector<double> &in) const = 0

Assign the local processor data (block plus halo) from global data.

Parameters:: in – The full set of degrees of freedom (size ndof).
Returns:: The local processor data (size nproc).

Access data

virtual int getBlock(int loc) const = 0

Get the block (processor) that owns a given global index.

Parameters:: loc – The global index of the degree of freedom.
Returns:: The index of the owning block.

virtual int getLocalIdx(int loc, int block = -1) const = 0

Get the local index of a given global index, or -1 if the degree of freedom is not held by this processor.

Parameters:

loc – The global index of the degree of freedom.
block – The block to look within (default: the block owning the given index).

Returns:

The local index, or -1 if not held by this processor.

double get(const vector<double> &vector, int loc) const

Get the value at a global index from local (processor) data, communicating with the owning processor if needed.

Parameters:

vector – The local processor data.
loc – The global index of the degree of freedom.

Returns:

The value at the given index.

Communication

void communicate(vector<double> &vector) const

Communicate the halo regions of the local data.

Sends the edges of this processor’s block to the halo regions of the neighbouring processors, and receives this processor’s halo from the neighbours’ block edges.

Parameters:: vector – The local data whose halo is to be updated.

void communicateAccumulate(vector<double> &vector) const

Communicate the halo regions, accumulating into the existing values.

Adds the values of this processor’s halo region onto the corresponding block locations of the neighbouring processors. Used to accumulate contributions to the gradient from neighbouring processors.

Parameters:: vector – The local data to accumulate from.

vector<double> gather(const vector<double> &block, int root = -1) const

Gather the local blocks to the given root processor (default: all).

Parameters:

block – The local processor data; only the block region is used.
root – The rank to gather onto (default: all ranks).

Returns:

The full data (size ndof) on the gathering ranks, empty on the others.

vector<double> scatter(const vector<double> &data, int root = -1) const

Scatter global data to the local processor data from the given root processor.

If root is -1 (default), the data must already be present on all ranks, and each rank extracts its own portion without communication.

Parameters:

data – The full data (size ndof).
root – The rank holding the data (default: all ranks).

Returns:

The local processor data (size nproc).

void bcast(int &value, int root = 0) const: Broadcast an integer from the given root processor.

void bcast(double &value, int root = 0) const: Broadcast a double from the given root processor.

void bcast(vector<double> &value, int root = 0) const: Broadcast a vector of doubles from the given root processor.

MPI reduction functions

Perform operations on local vectors before reducing the result across all the processors used by the communicator.

Every rank receives the result.

double sum(double a) const: Sum a value over all processors.

double sum(const vector<double> &a) const

Sum the elements of a vector over all processors.

Note that the whole local vector is summed, including any halo entries.

double norm(const vector<double> &a) const: Compute the L2 norm of the vector over all processors.

virtual double dotProduct(const vector<double> &a, const vector<double> &b) const

Compute the dot product of two vectors over all processors.

Only the block entries contribute; halo entries are excluded (the default implementation uses the whole vector).

Communicator subclasses

The communicator type is chosen by the potential: grid potentials use CommGrid, while element-based potentials use CommUnstructured.

CommGrid

class CommGrid : public minim::Communicator

A communicator that distributes grid data over MPI processes.

The grid is split into a commArray of sub-grids, one per processor, each surrounded by a halo of the given width. The final grid dimension is not parallelised.

Subclassed by minim::CommGrid2, minim::CommGrid3

Public Functions

virtual vector<int> assignBlock(const vector<int> &in) const override: Assign the local block from global or block data.

virtual vector<char> assignBlock(const vector<char> &in) const override: Assign the local block from global or block data.

virtual vector<double> assignBlock(const vector<double> &in) const override: Assign the local block from global or block data.

virtual vector<int> assignProc(const vector<int> &in) const override: Assign the local processor data (block plus halo) from global data.

virtual vector<char> assignProc(const vector<char> &in) const override: Assign the local processor data (block plus halo) from global data.

virtual vector<double> assignProc(const vector<double> &in) const override: Assign the local processor data (block plus halo) from global data.

virtual int getBlock(int loc) const override: Get the block (processor) that owns a given global index.

virtual int getLocalIdx(int loc, int block = -1) const override: Get the local index of a given global index, or -1 if not held by this processor.

virtual double dotProduct(const vector<double> &a, const vector<double> &b) const override: Compute the dot product of two vectors over all processors, excluding the halo entries.

CommGrid(int haloWidth): Construct a grid communicator with the given halo width.

virtual std::unique_ptr<Communicator> clone() const override: Create a deep copy of the communicator.

virtual void setup(Potential &pot, size_t ndof, vector<int> ranks) override: Set up the distribution of the given potential over the given ranks.

Public Members

int nDim: The number of grid dimensions.

int haloWidth: The width of the halo region in grid nodes.

CommGrid2

class CommGrid2 : public minim::CommGrid

A CommGrid with dimension-specific optimisations for 2D grids.

Used automatically for 2D grids; it specialises the dot product with an OpenMP-parallel loop collapsed over the two dimensions.

Public Functions

CommGrid2(int haloWidth): Construct a 2D grid communicator with the given halo width.

virtual std::unique_ptr<Communicator> clone() const override: Create a deep copy of the communicator.

virtual double dotProduct(const vector<double> &a, const vector<double> &b) const override: Compute the dot product of two vectors over all processors, excluding the halo entries.

CommGrid3

class CommGrid3 : public minim::CommGrid

A CommGrid with dimension-specific optimisations for 3D grids.

Used automatically for 3D grids; it specialises the dot product with an OpenMP-parallel loop collapsed over the three dimensions.

Public Functions

CommGrid3(int haloWidth): Construct a 3D grid communicator with the given halo width.

virtual std::unique_ptr<Communicator> clone() const override: Create a deep copy of the communicator.

virtual double dotProduct(const vector<double> &a, const vector<double> &b) const override: Compute the dot product of two vectors over all processors, excluding the halo entries.

CommUnstructured

class CommUnstructured : public minim::Communicator

A communicator that distributes unstructured data over MPI processes.

Used by element-based potentials. The degrees of freedom are split into equal-sized blocks in the order they are given, and the potential’s energy elements are each assigned to the processor that holds the most of their degrees of freedom. Any remaining degrees of freedom of an element lie in the halo.

Public Functions

virtual vector<int> assignBlock(const vector<int> &in) const override: Assign the local block from global or block data.

virtual vector<char> assignBlock(const vector<char> &in) const override: Assign the local block from global or block data.

virtual vector<double> assignBlock(const vector<double> &in) const override: Assign the local block from global or block data.

virtual vector<int> assignProc(const vector<int> &in) const override: Assign the local processor data (block plus halo) from global data.

virtual vector<char> assignProc(const vector<char> &in) const override: Assign the local processor data (block plus halo) from global data.

virtual vector<double> assignProc(const vector<double> &in) const override: Assign the local processor data (block plus halo) from global data.

virtual int getBlock(int loc) const override: Get the block (processor) that owns a given global index.

virtual int getLocalIdx(int loc, int block = -1) const override: Get the local index of a given global index, or -1 if not held by this processor.

virtual double dotProduct(const vector<double> &a, const vector<double> &b) const override: Compute the dot product of two vectors over all processors, using only the block entries.

CommUnstructured(): Construct an unstructured communicator.

virtual std::unique_ptr<Communicator> clone() const override: Create a deep copy of the communicator.

virtual void setup(Potential &pot, size_t ndof, vector<int> ranks) override: Set up the distribution of the given potential over the given ranks.