- MPI_Gatherv requires contiguous data, but a byte-wise transfer can
quickly exceed the 'int' limits used for MPI sizes/offsets. Thus
gather label/scalar components when possible to increase the
effective size limit.
For non-contiguous types (or large contiguous data) now also
reverts to manual handling
ENH: handle contiguous data in GAMGAgglomeration gather values
- delegate to globalIndex::gatherValues static method (new)
- used low-level MPI gather, but the wrapping routine contains an
additional safety check for is_contiguous which is not defined for
various std::pair<..> combination.
So std::pair<label,vector> (which is actually contiguous, but not
declared as is_contiguous) would falsely trip the check.
Avoid by simply gathering unbundled values instead.
- UPstream::mpiGather (MPI_Gather) - used by Pstream::listGatherValues
- UPstream::mpiScatter (MPI_Scatter) - used by Pstream::listScatterValues
These are much simpler forms for gather/scatter of fixed-sized
contiguous types data types (eg, primitives, simple pairs etc).
In the gather form, creates a list of gathered values on the master
process. The subranks have a list size of zero.
Similarly, scatter will distribute a list of values to single values
on each process.
Instead of
labelList sendSizes(Pstream::nProcs());
sendSizes[Pstream::myProcNo()] = sendData.size();
Pstream::gatherList(sendSizes);
Can write
const labelList sendSizes
(
UPstream::listGatherValues<label>(sendData.size())
);
// Less code, lower overhead and list can be const.
For scattering an individual value only,
instead of
labelList someValues;
if (Pstream::master()) someValues = ...;
Pstream::gatherList(sendSizes);
const label localValue
(
someValues[Pstream::myProcNo()]
);
Can write
labelList someValues;
if (Pstream::master()) someValues = ...;
Pstream::gatherList(sendSizes);
const label localValue
(
UPstream::listScatterValues<label>(someValues)
);
Can of course also mix listGatherValues to assemble a list on master
and use Pstream::scatterList to distribute.
ENH: adjusted globalIndex gather methods
- added mpiGather() method [contiguous data only] using MPI_Gatherv
- respect localSize if gathering master data to ensure that a
request for 0 master elements is properly handled.
