- the '-no-core' to limit coredumps to zero size
- the '-quick' option, which changes valgrind --leak-check from "full"
to "summary", and implies -no-core as well.
- enforce tcp libfabrics provider under valgrind since valgrind
does not otherwie work nicely with RMA
- fails with MPI_ARG_ERR.
Do not assume that any vendors actually support in-place handling
for MPI_Reduce(), regardless of what their documentation may claim.
- includes intrinsic MPI types, but no component aggregates since we
barely wish to use gatherv/scatterv (slow!) in the first place
and it makes no sense to recalculate the list of counts for
component aggregates which we will never use.
- for known data types (and component aggregates),
and intrisic reduction operation can now use
UPstream::mpiReduce(), UPstream::mpiAllReduce().
This change permits more operations to use MPI calls directly.
eg,
reduce(vec, sumOp<vector>);
now calls mpiAllReduce(..., op_sum) instead of point-to-point,
followed by a broadcast.
Similarly, when called as a simple reduction (not all-reduce)
Pstream::gather(vec, sumOp<vector>);
now calls mpiReduce(..., op_sum) instead of point-to-point
- extend use of MPI calls to list-wise reductions as well
- extend sumReduce() to bundle/unbundle vector-space types,
which lowers overall communication.
Before:
1) send/recv + binary op through a communication tree
2) broadcast
3) all-reduce of the count
Now:
1) pack into a local bundle
2) all-reduce
3) unpack the bundle
- support send/recv of basic types (int32, float, double, ...) in
addition to common OpenFOAM vectorspace types (floatVector,
doubleVector, ...)
This permits sending NUM items of the given types instead of sending
NUM sizeof() bytes.
For a vector (as double): 1 item instead of 24 items (3*8 bytes).
For a tensor (as double): 1 item instead of 72 items (9*8 bytes).
- simplify and rationalize some of the broadcast methods for more code
reuse.
The bottom level UPstream::broadcast is now always to/from "root=0".
This was previously passed as a default parameter, but never used
anything other than '0' in the code. Fixing it as '0' makes it
consistent with the 'top-down' logical for node-based broadcast.
- now distinguish between basic MPI types and user-defined types.
The new front-facing trait UPstream_basic_dataType unwinds components
and other types, but only for MPI fundamental types
(including any aliases)
- additional helper to combine a test for binary operator validity and
basic data type validity, which better expresses intent:
template<class BinaryOp, class T>
UPstream_data_opType;
- relax bit-wise operators to also accept signed integrals
and 'void' generic
- remove unused/unusable broadcast stream constructors/methods
- provide OPBstream::sends() and IPBstream::recvs() methods,
refactored from Pstream::broadcasts. These will always use
serializations, even for contiguous content.
- additional methods to support special handling of zero-sized lists.
For example,
if (UPstream::master(comm))
{
if (list.empty()) OPBstream::send(Foam::zero, comm);
else OPBstream::send(list, comm);
}
else
{
IPBstream is(comm);
if (is.remaining()) { is >> list; }
else { list.clear(); }
}
This avoids serialization of an empty list and the resulting double
broadcast (size + content), using instead a single broadcast (size).
STYLE: more consistency in communicator types (int vs label)
- returns the (start/size) range of the commLocalNode ranks in terms
of the (const) world communicator processors. This allows mapping
back into lists defined in terms of the world ranks.
- similar to what std::copy_n and std::fill_n would do, except with
templated loops. This allows compile-time transcribing with loop
unrolling. For example,
vector vec1 = ..., vec2 = ...;
FixedList<scalar, 6> values;
VectorSpaceOps<3>::copy_n(vec1.begin(), values.begin());
VectorSpaceOps<3>::copy_n(vec2.begin(), values.begin(3))
// do something with all of these values
STYLE: make start index of VectorSpaceOps optional
ENH: add clamped begin(int) versions to FixedList as per UList
The front-end traits:
- UPstream_dataType trait:
This wrapper is unwinds the type to check against base/alias, but also
checks if it is a component aggregate of a supported UPstream data type.
This will be that main entry point for usage.
- UPstream_opType trait:
Provides a mapping of OpenFOAM ops to their MPI equivalent.
The \c opcode_id is the corresponding internal representation.
The lower-level traits (not normally used within coding)
- UPstream_base_dataType trait:
Tests true/false if the specified data type has an internal MPI equivalent.
The \c datatype_id is the corresponding internal enumeration.
Even if this tests as false, it will always return \c type_byte as the
fallback for general contiguous data
- UPstream_alias_dataType trait:
Provides mapping for <int/long/long long,...> to the fundamental 32/64
integrals, since <int/long/long long,...> may not otherwise directly map
on all systems.
NOTE: can use the updates Test-machine-sizes test application to
determine if all data types and aliases are properly defined on
different systems
- this will allow send/recv/broadcast of various data types directly
without reinterpreting as bytes. No performance benefit, but makes
programming more flexible. In addition to the normal MPI types, also
provide some convenient groupings such as double[3], double[9] etc.
- the additional user-defined data types are created on the start of
MPI and removed before shutdown
- previously there were two places that used nProcsSimpleSum for
defining the communication pattern. However, this meant in practice
that filling of linearCommunication_ and treeCommunication_ themselves
where globally controlled.
Now retain the state within commsStructList and use linear/tree
when populating.
- deprecated and replaced (JAN-2023) with a version that uses
UPstream::Request to track the request. This avoids placing a
request on the global list and then having difficulty isolating it
later (or having it cleared out by someone else).
ENH: UPstream::addRequest() now ignores null requests
- avoids placing useless requests on the global list
- replace with plusOp for reductions. The old use didn't necessarily work
well with compiler resolution in all cases.
Simplify to use plusOp (and removed the old version).
[Does not affect very much code...]
COMP: incorrect include ordering for GeometricFieldFunctions
- header was included after the template code
REGRESSION: combineAllGather mapped to incorrect method (81fa7d08ee)
STYLE: use UPstream::commWarn(...) setter method
- eliminates nearly identical code between 'gather' and 'combineGather'
* Normal gather updates by assigning the result of the binary operation.
* Combine gather updates by using a binary operator that modifies
its first parameter in-place
By-product of this refactoring are these new variants:
listGather(), listGatherReduce()
mapGather(), mapGatherReduce()
that mirror the previously existing ones
listCombineGather(), listCombineReduce()
mapCombineGather(), mapCombineReduce()
except that they use the 'regular' binary operator
- the general OpenFOAM way of collecting data often looks like this:
List<scalar> allValues(numProcs);
allValues[myProc] = localValue;
Pstream::gatherList(allValues);
// Now possible like this
List<scalar> allValues(numProcs);
allValues[myProc] = localValue;
UPstream::mpiGather(nullptr, allValues.data_bytes(), sizeof(scalar));
- adjusted Pstream::gatherList accordingly.
Split out the manual implementations, give them new names
(..._tree_algorithm) and make them private.
STYLE: rename UPstream::gather -> UPstream::mpiGatherv
- easier to identify and establishes the connection to the MPI call
- additional startup guard for inter-node/local-node queries (UPstream)
- impose linear communication tree for inter-node/local-node
communicators. Was previously defaulted to a basic tree, but more
consistent to have flat addressing for these types of connections.
- demand-driven UPstream::interNode_offsets() for walking
inter-node ranges instead of creating it manually in various places.
- (style): List<int> instead of labelList for internal commsStruct
since the communication structures are tied to MPI sizes
and not to the OpenFOAM label sizes
- reduce the number of intermediate buffer allocations within
gatherList, scatterList.
- replace gatherValues() with listGatherValues(), which returns the
gathered values instead of passing by argument. This makes it less
fragile and more convenient. Naming as per
Pstream::listGatherValues(), but still retaining the original
parameter order.
- rename inplace 'gather' variant as 'gatherInplace' for clarity.
- changed signature of lowest-level globalIndex::gather routines from
List<Type> to UList<Type> for the output and removed any resizing.
This means that the caller is responsible for ensuring that the
receiving field is adequately sized (on master). This change allows
potential reuse of 'work' arrays etc.
The only code change for using this low-level gather was in
GAMGAgglomeration.
