The front-end traits:
- UPstream_dataType trait:
This wrapper is unwinds the type to check against base/alias, but also
checks if it is a component aggregate of a supported UPstream data type.
This will be that main entry point for usage.
- UPstream_opType trait:
Provides a mapping of OpenFOAM ops to their MPI equivalent.
The \c opcode_id is the corresponding internal representation.
The lower-level traits (not normally used within coding)
- UPstream_base_dataType trait:
Tests true/false if the specified data type has an internal MPI equivalent.
The \c datatype_id is the corresponding internal enumeration.
Even if this tests as false, it will always return \c type_byte as the
fallback for general contiguous data
- UPstream_alias_dataType trait:
Provides mapping for <int/long/long long,...> to the fundamental 32/64
integrals, since <int/long/long long,...> may not otherwise directly map
on all systems.
NOTE: can use the updates Test-machine-sizes test application to
determine if all data types and aliases are properly defined on
different systems
- this will allow send/recv/broadcast of various data types directly
without reinterpreting as bytes. No performance benefit, but makes
programming more flexible. In addition to the normal MPI types, also
provide some convenient groupings such as double[3], double[9] etc.
- the additional user-defined data types are created on the start of
MPI and removed before shutdown
- previously there were two places that used nProcsSimpleSum for
defining the communication pattern. However, this meant in practice
that filling of linearCommunication_ and treeCommunication_ themselves
where globally controlled.
Now retain the state within commsStructList and use linear/tree
when populating.
- deprecated and replaced (JAN-2023) with a version that uses
UPstream::Request to track the request. This avoids placing a
request on the global list and then having difficulty isolating it
later (or having it cleared out by someone else).
ENH: UPstream::addRequest() now ignores null requests
- avoids placing useless requests on the global list
- replace with plusOp for reductions. The old use didn't necessarily work
well with compiler resolution in all cases.
Simplify to use plusOp (and removed the old version).
[Does not affect very much code...]
COMP: incorrect include ordering for GeometricFieldFunctions
- header was included after the template code
REGRESSION: combineAllGather mapped to incorrect method (81fa7d08ee)
STYLE: use UPstream::commWarn(...) setter method
- eliminates nearly identical code between 'gather' and 'combineGather'
* Normal gather updates by assigning the result of the binary operation.
* Combine gather updates by using a binary operator that modifies
its first parameter in-place
By-product of this refactoring are these new variants:
listGather(), listGatherReduce()
mapGather(), mapGatherReduce()
that mirror the previously existing ones
listCombineGather(), listCombineReduce()
mapCombineGather(), mapCombineReduce()
except that they use the 'regular' binary operator
- the general OpenFOAM way of collecting data often looks like this:
List<scalar> allValues(numProcs);
allValues[myProc] = localValue;
Pstream::gatherList(allValues);
// Now possible like this
List<scalar> allValues(numProcs);
allValues[myProc] = localValue;
UPstream::mpiGather(nullptr, allValues.data_bytes(), sizeof(scalar));
- adjusted Pstream::gatherList accordingly.
Split out the manual implementations, give them new names
(..._tree_algorithm) and make them private.
STYLE: rename UPstream::gather -> UPstream::mpiGatherv
- easier to identify and establishes the connection to the MPI call
- additional startup guard for inter-node/local-node queries (UPstream)
- impose linear communication tree for inter-node/local-node
communicators. Was previously defaulted to a basic tree, but more
consistent to have flat addressing for these types of connections.
- demand-driven UPstream::interNode_offsets() for walking
inter-node ranges instead of creating it manually in various places.
- (style): List<int> instead of labelList for internal commsStruct
since the communication structures are tied to MPI sizes
and not to the OpenFOAM label sizes
- reduce the number of intermediate buffer allocations within
gatherList, scatterList.
- replace gatherValues() with listGatherValues(), which returns the
gathered values instead of passing by argument. This makes it less
fragile and more convenient. Naming as per
Pstream::listGatherValues(), but still retaining the original
parameter order.
- rename inplace 'gather' variant as 'gatherInplace' for clarity.
- changed signature of lowest-level globalIndex::gather routines from
List<Type> to UList<Type> for the output and removed any resizing.
This means that the caller is responsible for ensuring that the
receiving field is adequately sized (on master). This change allows
potential reuse of 'work' arrays etc.
The only code change for using this low-level gather was in
GAMGAgglomeration.
- split/duplicate functionality
- rework inter-node/intra-node handling to allow selection of
splitting based on 'shared' or hostname (default).
- always creates node communicators at startup:
* commInterNode() - between nodes
* commLocalNode() - within a node
- world-comm is now always a duplicate of MPI_COMM_WORLD to provide
better separation from other processes.
NB:
the inter-node comm is a slight exception to other communicators
in that we always retain its list of (global) ranks, even if the
local process is not in that communicator.
This can help when constructing topology-aware patterns.
FIX: non-participating ranks still had knowledge of their potential siblings
- after create by group, the procIDs_ of non-participating ranks
should be empty (except for the inter-node exception)
- using the List containers, and not their low-level data_bytes(),
size_bytes() methods is more convenient and allows future
adjustments to be centralized
ENH: trivial intptr_t wrapper for MPI_Win
STYLE: minor adjustments to mpirunDebug
- 'if constexpr (...)'
* instead of std::enable_if
* terminate template recursion
* compile-time elimination of code
- use C++14 '_t', '_v' versions,
eg, std::is_integral_v<T> instead of std::is_integral<T>::value
- std::begin, std::end, std::void_t instead of prev stdFoam versions
- provide is_contiguous_v<..> as short form of is_contiguous<..>::value
with the additional benefit of removing any cv qualifiers.
ENH: include is_rotational_vectorspace trait
- tests for vector-space and nComponents > 1 (ie, not sphericalTensor)
ENH: improve robustness of pTraits_.. tests by removing cv qualifiers
