- the maxCommsSize variable is used to 'chunk' large data transfers
(eg, with PstreamBuffers) into a multi-pass send/recv sequence.
The send/recv windows for chunk-wise transfers:
iter data window
---- -----------
0 [0, 1*chunk]
1 [1*chunk, 2*chunk]
2 [2*chunk, 3*chunk]
...
Since we mostly send/recv in bytes, the current internal limit
for MPI counts (INT_MAX) can be hit rather quickly.
The chunking limit should thus also be INT_MAX, but since it is
rather tedious to specify such large numbers, can instead use
maxCommsSize = -1
to specify (INT_MAX-1) as the limit.
The default value of maxCommsSize = 0 (ie, no chunking).
Note
~~~~
In previous versions, the number of chunks was determined by the
sender sizes. This required an additional MPI_Allreduce to establish
an overall consistent number of chunks to walk. This additional
overhead each time meant that maxCommsSize was rarely actually
enabled.
We can, however, instead rely on the send/recv buffers having been
consistently sized and simply walk through the local send/recvs until
no further chunks need to be exchanged. As an additional enhancement,
the message tags are connected to chunking iteration, which allows
the setup of all send/recvs without an intermediate Allwait.
ENH: extend UPstream::probeMessage to use int64 instead of int for sizes
- "buffered" corresponds to MPI_Bsend (buffered send),
whereas the old name "blocking" is misleading since the
regular MPI_Send also blocks until completion
(ie, buffer can be reused).
ENH: IPstream::read() returns std::streamsize instead of label (#3152)
- previously returned a 'label' but std::streamsize is consistent with
the input parameter and will help with later adjustments.
- use <label> instead of <int> for internal accounting of the message
size, for consistency with the underyling List<char> buffers used.
- improve handling for corner case of IPstream receive with
non-blocking, although this combination is not used anywhere
- provide no_topology() characteristic to avoid triggering potentially
expensive mesh connectivity calculations when they are not required.
- remove/deprecate unused pointField references from the renumber
methods. These appear to have crept in from outer similarities
with decompositionMethod, but have no meaning for renumbering.
- remove/deprecate various unused aggregation renumberings since these
have been previously replaced by pre-calling calcCellCells, or
using bandCompression directly.
- make regionFaceOrder for block-wise renumbering optional and
treat as experimental (ie, default is now disabled).
The original idea was to sort the intra-region and inter-region faces
separately. However, this will mostly lead to non-upper triangular
ordering between regions, which checkMesh and others don't really like.
ENH: add timing information for various renumberMesh stages
ENH: add reset of clockTime and cpuTime increment
- simplifies section-wise timings
ENH: add globalIndex::null() and fieldTypes::processorType conveniences
- provides more central management of these characteristics
- renumberMesh now has -dry-run, -write-maps, -no-fields,
-renumber-method, -renumber-coeffs options.
* Use -dry-run with -write-maps to visualize the before/after
effects of renumbering (creates a VTK file).
* -no-fields to renumber the mesh only.
This is useful and faster when the input fields are uniform
and the -overwrite option is specified.
* -renumber-method allows a quick means of specifying a different
default renumber method (instead of Cuthill-McKee).
The -renumber-coeffs option allows passing of dictionary content
for the method.
Examples,
// Different ways to specify reverse Cuthill-McKee
* -renumber-method RCM
* -renumber-coeffs 'reverse true;'
* -renumber-method CuthillMcKee
* -renumber-coeffs 'reverse true;'
* -renumber-coeffs 'method CuthillMcKee; reverse true;'
// Other (without dictionary coefficients)
* renumberMesh -renumber-method random
// Other (with dictionary coefficients)
renumberMesh \
-renumber-method spring \
-renumber-coeffs 'maxCo 0.1; maxIter 1000; freezeFraction 0.99;'
// Other (with additional libraries)
renumberMesh -renumber-method zoltan -lib zoltanRenumber
COMP: build zoltan renumbering to MPI-specific location
- zoltan and Sloan renumbering are now longer automatically linked to
the renumberMesh utility but must be separately loaded by a
command-line option or through a dictionary "libs" entry.
ENH: add output cellID for decomposePar -dry-run -cellDist
- for clang-based compilers the default linker may be lld or simply ld.
Support '+link-ld' to explicitly select use of the ld linker.
- consolidate linker rules into single files
STYLE: adjust SPDX Identifier
- was previously limited to 'char' whereas gatherv/scatterv
already supported various integer and float types
STYLE: rebundle allToAll declarations with macros
ENH: provide a version of allToAllConsensus returning the Map
- simplifies use and avoids ambiguities in the send/recv parameters
- the Map version will now also transmit zero value data if they exist
in the Map. Unlike the List version, zero values are not necessary to
signal connectivity with a Map.
COMP: forwarding template parameters for NBX routines
ENH: consolidate PstreamBuffers size exchange options
- had a variety of nearly identical backends for all-to-all,
gather/scatter. Now combined internally with a dispatch enumeration
which provides better control over which size exchange algorithm
is used.
DEFEATURE: remove experimental full-NBX PstreamBuffers variant
- no advantages seen compared to the hybrid NBX/PEX approach.
Removal reduces some code cruft.
DEFEATURE: remove experimental "double non-blocking" NBX version
- the idea was to avoid blocking receives for very large data transfers,
but that is usually better accomplished with a hybrid NBX/PEX approach
like PstreamBuffers allows
- retain group information when copying zones
- support construct empty (add details later)
- improve consistency for zone and boundaryMesh construction
- support front/back/both selection for faceZoneToCell
STYLE: prefer faceZone patch() method instead of operator()
STYLE: use std::unique_ptr instead of manual pointer management
- for zones and core patch types.
Easier data management, allows default destructors (for example)
- selected with '+strict' in WM_COMPILE_CONTROL or 'wmake -strict', it
enables the FOAM_DEPRECATED_STRICT() macro, which can be used to
mark methods that are implicitly deprecated, but are not yet marked
as full deprecated (eg, API modification is too recent, generates
too many warnings). Can be considered a developer option.
- since the Apple SIP (System Integrity Protection) clears environment
variables such as DYLD_LIBRARY_PATH, a number of workarounds have
been used to provide shadow values. However, for a more robust
installation using -rpath at compilation time appears to be the
better solution.
In addition to the usual -rpath specification with absolute file
paths, MacOS supports (@loader_path, @executable_path) as well.
Now default to link with rpath information for MacOS, which can be
disabled by adding `~rpath` in WM_COMPILE_CONTROL
Explicit library paths handled:
- FOAM_FOAM_EXT_LIBBIN, FOAM_EXT_LIBBIN/FOAM_MPI
The executable rpaths are handled assuming a structure of
install-path/bin
install-path/lib/$(FOAM_MPI)
install-path/lib
Absolute compile-time paths for FOAM_USER_LIBBIN, FOAM_SITE_LIBBIN
and FOAM_LIBBIN are not handled since these are either too fragile
(FOAM_USER_LIBBIN and FOAM_SITE_LIBBIN values) or covered via
@loader_path anyhow (FOAM_LIBBIN).
Since the value of FOAM_MPI is a compile-time value, this rpath
treatment makes the installation less suitable for runtime changes
to the MPI vendor/version.
Note: no rpath added for c-only compilations since there are
currently no c-only libraries or executables with dynamic loading
- the special MacOS dlopen handling (commit f584ec97d0)
did not fully solve the problem with SIP clearing.
Eg, sourcing the RunFunctions (for runParallel) triggers SIP and
clears DYLD_LIBRARY_PATH. With the cleared path it finds the dummy
libraries: the dummy Pstream::init() fails.
- for simulations where the yPlus is needed for other purposes or
just for obtaining information on the patches it can be useful
to disable field writing and save disk space.
The 'writeFields' flag (as per some other function objects)
has been added control writing the yPlus volume field.
If unspecified, the default value is 'true' so that the yPlus
function object continues to work as before.
However, this default may change to 'false' in the future to align
with other function objects.
ENH: wallShearStress: support disable of field writing
- similar to yPlus, the write() method combines writing information
and writing the fields. The 'writeFields' flag allows some
separation of that logic.
