- When OpenFOAM is under git control and a 'debian/' directory exists,
this could mean two things:
1) Additional debian control has been added to OpenFOAM
2) OpenFOAM has been imported into a debian project
For the case that OpenFOAM has been imported into a debian project,
using the git information would be highly misleading. There will be no
OpenFOAM SHA1 correspondence.
However, if additional debian control has been added to OpenFOAM the
SHA1 will be valid.
The ad hoc solution is to use an additional "openfoam.debian"
directory to flag the addition of debian controls into openfoam.
When a "debian/" directory exists without a "openfoam.debian", assume
that the OpenFOAM has been imported into debian and do not use the SHA1.
- when installed in-source, use PETSC_ARCH to find additional include
directory and the correct library directory
CONFIG: bump to new hypre version
- add -hint option for have_adios2, have_hypre, have_petsc
- When compiling additional modules or user code, we need more control
for the installation locations beyond the usual FOAM_USER_LIBBIN,
FOAM_SITE_LIBBIN, FOAM_LIBBIN, and wish to have these values be
modifiable without editing files.
- provide wmake rules for handling standard defaults:
* GENERAL_RULES/module-path-user
* GENERAL_RULES/module-path-group
* GENERAL_RULES/module-path-project
which are incorporated as follows:
Make/options:
include $(GENERAL_RULES)/module-path-user
Make/files:
LIB = $(FOAM_MODULE_LIBBIN)/libMyLibrary
By default these would compile into FOAM_USER_{APPBIN,LIBBIN} but
could be adjusted at compilation time. For example,
```
wmake -module-prefix=/path/my-install-location
```
Or
```
./Allwmake -module-prefix=/path/my-install-location
./Allwmake -prefix=/path/my-install-location
```
Or
```
FOAM_MODULE_PREFIX=/path/my-install-location ./Allwmake
```
ENH: add -no-recursion option for AllwmakeParseArguments
- more descriptive naming than the -fromWmake option (still supported)
- remove wmake/scripts/wmake.{cmake,wmake}-args since the -prefix
handling and -no-recursion is now directly handled by AllwmakeParseArguments
- scripts/wmake.wmake-args partial logic for Allwmake scripts.
- handle '-quiet' as synonym for '-silent'
- Do not specify '-j' option for wrapped cmake creation to avoid
the warning:
make[1]: warning: -jN forced in submake: disabling jobserver mode.
- dependency handling relocated from cmakeFunctions to wmakeFunctions
and reused for mpi-versioned builds. This allows more checks for
configuration parameters and removes hard-code build path
information.
CONFIG: remove spurious mplibHPMPI entries
CONFIG: remove ADIOS1 rules (antiquated)
- initial split of wmake-related commands into "plumbing" and
"porcelain" akin to how git handles things.
- wmakeBuildInfo (very low-level), now relocated to the wmake/scripts
and accessible for the user as "wmake -build-info".
This satisfies a long-standing desire to access build information
in a fashion similar to the api/patch information.
CONFIG: avoid git information when building with a debian/ directory
- when a 'debian/' directory exists, there is a high probability that
the '.git/' directory is from debian and not from OpenFOAM (ie,
useless here). This corresponds to an implicit '-no-git', which has
no effect when building from pristine sources.
ENH: wmakeCheckPwd becomes scripts/wmake-check-dir
- accessible for the user as "wmake -check-dir" and with 1 or 2
directory names. A wmakeCheckPwd symlink left for compatibility.
- adds into the include-quoted search list instead the general (-Idir)
search list.
* makes it less subject to ordering (since it will now generally be
searched first) and makes it less subject to how duplicate removal
is implemented. In some compilers (#1627), the last instance of
a duplicate directory would be used and not the first instance.
* removes clutter in some Make/options files
COMP: add missing linkage libraries
- improve handling of changes in ParaView/VTK or cmake parameters (#1693)
* adjust internals to support recording of an unlimited number of
configuration parameters and use file `cmp` instead of trying
to check strings ourselves.
ENH: new wmake/scripts/wmake.cmake-args handler
- additional handling of -prefix=... as CMAKE_INSTALL_PREFIX export.
- in some contexts, can use instead of AllwmakeParseArguments
- bin/tools/create-mpi-config to query/write values for system openmpi.
In some cases this can be used to avoid an mpicc requirement at runtime.
- adjust openfoam session to include -test-tutorial forwarding to the
tutorials/AutoTest. This helps with writing installation tests.
- adjust foamConfigurePaths to latest version
- removal of gperftools default config, as per develop
- align wmake have_* scripts to support version query as per current
develop branch
- use config.sh/ fallbacks when the corresponding *_ARCH_PATH is empty
(eg, BOOST, CGAL, FFTW).
This aids when building outside of the regular OpenFOAM environment.
- add '[-+.~]' to the recognized qualifiers.
This allows simple readable names such as
WM_COMPILER=Clang-vendor
but also opens the FUTURE (not yet supported) possibility of
combining in additional information. For example,
WM_COMPILER=Clang~openmp
WM_COMPILER=Clang+cuda~openmp
by using '+' (add) and '~' (subtract) notation similar to what
spack uses.
CONFIG: support 'override' rules
- if present, compiler-family 'override' rules are included after
compiler-family 'general' rules have been included. This allows a
central means for including dynamically generated content to
override some values.
Some examples:
To handle different gcc versions (system compiler):
wmake/rules/...Gcc/override
```
ifneq (,$(findstring 9, $(WM_COMPILER)))
cc := gcc-9
CC := g++-9 -std=c++11
endif
```
To handle different openmp on Darwin (#1656):
wmake/rules/darwin64Clang/override
```
# Use libomp (not libgomp) unless openmp is disabled
ifeq (,$(findstring "~openmp", "$(WM_COMPILER)"))
COMP_OPENMP = -DUSE_OMP -Xpreprocessor -fopenmp
LINK_OPENMP = -lomp
else
include $(GENERAL_RULES)/no-openmp
endif
```
This treatment arguably fits into wmake/rules/darwin64Clang/general,
but it serves to illustrate a possible use case.
- preferentially handle Allwmake.override, which allows packaging
tools to define alternative make scripts, or selectively disable
components.
- remove legacy handling of 'Optional' directory.
Conditionals have since migrated into scripts themselves and/or
use the wmake/scripts/have_* framework.
BUG: missed passing -debug for Allwmake scripts
- missed detection of system libraries when installed with multiarch
paths like /usr/lib/x86_64-linux-gnu
CONFIG: improve handling of group/user config files (#928)
- changed bashrc handling of FOAM_CONFIG_NOUSER to use
FOAM_CONFIG_MODE instead. Propagate into foamEtcFile to make this
a stickier control.
This change allows better control, but also enables cluster
installations to define their own value within the OpenFOAM prefs.sh
file to prevent users accidentally mis-configuring things if
necessary.
- remove undocumented handling of an (a)ll mode in foamEtcFile to
avoid potential pitfalls.
- add support for FOAM_CONFIG_ETC handling.
This allows injection of an extra search layer when finding
project etc files
ENH: improvements to foamConfigurePaths (#928)
- handle FOAM_CONFIG_ETC implicitly, or explicitly with the new
-etc option.
STYLE: more explicit wording in foamConfigurePaths usage (#1602)
- document that an absolute path (eg, -scotch-path) overrides/ignores
the equivalent ThirdParty setting (eg, -scotch)
- longer options -system-compiler and -third-compiler for -system
and -third, respectively. Clearer as to their purpose.
- adjust the location sanity check to look for META-INFO directory.
- missed detection of system libraries when installed with multiarch
paths like /usr/lib/x86_64-linux-gnu
CONFIG: improve handling of group/user config files (#928)
- changed bashrc handling of FOAM_CONFIG_NOUSER to use
FOAM_CONFIG_MODE instead. Propagate into foamEtcFile to make this
a stickier control.
This change allows better control, but also enables cluster
installations to define their own value within the OpenFOAM prefs.sh
file to prevent users accidentally mis-configuring things if
necessary.
- remove undocumented handling of an (a)ll mode in foamEtcFile to
avoid potential pitfalls.
- add support for FOAM_CONFIG_ETC handling.
This allows injection of an extra search layer when finding
project etc files
ENH: improvements to foamConfigurePaths (#928)
- handle FOAM_CONFIG_ETC implicitly, or explicitly with the new
-etc option.
STYLE: more explicit wording in foamConfigurePaths usage (#1602)
- document that an absolute path (eg, -scotch-path) overrides/ignores
the equivalent ThirdParty setting (eg, -scotch)
- longer options -system-compiler and -third-compiler for -system
and -third, respectively. Clearer as to their purpose.
- adjust the location sanity check to look for META-INFO directory.
- simplifies code, covers most cases.
Can use wmake -show-api or wmakeBuildInfo to query the make rules.
STYLE: Allwmake script adjustments
- use bin/foamEtcFile instead of relying on PATH.
The make environment may not have the OpenFOAM bin/ in it.
- simpler shell syntax
- locate where the user is less tempted to change it (#1515).
It really should be considered an invariant environment variable.
STYLE: wmake -help information to stdout, die errors to stderr
- The wmake -show-path-{c,cxx} options return the fully qualified
paths to the respective compilers. This can be useful when verifying
that the correct compiler is indeed configured.
- The -help-full to display the "advanced" options, but in the normal
case just show the basic options.
- The -Wno-deprecated-copy flag for gcc-9.2.0
In the future we may indeed wish to explicitly request default
generated constructors and assignment operators, but at the moment
these are still acceptable.
- The -Wno-alloc-size-larger-than flag for mingw compilations
Related to differences in PTRDIFF_MAX vs SIZE_MAX on the target.
Several issues related to this can be found in the gcc bug reports
and on stackoverflow etc.
Set the m4 -I include accordingly to have the folllowing:
- the directory of the parser.
- include/ in the top-level source tree of the current target
(eg, src/finiteVolume/include-m4/ when compiling libfiniteVolume)
- include/ from OpenFOAM
Additional -dry-run option for makeParser, wrap-lemon for expanding m4
only.
Extend m4 wrapping support to include bison as well.
- add additional rule (ending '.ll') for the combination of
flex for C-code but compiling the result as C++.
This can be needed for re-entrant parsers.
- update bison rule to handle renaming of skeleton files.
Use a wrap-bison script to manage this.
- with 1906, OpenFOAM is purely C++ code and the last toolchain bits C
code have been avoided. Thus relax the wmake C rule for including
WM_COMPILE_OPTION. This makes it much easier to add different
compiler options with fewer files.
For example, for Broadwell-specific options:
cd wmake/rules/linux64Gcc
cp c++Opt c++OptBdw
edit this file and then use WM_COMPILE_OPTION=OptBdw
- ensure that WM_COMPILE_OPTION is always non-empty
- When building OpenFOAM for different platform combinations
(single/double, int32/int64) the build information that is compiled
into the OpenFOAM lib can become out of sync.
This is because the update trigger (wmakeBuildInfo -check) is
independent of the targetted platform. The added file
'src/OpenFOAM/Alltouch' provides a direct means of forcing a rebuild
of the version information. Eg,
src/OpenFOAM/Alltouch
wmake src/OpenFOAM
Also provide an additional 'wmakeBuildInfo -remove' to forcibly
remove META-INFO/build-info, if that is desired.
- now only needed when specify compiling -m32 on a 64-bit system.
Internally use the __SIZEOF_LONG__ compiler macro (gcc, icc, llvm)
to define when long is actually an int32_t.
- remove make target for wmdep (flex-based scanner), which eliminates
a bootstrap dependency on flex.
As of OpenFOAM-v1806, wmdep has been superseded by wmdepend
(ragel-based scanner).
- replace dirToString binary with shell/awk equivalent for simpler
maintenance. The utility is very rarely used (auto scanning to
create Make/files) so there is no performance difference.
- forces c++DBUG='-DFULLDEBUG -g -O0' for the compilation, to allow
localized debugging during development without file editing and
while retaining the WM_COMPILE_OPTION (eg, Opt)
Note that switching between 'wmake' and 'wmake -debug' will not
cause existing targets to be rebuilt. As before, these are driven by
the dependencies. An intermediate wclean may thus be required.
- when windows portable executables (.exe or .dll) files are loaded,
their dependent libraries not fully loaded. For OpenFOAM this means
that the static constructors which are responsible for populating
run-time selection tables are not triggered, and most of the run-time
selectable models will simply not be available.
Possible Solution
=================
Avoid this problem by defining an additional library symbol such as
the following:
extern "C" void libName_Load() {}
in the respective library, and tag this symbol as 'unresolved' for
the linker so that it will attempt to resolve it at run-time by
loading the known libraries until it finds it. The link line would
resemble the following:
-L/some/path -llibName -ulibName_Load
Pros:
- Allows precise control of forced library loading
Cons:
- Moderately verbose adjustment of some source files (even with macro
wrapping for the declaration).
- Adjustment of numerous Make/options files and somewhat ad hoc
in nature.
- Requires additional care when implementing future libraries and/or
applications.
- This is the solution taken by the symscape patches (Richard Smith)
Possible Solution
=================
Avoid this problem by simply force loading all linked libraries.
This is done by "scraping" the information out of the respective
Make/options file (after pre-processing) and using that to define
the library list that will be passed to Foam::dlOpen() at run-time.
Pros:
- One-time (very) minimal adjustment of the sources and wmake toolchain
- Automatically applies to future applications
Cons:
- Possibly larger memory footprint of application (since all dependent
libraries are loaded).
- Possible impact on startup time (while loading libraries)
- More sensitive to build failures. Since the options files are
read and modified based on the existence of the dependent
libraries as a preprocessor step, if the libraries are initially
unavailable for the first attempt at building the application,
the dependencies will be inaccurate for later (successful) builds.
- This is solution taken by the bluecape patches (Bruno Santos)
Adopted Solution
================
The approach taken by Bruno was adopted in a modified form since
this appears to be the most easily maintained.
Additional Notes
================
It is always possible to solve this problem by defining a corresponding
'libs (...)' entry in the case system/controlDict, which forces a dlOpen
of the listed libraries. This is obviously less than ideal for large-scale
changes, but can work to resolve an individual problem.
The peldd utility (https://github.com/gsauthof/pe-util), which is
also packaged as part of MXE could provide yet another alternative.
Like ldd it can be used to determine the library dependencies of
binaries or libraries. This information could be used to define an
additional load layer for Windows.
- Eg, with surface writers now in surfMesh, there are fewer libraries
depending on conversion and sampling.
COMP: regularize linkage ordering and avoid some implicit linkage (#1238)
- fix typo in makefiles/info that affected wmake -show-compile-c
- additional safeguard in src/OpenFOAM/Make/options against self-linking.
This is not normally required unless PROJECT_LIBS has been added into
the link stage.
solaris:
- rename WM_ARCH from SunOS to solaris64 for consistency with wmake/rules
- drop non-64 solaris from wmake/rules
- remove automatic selection of FJMPI. This should be done in the bashrc
or prefs.sh file instead.
- remove old (likely inaccurate) exported flags, rely on wmake -show-xyz
or user config instead
darwin:
- remove '-Ddarwin' from the exported WM_CFLAGS, WM_CXXFLAGS.
Not used elsewhere (ThirdParty)
- with the wmake rules we may have some compiler options bound to the
internal compiler variable. For example,
CC = g++ -std=c++11 -m64
c++FLAGS = ...
So shift any flags from CC to CXXFLAGS for the output of
'wmake -show-cxx', 'wmake -show-cxxflags', etc.
This makes it much easier to handle the values correctly elsewhere.
Eg,
CXX="$(wmake -show-cxx)" CXXFLAGS="$(wmake -show-cxxflags)" \
./configure
- provide dedicated detection 'have_ptscotch' function that can be
used after the 'have_scotch' function.
It sets the PTSCOTCH_ARCH_PATH, PTSCOTCH_INC_DIR, PTSCOTCH_LIB_DIR
and helps when the serial and parallel versions are located with
different logic.
- can be useful for retrieving the compilation flags for use with other
make systems (eg, cmake)
* wmake -show-compile (C++ not C)
* wmake -show-cxx
* wmake -show-cxxflags
- with -mcpu=native for automatic detection and -armpl for linking in
the performance libraries
STYLE: relocate -mcpu into compiler instead of compiler-flags (#1225)
- in addition to managing different vendors and versions, it may also
be necessary or desirable to have a particular variant
(eg, profiling, release, etc).
Devise a new meaningful name for the variant and create a
corresponding wmake rule.
Eg, SYSTEMOPENMPI-profiling with a corresponding
"wmake/rules/linux64Gcc/mplibSYSTEMOPENMPI-profiling" file
that has suitable content for your system.
CONFIG: intel-mpi use intel64/ paths only for config and wmake rules (#1153)
- previously adjusted the config files, but missed the changes
required for the wmake rules too.
Now simply migrate to using "intel64/{include,bin,lib}"
instead of the older naming "{include,bin,lib}64"
These changes work since at least intel-mpi 2015 (5.x), but possibly
earlier as well
- was WM_PROJECT_API in the environment and FOAM_API in dictionaries.
Make these both consistently FOAM_API.
This is a non-breaking change, since the value of WM_PROJECT_API
(added in 1812) and/or FOAM_API is purely informative.
For the current correct values, always use
* foamEtcFile -show-api
* wmakeBuildInfo -show-api
- Note: mpich now builds libmpi.so instead of libmpich.so
- define both -DMPICH_SKIP_MPICXX and -DOMPI_SKIP_MPICXX regardless of
using openmpi or mpich. This simplifies the files and does not harm.
- instead of
WM_COMPILER=GccKNL WM_COMPILE_OPTION=Opt
-> linux64GccKNLDPInt32Opt
now specify
WM_COMPILER=Gcc WM_COMPILE_OPTION=OptKNL
-> linux64GccDPInt32OptKNL
This makes it easier (and more obvious) for adding different tweaks
without needing to generate too many files.
Eg,
cd wmake/rules/linux64Gcc
cp cOpt cOptBdw
cp c++Opt c++OptBdw
edit these two files and then use WM_COMPILE_OPTION=OptBdw
CONFIG: provide some default c/c++ flags in General compiler rules
- can make is easier when deriving new compile options, and ensures
that '-02' is enabled as an initial default.
- finds the correct root directory location before creating
the lnInclude directory
Eg,
from within something like src/finiteVolume/fields/fvPatchFields/..
wmakeLnInclude -update -root
it backtracks to find the top-level directory with Make/
and makes the lnInclude directory there:
Using /home/mark/openfoam/OpenFOAM-plus/src/finiteVolume
ln: /home/mark/openfoam/OpenFOAM-plus/src/finiteVolume/lnInclude
- was for parallel compilation across multiple hosts, but less useful
with modern CPUs with higher number of cores and/or hyperthreading.
Fragile use and dependent on a 'lockfile' utility that is not often
installed.
- was WM_PROJECT_API in the environment and FOAM_API in dictionaries.
Make these both consistently FOAM_API.
This is a non-breaking change, since the value of WM_PROJECT_API
(added in 1812) and/or FOAM_API is purely informative.
For the current correct values, always use
* foamEtcFile -show-api
* wmakeBuildInfo -show-api
- looks slightly messier without associative arrays, but improves
portability. Should now also work with dash.
STYLE: support wmakeBuildInfo -cmp and -check options as equivalent
- Use the OPENFOAM define (eg, 1806, 1812), which normally corresponds
to a major release, to define an API level. This remains consistent
within a release cycle and means that it is possible to manage
several sub-versions and continue to have a consistent lookup.
The current API value is updated automatically during the build
and cached as meta data for later use, even when the wmake/ directory
is missing or OpenFOAM has not yet be initialized.
The version information reported on program start or with -help
usage adjusted to reflect this. The build tag from git now also
carries the date as being more meaningful to trace than a hash
value.
- Update etc/bashrc and etc/cshrc to obtain the project directory
directly instead of via its prefix directory. The value obtained
corresponds to an absolute path, from which the prefix directory
can be obtained.
The combination of these changes removes the reliance on any
particular directory naming convention.
For example,
With an 1812 version (API level):
WM_PROJECT_VERSION=myVersion
installed as /some/path/somewhere/openfoam-mySandbox
This makes the -prefix, -foamInstall, -projectVersion, -version
values of foamEtcFiles, and similar entries for foamConfigurePaths
superfluous.
WM_PROJECT_INST_DIR is no longer required or used
ENH: improve handling and discovery of ThirdParty
- improve the flexibility and reusability of ThirdParty packs to cover
various standard use cases:
1. Unpacking initial release tar files with two parallel directories
- OpenFOAM-v1812/
- ThirdParty-v1812/
2. With an adjusted OpenFOAM directory name, for whatever reason
- OpenFOAM-v1812-myCustom/
- openfoam-1812-other-info/
3. Operating with/without ThirdParty directory
To handle these use cases, the following discovery is used.
Note PROJECT = the OpenFOAM directory `$WM_PROJECT_DIR`
PREFIX = the parent directory
VERSION = `$WM_PROJECT_VERSION`
API = `$WM_PROJECT_API`, as per `foamEtcFiles -show-api`
0. PROJECT/ThirdParty
- for single-directory installations
1. PREFIX/ThirdParty-VERSION
- this corresponds to the traditional approach
2. PREFIX/ThirdParty-vAPI
- allows for an updated value of VERSION (eg, v1812-myCustom)
without requiring a renamed ThirdParty. The API value
would still be '1812' and the original ThirdParty-v1812/
would be found.
3. PREFIX/ThirdParty-API
- this is the same as the previous example, but using an unadorned
API value. This also makes sense if the chosen version name also
uses the unadorned API value in its naming
(eg, 1812-patch190131, 1812.19W03)
4. PREFIX/ThirdParty-common
- permits maximum reuse for various versions, but only for
experienced user who are aware of potential version
incompatibilities
Directory existence is checked as is the presence of an Allwmake file
or a platforms/ directory. This reduces the potential of false positive
matches and limits the selection to directories that are either
with sources (has the Allwmake file), or pre-compiled binaries (has
the platforms/ directory).
If none of the explored directories are found to be suitable,
it reverts to using a PROJECT/ThirdParty dummy location since
this is within the project source tree and can be trusted to
have no negative side-effects.
ENH: add csh support to foamConfigurePaths
- this removes the previously experienced inconsistence in config file
contents.
REMOVED: foamExec
- was previously used when switching versions and before the
bashrc/cshrc discovery logic was added. It is now obsolete.
- provide default WM_DIR if not already set, to improve robustness if a
reduced environment is used
- add etc/ to WM_PROJECT_SITE search. This makes the site directory
structure consistent with the OpenFOAM structure.
Eg,
WM_PROJECT_SITE/etc/..
WM_PROJECT_SITE/bin/..
WM_PROJECT_SITE/platforms/..
- Don't set/export WM_OSTYPE. The default is POSIX and is properly
defaulted throughout, including in CMakeLists-OpenFOAM.txt (also for
Catalyst)
- append the commit hash value with the commit date when creating
the build string information and drop the version prefix.
This provides an immediate overview of when the code was last
changed. The prefixed version information can be dropped from
the build string, since it is readily available in other forms.
- removed reliance on ParaView_INCLUDE_DIR variable for conveying the
major.minor version information when compiling. This can be somewhat
fragile and also adds variable that is an unnecessary when running
(only used when compiling).
Instead use `have_pvplugin_support` function in paraviewFunctions
wmake script to determine the maj.min from the PV_PLUGIN_PATH
since we have already defined the output path there with paraview
maj.min numbering.
Can now build with paraview from the operating system,
provided that it has develop headers available.
ParaView_VERSION=system
In the etc/config.sh/paraview setup, the maj.min is taken from
the corresponding `paraview --version` output and used when
defining the PV_PLUGIN_PATH.
During the build, the include path taken from `paraview-config`
for a system installation, from the guess installation root
of the paraview binary, or ParaView_DIR otherwise.
NB: using a system ParaView for building runTimePostProcessing is unsupported.
- these types of builds appear to have various library resolution issues
(eg, libexpat not being loaded). Additionally, the build logic does
not yet cover this type of use case.
