- missed detection of system libraries when installed with multiarch
paths like /usr/lib/x86_64-linux-gnu
CONFIG: improve handling of group/user config files (#928)
- changed bashrc handling of FOAM_CONFIG_NOUSER to use
FOAM_CONFIG_MODE instead. Propagate into foamEtcFile to make this
a stickier control.
This change allows better control, but also enables cluster
installations to define their own value within the OpenFOAM prefs.sh
file to prevent users accidentally mis-configuring things if
necessary.
- remove undocumented handling of an (a)ll mode in foamEtcFile to
avoid potential pitfalls.
- add support for FOAM_CONFIG_ETC handling.
This allows injection of an extra search layer when finding
project etc files
ENH: improvements to foamConfigurePaths (#928)
- handle FOAM_CONFIG_ETC implicitly, or explicitly with the new
-etc option.
STYLE: more explicit wording in foamConfigurePaths usage (#1602)
- document that an absolute path (eg, -scotch-path) overrides/ignores
the equivalent ThirdParty setting (eg, -scotch)
- longer options -system-compiler and -third-compiler for -system
and -third, respectively. Clearer as to their purpose.
- adjust the location sanity check to look for META-INFO directory.
- simplifies code, covers most cases.
Can use wmake -show-api or wmakeBuildInfo to query the make rules.
STYLE: Allwmake script adjustments
- use bin/foamEtcFile instead of relying on PATH.
The make environment may not have the OpenFOAM bin/ in it.
- simpler shell syntax
Set the m4 -I include accordingly to have the folllowing:
- the directory of the parser.
- include/ in the top-level source tree of the current target
(eg, src/finiteVolume/include-m4/ when compiling libfiniteVolume)
- include/ from OpenFOAM
Additional -dry-run option for makeParser, wrap-lemon for expanding m4
only.
Extend m4 wrapping support to include bison as well.
- add additional rule (ending '.ll') for the combination of
flex for C-code but compiling the result as C++.
This can be needed for re-entrant parsers.
- update bison rule to handle renaming of skeleton files.
Use a wrap-bison script to manage this.
- remove make target for wmdep (flex-based scanner), which eliminates
a bootstrap dependency on flex.
As of OpenFOAM-v1806, wmdep has been superseded by wmdepend
(ragel-based scanner).
- replace dirToString binary with shell/awk equivalent for simpler
maintenance. The utility is very rarely used (auto scanning to
create Make/files) so there is no performance difference.
- forces c++DBUG='-DFULLDEBUG -g -O0' for the compilation, to allow
localized debugging during development without file editing and
while retaining the WM_COMPILE_OPTION (eg, Opt)
Note that switching between 'wmake' and 'wmake -debug' will not
cause existing targets to be rebuilt. As before, these are driven by
the dependencies. An intermediate wclean may thus be required.
- when windows portable executables (.exe or .dll) files are loaded,
their dependent libraries not fully loaded. For OpenFOAM this means
that the static constructors which are responsible for populating
run-time selection tables are not triggered, and most of the run-time
selectable models will simply not be available.
Possible Solution
=================
Avoid this problem by defining an additional library symbol such as
the following:
extern "C" void libName_Load() {}
in the respective library, and tag this symbol as 'unresolved' for
the linker so that it will attempt to resolve it at run-time by
loading the known libraries until it finds it. The link line would
resemble the following:
-L/some/path -llibName -ulibName_Load
Pros:
- Allows precise control of forced library loading
Cons:
- Moderately verbose adjustment of some source files (even with macro
wrapping for the declaration).
- Adjustment of numerous Make/options files and somewhat ad hoc
in nature.
- Requires additional care when implementing future libraries and/or
applications.
- This is the solution taken by the symscape patches (Richard Smith)
Possible Solution
=================
Avoid this problem by simply force loading all linked libraries.
This is done by "scraping" the information out of the respective
Make/options file (after pre-processing) and using that to define
the library list that will be passed to Foam::dlOpen() at run-time.
Pros:
- One-time (very) minimal adjustment of the sources and wmake toolchain
- Automatically applies to future applications
Cons:
- Possibly larger memory footprint of application (since all dependent
libraries are loaded).
- Possible impact on startup time (while loading libraries)
- More sensitive to build failures. Since the options files are
read and modified based on the existence of the dependent
libraries as a preprocessor step, if the libraries are initially
unavailable for the first attempt at building the application,
the dependencies will be inaccurate for later (successful) builds.
- This is solution taken by the bluecape patches (Bruno Santos)
Adopted Solution
================
The approach taken by Bruno was adopted in a modified form since
this appears to be the most easily maintained.
Additional Notes
================
It is always possible to solve this problem by defining a corresponding
'libs (...)' entry in the case system/controlDict, which forces a dlOpen
of the listed libraries. This is obviously less than ideal for large-scale
changes, but can work to resolve an individual problem.
The peldd utility (https://github.com/gsauthof/pe-util), which is
also packaged as part of MXE could provide yet another alternative.
Like ldd it can be used to determine the library dependencies of
binaries or libraries. This information could be used to define an
additional load layer for Windows.
- Eg, with surface writers now in surfMesh, there are fewer libraries
depending on conversion and sampling.
COMP: regularize linkage ordering and avoid some implicit linkage (#1238)
- provide dedicated detection 'have_ptscotch' function that can be
used after the 'have_scotch' function.
It sets the PTSCOTCH_ARCH_PATH, PTSCOTCH_INC_DIR, PTSCOTCH_LIB_DIR
and helps when the serial and parallel versions are located with
different logic.
- was for parallel compilation across multiple hosts, but less useful
with modern CPUs with higher number of cores and/or hyperthreading.
Fragile use and dependent on a 'lockfile' utility that is not often
installed.
- provide default WM_DIR if not already set, to improve robustness if a
reduced environment is used
- add etc/ to WM_PROJECT_SITE search. This makes the site directory
structure consistent with the OpenFOAM structure.
Eg,
WM_PROJECT_SITE/etc/..
WM_PROJECT_SITE/bin/..
WM_PROJECT_SITE/platforms/..
- Don't set/export WM_OSTYPE. The default is POSIX and is properly
defaulted throughout, including in CMakeLists-OpenFOAM.txt (also for
Catalyst)
- removed reliance on ParaView_INCLUDE_DIR variable for conveying the
major.minor version information when compiling. This can be somewhat
fragile and also adds variable that is an unnecessary when running
(only used when compiling).
Instead use `have_pvplugin_support` function in paraviewFunctions
wmake script to determine the maj.min from the PV_PLUGIN_PATH
since we have already defined the output path there with paraview
maj.min numbering.
Can now build with paraview from the operating system,
provided that it has develop headers available.
ParaView_VERSION=system
In the etc/config.sh/paraview setup, the maj.min is taken from
the corresponding `paraview --version` output and used when
defining the PV_PLUGIN_PATH.
During the build, the include path taken from `paraview-config`
for a system installation, from the guess installation root
of the paraview binary, or ParaView_DIR otherwise.
NB: using a system ParaView for building runTimePostProcessing is unsupported.
- these types of builds appear to have various library resolution issues
(eg, libexpat not being loaded). Additionally, the build logic does
not yet cover this type of use case.
- if FOAM_EXT_LIBBIN is unset and some scripts set this to /usr/lib*
as a fallback (eg, to avoid an undefined value) this will cause a
system library to be found before appropriate *_ARCH_PATH entry.
This was noticed during a scotch compilation without third-party:
resulting in the system library (/usr/lib64/libscotch.so) to be found
instead of the SCOTCH_ARCH_PATH location
(/usr/lib64/mpi/gcc/openmpi/lib64/).
Simply changing the search order doesn't work for use, since we wish
to retain a preference for any dynamic libraries discovered in a
real FOAM_EXT_LIBBIN.
Circumvent these issues by only taking libraries from
FOAM_EXT_LIBBIN if it also points to a location within ThirdParty.
- comments
- avoid egrep for getting processor count.
- wcleanBuild, wcleanPlatform with shorter form '-curr' instead of '-c'
to avoid any potential user confusion with '-comp'
- handling of dead links (find -L -delete unsupported)
- remove ignore case flag on 's/../../i' used in have_scotch script.
It is unneeded and not tolerated by Darwin's sed.
- avoid embedded comments in EXE_INC (Make/options files), which do
not work well with the OSX LLVM cpp.
It strips out the comments but also removes the continuation char.
STYLE: adjust notes about paraview library locations
- generalize some of the library extensions (.so vs .dylib).
Provide as wmake 'sysFunctions'
- added note about unsupported/incomplete system support
- centralize detection of ThirdParty packages into wmake/ subdirectory
by providing a series of scripts in the spirit of GNU autoconfig.
For example,
have_boost, have_readline, have_scotch, ...
Each of the `have_<package>` scripts will generally provide the
following type of functions:
have_<package> # detection
no_<package> # reset
echo_<package> # echoing
and the following type of variables:
HAVE_<package> # unset or 'true'
<package>_ARCH_PATH # root for <package>
<package>_INC_DIR # include directory for <package>
<package>_LIB_DIR # library directory for <package>
This simplifies the calling scripts:
if have_metis
then
wmake metisDecomp
fi
As well as reducing clutter in the corresponding Make/options:
EXE_INC = \
-I$(METIS_INC_DIR) \
-I../decompositionMethods/lnInclude
LIB_LIBS = \
-L$(METIS_LIB_DIR) -lmetis
Any additional modifications (platform-specific or for an external build
system) can now be made centrally.
- However, the new ragel-based parser is much faster
than the others, and does not cause 'too many open files' error
that the flex-based parser does (issue #784).
The timings (using src/sampling as being somewhat representative)
$ wclean; wmakeLnInclude -u .; time wmake -s dep
3.4s wmkdepend (ragel) [now default]
5.7s wmkdep (flex)
6.1s cpp -M
- The makeDepend script is for testing purposes only, but could used as
a hook for other dependency generation systems (eg, ninja).
It simply wraps 'cpp -M' in a form that is calling compatible with
wmkdepend.
BUG: wmkdepend parser was missing optional leading space on #include match
STYLE: use -G2 (goto-based) option for wmkdepend state machine
- the machine is compact with few states and lends itself to this
- unknown options or missing option values now emit a shorter message
without the entire usage. This makes it easier to identify the errors
and is better aligned with the behaviour of GNU system tools.
====
$ simpleFoam -case
Using: OpenFOAM-plus (see www.OpenFOAM.com)
Build: plus-01234
Error: option '-case' requires an argument
See 'simpleFoam -help' for usage
====
- provide for reduced (-help) and full (-help-full) usage information.
In many cases the reduced usage provided sufficient and easier
to find the information.
- make -srcDoc an alias for -doc-source
- no warnings about option aliases for future dates.
- add -compiler=NAME option to remove a build or platforms directory
corresponding to any specified compiler on the current arch.
- when -compiler or -compiler=NAME is specified, also clean related
sub-directories as well. This will cleanup mpi-related directory.
- suppress error messages that appear with zsh.
According to unset(1p), 'unset -f' unsets a function.
If the function was not previously defined, this is a no-op.
This is similar for zsh, but there it emits a warning if the
function was not previously defined.
- avoid 'local' in functions sources from etc/bashrc.
ksh does not support this.
- use 'command' shell builtin instead of 'type'.
Seems to be more consistent between shell flavours.
- handle sourcing bashrc with a relative path (issue #383)
- handle sourcing from bash and zsh.
Still need manual intervention when sourcing dash, sh, or ksh.
- replace grep in etc/cshrc with sed only
- logical instead of physical path for WM_PROJECT_DIR (issue #431).
Doesn't seem to be possible for csh/tcsh.
* Continue using physical locations when comparing directories,
but not for the top-level FOAM_INST_DIR, WM_PROJECT_DIR.
- relocate WM_CC, WM_CXX overrides from etc/config.*/compiler
to etc/config.*/settings to ensure that they are left untouched
when etc/config.sh/compiler is sourced while making third-party
packages (eg, gcc, llvm, CGAL).
- provide fallback FOAM_TUTORIALS setting in RunFunctions
STYLE: remove "~OpenFOAM" fallback as being too rare, non-obvious
- Could be related to interrupted builds.
So if there are any parts of the build that rely on an explicit
'wmakeLnInclude', make sure that the contents are properly updated.
--
ENH: improved feedback from top-level Allwmake
- Report which section (libraries, applications) is being built.
- Provide final summary of date, version, etc, which can be helpful
for later diagnosis or record keeping.
- The -log=XXX option for Allwmake now accepts a directory name
and automatically appends an appropriate log name.
Eg,
./Allwmake -log=logs/ ->> logs/log.linux64GccDPInt32Opt
The default name is built from the value of WM_OPTIONS.
--
BUG: shell not exiting properly in combination with -log option
- the use of 'tee' causes the shell to hang around.
Added an explicit exit to catch this.
--
- Detecting the '-k' (-non-stop) option at the top-level Allwmake, which
may improve robustness.
- Explicit continue-on-error for foamyMesh (as optional component)
- unify format of script messages for better readability
COMP: reduce warnings when building Pstream (old-style casts in openmpi)
and added support for queue scheduling option '-q', '-queue'
Now the 'Allwmake' scripts execute 'wmake -all' to handle parallel
processing in a general way, avoiding code duplication.
Updates lnInclude directories and dep files before compilation. This is
useful to apply following a "git pull" to ensure consistency between the
source files, dep files and links.
