http://www.openfoam.com
Copyright (c) 2011 OpenCFD Ltd.

Cleanup of automatic regIOobject rereading.

- all files (usually only IOdictionary) that need to be monitored
should be registered using MUST_READ_IF_MODIFIED. The MUST_READ should
be used for objects that do not need to be re-read (e.g. fields).
In the old system it would actually monitor e.g. 0/U and constant/polyMesh
files.
I've temporarily added a warning in IOdictionary if constructed with MUST_READ.
Same for IOList,IOField,IOMap if constructed with MUST_READ_IF_MODIFIED
(or is rereading supported?). Please let me know if something does not work or
you see the warning
   "Dictionary constructed with IOobject::MUST_READ instead of IOobject::MUST_READ_IF_MODIFIED." << nl


- any monitored and modified file will get reloaded from the exact path
that was monitored. In the old system it would/could do a re-search through all
times.


- all reductions to synchronise status on different processors are done with
a single reduction instead of one reduction per registered object. This could
be quite a gain on large numbers of processors.


- all file monitoring is done by an instance of 'fileMonitor' in the Time
class. The fileMonitor class can be found in OSspecific. It uses either
timestamps as before or the (linux-specific) 'inotify' system framework
(available only if compiled with -DFOAM_USE_INOTIFY).


- the monitoring can be done in one of four modes as set by
    OptimisationSwitches::fileModificationChecking

    - timeStamp : old behaviour : all nodes check the timestamp
    - inotify   : using inotify instead of timestamps
    - timeStampMaster,inotifyMaster : only the master node checks the file
                and only the master node reads it and distribute it to the
                slaves. This makes runTimeModifiable possible on distributed
                running (see below).

- distributed running:
    - set fileModificationChecking to e.g. timeStampMaster
    - decompose a case, e.g. cavity
    - copy system and constant to processor0/
    - put the all the processor* directories on the wanted nodes inside
      the case directory. E.g.
        - on master have /tmp/cavity/processor0
        - on slaveN have /tmp/cavity/processorN
    - so to reiterate:
        - there is no need for cavity/constant or cavity/system, all the
          dictionaries are only in processor0/constant or processor0/system
        - the slave processor directories have no system directory and the
          constant directory only contains the mesh.
    - start the job in distributed mode by specifying the slave roots
      (so one less than the number of processors) with
      the -roots command line option:

        mpirun -np 2 icoFoam -roots '("/tmp")' -parallel

    - the alternative to the -roots option is to have a
      cavity/system/decomposeParDict on the master with
            distributed     yes;
            roots           ("/tmp");


Details:
- timeStampMaster, inotifyMaster : this works only for IOdictionaries that
are READ_IF_MODIFIED. It means that slaves read exactly the same dictionary
as the master so cannot be used for dictionaries that contain e.g. mesh
specific information.

- note: even if the file does not exist (e.g. when timeStampMaster) it
will still register a local file with the fileMonitor. This is so fileMonitor
stays synchronised. So take care when reading/creating non-parallel dictionary.

- inotify is a monitoring framework used to monitor changes in
lots of files (e.g. used in desktop search engines like beagle). You specify
files to monitor and then get warned for any changes to these files.
It does not need timestamps. There is no need for fileModificationSkew
to allow for time differences. (there can still temporarily be a difference
in modified status between different processors due to nfs lagging). The big
problem is that it does not work over nfs3 (not sure about nfs4).

- fileMonitor stores two hashtables per file so there is a small overhead
adding and removing files from monitoring.

- if runTimeModifiable is false at start of run no files will get monitored,
however if runTimeModified gets set to false during the run the files
will still get monitored (though never reloaded). This is only a hypothetical
problem in that the kernel still stores events for the monitored files. However
inotify is very efficient - e.g. it gets used to track changes on file systems
for desktop search engines.

- in the old system one could call modified() on any object and get
and uptodate state. In the new system it will return the state from
the last runTime++ (which if it triggered any re-reads will have reset the
state anyway).