[Unison-hackers] Update detection optimization

[Jerome Vouillon]

On Thu, Jul 16, 2009 at 03:37:49PM -0400, Benjamin Pierce wrote:
> > * Experimental update detection optimization:
> >  do not read the contents of unchanged directories
> 
> Nice idea!
> 
> Just to make sure I understand... If a child of a directory is also a  
> directory, then the fast check is disabled for the parent?

No, it is not disabled.  Unison still has to scan recursively the
children of a directory (files included) for correctness.  But it gets
the list of children from the archive.  It does not read the directory
contents, reorder it, check that there is not bad filenames or
duplicated filenames and finally match the directory contents with the
archive contents.  So, one gets a significant speed-up even in this
case.

The speed-up is most dramatic with immutable directories, as the
system does not have to read their contents from disk at all.

For this to work, one has to make sure that the archive contents
reflects the directory contents, so a directory is marked as unchanged
only when it has no new or deleted childs.  Also, the optimization has
to be disabled whenever a preference (follow, ignore, ...) that may
affect Unison view of the filesystem is changed.  So, Unison keeps
track of these preferences in the archive.

I have the following (approximate) numbers for my home directory.
"Cold cache" means when everything has to be read from the disk, while
"hot cache" means that everything is already cached in memory and
there is no disk access.  These number are for fully synchronized
replicas, when the optimization is most effective.  Also, I have large
maildir directories synchronized with the "immutable" preference.
Finally, this is with xferbycopying disabled, otherwise Unison takes
70% of its time filling the hash tables :-(.

Laptop with SSD drive
              With optimization   Without optimization   Speed-up
  Cold cache          2.8                  5.8            x 2.0
  Hot cache           0.42                 2.12           x 5.0

Desktop computer with hard drive
              With optimization   Without optimization
  Cold cache         11.4                 19.3            x 1.7
  Hot cache           0.28                 1.32           x 4.7

-- Jerome