[Unison-hackers] filesystem watching with stable unison and dev version (was: [unison-svn] r506 - in trunk/src: . fsmonitor ...)

[cgat at arcor.de]

Hello unison-hackers!

Am Thu,  9 Aug 2012 10:30:22 -0400
schrieb vouillon at seas.upenn.edu:

> Log:
> * More robust file watching helper programs for Windows and Linux.
>   They communicate with Unison through pipes (Unison redirects stdin
>   and stdout), using a race-free protocol.

Wow, very nice.

Maybe you have already seen my anouncment on unison-users:
A bash script that supports filesystem watch based syncing with the
current stable unison versions, and can provide a system tray icon.
See https://launchpad.net/sucsynct

Of course, I would also like it to support the new "-repeat watch"
option in the future.

While developing the script I had written down some thoughts, that may
be interesting to discuss here:

---
Ideally, event based syncing will only trigger a sync after a 
new change event occured, and each sync should only cover 
those files that have changed. However, two circumstances 
force us to do some additional change detection.

1) The recursive establishment of inotify watches may miss 
file creations in new subdirectories, if files are created 
before a watch could be established on the new subdirectory. 
Consequently, when a new directory is created, we need to 
re-scan the whole sub-tree for changes, instead of just 
relying on the event records. Fortunately, unison can do 
very fast change detection as it can rely on its database 
and the filesystem meta-data (at least on non-windows 
filesystems).

2) An external program, like sucsynct, can not identify 
the new changes that occur while a sync is in progress, 
without extensive logfile parsing. There is thus a 
potenial risk to miss new changes made during syncs. 
Consequently, external programms usually do periodic 
syncs in short intervalls (polling), to minimize the 
delay in change propagation, and with this the risk 
that conflicting changes may be made concurrently on 
both replicas.

Because the whole point of event based syncing is to 
avoid needless periodic syncing, some additional attention
was necessary to solve 2) in a way that can truly suspend 
syncing whithout risking to miss a change.

The basic concept taken here is to suspended syncing 
only after a waker could be set up successfully, that is 
without new changes occuring in the meantime, and to 
verify this with one extra syncing cycle.

What about unison's upcoming "-repeat watch" 
fsmonitor.py feature? Issue 1 probably applies as well. 
For 2, unison could filter the first and second occurence 
of its transfers from the event log an be very carefull 
with the timing, to avoid a full verification sync.
Unison may further keep its data in ram in between syncs, 
and avoid continous periodic polling of the event file. 
It could suspend until triggered to read the event file 
again, once it finds no new events in the event buffer.
(Yet, similarly to sucsynct, unison will probably also 
have to ensure that the separate waker pipe for the event 
buffer could be set up without missing events, before 
blocking on reading the pipe.)
---


Kind Regards,
Christian