[Unison-hackers] inotify

[Benjamin Pierce]

Mitch,

I'm delighted to hear that someone is interested in taking this on!   
It's a feature that I've wanted forever.

I believe that, for the sake of portability, it's best to keep the  
inotify part separate from Unison itself -- i.e., rather than making  
Unison call inotify directly, there should be a little external  
utility program that calls inotify, finds out what files are  
changing, and tells Unison about them in some simple, portable way.

Roughly:

    * On each host, there is a filesystem watcher program running; it  
uses inotify (or whatever other library is appropriate) to find out  
what's changing and, whenever it sees a change, it writes a line  
containing the full pathname of this file to some file called, say,  
CHANGES.

    * When started with a special flag (e.g., "-watch CHANGES"),  
Unison will
        - read CHANGES on both hosts to find out what paths might be  
modified
        - synchronize these paths (i.e., behave as if it had been  
started with command-line arguments "-path P" for each P listed in  
CHANGES)
        - repeat

Of course, there are plenty of details that need to be fleshed out  
here.  E.g.
    - how do we lock CHANGES properly, so that Unison doesn't read  
partially-written data from the watcher?
    - how do we prevent CHANGES from growing without bound?
    - what should happen in case Unison discovered conflicts or  
encounters other sorts of errors?
    - Can we use the standard text UI (in -batch mode, of course), or  
does this mode of usage require something more special?

Does that sound generally reasonable?

Regards,

     - Benjamin




On Oct 1, 2006, at 2:15 PM, Mitch Skinner wrote:

> Hello,
>
> I've started working on inotify support.
>
> For those not familiar with inotify, it's a Linux feature that allows
> programs to watch directories for changes.  Similar functionality  
> exists
> on other platforms; on windows there's ReadDirChangesW, on BSD
> apparently you can use kqueues, I'm sure there's something on OS X  
> but I
> don't know what it is.  So far what I have is linux-specific; if  
> people
> want to do the work for other platforms then maybe we could design  
> some
> kind of generic interface, but that sounds like a lot of work so  
> for the
> first iteration I'd like to just focus on the platform I have handy.
>
> So far, all I've got is a bare-bones O'Caml wrapper for the inotify C
> interface.  That appears to be working well, but now I'm trying to
> figure out how to integrate it with unison, and I have to admit I'm a
> bit stumped.
>
> As I see it, two main changes need to be made:
> 1. The code that walks the directories needs to register the  
> directories
> of interest for watching.  I believe this needs to happen before  
> update
> detection, to avoid missing changes.  My current best guess is that  
> this
> belongs somewhere in update.ml (findLocal? buildUpdate*?).
> 2. There needs to be some kind of driver that processes the change
> notifications from inotify.  Inotify messages come from the kernel  
> via a
> file descriptor; you select() on it or do a blocking read.  I think
> there's a way to specify a file descriptor for the GTK mainloop to
> monitor but I don't know the details.  Alternatively, maybe we could
> have a separate thread do blocking reads and make the results  
> available
> through some kind of queue?  I'm an O'Caml newbie, so I'd love some
> advice there.
>
> #2 is really the meat of the whole thing.  I'm willing to put in the
> elbow grease to see it through (to whatever standards y'all want), but
> I'd like to get some advice on how to approach it.  Should it sync
> changes periodically, or as they happen?  How should the UI work?
>
> I've put up my inotify O'Caml wrapper here, in case anyone is curious
> about the details of how inotify works:
> http://arctur.us/unison/
>
> I've pasted in a transcript below showing how the inotify test program
> works.  Ctrl-C exits.
>
> Thanks for any advice, I'm a newbie to almost all of this,
> Mitch
>
> [mitch at firebolt inotify_caml]$ ls
> compile         inotify.h   inotify.mli         inotify_test.ml
> inotify_caml.c  inotify.ml  inotify-syscalls.h
> [mitch at firebolt inotify_caml]$ . compile
> In file included from inotify_caml.c:5:
> inotify-syscalls.h: In function ‘inotify_init’:
> inotify-syscalls.h:48: warning: implicit declaration of function  
> ‘syscall’
> [mitch at firebolt inotify_caml]$ mkdir testdir
> [mitch at firebolt inotify_caml]$ sleep 10 && touch testdir/foo &
> [1] 20884
> [mitch at firebolt inotify_caml]$ ./inotify_test testdir/
> **Events received** (32 bytes)
> Event:
>   Watch descriptor: 1
>   Mask: IN_CREATE
>   Cookie: 0
>   Name: foo
>
> **Events received** (32 bytes)
> Event:
>   Watch descriptor: 1
>   Mask: IN_OPEN
>   Cookie: 0
>   Name: foo
>
> **Events received** (32 bytes)
> Event:
>   Watch descriptor: 1
>   Mask: IN_CLOSE_WRITE, IN_CLOSE
>   Cookie: 0
>   Name: foo
>
> **Events received** (32 bytes)
> Event:
>   Watch descriptor: 1
>   Mask: IN_ATTRIB
>   Cookie: 0
>   Name: foo
>
>
> [1]+  Done                    sleep 10 && touch testdir/foo
> [mitch at firebolt inotify_caml]$
>
>
> _______________________________________________
> Unison-hackers mailing list
> Unison-hackers at lists.seas.upenn.edu
> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers
>