From lists at arctur.us Sun Oct 1 14:15:30 2006 From: lists at arctur.us (Mitch Skinner) Date: Sun, 01 Oct 2006 11:15:30 -0700 Subject: [Unison-hackers] inotify Message-ID: <1159726530.9665.49.camel@firebolt> Hello, I've started working on inotify support. For those not familiar with inotify, it's a Linux feature that allows programs to watch directories for changes. Similar functionality exists on other platforms; on windows there's ReadDirChangesW, on BSD apparently you can use kqueues, I'm sure there's something on OS X but I don't know what it is. So far what I have is linux-specific; if people want to do the work for other platforms then maybe we could design some kind of generic interface, but that sounds like a lot of work so for the first iteration I'd like to just focus on the platform I have handy. So far, all I've got is a bare-bones O'Caml wrapper for the inotify C interface. That appears to be working well, but now I'm trying to figure out how to integrate it with unison, and I have to admit I'm a bit stumped. As I see it, two main changes need to be made: 1. The code that walks the directories needs to register the directories of interest for watching. I believe this needs to happen before update detection, to avoid missing changes. My current best guess is that this belongs somewhere in update.ml (findLocal? buildUpdate*?). 2. There needs to be some kind of driver that processes the change notifications from inotify. Inotify messages come from the kernel via a file descriptor; you select() on it or do a blocking read. I think there's a way to specify a file descriptor for the GTK mainloop to monitor but I don't know the details. Alternatively, maybe we could have a separate thread do blocking reads and make the results available through some kind of queue? I'm an O'Caml newbie, so I'd love some advice there. #2 is really the meat of the whole thing. I'm willing to put in the elbow grease to see it through (to whatever standards y'all want), but I'd like to get some advice on how to approach it. Should it sync changes periodically, or as they happen? How should the UI work? I've put up my inotify O'Caml wrapper here, in case anyone is curious about the details of how inotify works: http://arctur.us/unison/ I've pasted in a transcript below showing how the inotify test program works. Ctrl-C exits. Thanks for any advice, I'm a newbie to almost all of this, Mitch [mitch at firebolt inotify_caml]$ ls compile inotify.h inotify.mli inotify_test.ml inotify_caml.c inotify.ml inotify-syscalls.h [mitch at firebolt inotify_caml]$ . compile In file included from inotify_caml.c:5: inotify-syscalls.h: In function ?inotify_init?: inotify-syscalls.h:48: warning: implicit declaration of function ?syscall? [mitch at firebolt inotify_caml]$ mkdir testdir [mitch at firebolt inotify_caml]$ sleep 10 && touch testdir/foo & [1] 20884 [mitch at firebolt inotify_caml]$ ./inotify_test testdir/ **Events received** (32 bytes) Event: Watch descriptor: 1 Mask: IN_CREATE Cookie: 0 Name: foo **Events received** (32 bytes) Event: Watch descriptor: 1 Mask: IN_OPEN Cookie: 0 Name: foo **Events received** (32 bytes) Event: Watch descriptor: 1 Mask: IN_CLOSE_WRITE, IN_CLOSE Cookie: 0 Name: foo **Events received** (32 bytes) Event: Watch descriptor: 1 Mask: IN_ATTRIB Cookie: 0 Name: foo [1]+ Done sleep 10 && touch testdir/foo [mitch at firebolt inotify_caml]$ From bcpierce at cis.upenn.edu Wed Oct 4 10:59:56 2006 From: bcpierce at cis.upenn.edu (Benjamin Pierce) Date: Wed, 4 Oct 2006 10:59:56 -0400 Subject: [Unison-hackers] inotify In-Reply-To: <1159726530.9665.49.camel@firebolt> References: <1159726530.9665.49.camel@firebolt> Message-ID: <7636E9FE-236D-4B32-86C9-FA9DCC0DA3D0@cis.upenn.edu> Mitch, I'm delighted to hear that someone is interested in taking this on! It's a feature that I've wanted forever. I believe that, for the sake of portability, it's best to keep the inotify part separate from Unison itself -- i.e., rather than making Unison call inotify directly, there should be a little external utility program that calls inotify, finds out what files are changing, and tells Unison about them in some simple, portable way. Roughly: * On each host, there is a filesystem watcher program running; it uses inotify (or whatever other library is appropriate) to find out what's changing and, whenever it sees a change, it writes a line containing the full pathname of this file to some file called, say, CHANGES. * When started with a special flag (e.g., "-watch CHANGES"), Unison will - read CHANGES on both hosts to find out what paths might be modified - synchronize these paths (i.e., behave as if it had been started with command-line arguments "-path P" for each P listed in CHANGES) - repeat Of course, there are plenty of details that need to be fleshed out here. E.g. - how do we lock CHANGES properly, so that Unison doesn't read partially-written data from the watcher? - how do we prevent CHANGES from growing without bound? - what should happen in case Unison discovered conflicts or encounters other sorts of errors? - Can we use the standard text UI (in -batch mode, of course), or does this mode of usage require something more special? Does that sound generally reasonable? Regards, - Benjamin On Oct 1, 2006, at 2:15 PM, Mitch Skinner wrote: > Hello, > > I've started working on inotify support. > > For those not familiar with inotify, it's a Linux feature that allows > programs to watch directories for changes. Similar functionality > exists > on other platforms; on windows there's ReadDirChangesW, on BSD > apparently you can use kqueues, I'm sure there's something on OS X > but I > don't know what it is. So far what I have is linux-specific; if > people > want to do the work for other platforms then maybe we could design > some > kind of generic interface, but that sounds like a lot of work so > for the > first iteration I'd like to just focus on the platform I have handy. > > So far, all I've got is a bare-bones O'Caml wrapper for the inotify C > interface. That appears to be working well, but now I'm trying to > figure out how to integrate it with unison, and I have to admit I'm a > bit stumped. > > As I see it, two main changes need to be made: > 1. The code that walks the directories needs to register the > directories > of interest for watching. I believe this needs to happen before > update > detection, to avoid missing changes. My current best guess is that > this > belongs somewhere in update.ml (findLocal? buildUpdate*?). > 2. There needs to be some kind of driver that processes the change > notifications from inotify. Inotify messages come from the kernel > via a > file descriptor; you select() on it or do a blocking read. I think > there's a way to specify a file descriptor for the GTK mainloop to > monitor but I don't know the details. Alternatively, maybe we could > have a separate thread do blocking reads and make the results > available > through some kind of queue? I'm an O'Caml newbie, so I'd love some > advice there. > > #2 is really the meat of the whole thing. I'm willing to put in the > elbow grease to see it through (to whatever standards y'all want), but > I'd like to get some advice on how to approach it. Should it sync > changes periodically, or as they happen? How should the UI work? > > I've put up my inotify O'Caml wrapper here, in case anyone is curious > about the details of how inotify works: > http://arctur.us/unison/ > > I've pasted in a transcript below showing how the inotify test program > works. Ctrl-C exits. > > Thanks for any advice, I'm a newbie to almost all of this, > Mitch > > [mitch at firebolt inotify_caml]$ ls > compile inotify.h inotify.mli inotify_test.ml > inotify_caml.c inotify.ml inotify-syscalls.h > [mitch at firebolt inotify_caml]$ . compile > In file included from inotify_caml.c:5: > inotify-syscalls.h: In function ?inotify_init?: > inotify-syscalls.h:48: warning: implicit declaration of function > ?syscall? > [mitch at firebolt inotify_caml]$ mkdir testdir > [mitch at firebolt inotify_caml]$ sleep 10 && touch testdir/foo & > [1] 20884 > [mitch at firebolt inotify_caml]$ ./inotify_test testdir/ > **Events received** (32 bytes) > Event: > Watch descriptor: 1 > Mask: IN_CREATE > Cookie: 0 > Name: foo > > **Events received** (32 bytes) > Event: > Watch descriptor: 1 > Mask: IN_OPEN > Cookie: 0 > Name: foo > > **Events received** (32 bytes) > Event: > Watch descriptor: 1 > Mask: IN_CLOSE_WRITE, IN_CLOSE > Cookie: 0 > Name: foo > > **Events received** (32 bytes) > Event: > Watch descriptor: 1 > Mask: IN_ATTRIB > Cookie: 0 > Name: foo > > > [1]+ Done sleep 10 && touch testdir/foo > [mitch at firebolt inotify_caml]$ > > > _______________________________________________ > Unison-hackers mailing list > Unison-hackers at lists.seas.upenn.edu > http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers > From gildor at debian.org Wed Oct 4 18:29:25 2006 From: gildor at debian.org (Sylvain Le Gall) Date: Wed, 4 Oct 2006 22:29:25 +0000 (UTC) Subject: [Unison-hackers] Upgrade the unison stable version Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I am the debian maintainer of unison. I have a grave bug concerning unison (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=349674). I think the bug is solved, since the bug reporter also submit the bug to the unison mailing list. But, in order to be able to close it, i need a newer version of unison. But i only package unison stable version... So, here is my question : Is a stable release planned ? Debian is in the process of releasing a new major version (etch) by december. It would be great to have a unison version in it (the actual version is 2.9.1, the latest stable without grave bug...). The freeze will happen somewhere in the middle of october. If a unison version is promoted stable before this date, it will be great ! Kind regard && thanks for unison Sylvain Le Gall -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) iD8DBQFFJDXrir2bofsN/psRAi8kAKCOHL7ygFOxzv2PwiN4llWpqUyJ7gCgjZTv evI+3/vWchgQCSQeubfQiIw= =TI8a -----END PGP SIGNATURE----- From bcpierce at cis.upenn.edu Wed Oct 4 21:40:49 2006 From: bcpierce at cis.upenn.edu (Benjamin Pierce) Date: Wed, 4 Oct 2006 21:40:49 -0400 Subject: [Unison-hackers] Upgrade the unison stable version In-Reply-To: References: Message-ID: <0287292E-7D8D-49D6-ABC0-AED1176A222F@cis.upenn.edu> > I am the debian maintainer of unison. I have a grave bug concerning > unison (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=349674). I > think the bug is solved, since the bug reporter also submit the bug to > the unison mailing list. The flaw reported in this message is present in *all* versions of Unison. A design for a fix has been discussed, but it has not been implemented yet. > Debian is in the process of releasing a new major version (etch) by > december. It would be great to have a unison version in it (the actual > version is 2.9.1, the latest stable without grave bug...). It would be great to have Debian distributing a version newer than 2.9.1, which is really old now! However... > The freeze will happen somewhere in the middle of october. If a unison > version is promoted stable before this date, it will be great ! ... middle of October is really soon. I don't know if it's possible to get the current development version polished up enough to become not just a beta but a new stable release in that timeframe. - Benjamin > > Kind regard && thanks for unison > Sylvain Le Gall > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.5 (GNU/Linux) > > iD8DBQFFJDXrir2bofsN/psRAi8kAKCOHL7ygFOxzv2PwiN4llWpqUyJ7gCgjZTv > evI+3/vWchgQCSQeubfQiIw= > =TI8a > -----END PGP SIGNATURE----- > > _______________________________________________ > Unison-hackers mailing list > Unison-hackers at lists.seas.upenn.edu > http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers From gildor at debian.org Thu Oct 5 17:38:54 2006 From: gildor at debian.org (Sylvain Le Gall) Date: Thu, 5 Oct 2006 21:38:54 +0000 (UTC) Subject: [Unison-hackers] Upgrade the unison stable version References: <0287292E-7D8D-49D6-ABC0-AED1176A222F@cis.upenn.edu> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, On 05-10-2006, Benjamin Pierce wrote: >> I am the debian maintainer of unison. I have a grave bug concerning >> unison (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=349674). I >> think the bug is solved, since the bug reporter also submit the bug to >> the unison mailing list. > > The flaw reported in this message is present in *all* versions of > Unison. A design for a fix has been discussed, but it has not been > implemented yet. > I know... but the bug is on the unison package, if i duplicate it to unison2.9.1 it will block the two version... This is what i don't want. >> Debian is in the process of releasing a new major version (etch) by >> december. It would be great to have a unison version in it (the actual >> version is 2.9.1, the latest stable without grave bug...). > > It would be great to have Debian distributing a version newer than > 2.9.1, which is really old now! However... > >> The freeze will happen somewhere in the middle of october. If a unison >> version is promoted stable before this date, it will be great ! > > ... middle of October is really soon. I don't know if it's possible > to get the current development version polished up enough to become > not just a beta but a new stable release in that timeframe. > OK, i know this will be your answer ;-) Do you have an idea for a quick fix, for the 2.13.16 version ? Thanks Kind regard Sylvain Le Gall -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) iD8DBQFFJXuTir2bofsN/psRAmdyAJsHAvDHo0bL4We533x/7I5upEQeDQCfS1I6 KIZNXyeL5fy2PhyJEPIy5KI= =LgP5 -----END PGP SIGNATURE----- From bcpierce at cis.upenn.edu Thu Oct 5 23:30:37 2006 From: bcpierce at cis.upenn.edu (Benjamin Pierce) Date: Thu, 5 Oct 2006 23:30:37 -0400 Subject: [Unison-hackers] Upgrade the unison stable version In-Reply-To: References: <0287292E-7D8D-49D6-ABC0-AED1176A222F@cis.upenn.edu> Message-ID: >>> The freeze will happen somewhere in the middle of october. If a >>> unison >>> version is promoted stable before this date, it will be great ! >> >> ... middle of October is really soon. I don't know if it's possible >> to get the current development version polished up enough to become >> not just a beta but a new stable release in that timeframe. >> > > OK, i know this will be your answer ;-) > > Do you have an idea for a quick fix, for the 2.13.16 version ? Yes: Document the fact that Unison is not safe to use with removable media. Regards, - Benjamin From gildor at debian.org Fri Oct 6 02:21:21 2006 From: gildor at debian.org (Sylvain Le Gall) Date: Fri, 6 Oct 2006 06:21:21 +0000 (UTC) Subject: [Unison-hackers] Upgrade the unison stable version References: <0287292E-7D8D-49D6-ABC0-AED1176A222F@cis.upenn.edu> Message-ID: Hello, On 06-10-2006, Benjamin Pierce wrote: >>>> The freeze will happen somewhere in the middle of october. If a >>>> unison >>>> version is promoted stable before this date, it will be great ! >>> >>> ... middle of October is really soon. I don't know if it's possible >>> to get the current development version polished up enough to become >>> not just a beta but a new stable release in that timeframe. >>> >> >> OK, i know this will be your answer ;-) >> >> Do you have an idea for a quick fix, for the 2.13.16 version ? > > Yes: Document the fact that Unison is not safe to use with removable > media. > Well, i could do that, but it is not really fair regarding the bug... If it is a question of time for working on the issue, i think i don't made myself clear enough: i am not asking for a new version (because you told me that it was not possible). I just need some information/idea in order for me to fix this. If unison just crash when encounting the error on Unix/Linux, it will be enough for me... So if you can just give me some idea, i could try to fix this myself this week-end. Well, if you don't have any idea, i will have to find another solution myself (but it won't be as good as your). Regard Sylvain Le Gall From ralph-lehmann at gmx.net Fri Oct 6 03:41:20 2006 From: ralph-lehmann at gmx.net (Ralph Lehmann) Date: Fri, 06 Oct 2006 09:41:20 +0200 Subject: [Unison-hackers] Upgrade the unison stable version In-Reply-To: References: <0287292E-7D8D-49D6-ABC0-AED1176A222F@cis.upenn.edu> Message-ID: <452608A0.70306@gmx.net> Benjamin Pierce schrieb: >>>> The freeze will happen somewhere in the middle of october. If a >>>> unison >>>> version is promoted stable before this date, it will be great ! >>> >>> ... middle of October is really soon. I don't know if it's possible >>> to get the current development version polished up enough to become >>> not just a beta but a new stable release in that timeframe. >>> >> >> OK, i know this will be your answer ;-) >> >> Do you have an idea for a quick fix, for the 2.13.16 version ? > > Yes: Document the fact that Unison is not safe to use with removable > media. Another suggestion: Document the fact that Unison is not safe to use with removable media without a backup preference. And/Or: Recommend, that the user should place the unison directory (with his archive files) on the removable disk itself. ciao Ralph From bcpierce at cis.upenn.edu Fri Oct 6 08:39:37 2006 From: bcpierce at cis.upenn.edu (Benjamin Pierce) Date: Fri, 6 Oct 2006 08:39:37 -0400 Subject: [Unison-hackers] Upgrade the unison stable version In-Reply-To: References: <0287292E-7D8D-49D6-ABC0-AED1176A222F@cis.upenn.edu> Message-ID: <013A6354-91E0-4E20-A60D-313735CD1BB5@cis.upenn.edu> >>> Do you have an idea for a quick fix, for the 2.13.16 version ? >> >> Yes: Document the fact that Unison is not safe to use with removable >> media. >> > > Well, i could do that, but it is not really fair regarding the bug... I don't understand why you say that. Unison is behaving exactly as it is designed to behave and exactly as its documentation specifies. The issue here is that, when a piece of removable storage goes offline, the filesystem will tell Unison that all the files on it are gone, and Unison will *correctly* (though, I agree, surprisingly) delete all of them from the other replica. > If it is a question of time for working on the issue, i think i don't > made myself clear enough: i am not asking for a new version > (because you > told me that it was not possible). I just need some information/ > idea in > order for me to fix this. If unison just crash when encounting the > error on Unix/Linux, it will be enough for me... There was extensive discussion of possible solutions on this list a few weeks ago. The final proposed design was to add a preference for specifying a "mount point" within the replicas (the default would be just the empty path) that is expected always to exist on both hosts; before any changes are transferred, this will be checked, and the run will be aborted if it is missing. Implementation of this idea should be pretty easy. Regards, - Benjamin From bcpierce at cis.upenn.edu Fri Oct 6 08:41:30 2006 From: bcpierce at cis.upenn.edu (Benjamin Pierce) Date: Fri, 6 Oct 2006 08:41:30 -0400 Subject: [Unison-hackers] Upgrade the unison stable version In-Reply-To: <4525F4E4.6040801@web.de> References: <0287292E-7D8D-49D6-ABC0-AED1176A222F@cis.upenn.edu> <4525F4E4.6040801@web.de> Message-ID: <45C96248-15F2-4AA0-9231-A3A7E2F49111@cis.upenn.edu> > Another suggestion: Document the fact that Unison is not safe to use > with removable media without a backup preference. And/Or: Recommend, > that the user should place the unison directory (with his archive > files) > on the removable disk itself. This is not a sufficient fix: If the removable device goes offline *while* Unison is checking for updates (but after it has read the archive), unfortunate behavior will still result. - Benjamin From bcpierce at cis.upenn.edu Fri Oct 6 14:16:23 2006 From: bcpierce at cis.upenn.edu (Benjamin Pierce) Date: Fri, 6 Oct 2006 14:16:23 -0400 Subject: [Unison-hackers] Upgrade the unison stable version In-Reply-To: <45266CFD.8080403@web.de> References: <0287292E-7D8D-49D6-ABC0-AED1176A222F@cis.upenn.edu> <4525F4E4.6040801@web.de> <45C96248-15F2-4AA0-9231-A3A7E2F49111@cis.upenn.edu> <45266CFD.8080403@web.de> Message-ID: >>> Another suggestion: Document the fact that Unison is not safe to use >>> with removable media without a backup preference. And/Or: Recommend, >>> that the user should place the unison directory (with his archive >>> files) >>> on the removable disk itself. >> >> This is not a sufficient fix: If the removable device goes offline >> *while* Unison is checking for updates (but after it has read the >> archive), unfortunate behavior will still result. > > Your're right. :-( > > Then, consider to check the existence of archive files again _after_ > looking for updates. This should works, if these files are stored > on the > USB-device, and it's should not to hard to implement. Yes, this would be more effective. But it depends on putting the archives on the removable device. I like the other proposed fix better. - Benjamin From gildor at debian.org Sat Oct 7 03:58:35 2006 From: gildor at debian.org (Sylvain Le Gall) Date: Sat, 7 Oct 2006 07:58:35 +0000 (UTC) Subject: [Unison-hackers] Upgrade the unison stable version References: <0287292E-7D8D-49D6-ABC0-AED1176A222F@cis.upenn.edu> <013A6354-91E0-4E20-A60D-313735CD1BB5@cis.upenn.edu> Message-ID: Hello, On 06-10-2006, Benjamin Pierce wrote: >>>> Do you have an idea for a quick fix, for the 2.13.16 version ? >>> >>> Yes: Document the fact that Unison is not safe to use with removable >>> media. >>> >> >> Well, i could do that, but it is not really fair regarding the bug... > > I don't understand why you say that. Unison is behaving exactly as > it is designed to behave and exactly as its documentation specifies. > The issue here is that, when a piece of removable storage goes > offline, the filesystem will tell Unison that all the files on it are > gone, and Unison will *correctly* (though, I agree, surprisingly) > delete all of them from the other replica. > Indeed, you are right, i am not saying that anything is wrong regarding unison, i just try to say that it won't really solve the problem (even if the problem is a correct behavior). >> If it is a question of time for working on the issue, i think i don't >> made myself clear enough: i am not asking for a new version >> (because you >> told me that it was not possible). I just need some information/ >> idea in >> order for me to fix this. If unison just crash when encounting the >> error on Unix/Linux, it will be enough for me... > > There was extensive discussion of possible solutions on this list a > few weeks ago. The final proposed design was to add a preference for > specifying a "mount point" within the replicas (the default would be > just the empty path) that is expected always to exist on both hosts; > before any changes are transferred, this will be checked, and the run > will be aborted if it is missing. Implementation of this idea should > be pretty easy. > OK, i go through the archive and try to make a plan to solve this bug. Here it is : - add an option to check presence of files/directories in the two root in the configuration file + command line, - if a file is detected as deleted, check the presence of files/directories, to be sure the deletion doesn't come from a media disappearance, - if one of the files/directories is not here -> abort, - otherwise, continue, There was also another approach where while scanning, you try to detect the device number and remember it... Maybe it could also be used, as a way to generate the files/directories option automatically ? I think that unison open a dir before scanning the file, which could be used to check the device number. So if a file under a particular mount point need to be deleted, unison can check that the media is not gone. If all the file are on the same mount point, it will default to the root dir. So, i need to know what option name you want ? If i produce a patch, do you think you should integrate it in unison ? Any comments, suggestions ? Regard Sylvain Le Gall ps: i hope you don't get upset about this, i really appreciate your work on unison, which is a great software -- hope the content of my mail doesn't suggest anything else.