[Unison-hackers] unison more memory efficient than rsync?

[Geoffrey Alan Washburn]

Benjamin Pierce wrote:
> I regularly synchronize 120Gb disks with, among other things, lots  
> and lots of mail files.  Update detection is a bit slower than I'd  
> like because of all the inodes to be scanned, but not too bad.   
> Others have reported success with much larger replicas.
> 
> Unison's overall memory usage scales more or less linearly with the  
> number of files -- but with a fairly small constant factor.  Its peak  
> memory usage scales linearly with the size of the largest individual  
> file.  There have been some reports of problems with really gigantic  
> (multi-gig) individual files.

> I don't know whether the current rsync implementation calculates  
> block fingerprints only within each file (as Unison does), or across  
> a whole filesystem.  If it's just per file, then the memory usage may  
> be comparable, since the core algorithms are very similar.

	Well, there is the "rsync transfer algorithm" and there is the rsync 
tool, and I'm pretty sure the problem is in the tool and not the core 
algorithm.  When I use the rsync tool to do a backup, scans the entire 
disk first and then goes about transferring deleting/transferring.  At 
that point the rsync process has begun to consume a few hundred 
megabytes.  As far as I've been able to tell there is no command-line 
switch to make it not scan the entire disk first (for whatever it is 
doing).

	So I guess maybe a better question to ask, is when synchronizing those 
120GB disks, have you observed how much memory unison uses at a maximum?

	The only real problem at the moment is that Apple's version of rsync 
understands extended attributes, and Unison doesn't, but I'm hoping to 
just sit down soon and finish writing my OCaml extended attributes 
library.  At that point I'm hoping someone with more experience with the 
Unison internals might be able to use the library.