[Unison-hackers] Sneaker Net or Incremental Backup

Sat Jan 10 17:05:51 EST 2009

An embedded and charset-unspecified text was scrubbed...
Name: warning1.txt
Url: http://lists.seas.upenn.edu/pipermail/unison-hackers/attachments/20090110/030c3086/warning1.txt
-------------- next part --------------
 > That's a nice hack -- I had no idea the new copyprog functionality
 > could be used that way.
That is what I love about linux/unix.  Lots of small programs that can 
be combined to do great things.

Didn't have quite as much free time as I though, but here are he results 
of my initial testing.  It mostly works, and for now works well enough 
for me.  I can now cron my syncs and not have to worry that some large 
file is going to kill our pipe for a few days :)  I'll report back again 
in a month or so, or sooner if I run into an issue and have to change 
something.

I have attached the shell script and the profile I used for testing. 
The important parts are:
copyprog      =   sneakernet.sh /home/Duane/test 
duane at 192.168.113.2:/home/duane/test /home/Duane/usbdrive
copyprogrest  =   sneakernet.sh

This replaces what would normally be rsync with the attached shell script.

I have only done limited testing so far, but I don't see anything that 
could be harmful.

To be useful you will need 2 profiles a normal and a sneakernet.  In 
your normal profile you should have
copyprog      =   rsync --inplace --compress --max-size=xxx
copyprogrest  =   rsync --partial --inplace --compress --max-size=xxx

Where xxx is the size you have determined is too much to transfer via 
network. For me it is 1073741824 = 1GB

The sneakernet profile should be identical except for the copyprog args
copyprog      =   sneakernet.sh LocalRoot RemoteRoot SneakerNetPath
copyprogrest  =   sneakernet.sh
LocalRoot is the path on the local machine you want to sync
RemoteRoot is not formatted the same as unison, it is formatted as 
passed by unison to rsync
SneakerNetPath is the path to a directory on some removable media

so now you will have 2 command to get everything synched
unison myprofile
unison myprofilesneakernet

Issues:
On copying from the modified machine, unison complains that the copy did 
not succeed and moves on, so only one file for each directory will be 
copied to the usb drive in a directory that is out of sync.  This is not 
a big deal for me, but it may be for others.  If it turns out to be a 
problem I'll probably dig through the source and come up with a real 
solution rather than this hack/workaround.

I have not done any testing with copyprogrest.  Since we are using local 
disks at all times, I don't know if this will be needed.

bcpierce at seas.upenn.edu wrote:
> That's a nice hack -- I had no idea the new copyprog functionality  
> could be used that way.  Will be interested to hear whether it works...
> 
>      - Benjamin
> 
> Quoting Duane McKinney <duane.mckinney at gmail.com>:
> 
>> Or from Version 2.31.4
>> copyprog xxx
>>      A string giving the name of an external program that can be used to
>> copy large files efficiently (plus command-line switches telling it to
>> copy files in-place). The default setting invokes rsync with appropriate
>> options?most users should not need to change it.
>>
>> copyprogrest xxx
>>      A variant of copyprog that names an external program that should be
>> used to continue the transfer of a large file that has already been
>> partially transferred. Typically, copyprogrest will just be copyprog
>> with one extra option (e.g., ?partial, for rsync). The default setting
>> invokes rsync with appropriate options?most users should not need to
>> change it.
>>
>> copythreshold n
>>      A number indicating above what filesize (in kilobytes) Unison
>> should use the external copying utility specified by copyprog.
>> Specifying 0 will cause all copies to use the external program; a
>> negative number will prevent any files from using it. The default is -1.
>> See the Making Unison Faster on Large Files section for more information.
>>
>> I guess I should have checked out the Beta Documentation 1st.  I'll try
>> messing around with using a shell script as the copyprog.  I would
>> assume that it would work just fine.  My plan is to write a shell script
>> which ignores the destination unison tells it, and instead copies it to
>> a usb drive.
>>
>> I'll probably report back on this in a week or so.  Currently I am
>> running 2.27, so I have some compiling and testing to do.
>>
>> Duane McKinney wrote:
>>> What about a flag that tells it to instead of executing the copies, just
>>> prints out a list of the files that would be copied?  A proper name is
>>> escaping me at the moment, but it is a common option on programs.
>>>
>>> Benjamin Pierce wrote:
>>>> Both of these features would be easy to implement using information
>>>> that Unison already has available.  If you want to give it a try, I
>>>> can tell you where to start.  (I'd be reluctant, though, to add this
>>>> code to the main sources -- Unison arguably has too many flags and
>>>> switches already!)
>>>>
>>>> Best,
>>>>
>>>>      - Benjamin
>>>>
>>>> On Dec 27, 2008, at 10:27 AM, Duane McKinney wrote:
>>>>
>>>>> I searched, but came up empty.  Can this be done.
>>>>> I sync two offices using unison.
>>>>> 1) (Optional)I would like to be able to set a preference that says,
>>>>> don't try to
>>>>> sync a file if it would require X bytes transferred
>>>>> 2) Get a list of changed files from a root.  That way I can copy the
>>>>> files to a
>>>>> removable drive and sync them when I get to the other office.
>>>>>
>>>>> Here is my reasoning.  Most of the time, root changes are very
>>>>> small, a few MB.
>>>>> But Let's say that I download a new CENTOS release and place it on
>>>>> the file
>>>>> server.  This may be a poor example, because I could retrieve it
>>>>> over the
>>>>> internet again, but just bear with me.  I would like for unison, to
>>>>> instead of
>>>>> trying to sync this file over the network to skip it.  The I would
>>>>> like to be
>>>>> able to say once a week, run a job, that would take the differences
>>>>> in the root
>>>>> and sent them to a USB drive.  I can then carry this drive to
>>>>> location 2, and
>>>>> update the root there.   Then from my understanding, the network
>>>>> sync would
>>>>> detect that both roots are identical.
>>>>>
>>>>> That way, our bandwidth isn't being eaten up for hours/days, trying
>>>>> to perform a
>>>>> sync that will most likely fail because it will take so long.
>>>>>
>>>>> Also, this would be more useful, than synchronizing the whole root
>>>>> to a usb
>>>>> drive, because, the total of the data that I am synchronizing is >
>>>>> 1TB.  I would
>>>>> not mind moving a few GB (<100) via USB.  I am trying to avoid
>>>>> needing either, a
>>>>> lot of time, or a bunch of USB drives (one for each root).
>>>>>
>>>>> I have not yet looked at the source, but I would assume that most of
>>>>> the items
>>>>> required for this feature are already in place.  Is it feasible?  Is
>>>>> there
>>>>> already a way?
>>>>>
>>>>> _______________________________________________
>>>>> Unison-hackers mailing list
>>>>> Unison-hackers at lists.seas.upenn.edu
>>>>> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers
>> _______________________________________________
>> Unison-hackers mailing list
>> Unison-hackers at lists.seas.upenn.edu
>> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers
>>
>>
> 
> 
> _______________________________________________
> Unison-hackers mailing list
> Unison-hackers at lists.seas.upenn.edu
> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.prf
Type: application/pics-rules
Size: 535 bytes
Desc: not available
Url : http://lists.seas.upenn.edu/pipermail/unison-hackers/attachments/20090110/030c3086/test.bin