[Unison-hackers] Sneaker Net or Incremental Backup

Benjamin Pierce bcpierce at cis.upenn.edu
Sat Jan 10 17:15:03 EST 2009 

Hi Duane,

Could you re-post this with the shell script appended in-line instead  
of attached?  And/or perhaps just put this explanation in the FAQ wiki

          http://alliance.seas.upenn.edu/~bcpierce/wiki/index.php

Thanks!

     - Benjamin


On Jan 10, 2009, at 5:05 PM, Duane McKinney wrote:

> This message has been automatically modified by CETS's antivirus/ 
> antispam
> filter. If you have questions about this, please send mail to cets.
>
> An attachment named sneakernet.sh was removed from this document as it
> constituted a security hazard. Send mail to cets if you have  
> questions about this policy.
>
> > That's a nice hack -- I had no idea the new copyprog functionality
> > could be used that way.
> That is what I love about linux/unix.  Lots of small programs that  
> can be combined to do great things.
>
>
> Didn't have quite as much free time as I though, but here are he  
> results of my initial testing.  It mostly works, and for now works  
> well enough for me.  I can now cron my syncs and not have to worry  
> that some large file is going to kill our pipe for a few days :)   
> I'll report back again in a month or so, or sooner if I run into an  
> issue and have to change something.
>
> I have attached the shell script and the profile I used for testing.  
> The important parts are:
> copyprog      =   sneakernet.sh /home/Duane/test  
> duane at 192.168.113.2:/home/duane/test /home/Duane/usbdrive
> copyprogrest  =   sneakernet.sh
>
> This replaces what would normally be rsync with the attached shell  
> script.
>
> I have only done limited testing so far, but I don't see anything  
> that could be harmful.
>
> To be useful you will need 2 profiles a normal and a sneakernet.  In  
> your normal profile you should have
> copyprog      =   rsync --inplace --compress --max-size=xxx
> copyprogrest  =   rsync --partial --inplace --compress --max-size=xxx
>
> Where xxx is the size you have determined is too much to transfer  
> via network. For me it is 1073741824 = 1GB
>
> The sneakernet profile should be identical except for the copyprog  
> args
> copyprog      =   sneakernet.sh LocalRoot RemoteRoot SneakerNetPath
> copyprogrest  =   sneakernet.sh
> LocalRoot is the path on the local machine you want to sync
> RemoteRoot is not formatted the same as unison, it is formatted as  
> passed by unison to rsync
> SneakerNetPath is the path to a directory on some removable media
>
> so now you will have 2 command to get everything synched
> unison myprofile
> unison myprofilesneakernet
>
> Issues:
> On copying from the modified machine, unison complains that the copy  
> did not succeed and moves on, so only one file for each directory  
> will be copied to the usb drive in a directory that is out of sync.   
> This is not a big deal for me, but it may be for others.  If it  
> turns out to be a problem I'll probably dig through the source and  
> come up with a real solution rather than this hack/workaround.
>
> I have not done any testing with copyprogrest.  Since we are using  
> local disks at all times, I don't know if this will be needed.
>
> bcpierce at seas.upenn.edu wrote:
>> That's a nice hack -- I had no idea the new copyprog functionality   
>> could be used that way.  Will be interested to hear whether it  
>> works...
>>     - Benjamin
>> Quoting Duane McKinney <duane.mckinney at gmail.com>:
>>> Or from Version 2.31.4
>>> copyprog xxx
>>>     A string giving the name of an external program that can be  
>>> used to
>>> copy large files efficiently (plus command-line switches telling  
>>> it to
>>> copy files in-place). The default setting invokes rsync with  
>>> appropriate
>>> options—most users should not need to change it.
>>>
>>> copyprogrest xxx
>>>     A variant of copyprog that names an external program that  
>>> should be
>>> used to continue the transfer of a large file that has already been
>>> partially transferred. Typically, copyprogrest will just be copyprog
>>> with one extra option (e.g., –partial, for rsync). The default  
>>> setting
>>> invokes rsync with appropriate options—most users should not need to
>>> change it.
>>>
>>> copythreshold n
>>>     A number indicating above what filesize (in kilobytes) Unison
>>> should use the external copying utility specified by copyprog.
>>> Specifying 0 will cause all copies to use the external program; a
>>> negative number will prevent any files from using it. The default  
>>> is -1.
>>> See the Making Unison Faster on Large Files section for more  
>>> information.
>>>
>>> I guess I should have checked out the Beta Documentation 1st.   
>>> I'll try
>>> messing around with using a shell script as the copyprog.  I would
>>> assume that it would work just fine.  My plan is to write a shell  
>>> script
>>> which ignores the destination unison tells it, and instead copies  
>>> it to
>>> a usb drive.
>>>
>>> I'll probably report back on this in a week or so.  Currently I am
>>> running 2.27, so I have some compiling and testing to do.
>>>
>>> Duane McKinney wrote:
>>>> What about a flag that tells it to instead of executing the  
>>>> copies, just
>>>> prints out a list of the files that would be copied?  A proper  
>>>> name is
>>>> escaping me at the moment, but it is a common option on programs.
>>>>
>>>> Benjamin Pierce wrote:
>>>>> Both of these features would be easy to implement using  
>>>>> information
>>>>> that Unison already has available.  If you want to give it a  
>>>>> try, I
>>>>> can tell you where to start.  (I'd be reluctant, though, to add  
>>>>> this
>>>>> code to the main sources -- Unison arguably has too many flags and
>>>>> switches already!)
>>>>>
>>>>> Best,
>>>>>
>>>>>     - Benjamin
>>>>>
>>>>> On Dec 27, 2008, at 10:27 AM, Duane McKinney wrote:
>>>>>
>>>>>> I searched, but came up empty.  Can this be done.
>>>>>> I sync two offices using unison.
>>>>>> 1) (Optional)I would like to be able to set a preference that  
>>>>>> says,
>>>>>> don't try to
>>>>>> sync a file if it would require X bytes transferred
>>>>>> 2) Get a list of changed files from a root.  That way I can  
>>>>>> copy the
>>>>>> files to a
>>>>>> removable drive and sync them when I get to the other office.
>>>>>>
>>>>>> Here is my reasoning.  Most of the time, root changes are very
>>>>>> small, a few MB.
>>>>>> But Let's say that I download a new CENTOS release and place it  
>>>>>> on
>>>>>> the file
>>>>>> server.  This may be a poor example, because I could retrieve it
>>>>>> over the
>>>>>> internet again, but just bear with me.  I would like for  
>>>>>> unison, to
>>>>>> instead of
>>>>>> trying to sync this file over the network to skip it.  The I  
>>>>>> would
>>>>>> like to be
>>>>>> able to say once a week, run a job, that would take the  
>>>>>> differences
>>>>>> in the root
>>>>>> and sent them to a USB drive.  I can then carry this drive to
>>>>>> location 2, and
>>>>>> update the root there.   Then from my understanding, the network
>>>>>> sync would
>>>>>> detect that both roots are identical.
>>>>>>
>>>>>> That way, our bandwidth isn't being eaten up for hours/days,  
>>>>>> trying
>>>>>> to perform a
>>>>>> sync that will most likely fail because it will take so long.
>>>>>>
>>>>>> Also, this would be more useful, than synchronizing the whole  
>>>>>> root
>>>>>> to a usb
>>>>>> drive, because, the total of the data that I am synchronizing  
>>>>>> is >
>>>>>> 1TB.  I would
>>>>>> not mind moving a few GB (<100) via USB.  I am trying to avoid
>>>>>> needing either, a
>>>>>> lot of time, or a bunch of USB drives (one for each root).
>>>>>>
>>>>>> I have not yet looked at the source, but I would assume that  
>>>>>> most of
>>>>>> the items
>>>>>> required for this feature are already in place.  Is it  
>>>>>> feasible?  Is
>>>>>> there
>>>>>> already a way?
>>>>>>
>>>>>> _______________________________________________
>>>>>> Unison-hackers mailing list
>>>>>> Unison-hackers at lists.seas.upenn.edu
>>>>>> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers
>>> _______________________________________________
>>> Unison-hackers mailing list
>>> Unison-hackers at lists.seas.upenn.edu
>>> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers
>>>
>>>
>> _______________________________________________
>> Unison-hackers mailing list
>> Unison-hackers at lists.seas.upenn.edu
>> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers
> <test.prf>_______________________________________________
> Unison-hackers mailing list
> Unison-hackers at lists.seas.upenn.edu
> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers

More information about the Unison-hackers mailing list