[Unison-hackers] Sneaker Net or Incremental Backup
Benjamin Pierce
bcpierce at cis.upenn.edu
Tue Jan 13 08:44:40 EST 2009
Thanks, Duane! (Also for the FAQ entry!)
On Jan 13, 2009, at 7:08 AM, Duane McKinney wrote:
> $ cat sneakernet.sh
> #!/bin/sh
>
> # Usage
> # $1 = the local root
> # $2 = the remote path that unison will pass to rsync
> # $3 = the path used for the sneakernet,
> # ie the path to the folder on a usb drive moved between
> locations
> # $4 = Souce of the copy as sent by unison
> # $5 = Destination of the copy as sent by unison
>
>
> # Debuggin stuff
> OUTFILE=/dev/null
>
> SOURCEFILE=$4
> DESTFILE=`echo $4 | sed "s|$1|$3|"`
> DESTDIR=`echo $DESTFILE | sed 's:\(/.*/\).*:\1:'`
>
> echo $SOURCEFILE
> echo $DESTFILE
> echo $DESTDIR
>
> if [ -s $SOURCEFILE ]
> then
> #echo Copying from local
> mkdir -p $DESTDIR >> $OUTFILE
> cp -af $SOURCEFILE $DESTFILE >> $OUTFILE
> else
> SOURCEFILE2=`echo $4 | sed "s|$2|$3|"`
> DESTFILE2=$5
> DESTDIR2=`echo $DESTFILE2 | sed 's:\(/.*/\).*:\1:'`
>
> #echo $SOURCEFILE2
> #echo $DESTFILE2
>
> if [ -s $SOURCEFILE2 ]
> then
> mkdir -p $DESTDIR2 >> $OUTFILE
> cp -af $SOURCEFILE2 $DESTFILE2 >> $OUTFILE
> fi
> fi
>
>
> Benjamin Pierce wrote:
>> Hi Duane,
>>
>> Could you re-post this with the shell script appended in-line instead
>> of attached? And/or perhaps just put this explanation in the FAQ
>> wiki
>>
>> http://alliance.seas.upenn.edu/~bcpierce/wiki/index.php
>>
>> Thanks!
>>
>> - Benjamin
>>
>>
>> On Jan 10, 2009, at 5:05 PM, Duane McKinney wrote:
>>
>>> This message has been automatically modified by CETS's antivirus/
>>> antispam
>>> filter. If you have questions about this, please send mail to cets.
>>>
>>> An attachment named sneakernet.sh was removed from this document
>>> as it
>>> constituted a security hazard. Send mail to cets if you have
>>> questions about this policy.
>>>
>>>> That's a nice hack -- I had no idea the new copyprog functionality
>>>> could be used that way.
>>> That is what I love about linux/unix. Lots of small programs that
>>> can be combined to do great things.
>>>
>>>
>>> Didn't have quite as much free time as I though, but here are he
>>> results of my initial testing. It mostly works, and for now works
>>> well enough for me. I can now cron my syncs and not have to worry
>>> that some large file is going to kill our pipe for a few days :)
>>> I'll report back again in a month or so, or sooner if I run into an
>>> issue and have to change something.
>>>
>>> I have attached the shell script and the profile I used for testing.
>>> The important parts are:
>>> copyprog = sneakernet.sh /home/Duane/test
>>> duane at 192.168.113.2:/home/duane/test /home/Duane/usbdrive
>>> copyprogrest = sneakernet.sh
>>>
>>> This replaces what would normally be rsync with the attached shell
>>> script.
>>>
>>> I have only done limited testing so far, but I don't see anything
>>> that could be harmful.
>>>
>>> To be useful you will need 2 profiles a normal and a sneakernet. In
>>> your normal profile you should have
>>> copyprog = rsync --inplace --compress --max-size=xxx
>>> copyprogrest = rsync --partial --inplace --compress --max-
>>> size=xxx
>>>
>>> Where xxx is the size you have determined is too much to transfer
>>> via network. For me it is 1073741824 = 1GB
>>>
>>> The sneakernet profile should be identical except for the copyprog
>>> args
>>> copyprog = sneakernet.sh LocalRoot RemoteRoot SneakerNetPath
>>> copyprogrest = sneakernet.sh
>>> LocalRoot is the path on the local machine you want to sync
>>> RemoteRoot is not formatted the same as unison, it is formatted as
>>> passed by unison to rsync
>>> SneakerNetPath is the path to a directory on some removable media
>>>
>>> so now you will have 2 command to get everything synched
>>> unison myprofile
>>> unison myprofilesneakernet
>>>
>>> Issues:
>>> On copying from the modified machine, unison complains that the copy
>>> did not succeed and moves on, so only one file for each directory
>>> will be copied to the usb drive in a directory that is out of sync.
>>> This is not a big deal for me, but it may be for others. If it
>>> turns out to be a problem I'll probably dig through the source and
>>> come up with a real solution rather than this hack/workaround.
>>>
>>> I have not done any testing with copyprogrest. Since we are using
>>> local disks at all times, I don't know if this will be needed.
>>>
>>> bcpierce at seas.upenn.edu wrote:
>>>> That's a nice hack -- I had no idea the new copyprog functionality
>>>> could be used that way. Will be interested to hear whether it
>>>> works...
>>>> - Benjamin
>>>> Quoting Duane McKinney <duane.mckinney at gmail.com>:
>>>>> Or from Version 2.31.4
>>>>> copyprog xxx
>>>>> A string giving the name of an external program that can be
>>>>> used to
>>>>> copy large files efficiently (plus command-line switches telling
>>>>> it to
>>>>> copy files in-place). The default setting invokes rsync with
>>>>> appropriate
>>>>> options—most users should not need to change it.
>>>>>
>>>>> copyprogrest xxx
>>>>> A variant of copyprog that names an external program that
>>>>> should be
>>>>> used to continue the transfer of a large file that has already
>>>>> been
>>>>> partially transferred. Typically, copyprogrest will just be
>>>>> copyprog
>>>>> with one extra option (e.g., –partial, for rsync). The default
>>>>> setting
>>>>> invokes rsync with appropriate options—most users should not
>>>>> need to
>>>>> change it.
>>>>>
>>>>> copythreshold n
>>>>> A number indicating above what filesize (in kilobytes) Unison
>>>>> should use the external copying utility specified by copyprog.
>>>>> Specifying 0 will cause all copies to use the external program; a
>>>>> negative number will prevent any files from using it. The default
>>>>> is -1.
>>>>> See the Making Unison Faster on Large Files section for more
>>>>> information.
>>>>>
>>>>> I guess I should have checked out the Beta Documentation 1st.
>>>>> I'll try
>>>>> messing around with using a shell script as the copyprog. I would
>>>>> assume that it would work just fine. My plan is to write a shell
>>>>> script
>>>>> which ignores the destination unison tells it, and instead copies
>>>>> it to
>>>>> a usb drive.
>>>>>
>>>>> I'll probably report back on this in a week or so. Currently I am
>>>>> running 2.27, so I have some compiling and testing to do.
>>>>>
>>>>> Duane McKinney wrote:
>>>>>> What about a flag that tells it to instead of executing the
>>>>>> copies, just
>>>>>> prints out a list of the files that would be copied? A proper
>>>>>> name is
>>>>>> escaping me at the moment, but it is a common option on programs.
>>>>>>
>>>>>> Benjamin Pierce wrote:
>>>>>>> Both of these features would be easy to implement using
>>>>>>> information
>>>>>>> that Unison already has available. If you want to give it a
>>>>>>> try, I
>>>>>>> can tell you where to start. (I'd be reluctant, though, to add
>>>>>>> this
>>>>>>> code to the main sources -- Unison arguably has too many flags
>>>>>>> and
>>>>>>> switches already!)
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> - Benjamin
>>>>>>>
>>>>>>> On Dec 27, 2008, at 10:27 AM, Duane McKinney wrote:
>>>>>>>
>>>>>>>> I searched, but came up empty. Can this be done.
>>>>>>>> I sync two offices using unison.
>>>>>>>> 1) (Optional)I would like to be able to set a preference that
>>>>>>>> says,
>>>>>>>> don't try to
>>>>>>>> sync a file if it would require X bytes transferred
>>>>>>>> 2) Get a list of changed files from a root. That way I can
>>>>>>>> copy the
>>>>>>>> files to a
>>>>>>>> removable drive and sync them when I get to the other office.
>>>>>>>>
>>>>>>>> Here is my reasoning. Most of the time, root changes are very
>>>>>>>> small, a few MB.
>>>>>>>> But Let's say that I download a new CENTOS release and place it
>>>>>>>> on
>>>>>>>> the file
>>>>>>>> server. This may be a poor example, because I could retrieve
>>>>>>>> it
>>>>>>>> over the
>>>>>>>> internet again, but just bear with me. I would like for
>>>>>>>> unison, to
>>>>>>>> instead of
>>>>>>>> trying to sync this file over the network to skip it. The I
>>>>>>>> would
>>>>>>>> like to be
>>>>>>>> able to say once a week, run a job, that would take the
>>>>>>>> differences
>>>>>>>> in the root
>>>>>>>> and sent them to a USB drive. I can then carry this drive to
>>>>>>>> location 2, and
>>>>>>>> update the root there. Then from my understanding, the
>>>>>>>> network
>>>>>>>> sync would
>>>>>>>> detect that both roots are identical.
>>>>>>>>
>>>>>>>> That way, our bandwidth isn't being eaten up for hours/days,
>>>>>>>> trying
>>>>>>>> to perform a
>>>>>>>> sync that will most likely fail because it will take so long.
>>>>>>>>
>>>>>>>> Also, this would be more useful, than synchronizing the whole
>>>>>>>> root
>>>>>>>> to a usb
>>>>>>>> drive, because, the total of the data that I am synchronizing
>>>>>>>> is >
>>>>>>>> 1TB. I would
>>>>>>>> not mind moving a few GB (<100) via USB. I am trying to avoid
>>>>>>>> needing either, a
>>>>>>>> lot of time, or a bunch of USB drives (one for each root).
>>>>>>>>
>>>>>>>> I have not yet looked at the source, but I would assume that
>>>>>>>> most of
>>>>>>>> the items
>>>>>>>> required for this feature are already in place. Is it
>>>>>>>> feasible? Is
>>>>>>>> there
>>>>>>>> already a way?
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Unison-hackers mailing list
>>>>>>>> Unison-hackers at lists.seas.upenn.edu
>>>>>>>> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers
>>>>> _______________________________________________
>>>>> Unison-hackers mailing list
>>>>> Unison-hackers at lists.seas.upenn.edu
>>>>> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Unison-hackers mailing list
>>>> Unison-hackers at lists.seas.upenn.edu
>>>> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers
>>> <test.prf>_______________________________________________
>>> Unison-hackers mailing list
>>> Unison-hackers at lists.seas.upenn.edu
>>> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers
>
> _______________________________________________
> Unison-hackers mailing list
> Unison-hackers at lists.seas.upenn.edu
> http://lists.seas.upenn.edu/mailman/listinfo/unison-hackers
>
More information about the Unison-hackers
mailing list