<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">On 05/01/2025 15:39, Greg Troxel wrote:<br>
</div>
<blockquote type="cite" cite="mid:rmipll1idlu.fsf@s1.lexort.com">
<pre wrap="" class="moz-quote-pre">What are you doing about swap space? Does your system page? If you are
running a RPI3 in aarch64 mode, you could configure 32GB of swap space
on a USB-attached SSD, and then unison should run. Have you tried that?</pre>
</blockquote>
I originally had 3 GB of swap space on a USB-attached magnetic disk.
The result is that, as soon as physical memory is exhausted, the
system becomes unusable because it spends most if its time waiting
for swapping operations to complete. An SSD might be faster, but the
connection is still over USB 2.0, so it might not even make a
difference at all.<span style="white-space: pre-wrap">
</span>
<blockquote type="cite" cite="mid:rmipll1idlu.fsf@s1.lexort.com">
<blockquote type="cite">
<pre wrap="" class="moz-quote-pre">I’d be curious to know what the “average” use case for Unison looks
like. Does anyone happen to have an idea of how people out there are
using Unison?
</pre>
</blockquote>
<pre wrap="" class="moz-quote-pre">
There are about 1000 people on the users list, so I'd guess there are at
least 10K users. But I'm guessing. Unison, being honorable Free
Software, does not contain tracking code to report.</pre>
</blockquote>
Thank God there’s no tracking code in Unison :-) that’s what I
expected, but I thought you guys might have an idea based on
discussions on this lists, bug reports etc. You’ve probably seen a
lot more usage scenarios than I have.<br>
<blockquote type="cite" cite="mid:rmipll1idlu.fsf@s1.lexort.com">
<blockquote type="cite">
<pre wrap="" class="moz-quote-pre">For my use case, around a terabyte of data in a million files, accessed
by a handful of users and fairly static, would be well within the means
of a Pi 3 with 1 GB of RAM – as long as we’re just talking about file
sharing via CIFS or SFTP. It‘s only when Unison gets involved that the
system reaches its limit.
</pre>
</blockquote>
<pre wrap="" class="moz-quote-pre">
CIFS/SFTP is not a fair comparison, because that is access not sync.</pre>
</blockquote>
This was not meant to imply that others do the job better, but to
illustrate the perspective of a user who adds sync capabilities to
their existing file server: without sync, 1 GB (probably even less)
will work just fine. But as soon as sync enters the picture, a lot
more memory is suddenly needed. Other tools might have similar
issues – except that the “competitors” (Syncthing, git etc.) work
quite differently from Unison and are definitely not a drop-in
replacement.<br>
<blockquote type="cite" cite="mid:rmipll1idlu.fsf@s1.lexort.com">
<pre wrap="" class="moz-quote-pre">You could invent virtual memory and use it :-)</pre>
</blockquote>
See above – disk access times make this impractical.<br>
<blockquote type="cite" cite="mid:rmipll1idlu.fsf@s1.lexort.com">
<pre wrap="" class="moz-quote-pre">Or you could implement
application-specific virtual memory. This would look like on-disk
storage for archives, with a cache of objects in memory, and reading
them on demand.</pre>
</blockquote>
The general issue with any kind of virtual memory and swapping is to
predict what data will be needed, and what is OK to swap out.
Keeping the archive on disk and caching just parts in memory is a
kind of memory virtualization what swapping as well, except it
happens in the application and the code can be tuned to minimize
swap operations. For example, if archive entries are identified by
their hash code, one could make the order in which files are
compared dependent on hash codes, so each chunk of the archive needs
to be loaded into memory only once in the comparison step.<br>
<blockquote type="cite" cite="mid:rmipll1idlu.fsf@s1.lexort.com">
<pre wrap="" class="moz-quote-pre">You could also choose to generate profiles for subparts of the file tree
and run them sequentially. I have organized files by directory and sync
them separately anyway, because I want to control which directories get
synced to which subset of places, for various reasons not about unison.
</pre>
</blockquote>
<p>That’s what I did before. However, I sometimes move files between
these subparts (and incidentally, these tend to be on the larger
side, several GB per file). These files would then get copied over
the wire again, instead of the move operation just being mirrored
on the other end. And with a copy-on-write filesystem with
snapshots, there are further implications regarding disk space.<br>
</p>
</body>
</html>