[Unison-hackers] Memory exhaustion issue (#1068)

[Michael von Glasow]

In the meantime, I ran another scan (production, large roots) and left
it for about two hours. Seems it’s swapping – there’s a bit of a
learning curve with getting the right data out of `top`.

Memory usage of Unison is now at 155.2 MB res, 212.2 MB swap. That adds
up to 367.4 MB, roughly in the ballpark of what we had right at the end
of the scan (300.3 MB res, 68.2 MB swap, which adds up to 368.5 MB). The
only difference is that some memory got swapped out to disk, but GC
apparently did not recover any significant amounts of memory (1.1 MB).

I then started sync, this time with external rsync (default copyprog,
copythreshold around 160 MB). This spawns two rsync processes, whose
memory usage stays at 5.6 MB and 4.2 MB (res; swap is at 0) pretty much
the whole time, while Unison’s memory usage stays fairly constant as
well, give or take 1 MB. (The new file is in a new dir, which might
explain the two rsync processes – nothing else was transferred. The
rsync process with the smaller memory footprint seems to be the one that
handled the transfer.)

After transfer, Unison is at 94.3 MB res, 273.9 MB swap (total 368.2 MB,
so not much of a change). Just before sync finishes, memory usage is at
288.2 MB res, 88.5 MB swap (377.3 MB total).

On 24/11/2024 19:14, Tõivo Leedjärv wrote:
> On Sun, 24 Nov 2024 at 16:51, Michael von Glasow <michael at vonglasow.com> wrote:
>> What I do not understand: why would simply copying a file in its
>> entirety, without looking at it any further, consume any significant
>> amounts of memory? The transmitting end would read the file sequentially
> Except that it doesn't. If it would, that would be a bug.
Good, that is consistent with what I would expect, and with the behavior
I am seeing from external rsync. From the documentation, I would expect
Unison with `rsync = no` to behave in the same way, but I am seeing way
higher memory usage. Hopefully VM tests will shed some light on this.
> My testing with the minimal roots showed no such excessive memory
> usage (the few MB used are normal for GC working). My suspicion is
> still on the overall archive size (it's the number of items that
> matters, file sizes don't matter at all); that's why it would be good
> for you to repeat testing with empty roots.

OK, that makes testing a bit easier. In theory, an archive with some
700,000 files, each holding 1 KB of random data, should exhibit the same
behavior as my live setup.

That will take a bit of time to set up the VM though.