[Unison-hackers] Memory exhaustion issue (#1068)

[Michael von Glasow]

Update on rsync=false: the system became unresponsive again at about 4%
progress. So rsync=false slightly improves the situation (critical
amount of data is about 4 times what is was before), but ultimately not
enough.

On 24/11/2024 17:06, Tõivo Leedjärv wrote:
> On Sun, 24 Nov 2024 at 12:48, Greg Troxel <gdt at lexort.com> wrote:
>> It is probably a good idea to force gc when a unison server finishes
>> servicing a request and is going to sleep.  I don't think it's good to
>> poke gc all the time; there are reasons it was designed how it was.
> All true. We definitely want to let the GC do its thing. Just about
> the only case would be exactly this one, when we know a major activity
> has finished and there is no more activity coming up to trigger GC.

I would add another condition: after finishing a major activity and
Unison is using a lot of memory, measured as physical memory used by
Unison vs. free physical memory. It would cause a delay, but for the
sake of stability.

What I do not understand: why would simply copying a file in its
entirety, without looking at it any further, consume any significant
amounts of memory? The transmitting end would read the file sequentially
in chunks, send them across the network, and the receiving end would
write the data to disk as it arrives. That should require no more than a
buffer on each end, which would be of a fixed size, independent of the
size of the file being transferred. Larger files would just take more
passes of filling the buffer. Or am I overlooking something?