[Unison-hackers] Memory exhaustion issue (#1068)

[Jacques-Henri Jourdan]

Hi all,

I'm not an expert in what unison does with its memory nor have I the 
time right now to explore Michael's problem, but I'd like to mention 
that OCaml has been equipped with an efficient memory profiler that can 
help you understand memory usage.

There are two available front-ends for it:
https://urldefense.com/v3/__https://blog.janestreet.com/finding-memory-leaks-with-memtrace/__;!!IBzWLUs!Sh6AtzwZ_O-chwp4YIXz2sY9qnLhEoVnzGrZ-LrBdkUoaB1rhnVRr0dn92TjymnjlzPreWoJR6FxOxJQR6cMtpNeTSTfJZT8ettzErpgCA$ 
and 
https://urldefense.com/v3/__https://ocamlpro.com/blog/2020_12_01_memthol_exploring_program_profiling/__;!!IBzWLUs!Sh6AtzwZ_O-chwp4YIXz2sY9qnLhEoVnzGrZ-LrBdkUoaB1rhnVRr0dn92TjymnjlzPreWoJR6FxOxJQR6cMtpNeTSTfJZT8ets0mLfnQA$ 

Support for this has been dropped in 5.0 and resurrected recently 
(https://urldefense.com/v3/__https://github.com/ocaml/ocaml/pull/12923__;!!IBzWLUs!Sh6AtzwZ_O-chwp4YIXz2sY9qnLhEoVnzGrZ-LrBdkUoaB1rhnVRr0dn92TjymnjlzPreWoJR6FxOxJQR6cMtpNeTSTfJZT8etuPw_T5Uw$ , soon to be released in 5.3), 
so if you want to try it, you can either go back to 4.14 or use OCaml 
5.3.0 beta.

-- 
JH


Le 24/11/2024 à 20:11, Michael von Glasow a écrit :
> In the meantime, I ran another scan (production, large roots) and left
> it for about two hours. Seems it’s swapping – there’s a bit of a
> learning curve with getting the right data out of `top`.
>
> Memory usage of Unison is now at 155.2 MB res, 212.2 MB swap. That adds
> up to 367.4 MB, roughly in the ballpark of what we had right at the end
> of the scan (300.3 MB res, 68.2 MB swap, which adds up to 368.5 MB). The
> only difference is that some memory got swapped out to disk, but GC
> apparently did not recover any significant amounts of memory (1.1 MB).
>
> I then started sync, this time with external rsync (default copyprog,
> copythreshold around 160 MB). This spawns two rsync processes, whose
> memory usage stays at 5.6 MB and 4.2 MB (res; swap is at 0) pretty much
> the whole time, while Unison’s memory usage stays fairly constant as
> well, give or take 1 MB. (The new file is in a new dir, which might
> explain the two rsync processes – nothing else was transferred. The
> rsync process with the smaller memory footprint seems to be the one that
> handled the transfer.)
>
> After transfer, Unison is at 94.3 MB res, 273.9 MB swap (total 368.2 MB,
> so not much of a change). Just before sync finishes, memory usage is at
> 288.2 MB res, 88.5 MB swap (377.3 MB total).
>
> On 24/11/2024 19:14, Tõivo Leedjärv wrote:
>> On Sun, 24 Nov 2024 at 16:51, Michael von Glasow 
>> <michael at vonglasow.com> wrote:
>>> What I do not understand: why would simply copying a file in its
>>> entirety, without looking at it any further, consume any significant
>>> amounts of memory? The transmitting end would read the file 
>>> sequentially
>> Except that it doesn't. If it would, that would be a bug.
> Good, that is consistent with what I would expect, and with the behavior
> I am seeing from external rsync. From the documentation, I would expect
> Unison with `rsync = no` to behave in the same way, but I am seeing way
> higher memory usage. Hopefully VM tests will shed some light on this.
>> My testing with the minimal roots showed no such excessive memory
>> usage (the few MB used are normal for GC working). My suspicion is
>> still on the overall archive size (it's the number of items that
>> matters, file sizes don't matter at all); that's why it would be good
>> for you to repeat testing with empty roots.
>
> OK, that makes testing a bit easier. In theory, an archive with some
> 700,000 files, each holding 1 KB of random data, should exhibit the same
> behavior as my live setup.
>
> That will take a bit of time to set up the VM though.
>
> _______________________________________________
> Unison-hackers mailing list
> Unison-hackers at LISTS.SEAS.UPENN.EDU
> https://LISTS.SEAS.UPENN.EDU/mailman/listinfo/unison-hackers