[Unison-hackers] Memory exhaustion issue (#1068)

[Greg Troxel]

Michael von Glasow <michael at vonglasow.com> writes:

> On 23/11/2024 23:01, Greg Troxel wrote:
>> Michael von Glasow<michael at vonglasow.com> writes:
>>
>> Sorry, I guess I was unclear.  I am not really looking for just
>> sufficient to make your case, but the simplest possible way to reproduce
>> the problem, expressed programmatically so that others can run it (after
>> reading the code to feel it is safe).  So that's no GUI, no persistent
>> server, and everything in the profile to be synced created by the
>> script.
> I‘ll look into it, but reproducing would also involve creating a huge
> set of files to sync.

Yes, but it might not be the whole of what you are doing.  Still, if
others can't reproduce the bug, it is highly unlikely to get fixed.

You may also be able to reproduce not by running out, but showing the
effect of varying things on memory usage.

The minimization guidelines call for not running a server on the server,
too.

>> For comparing with other people, I think it will be more useful to talk
>> about KB or MB of memory usage vs %.  Lots of people will have different
>> amounts of RAM and other loads, although 1 GB such as RPI3 is pretty
>> common.
> In principle, +1. However, the tools I have spit out percentages. One %
> is roughly 10 MB, so the math is fairly easy.

On Raspberry Pi OS, ps alx will show KB. 

>> (or 6, but there is no reason to use 6 if you are compiling).
> You do you, but the trouble is that distro repositories are a bit slow
> to catch up. Part of the whole update exercise was so I could finally
> return to Unison from the distro repos and not have to build my own any
> more. Especially since building for non-Intel platforms is quite a
> hassle – I tried to get that on CI some time ago, but my attempts
> stalled as setting up a cross-compilation environment proved challenging.

Your choice to run old software is yours.  The license gives you the
freedom to do that - and I don't object to you doing it.  But the issue
tracker policy is firmly "latest release only".

I am boggled that it is hard to build.  I build unison on Raspberry Pi
under NetBSD all the time, via pkgsrc, and it's uneventful.

I have a RPI3 running Raspberry Pi OS (that I don't use unison on).  It
didn't have ocaml, but "apt install ocaml" ran uneventfully.  I cloned
the unison git repo, cd'd into it and typed gmake.  37s later I got a
shell prompt and

  $ src/unison -version
  unison version 2.53.7 (ocaml 4.13.1)

This is one version back for ocaml compared to the recommend 4.14.1, but
still it's pretty good for a few minutes of thinking and a few minutes
of waiting.

I suspect I'd need to install liblablgtk3-ocaml-dev and a few others and
could then build the gui, but you don't need the gui on the server.

>> And, that other memory uesd for scan/etc. is reused.
> It is roughly proportional to the size of the transferred file, although
> 16000M uses more in comparison than 12800M does (roughly 55M vs. 34M).
> Also, archive file size seems to matter.

I suspect there's a lot of randomness and paging variation.

>>> Looking at the docs, what comes to mind is:
>>>
>>> - copyprog, copyprogthreshold (use external program <copyprog> for
>>> copying files larger than <copyprogthreshold> kB)
>> That is about to be deleted.
>
> That would be too bad, because that seems to be a valid workaround. I
> added `copythreshold = 163840` (1% of 16G, roughly where the issues
> started) to my profile and was able to sync.

> For “cut the smartness above file size X”, `copythreshold` (not
> `copyprogthreshold`, that was a typo) seems to do the trick. Please keep
> that feature until memory efficiency is improved, unless there is an
> alternative.

There was an issue open for a really long time and there has been lots
of "it might help" and no data that it does.  The decision has been made
and a deprecation warning hoisted.

  https://urldefense.com/v3/__https://github.com/bcpierce00/unison/issues/871__;!!IBzWLUs!RH1lsaGfwyT_LsBRbiPLiglTxSLfRBl6SZfiNOzaWFor9GsojdEHXYocECmrmJMJ7EeKJswO6lRsja6Mbov-lMaI$ 

We don't have a valid bug report on the table about high memory usage,
as you're only telling us about old versions.

Fundamentally, I don't think there are any good reasons why unison
should use memory more than rsync.  It's going to take someone motivated
to figure it out though.