[Unison-hackers] Memory exhaustion issue (#1068)

[Greg Troxel]

Michael von Glasow <michael at vonglasow.com> writes:

> I wrote a script to generate a directory of 700,000 files on my local
> hard disk, each containing 1 kB of random data. Then I attempted to sync
> that to the test machine, while running `top` to monitor resource usage.
> At that time, around 430 MB of memory were used (not counting
> buffers/cache) on the VM.

I would suggest using ps alx to look at the size of unison, rather than
total on the machine.

> During most of the initial scan, Unison on the server was at 8.1 MB res;
> when the scan finished, it was at 8.5 MB.
>
> As I initiated sync, memory went up. (File counts are from the file
> displayed by the Unison GUI; files are numbered 00001 to 70000 and were
> created in that order.)
>
> At 35000 files: 33.5M
> At 70000 files: 51.6M
> At 140000 files: 89.7M
> At 210000 files: 126.0M
> At 280000 files: 170.1M
> At 350000 files: 213.0M
> At 420000 files: 250.1M
> At 490000 files: 290.3M
> At 560000 files: 339.1M
> At 630000 files: 377.3M
> At 686000 files: 478M
> At 700000 files: 523.5M
>
> That is, initially memory consumption increases fairly constantly with
> the number of files, about 40 MB per 70,000 files, which would be some
> 570 bytes per file. After reaching 490,000 files (70%), this goes up
> slightly (the next 70,000 files occupy about 50 MB), then returns to the
> old rate until, above 630,000 files, memory usage increases
> dramatically: the next 56,000 files occupy around 100 MB of memory,
> which would correspond to some 1.8 kB per file, almost twice the size of
> a file. That is, unless one of these figures lags behind the other. On
> average, one file occupies around 730 bytes of memory.

It's really hard to conclude fine points, but 600ish bytes of memory per
file does not sound implausible.

> Unfortunately, sync did not finish but aborted with an error message
> after (reportedly) transferring file 700,000:

You got a hint about this which sounds right to me.

Once you fix this, you can probably continue to experiment.