[Unison-hackers] Memory exhaustion issue (#1068)

[Jacques-Henri Jourdan]

Could you have reached the maximum inodes number ?


Le 27/11/2024 à 20:40, Michael von Glasow a écrit :
> I just attempted to set up a test environment to reproduce my
> observations. The test system is a VirtualBox VM running Ubuntu Server
> 24.04.1, with 1 GB memory (961.7 MB as reported by top). Swap was
> disabled for the test. Unison version is 2.53.3 (ocaml 4.14.1), from the
> Ubuntu repositories.
>
> I wrote a script to generate a directory of 700,000 files on my local
> hard disk, each containing 1 kB of random data. Then I attempted to sync
> that to the test machine, while running `top` to monitor resource usage.
> At that time, around 430 MB of memory were used (not counting
> buffers/cache) on the VM.
>
> During most of the initial scan, Unison on the server was at 8.1 MB res;
> when the scan finished, it was at 8.5 MB.
>
> As I initiated sync, memory went up. (File counts are from the file
> displayed by the Unison GUI; files are numbered 00001 to 70000 and were
> created in that order.)
>
> At 35000 files: 33.5M
> At 70000 files: 51.6M
> At 140000 files: 89.7M
> At 210000 files: 126.0M
> At 280000 files: 170.1M
> At 350000 files: 213.0M
> At 420000 files: 250.1M
> At 490000 files: 290.3M
> At 560000 files: 339.1M
> At 630000 files: 377.3M
> At 686000 files: 478M
> At 700000 files: 523.5M
>
> That is, initially memory consumption increases fairly constantly with
> the number of files, about 40 MB per 70,000 files, which would be some
> 570 bytes per file. After reaching 490,000 files (70%), this goes up
> slightly (the next 70,000 files occupy about 50 MB), then returns to the
> old rate until, above 630,000 files, memory usage increases
> dramatically: the next 56,000 files occupy around 100 MB of memory,
> which would correspond to some 1.8 kB per file, almost twice the size of
> a file. That is, unless one of these figures lags behind the other. On
> average, one file occupies around 730 bytes of memory.
>
> Unfortunately, sync did not finish but aborted with an error message
> after (reportedly) transferring file 700,000:
>
> Fatal error – Error in Lock.acquire: No space left on device
>
> The path to the lock file is the one on the server (the hex value after
> `lk` matches up with the `ar and `fp` files).
>
> The server disk has some 4 GB of free space, so it looks like I hit some
> other limit, not sure which. Filesystem is ext4, same as on the host 
> system.
>
> Files on the server seem to have been synchronized until 659751. The ar*
> file is 655 bytes in size after the failed sync, the fp* file is 34
> bytes in size. Memory usage of the Unison process on the server was at
> 524.2 MB, meaning we were close to running out of memory.
>
> If I retry sync (with a fresh Unison process on both ends), I get the
> same error message right after selecting the profile.
>
> _______________________________________________
> Unison-hackers mailing list
> Unison-hackers at LISTS.SEAS.UPENN.EDU
> https://LISTS.SEAS.UPENN.EDU/mailman/listinfo/unison-hackers