From gdt at lexort.com Tue Nov 15 09:02:03 2022 From: gdt at lexort.com (Greg Troxel) Date: Tue, 15 Nov 2022 09:02:03 -0500 Subject: [Unison-hackers] dune and opam issues: a summary of my fuzzy understanding Message-ID: * meta I am continuing to struggle with proposed changes surrounding dune and opam for a number of reasons. This email attempts to summarize where we are and my current understanding and lack thereof. ** email I'd appreciate trimming of things that aren't being replied to and not top posting :-) ** an aside about make Unison's official build system is make, and I don't see that changing soon -- and definitely not as part of this discussion. Before that question can be entertained, we need to ensure that: - The dune/opam world is in order and there is broad agreement about it. - The dune build works on every system that can build unison now. (There was a comment about "IMHO the key point is what OS / architectures unison wants to support. If that set is covered by dune, there are few resons to keep using make." and that is in my view a red flag.) - The dune build deals with documentation and everything else built by the current system. - "dune clean" restores the repo to original state - The dune build has support for tests, and can be extended (or perhaps wrapped) to do everything that our current CI does. - Packaging systems are ok with the dune build. - The active maintainers/contributors have the dune/opam stuff paged in. Maybe this is more ok than I fear, but I have seen way too many instances of proposals to change build systems that claim they will be strictly better, but then aren't, to trust anything, and to do anything other than a written-requirements type process. I don't want to discuss this now, as it's a distraction from what to do about dune-project and opam changes, and I'd like to get to closure on those before 2.53.1. * my understanding (and lack thereof!) ** opam opam is an ocaml-native packaging system, in which there is a file `foo.opam` that decribes a package and how to build it. Sometimes these control files are in https://github.com/ocaml/opam-repository and sometimes they are in the package repositories (and perhaps in released tarballs also). What I don't understand: - Why are control files sometimes in the above repo and sometimes in the project repos? - Which location is preferred and why? - Does the file have to be at top-level in the project, if it's there (there are N packaging systems and it's not really ok for each to demand stuff in the top-level directory)? - When there are multiple files for multiple build flavors, can one build them without colliding? Or is this really a dune issue? ** dune https://github.com/ocaml/dune dune is an ocaml-specific build system. It works by having a file `dune-project` that provides much the same information as an opam file, and files `dune` in various directories that describe how to build. A `dune-project` file duplicates the contents of opam files, and dune can write opam files. What I don't understand: - Is the definition of projects necessary to use dune? - Why are there magic numbers in dune files, e.g. "-w -3-6-9-10-26-27-32-34-35-38-39-50-52". ** dune/opam relationship Some people seem to think opam files should be hand maintained, and some don't. What I don't understand: - What are the arguments for and against the two styles (dune-generated vs hand-maintained)? - If opam files are in the opam repository, and also generated, how does that work? - If files are generated, then mostly-obviously they shouldn't be checked in. Do people think they should be? If they are, are we sure they will be bit-for-bit identical when regenerated on different systems? * where we are ** opam Some changes have been applied to unison.opam (because I was able to convince myself they were ok, by a combination of understanding and having the questions I asked be answered (thank you kit-ty-kate!)). - complaint about opam and not having a CLI version https://github.com/bcpierce00/unison/issues/793 I don't understand; it seems the unison.opam file requires lablgtk3 and then builds a CLI-only version. While that's wrong, the issue says something else. - complaint (valid!) about status of opam packages and the relationship to the opam repository https://github.com/bcpierce00/unison/issues/827 This is validly mostly duplicative with #793, and also has a lot of broad-ranging discussion that belongs here. ** dune There are no open issues/PRs or mailinglist complaints that seem current. There is - c_names issue in 2.51.2 https://github.com/bcpierce00/unison/issues/317 I think this is now history and should be closed. I pinged the issue and will close it if I don't hear back. * dune-project We have a PR on the table to rototill the dune-project setup: https://github.com/bcpierce00/unison/pull/671 This does two unrelated things: - Updates dependencies: dune 2.7, lablgtk3. - Converts to generating opam files which overwrites 1 and adds 2. The new files are similar, but I see a new dependency on odoc, which is not currently known to me to be a unison dependency. I have problems with this PR, some of which I've tried to ask about, and some of which are new from understanding gained while writing this message. - There is not a reasonable commit message. (A commit message should explain why the changes are being made, and should be understandable to those who are not dune experts.) - I'm not sure that everyone is ok with generating opam files. - It's not clear that opam files, if generated, should be checked in. - Regeneration of opam files is in the same commit is not ok; this is of course easy to fix. * way forward At this point, my inclination is: - Reset opam maintainer to `unison-hackers at lists.seas.upenn.edu` because the current maintainer has not answered mail, and we don't really have another plan. This is saying that the file is maintained by the project, just like every other file in the repo. - Apply the lablgtk3 and dune 2.7 changes from #671, basically cherrypicking the parts I understand and agree with, and writing a commit message. - Consider turning on generation, and removing unison.opam from the repo, pending feedback, probably via a PR that does only that. - See if there are things to be done other than the above 3. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 194 bytes Desc: not available URL: From jacques-henri.jourdan at cnrs.fr Tue Nov 15 11:38:06 2022 From: jacques-henri.jourdan at cnrs.fr (Jacques-Henri Jourdan) Date: Tue, 15 Nov 2022 17:38:06 +0100 Subject: [Unison-hackers] dune and opam issues: a summary of my fuzzy understanding Message-ID: <32c0d40b-72a2-8ab8-b537-43bb5996eb24@cnrs.fr> Hi all, Let me first give my answers to the questions you asked: > - Why are control files sometimes in the above repo and sometimes in > the project repos? On the one hand, the opam file should be published in opam repository so that opam, as a package manager, can be used to install unison easily, i.e., by simply typing "opam install unison". Users of opam need not explicitly give the URL of the project repository: opam will lookup in opam-repository the opam file and use it to get all the information it needs. Also, the opam-repository repo contains one opam file for each release, so that a user can explicitly ask to install e.g., unison 2.53.0 via "opam install unison.2.53.0", just like we would do with a package manager from a Linux distribution. Thanks to this repository, the command "opam upgrade" allows to upgrade the version of unison, but only when a new release is published (a regular user doesn't want to upgrade unison at every commit, because the content of the repository between two releases can be bogus/unstable). On the other hand, the project repository contains an opam file, so that its contents can be kept in sync when details specific to the software changes (e.g., which build system, configuration options, dependencies). This file is also useful when people want to install a development version of the software, e.g., for testing an unreleased feature. If they want to do that, they need to issue special commands to opam to tell it that this package should be installed from the development repository. When doing that, opam will stop tracking the version of unison used (and instead use a generic "dev" version string which tells nothing), and "opam upgrade" will simply upgrade to the most recent content of the repository. When publishing a release, the release manager needs to copy the opam file from the development repository to opam-repository, and add a "url" section specifying where to find the sources of that particular release. > > - Which location is preferred and why? Both locations are useful, but for different purposes: - opam-repository is useful for end-users wanting to use released versions of unison, without invoking complex command to tell opam where to find the code - the dev repo is useful for easier maintenance, and to let advanced users install and test development versions > - Does the file have to be at top-level in the project, if it's there > (there are N packaging systems and it's not really ok for each to > demand stuff in the top-level directory)? If this file is used to install development versions, then opam *expects* it to be there. I imagine there might be options to tell opam where the opam file is, but this is non-standard and should be avoided. Of course, if the file is kept in the repo only for maintenance purposes, then it does not need to be at the root of the project. But maintaining an opam file in an exotic location is going to be cumbersome, because one will not be able to test it by installing the development version. > - When there are multiple files for multiple build flavors, can one > build them without colliding? Or is this really a dune issue? I'm not sure I understand your question: opam is really a "package manager", so the question is not really *building* the software in multiple flavors, but rather *installing* it. It is possible to have several opam files at the root of the same repository. They will correspond to different packages using the same source tree, for example because the software has been split into several packages (a library may be distributed using several packages representing different features set o the library...). But if these packages are installed simultaneously, then it's just like with "make install": they need to install a different set of files. Otherwise, they are conflicting, and this should be part of a "conflict" clause of the opam files. In the case of unison, we have several options: 1- have two packages "unison-cli" and "unison-gui", both installing the executable "unison", and declare the two packages in conflict. The user will have to choose between the two packages when typing "opam". The problem is perhaps that there is no "canonical" package for unison. 2- same as 1-, but with two different names for the executable. The user can install both packages, but once installed, she will have to type two different commands depending on whether she wants to use the version with the GUI or not: this is not the same behavior as with "make install" and thus I don't really like this solution. 3- use only one package, but with an optional dependency on lablgtk. The package will be built with two different configuration depending on whether lablgtk is installed, and an upgrade of lablgtk will automatically trigger a rebuild of unison. The disadvantage is that a user willing to install unison with the gui will type "opam install unison", but will not have the gui installed because lablgtk will not be installed by default; this can be confusing. 4- same as 3-, but with a pseudo-package "unison-gui" (i.e., without any code), with a dependency on both "unison" and "lablgtk", so that installing "unison-gui" will make sure the GUI is enabled. This is somewhat less confusing than 3-, because a user can install an actual package that says "I want unison with its GUI". > - Is the definition of projects necessary to use dune? Yes, but it can be minimal. It need not duplicate all the information of the opam file if we decide that we don't want to generate the opam file with dune. > - Why are there magic numbers in dune files, e.g. > "-w -3-6-9-10-26-27-32-34-35-38-39-50-52". By default, dune enables more warnings than the default OCaml compiler enables, and make those warnings errors. So we need to disable the warnings the are triggered by the code of OCaml, because they will be reported as errors. These "magic numbers" are simply identifiers of warnings. > - What are the arguments for and against the two styles > (dune-generated vs hand-maintained)? Of course, generating opam files automatically avoids duplicating information between dune and opam. So, if dune is the primary build system of a project, and if all the information that one wants to place in the opam file can be generated by dune, then this is the way to go. *However, opam can be instructed to use a different build system. There is no fundamental reason to use dune in the opam file. The opam file can very well tell opam to use the standard Makefile for both compilation and installation.* And I would argue that if the main unison developers are not using dune, then dune should not be used in the opam file, because if the dune build system is bogus (because not properly configured...), then the unison developers will not test it, and therefore they will not notice the bug. I don't see why the validity of the opam file should rely on the minority of developers using dune. > - If opam files are in the opam repository, and also generated, how does > that work? Each time someone changes the content of the dune-project file, she should regenerate the opam file(s), and commit both at the same time. It is a good idea to setup a CI job that checks that the commited opam file corresponds to the one that would be generated by dune. > - If files are generated, then mostly-obviously they shouldn't be > checked in. Do people think they should be? If they are, are we > sure they will be bit-for-bit identical when regenerated on > different systems? They should be, because the opam file needs to be present if one wants to use opam to install a development version of unison (as explained above). Yes, they should be bit-for-bit identical (otherwise, I consider this as a dune bug). > * way forward > > At this point, my inclination is: > > - Reset opam maintainer to `unison-hackers at lists.seas.upenn.edu ` > because the current maintainer has not answered mail, and we don't > really have another plan. This is saying that the file is > maintained by the project, just like every other file in the repo. Agreed. > - Apply the lablgtk3 and dune 2.7 changes from #671, basically > cherrypicking the parts I understand and agree with, and writing a > commit message. Agreed. > - Consider turning on generation, and removing unison.opam from the > repo, pending feedback, probably via a PR that does only that. > > - See if there are things to be done other than the above 3. I would rather suggest to change the opam file so that it uses the Makefile. If make is the primary build system, then the dune files should be maintained by their users.? As a result, a user of the main build system can maintain the opam files without understanding dune, which is mainly orthogonal. My 2 cents, best regards, -- JH Jourdan -------------- next part -------------- An HTML attachment was scrubbed... URL: From lys at alumni.upenn.edu Wed Nov 16 02:37:44 2022 From: lys at alumni.upenn.edu (Yishuai Li) Date: Wed, 16 Nov 2022 15:37:44 +0800 Subject: [Unison-hackers] dune and opam issues: a summary of my fuzzy understanding In-Reply-To: <32c0d40b-72a2-8ab8-b537-43bb5996eb24@cnrs.fr> References: <32c0d40b-72a2-8ab8-b537-43bb5996eb24@cnrs.fr> Message-ID: > - Consider turning on generation, and removing unison.opam from the > repo, pending feedback, probably via a PR that does only that. > > - See if there are things to be done other than the above 3. > > > I would rather suggest to change the opam file so that it uses the Makefile. If make is the primary build system, then the dune files should be maintained by their users. As a result, a user of the main build system can maintain the opam files without understanding dune, which is mainly orthogonal. Another reason for keeping the OPAM file(s) is to utilize [opam-publish] to streamline the publishing of the Unison package(s) to the OPAM repository. Once a version is tagged, and its OPAM files are up to date, then anyone may propose them to the OPAM repository by running `opam publish`, rather than handwriting an OPAM file for each version. [opam-publish]: https://urldefense.com/v3/__https://github.com/ocaml-opam/opam-publish__;!!IBzWLUs!Ril1nRCkGTzQADHbu5sFg58lhGaVotbey6hxC3rYKd3_RYrVkHOX5u69B35lmvJFBsPWHjIAp0YBCBM_JM12syfNNLH1-D0M$ Yishuai From lys at alumni.upenn.edu Wed Nov 16 03:08:41 2022 From: lys at alumni.upenn.edu (Yishuai Li) Date: Wed, 16 Nov 2022 16:08:41 +0800 Subject: [Unison-hackers] dune and opam issues: a summary of my fuzzy understanding In-Reply-To: <32c0d40b-72a2-8ab8-b537-43bb5996eb24@cnrs.fr> References: <32c0d40b-72a2-8ab8-b537-43bb5996eb24@cnrs.fr> Message-ID: > - When there are multiple files for multiple build flavors, can one > build them without colliding? Or is this really a dune issue? > > > I'm not sure I understand your question: opam is really a "package manager", so the question is not really *building* the software in multiple flavors, but rather *installing* it. > > It is possible to have several opam files at the root of the same repository. They will correspond to different packages using the same source tree, for example because the software has been split into several packages (a library may be distributed using several packages representing different features set o the library...). But if these packages are installed simultaneously, then it's just like with "make install": they need to install a different set of files. Otherwise, they are conflicting, and this should be part of a "conflict" clause of the opam files. > > In the case of unison, we have several options: > 1- have two packages "unison-cli" and "unison-gui", both installing the executable "unison", and declare the two packages in conflict. The user will have to choose between the two packages when typing "opam". The problem is perhaps that there is no "canonical" package for unison. > 2- same as 1-, but with two different names for the executable. The user can install both packages, but once installed, she will have to type two different commands depending on whether she wants to use the version with the GUI or not: this is not the same behavior as with "make install" and thus I don't really like this solution. > 3- use only one package, but with an optional dependency on lablgtk. The package will be built with two different configuration depending on whether lablgtk is installed, and an upgrade of lablgtk will automatically trigger a rebuild of unison. The disadvantage is that a user willing to install unison with the gui will type "opam install unison", but will not have the gui installed because lablgtk will not be installed by default; this can be confusing. > 4- same as 3-, but with a pseudo-package "unison-gui" (i.e., without any code), with a dependency on both "unison" and "lablgtk", so that installing "unison-gui" will make sure the GUI is enabled. This is somewhat less confusing than 3-, because a user can install an actual package that says "I want unison with its GUI". I agree that 4- is the right direction to go. In PR #671 I've just unlinked OPAM from Dune, but have less confidence in writing the build rules myself. JH do you feel like writing the Makefile-based rules to make the PR ready to merge? Yishuai From jacques-henri.jourdan at cnrs.fr Wed Nov 16 03:29:54 2022 From: jacques-henri.jourdan at cnrs.fr (Jacques-Henri Jourdan) Date: Wed, 16 Nov 2022 09:29:54 +0100 Subject: [Unison-hackers] dune and opam issues: a summary of my fuzzy understanding In-Reply-To: References: <32c0d40b-72a2-8ab8-b537-43bb5996eb24@cnrs.fr> Message-ID: <542ff3bc-d07f-0fb4-45cb-e895c8629bd2@cnrs.fr> > JH do you feel like writing the Makefile-based rules to make the PR > ready to merge? Sure. But before doing that, let's wait for a few days to see if everyone agrees this is the way to go. Also, there is another question to answer: should we streamline the dune-project file, given that we no longer use it to generate opam files. AFAICT, much of the information there is only used for generating opam files. This include synopsis, descriptions, license, authors, maintainers and dependencies. -- JH From gdt at lexort.com Wed Nov 16 06:35:45 2022 From: gdt at lexort.com (Greg Troxel) Date: Wed, 16 Nov 2022 06:35:45 -0500 Subject: [Unison-hackers] dune and opam issues: a summary of my fuzzy understanding In-Reply-To: <542ff3bc-d07f-0fb4-45cb-e895c8629bd2@cnrs.fr> (Jacques-Henri Jourdan's message of "Wed, 16 Nov 2022 09:29:54 +0100") References: <32c0d40b-72a2-8ab8-b537-43bb5996eb24@cnrs.fr> <542ff3bc-d07f-0fb4-45cb-e895c8629bd2@cnrs.fr> Message-ID: Jacques-Henri Jourdan writes: >> JH do you feel like writing the Makefile-based rules to make the PR >> ready to merge? > > Sure. But before doing that, let's wait for a few days to see if > everyone agrees this is the way to go. And I don't think we need to have one PR. I am inclined to fix separable things in chunks, and we already have two things that are agreed, so I just merged the maintainer change and closed the issue about it. > Also, there is another question to answer: should we streamline the > dune-project file, given that we no longer use it to generate opam > files. AFAICT, much of the information there is only used for > generating opam files. This include synopsis, descriptions, license, > authors, maintainers and dependencies. I am not at all sure we should do either of those things, because I think the future of what we are doing about opam is quite uncertain. I am not at all opposed to dune-based builds working, and I don't have a handle on the set of systems where ocaml works and unison makefiles work opam build of unison with make would work opam build of unison with dune works dune build of unison works My leaning is to let opam/dune-world people maintain the dune/opam things, assuming there are people that want to follow along and do that. Right now we have dune builds in CI, but the platform coverage of CI is very poor, and the future of unison CI is uncertain. There was a comment earlier about what was the set of systems and CPU types that unison targets. I don't immediately see anything restrictive about opam and dune, beyond needing ocaml which of course unison does anyway. So is there an argument that there is no portability loss? Can one build dune and opam with ocal 4.08 (and 4.09-4.13)? I know it works with 4.14 on my system, NetBSD 9 amd64. Related, how does the broader ocaml world deal with "Laggard Term Stable" packaging systems that intentionally ship old code, but have users that somehow expect to run new code? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 194 bytes Desc: not available URL: From jacques-henri.jourdan at cnrs.fr Wed Nov 16 08:18:32 2022 From: jacques-henri.jourdan at cnrs.fr (Jacques-Henri Jourdan) Date: Wed, 16 Nov 2022 14:18:32 +0100 Subject: [Unison-hackers] dune and opam issues: a summary of my fuzzy understanding In-Reply-To: References: <32c0d40b-72a2-8ab8-b537-43bb5996eb24@cnrs.fr> <542ff3bc-d07f-0fb4-45cb-e895c8629bd2@cnrs.fr> Message-ID: <78eec0c9-e4a8-fd86-ee9f-733a955eaacb@cnrs.fr> > Related, how does the broader ocaml world deal with "Laggard Term > Stable" packaging systems that intentionally ship old code, but have > users that somehow expect to run new code? I think that's one of the reason one wants to use opam : you can both work on debian old-stable with rather old system software, and at the same time locally install opam with recent versions of OCaml and all libraries.