[Unison-hackers] dune and opam issues: a summary of my fuzzy understanding

[Greg Troxel]

* meta

I am continuing to struggle with proposed changes surrounding dune and
opam for a number of reasons.  This email attempts to summarize where we
are and my current understanding and lack thereof.

** email

I'd appreciate trimming of things that aren't being replied to and not
top posting :-)

** an aside about make

Unison's official build system is make, and I don't see that changing
soon -- and definitely not as part of this discussion.  Before that
question can be entertained, we need to ensure that:

  - The dune/opam world is in order and there is broad agreement about it.

  - The dune build works on every system that can build unison now.
    (There was a comment about "IMHO the key point is what OS /
    architectures unison wants to support. If that set is covered by
    dune, there are few resons to keep using make." and that is in my
    view a red flag.)
    
  - The dune build deals with documentation and everything else built by
    the current system.

  - "dune clean" restores the repo to original state

  - The dune build has support for tests, and can be extended (or
    perhaps wrapped) to do everything that our current CI does.

  - Packaging systems are ok with the dune build.

  - The active maintainers/contributors have the dune/opam stuff paged
    in.

Maybe this is more ok than I fear, but I have seen way too many
instances of proposals to change build systems that claim they will be
strictly better, but then aren't, to trust anything, and to do anything
other than a written-requirements type process.

I don't want to discuss this now, as it's a distraction from what to do
about dune-project and opam changes, and I'd like to get to closure on
those before 2.53.1.

* my understanding (and lack thereof!)

** opam

opam is an ocaml-native packaging system, in which there is a file `foo.opam`
that decribes a package and how to build it.   Sometimes these
control files are in https://github.com/ocaml/opam-repository and
sometimes they are in the package repositories (and perhaps in released
tarballs also).

What I don't understand:

  - Why are control files sometimes in the above repo and sometimes in
    the project repos?

  - Which location is preferred and why?

  - Does the file have to be at top-level in the project, if it's there
    (there are N packaging systems and it's not really ok for each to
    demand stuff in the top-level directory)?

  - When there are multiple files for multiple build flavors, can one
    build them without colliding?   Or is this really a dune issue?

** dune https://github.com/ocaml/dune

dune is an ocaml-specific build system.  It works by having a file
`dune-project` that provides much the same information as an opam file,
and files `dune` in various directories that describe how to build.

A `dune-project` file duplicates the contents of opam files, and dune
can write opam files.

What I don't understand:
  - Is the definition of projects necessary to use dune?
  - Why are there magic numbers in dune files, e.g.
    "-w -3-6-9-10-26-27-32-34-35-38-39-50-52".

** dune/opam relationship

Some people seem to think opam files should be hand maintained, and some
don't.

What I don't understand:

  - What are the arguments for and against the two styles
    (dune-generated vs hand-maintained)?

  - If opam files are in the opam repository, and also generated, how does
    that work?

  - If files are generated, then mostly-obviously they shouldn't be
    checked in.  Do people think they should be?  If they are, are we
    sure they will be bit-for-bit identical when regenerated on
    different systems?

* where we are

** opam

Some changes have been applied to unison.opam (because I was able to
convince myself they were ok, by a combination of understanding and
having the questions I asked be answered (thank you kit-ty-kate!)).

  - complaint about opam and not having a CLI version
    https://github.com/bcpierce00/unison/issues/793

    I don't understand; it seems the unison.opam file requires lablgtk3
    and then builds a CLI-only version.  While that's wrong, the issue
    says something else.

  - complaint (valid!) about status of opam packages and the
    relationship to the opam repository
    https://github.com/bcpierce00/unison/issues/827

    This is validly mostly duplicative with #793, and also has a lot of
    broad-ranging discussion that belongs here.

** dune

There are no open issues/PRs or mailinglist complaints that seem
current.

There is

  - c_names issue in 2.51.2
    https://github.com/bcpierce00/unison/issues/317

    I think this is now history and should be closed.  I pinged the
    issue and will close it if I don't hear back.

* dune-project

We have a PR on the table to rototill the dune-project setup:
  https://github.com/bcpierce00/unison/pull/671

This does two unrelated things:

  - Updates dependencies: dune 2.7, lablgtk3.

  - Converts to generating opam files which overwrites 1 and adds 2.
    The new files are similar, but I see a new dependency on odoc, which
    is not currently known to me to be a unison dependency.

I have problems with this PR, some of which I've tried to ask about, and
some of which are new from understanding gained while writing this message.

  - There is not a reasonable commit message.  (A commit message should
    explain why the changes are being made, and should be understandable
    to those who are not dune experts.)

  - I'm not sure that everyone is ok with generating opam files.

  - It's not clear that opam files, if generated, should be checked in.

  - Regeneration of opam files is in the same commit is not ok; this is
    of course easy to fix.

* way forward

At this point, my inclination is:

  - Reset opam maintainer to `unison-hackers at lists.seas.upenn.edu`
    because the current maintainer has not answered mail, and we don't
    really have another plan.  This is saying that the file is
    maintained by the project, just like every other file in the repo.

  - Apply the lablgtk3 and dune 2.7 changes from #671, basically
    cherrypicking the parts I understand and agree with, and writing a
    commit message.

  - Consider turning on generation, and removing unison.opam from the
    repo, pending feedback, probably via a PR that does only that.

  - See if there are things to be done other than the above 3.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20221115/e802dd8d/attachment.asc>