From toivol at gmail.com  Thu Jul  1 08:08:06 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Thu, 1 Jul 2021 14:08:06 +0200
Subject: [Unison-hackers] version negotiation
In-Reply-To: <rmizgv77364.fsf@s1.lexort.com>
References: <rmik0mc9r7k.fsf@s1.lexort.com>
 <CAALvpZRg04fF+5vRNysbcSxqbMkMA==BEq71fKddS1DAzhOeWA@mail.gmail.com>
 <rmizgv77364.fsf@s1.lexort.com>
Message-ID: <CAALvpZRG8yvnBg0yOfO2RrnVeXL0JqfnOWmQAGGPiLSaQXTX3g@mail.gmail.com>

On Wed, Jun 30, 2021 at 6:00 PM Greg Troxel <gdt at lexort.com> wrote:
>
>   server sends vesrion string
>
>   server sends first message of existing protocol
>
>   if client wants to do old (because client is old, or because client
>   sees a version < 2.51.5), client speaks old protocol as it used to
>
>   if client is new and sees 2.51.5 or higher, then client knows server
>   can do the negotiation protocol.
[...]

I've now pushed an update at
https://github.com/bcpierce00/unison/pull/507/ (ignore the build
errors, these are CI infra errors).
Conceptually this works as proposed by Greg, but to make it actually
work it relies on tricks with the old protocol. Tricks that the new
versions will notice but old versions will silently process with no
negative side-effects.

Please test with various version/platform/connection combinations if you can.

From gdt at lexort.com  Tue Jul  6 13:34:22 2021
From: gdt at lexort.com (Greg Troxel)
Date: Tue, 06 Jul 2021 13:34:22 -0400
Subject: [Unison-hackers] review/test request for multiple listening sockets
 (socket mode)
Message-ID: <rmieecb9wip.fsf@s1.lexort.com>


Toivo has submitted a PR to fix the bug where unison listens on only one
address, rather than in the typical case 127.0.0.1 and ::1.  I don't use
socket mode, and would appreciate a code review and/or independent test.
As usual I'll merge it anyway after a while if there aren't any
reviews/tests.


https://github.com/bcpierce00/unison/pull/563
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210706/355aaa32/attachment.asc>

From toivol at gmail.com  Fri Jul  9 11:06:14 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Fri, 9 Jul 2021 17:06:14 +0200
Subject: [Unison-hackers] version negotiation
In-Reply-To: <CABgF7-MRWgQAmkSOJHpBy4ta1xSWqMTjfFD7AUGh4-pM+d3gNg@mail.gmail.com>
References: <rmik0mc9r7k.fsf@s1.lexort.com>
 <CABgF7-MRWgQAmkSOJHpBy4ta1xSWqMTjfFD7AUGh4-pM+d3gNg@mail.gmail.com>
Message-ID: <CAALvpZQ6_C-EyUGvALhO1WReo4p6Y9a+FhAU_6RzV8_Z+NJ8QQ@mail.gmail.com>

On Wed, Jun 30, 2021 at 3:51 PM Benjamin Pierce <bcpierce at cis.upenn.edu> wrote:
>
> You're probably already thinking about this, but one critical aspect of protocol negotiation is the set of preferences that each side supports.  We did take the step a few years ago of introducing "client-only" preferences, which can be added without bumping the protocol number.  But this scheme is getting fancier than that!

There is a solution to this at https://github.com/bcpierce00/unison/pull/509
This PR itself is a non-breaking change.

From toivol at gmail.com  Tue Jul 20 15:04:30 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Tue, 20 Jul 2021 21:04:30 +0200
Subject: [Unison-hackers] Keeping compatibility with 2.51 on type level
Message-ID: <CAALvpZTe-BvPTWoQboMZ9mLo+BzKFihwgvsszA=OXiKXGut-Ag@mail.gmail.com>

I have added two more commits to PR
https://github.com/bcpierce00/unison/pull/507

Since the PR is now fully compatible with 2.51, there is yet another
piece of the puzzle that must be solved if we want to keep that
compatibility in the future.

Due to 2.51 using OCaml's Marshal module for on-the-wire encoding then
in addition to requiring mostly matching OCaml versions, also the
types on both ends must match exactly. And this is precisely what
these two additional commits are about.

The code is there for review, this mail is just to clarify some of the
assumptions and design decisions I made. As a background for why this
matters and why it's needed: with this foundation I can make for
example the ACL and xattr patches non-breaking.

- The assumption is that the only way to remain compatible with 2.51
  is to keep unchanged the types that are exchanged between client
  and server.
- To that end, the types that are changed (due to new features, bug
  fixes, you name it) must be duplicated: one type for the actual code
  to function and the other type that is an exact copy of how it
  existed in 2.51 <= 2.51.4.
- That copy of a type is never used in the then-current code, it is
  there only to provide on-the-wire compatibility with 2.51. As such,
  the data will be converted between the types at the moment of
  marshaling and unmarshaling with the help of the `Marshal` module.
- Each RPC call takes type-specific conversion functions (optional,
  supplied only if the types have changed).
- While easily extendable to future version changes, I have foreseen
  this mechanism to be used only for compatibility with 2.51. Managing
  type changes between future versions (those using St?phane's Umarshal,
  for example) will have basically the same mechanism built-in (by
  making dynamic/conditional the definition of "schema" for marshaling
  and unmarshaling a certain type (it will be static for most types,
  since they don't change between versions)).
- In the code, I decided to keep the copies of the so-called
  compatibility types in the same module, next to the "current" types.
  Likewise with any functions that must be duplicated to work with the
  compatibility types (besides the conversion functions, these would
  be very rare). I did it mainly for simplicity and code locality
  (compatibility code is right next to the code it provides compatibility
  for). Perhaps a cleaner solution would be to add a V0 sub-module,
  but my OCaml skills don't see how to make it work in a simple way.

In the commits you can see an example with `Update.archive` and
`Props.t` preserved in their 2.51 state. Note that in addition to
plain data conversions, the `checkArchive` function must be preserved
as well as it's result on both ends must match and is directly
dependent on the entire data structure. I have verified that this
works and retains full compatibility with 2.51 even if the current
`Props.t` is changed.

And to be complete, it has to be mentioned that this approach only
works as long as the new code does not break with the data conversions
(which by definition are lossy).

From toivol at gmail.com  Sat Jul 24 13:34:55 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Sat, 24 Jul 2021 19:34:55 +0200
Subject: [Unison-hackers] Way forward with protocol change and compatibility
 PRs
Message-ID: <CAALvpZS_5emmRHwKxdCb+shaohfCPw4yNkODu2HKx7QWSV++BQ@mail.gmail.com>

This is a subtle reminder that reviewers are needed.

In my view, with these two PRs and St?phane's Umarshal work taken
together, we are so close to ticking all the important boxes.
https://github.com/bcpierce00/unison/pull/507
https://github.com/bcpierce00/unison/pull/509

After merging these changes, we would achieve the following:
- Full independence from OCaml version
- Full independence from Unison version
- Full backwards compatibility with 2.51.x versions
  (that one still depends on OCaml version)
- Backwards-compatible extensibility
- Smooth (invisible) upgrade from 2.51.x (and older!) versions

I will continue maintaining those PRs for the foreseeable future.

What is a realistic way forward to get these changes reviewed and
merged (or rejected)?

-- 
Toivo