From toivol at gmail.com  Sat Jun  5 15:15:51 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Sat, 5 Jun 2021 21:15:51 +0200
Subject: [Unison-hackers] Make "watch = false" the default
Message-ID: <CAALvpZQMwhW4YYEMzsMW3o7TY3c+utZ9-wj8S-xwMQHAp27KYQ@mail.gmail.com>

The "watch" preference defaulting to "true" has caused some issues
([1] [2] and some others) to users who may not even know they are
running with the watcher enabled. With some of these bugs unresolved
and watcher semantics still somewhat confusing and limited (see
below), I'd like to initiate a discussion to make "watch" preference
default to "false". This default can then be revised again once
critical bugs are fixed and semantics extended or better documented.

The exact proposal is this:
1. "watch = false" becomes the default for everyone.
2. "watch" is automatically flipped to "true" if "-repeat watch" is specified.

Reasoning for point 2 is simple: "-repeat watch" requires "watch =
true" and specifying "repeat" is the only request the user should make
or be forced to know about.

Reasoning for point 1 can now completely exclude scenarios where the
user has specified "-repeat watch". It is my assumption that using
"watch = true" without "-repeat watch" is currently almost always a
result of the defaults, not of user intention. There is no clear
documentation of the "watch" preference for users to base their
intentions on.

I have investigated the code a bit and my current (possibly limited or
incorrect) understanding is that a) the "watch" preference without
"-repeat watch" is only leveraged by the GUIs; and b) the effect of
the "watch = true" without "-repeat watch" is limited to non-existent.
Users (in general, but even more so when using the text UI) are not
getting any benefits from the default preference, yet they are exposed
to potential bugs.

The limited to non-existent effect claim is based on my understanding
that input from the fsmonitor process is used solely to filter the
(static) list of paths Unison is going to scan. The list of paths to
scan itself does not come from fsmonitor process, it is the list of
paths from the profile. For example, if the user has not specified any
explicit paths then the paths to scan are the roots. Here, the effect
from "watch = true" is binary, either not scan the root (no changes
reported by fsmonitor) or scan the entire root (any change reported by
fsmonitor). This may sound like somewhat of a benefit but it has to be
put in context: first, it currently works only in GUIs (for example by
clicking the "Rescan" button; but not when specifying "-repeat N" on
command line), and second, re-scanning even big roots tends to be very
(even extremely) fast (assuming much has been cached). It may be
slower on Windows (I have no empirical data) but again, the decision
not to scan is made only if fsmonitor has not reported any changes at
all.

Please do correct me if my understanding is incorrect at any point.

At the moment, the fsmonitor binary is not packaged on Windows due to [2].

[1] https://github.com/bcpierce00/unison/issues/329#issuecomment-708633505
[2] https://github.com/bcpierce00/unison/pull/493#issuecomment-823419018

From bcpierce at cis.upenn.edu  Sun Jun  6 06:44:11 2021
From: bcpierce at cis.upenn.edu (Benjamin Pierce)
Date: Sun, 6 Jun 2021 06:44:11 -0400
Subject: [Unison-hackers] Make "watch = false" the default
In-Reply-To: <CAALvpZQMwhW4YYEMzsMW3o7TY3c+utZ9-wj8S-xwMQHAp27KYQ@mail.gmail.com>
References: <CAALvpZQMwhW4YYEMzsMW3o7TY3c+utZ9-wj8S-xwMQHAp27KYQ@mail.gmail.com>
Message-ID: <CABgF7-Nhydas0upojSPnvjFuM9VZNW6XWYotrU_1qMh-fyLxGw@mail.gmail.com>

This matches my (somewhat hazy :-) recollection.

       - Benjamin

On Sat, Jun 5, 2021 at 3:16 PM T?ivo Leedj?rv <toivol at gmail.com> wrote:

> The "watch" preference defaulting to "true" has caused some issues
> ([1] [2] and some others) to users who may not even know they are
> running with the watcher enabled. With some of these bugs unresolved
> and watcher semantics still somewhat confusing and limited (see
> below), I'd like to initiate a discussion to make "watch" preference
> default to "false". This default can then be revised again once
> critical bugs are fixed and semantics extended or better documented.
>
> The exact proposal is this:
> 1. "watch = false" becomes the default for everyone.
> 2. "watch" is automatically flipped to "true" if "-repeat watch" is
> specified.
>
> Reasoning for point 2 is simple: "-repeat watch" requires "watch =
> true" and specifying "repeat" is the only request the user should make
> or be forced to know about.
>
> Reasoning for point 1 can now completely exclude scenarios where the
> user has specified "-repeat watch". It is my assumption that using
> "watch = true" without "-repeat watch" is currently almost always a
> result of the defaults, not of user intention. There is no clear
> documentation of the "watch" preference for users to base their
> intentions on.
>
> I have investigated the code a bit and my current (possibly limited or
> incorrect) understanding is that a) the "watch" preference without
> "-repeat watch" is only leveraged by the GUIs; and b) the effect of
> the "watch = true" without "-repeat watch" is limited to non-existent.
> Users (in general, but even more so when using the text UI) are not
> getting any benefits from the default preference, yet they are exposed
> to potential bugs.
>
> The limited to non-existent effect claim is based on my understanding
> that input from the fsmonitor process is used solely to filter the
> (static) list of paths Unison is going to scan. The list of paths to
> scan itself does not come from fsmonitor process, it is the list of
> paths from the profile. For example, if the user has not specified any
> explicit paths then the paths to scan are the roots. Here, the effect
> from "watch = true" is binary, either not scan the root (no changes
> reported by fsmonitor) or scan the entire root (any change reported by
> fsmonitor). This may sound like somewhat of a benefit but it has to be
> put in context: first, it currently works only in GUIs (for example by
> clicking the "Rescan" button; but not when specifying "-repeat N" on
> command line), and second, re-scanning even big roots tends to be very
> (even extremely) fast (assuming much has been cached). It may be
> slower on Windows (I have no empirical data) but again, the decision
> not to scan is made only if fsmonitor has not reported any changes at
> all.
>
> Please do correct me if my understanding is incorrect at any point.
>
> At the moment, the fsmonitor binary is not packaged on Windows due to [2].
>
> [1] https://github.com/bcpierce00/unison/issues/329#issuecomment-708633505
> [2] https://github.com/bcpierce00/unison/pull/493#issuecomment-823419018
> _______________________________________________
> Unison-hackers mailing list
> Unison-hackers at LISTS.SEAS.UPENN.EDU
> https://LISTS.SEAS.UPENN.EDU/mailman/listinfo/unison-hackers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210606/0dae15e3/attachment-0001.htm>

From gdt at lexort.com  Tue Jun  8 11:51:12 2021
From: gdt at lexort.com (Greg Troxel)
Date: Tue, 08 Jun 2021 11:51:12 -0400
Subject: [Unison-hackers] intent to merge "watch = false" default
In-Reply-To: <CAALvpZQMwhW4YYEMzsMW3o7TY3c+utZ9-wj8S-xwMQHAp27KYQ@mail.gmail.com>
 (=?utf-8?Q?=22T=C3=B5ivo_Leedj=C3=A4rv=22's?= message of "Sat, 5 Jun 2021
 21:15:51 +0200")
References: <CAALvpZQMwhW4YYEMzsMW3o7TY3c+utZ9-wj8S-xwMQHAp27KYQ@mail.gmail.com>
Message-ID: <rmih7i8s6bz.fsf@s1.lexort.com>


I intend to merge this:

  https://github.com/bcpierce00/unison/pull/528

some time after 18Z on 10 June (a bit over 48h from now).

Please speak up in the PR or here if you think that isn't a good idea.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210608/538fd772/attachment.asc>

From gdt at lexort.com  Mon Jun 14 18:33:18 2021
From: gdt at lexort.com (Greg Troxel)
Date: Mon, 14 Jun 2021 18:33:18 -0400
Subject: [Unison-hackers] draft NEWS, licensing clarification, release plan
Message-ID: <rmio8c83wlt.fsf@s1.lexort.com>


* NEWS

With help from T?ivo, I have prepared NEWS for the imminent (perhaps
even this month) 2.51.4 release.

(I had meant to put it on a feature branch in my fork, but somehow git
pushed it to the main repo, and I can't force-push the old ref, so I'm
just going with it.  I intend to normally make changes via the PR
process.)

So please have a look and tell me what's wrong/missing, including
replacement/new text I can use.

  https://github.com/bcpierce00/unison/blob/master/doc/changes.tex

* LICENSING

Prompted by some offlist discussions, I've added a description of
licensing.  This is meant to say how it is and has been, for people who
don't have a habit of listening to Free Software licensing podcasts for
fun, and is not meant to be a policy change.

I'd specifically like a nod (or objection) from Benjamin and everyone
else is of course welcome to comment.

https://github.com/bcpierce00/unison/pull/536
https://github.com/bcpierce00/unison/pull/536/commits/6b21600c75fa0a962b38a8216b2282d5732fb5b3

* RELEASE PLAN

  - a few manual edits (e.g. mailinglist info is old) that I'll just
    amke directly

  - merge the licensing PR or decide not to

  - tag an RC and ask for testing on unison-users@
    (If you are on hackers and not users, let me know, so I can send
    that request here too.)

  - adjust version number in manual and tag release

  - As long as this heads for happening soonish, I would like to avoid
    merging anything that isn't a bugfix relative to 2.51.3 or a doc
    fix.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210614/02c9edd2/attachment.asc>

From gdt at lexort.com  Mon Jun 14 19:06:56 2021
From: gdt at lexort.com (Greg Troxel)
Date: Mon, 14 Jun 2021 19:06:56 -0400
Subject: [Unison-hackers] branch cleanup
Message-ID: <rmilf7c2ghb.fsf@s1.lexort.com>


I noticed some branches in the repo.  I wonder if these should be
brought up to date and turned into PRs, or if they are no longer needed
because something else happened, or if they are simply so old they
should just be deleted.

* https://github.com/bcpierce00/unison/tree/i18n

$ git log --oneline origin/master..origin/i18n
e893803 (upstream/i18n, origin/i18n) Add ugettext and compile with it. Allow to disable/enable gettext through ENABLE_NLS variable.
daa9608 Add the po/ directory, which stores i18n information.
f643dcd Create new i18n branch for work on internationalization

* https://github.com/bcpierce00/unison/tree/MakefileNit

$ git  log --oneline upstream/master..upstream/MakefileNit
4efa55c (upstream/MakefileNit, origin/MakefileNit) Makefile nit
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210614/c3a805e9/attachment.asc>

From bcpierce at cis.upenn.edu  Mon Jun 14 20:22:10 2021
From: bcpierce at cis.upenn.edu (Benjamin Pierce)
Date: Mon, 14 Jun 2021 20:22:10 -0400
Subject: [Unison-hackers] branch cleanup
In-Reply-To: <rmilf7c2ghb.fsf@s1.lexort.com>
References: <rmilf7c2ghb.fsf@s1.lexort.com>
Message-ID: <CABgF7-NBB0BbbkFGd0zFuSayFpBPJzTaZw=LFj8MaobBfX=QbA@mail.gmail.com>

IIRC the i8n branch never got completed.  Don't remember what the makefile
nit was about.

   - B


>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210614/9e33800a/attachment.htm>

From bcpierce at cis.upenn.edu  Mon Jun 14 20:26:05 2021
From: bcpierce at cis.upenn.edu (Benjamin Pierce)
Date: Mon, 14 Jun 2021 20:26:05 -0400
Subject: [Unison-hackers] draft NEWS, licensing clarification,
 release plan
In-Reply-To: <rmio8c83wlt.fsf@s1.lexort.com>
References: <rmio8c83wlt.fsf@s1.lexort.com>
Message-ID: <CABgF7-OosvD6WBS6RVgyC0WWnhvJN955UYAJcUX8BUznygwa_A@mail.gmail.com>

Yes, the license / conventions explanation looks fine to me.

   - B

On Mon, Jun 14, 2021 at 7:04 PM Greg Troxel <gdt at lexort.com> wrote:

>
> * NEWS
>
> With help from T?ivo, I have prepared NEWS for the imminent (perhaps
> even this month) 2.51.4 release.
>
> (I had meant to put it on a feature branch in my fork, but somehow git
> pushed it to the main repo, and I can't force-push the old ref, so I'm
> just going with it.  I intend to normally make changes via the PR
> process.)
>
> So please have a look and tell me what's wrong/missing, including
> replacement/new text I can use.
>
>   https://github.com/bcpierce00/unison/blob/master/doc/changes.tex
>
> * LICENSING
>
> Prompted by some offlist discussions, I've added a description of
> licensing.  This is meant to say how it is and has been, for people who
> don't have a habit of listening to Free Software licensing podcasts for
> fun, and is not meant to be a policy change.
>
> I'd specifically like a nod (or objection) from Benjamin and everyone
> else is of course welcome to comment.
>
> https://github.com/bcpierce00/unison/pull/536
>
> https://github.com/bcpierce00/unison/pull/536/commits/6b21600c75fa0a962b38a8216b2282d5732fb5b3
>
> * RELEASE PLAN
>
>   - a few manual edits (e.g. mailinglist info is old) that I'll just
>     amke directly
>
>   - merge the licensing PR or decide not to
>
>   - tag an RC and ask for testing on unison-users@
>     (If you are on hackers and not users, let me know, so I can send
>     that request here too.)
>
>   - adjust version number in manual and tag release
>
>   - As long as this heads for happening soonish, I would like to avoid
>     merging anything that isn't a bugfix relative to 2.51.3 or a doc
>     fix.
> _______________________________________________
> Unison-hackers mailing list
> Unison-hackers at LISTS.SEAS.UPENN.EDU
> https://LISTS.SEAS.UPENN.EDU/mailman/listinfo/unison-hackers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210614/1259bdd0/attachment-0001.htm>

From gdt at lexort.com  Wed Jun 23 10:24:22 2021
From: gdt at lexort.com (Greg Troxel)
Date: Wed, 23 Jun 2021 10:24:22 -0400
Subject: [Unison-hackers] ready?
Message-ID: <rmiim241wx5.fsf@s1.lexort.com>


I just merged

  https://github.com/bcpierce00/unison/pull/548

and I think the other PRs are not in the 'needed for release' category.

So I am inclined to promote the current state,

  6eb758ba99ee50bd517a3a30dc976039dccf3aab

to release with only a version increase and doc rebuild.  I'll go ahead
and change version/doc and not tag quite yet, maybe waiting 24h, but if
you're inclined to check it, that would be good.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210623/ed71a5cc/attachment.asc>

From gdt at lexort.com  Thu Jun 24 13:09:52 2021
From: gdt at lexort.com (Greg Troxel)
Date: Thu, 24 Jun 2021 13:09:52 -0400
Subject: [Unison-hackers] [Greg Troxel] [unison-users] 2.51.4 released!
Message-ID: <rmifsx7rxy7.fsf@s1.lexort.com>

(I realize some of you on -hackers might not be subscribed.  I am not a
fan of crossposting, and will likely only duplicate messages for things
like releases.)


-------------- next part --------------
An embedded message was scrubbed...
From: Greg Troxel <gdt at lexort.com>
Subject: [unison-users] 2.51.4 released!
Date: Thu, 24 Jun 2021 12:53:47 -0400
Size: 12700
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210624/e85cbaf3/attachment.eml>

From gdt at lexort.com  Tue Jun 29 19:38:39 2021
From: gdt at lexort.com (Greg Troxel)
Date: Tue, 29 Jun 2021 19:38:39 -0400
Subject: [Unison-hackers] version negotiation
Message-ID: <rmik0mc9r7k.fsf@s1.lexort.com>


I just wrote to -users to ask if people really use socket mode, and if
so how it is sound from a security viewpoint.  I am guessing that the
answer is that some do, that it is not sound, but some people have
threat models that say their LAN is ok.

A PR to add version negotiation is on the table:

  https://github.com/bcpierce00/unison/pull/507


I am trying to meet the dual goals of allowing automatic compat and not
being complicated, with the following constraints:

  Eventually we should have a way for two unison instances to negotatiate
  and find a common version with no user/admin interaction.

  Right now there are some people running 2.48.  I am ignoring this.

  RIght now there are a lot of people running various 2.51.   If we can
  release a new 2.51.5 with the property that it works fine both ways with the
  existing 2.51.Y, Y <= 4, but can negotiate with itself (perhaps
  choosing the current protocol) and negotiate with some 2.52.0 which
  has the current protocol and a new one, all without config, that would
  be great.

  If the future 2.52 with the new protocol can also interop with
  2.51.<=4, that's nice, but I'm not sure it's realistic

With these properties, we can say to people

  update to 2.51.5, and don't worry it's safe.  packaging systems can do
  it, even in LTS, because is is just a new optional feature along 2.51

  once everything is 2.51.5, then you can update randomly to 2.52 and
  not have trouble



My understanding is that right now, unison simply assumes that the
remote side speaks the same protocol, and things deteriorate from there
if it is not true (because of a 2.48/2.51 mismatch).

#507, as I understand it, causes the server to start off with proto
version negotation instead, and the server can be configured not to, so
that old clients can connect.

So, I am wondering if instead, we say:

  2.51.5 will simply announce as 2.51.5

  2.51.5, if it sees the remote as 2.51.5 or higher, instead of starting
  the old protocol, will send the first message of the "determine
  supported proto versions negotiation protocol".  If it sees something
  earlier, it will just do the old protocol.


this makes a version higher than a magic number being sent from the
server be a clue to the client that it can do proto version negotiation
instead of just speaking the old protocol.  And maybe the client must do
this if it is 2.51.5 or higher, and must not otherwise.

This is a bit half baked, but I wanted to throw it out for discussion.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210629/a0d0932f/attachment.asc>

From bcpierce at cis.upenn.edu  Wed Jun 30 08:04:00 2021
From: bcpierce at cis.upenn.edu (Benjamin Pierce)
Date: Wed, 30 Jun 2021 08:04:00 -0400
Subject: [Unison-hackers] version negotiation
In-Reply-To: <rmik0mc9r7k.fsf@s1.lexort.com>
References: <rmik0mc9r7k.fsf@s1.lexort.com>
Message-ID: <CABgF7-MRWgQAmkSOJHpBy4ta1xSWqMTjfFD7AUGh4-pM+d3gNg@mail.gmail.com>

This sounds like generally a good direction to me, but I don't understand
the details yet.

One tiny correction:

My understanding is that right now, unison simply assumes that the
> remote side speaks the same protocol, and things deteriorate from there
> if it is not true (because of a 2.48/2.51 mismatch).
>

There's an explicit check for version number compatibility, not an
assumption.  So things don't deteriorate; they just stop.  :-)

Keeping socket mode seems like a good idea if possible.  I do think there
are people out there with useful use cases.  (Unison gets applied an an
astonishing range of situations.)

You're probably already thinking about this, but one critical aspect of
protocol negotiation is the set of preferences that each side supports.  We
did take the step a few years ago of introducing "client-only" preferences,
which can be added without bumping the protocol number.  But this scheme is
getting fancier than that!

  2.51.5 will simply announce as 2.51.5
>
>   2.51.5, if it sees the remote as 2.51.5 or higher, instead of starting
>   the old protocol, will send the first message of the "determine
>   supported proto versions negotiation protocol".  If it sees something
>   earlier, it will just do the old protocol.
>

I'm not 100% clear how this will work.  If the *type* of some data
structure changes, then how will the code be able to switch protocols
without some kind of translation layer?

   - B


>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210630/f18104c7/attachment-0001.htm>

From toivol at gmail.com  Wed Jun 30 08:19:19 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Wed, 30 Jun 2021 14:19:19 +0200
Subject: [Unison-hackers] version negotiation
In-Reply-To: <rmik0mc9r7k.fsf@s1.lexort.com>
References: <rmik0mc9r7k.fsf@s1.lexort.com>
Message-ID: <CAALvpZRg04fF+5vRNysbcSxqbMkMA==BEq71fKddS1DAzhOeWA@mail.gmail.com>

On Wed, Jun 30, 2021 at 1:38 AM Greg Troxel <gdt at lexort.com> wrote:
>
> My understanding is that right now, unison simply assumes that the
> remote side speaks the same protocol, and things deteriorate from there
> if it is not true (because of a 2.48/2.51 mismatch).

Pretty much, but there is a hello string sent by the server which the
client verifies. If there is a 2.48/2.51 mismatch then the connection
is terminated immediately. There is no OCaml version check (just that
it is >= 4.02 something). That may work or fail in mysterious ways.

> #507, as I understand it, causes the server to start off with proto
> version negotation instead, and the server can be configured not to, so
> that old clients can connect.

Correct. Or, in case of ssh, with the latest commit the client can
actually announce that it supports proto version negotiation and if
not, the server will treat it as an old client.

As it is, only socket server users will have to do anything, and only
if they know they have to support clients < 2.51.5. For all other
scenarios, compatibility is ensured automatically.

> So, I am wondering if instead, we say:
>
>   2.51.5 will simply announce as 2.51.5

Right now, only server announces itself. Clients don't. And, right
now, all existing clients expect to receive a _fixed_ hello string
(which is "Unison 2.51 with OCaml >= 4.01.2\n").

>   2.51.5, if it sees the remote as 2.51.5 or higher, instead of starting
>   the old protocol, will send the first message of the "determine
>   supported proto versions negotiation protocol".  If it sees something
>   earlier, it will just do the old protocol.

Right now, the server announces itself and then immediately starts the
OCaml-dependent protocol. This makes it very difficult for client to
say anything. This, combined with the fixed hello string above, makes
adding any auto-detection rather difficult. That's why the
auto-detection trick over ssh uses command line arguments.

I thought about this approach -- fine, let the server start its
protocol. And then the client announces itself/negotiates _within_ the
existing protocol. Alas, this won't work because the existing protocol
is OCaml-dependent and so it can't be relied on for even the simplest
step like announcing the client.

From toivol at gmail.com  Wed Jun 30 10:19:51 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Wed, 30 Jun 2021 16:19:51 +0200
Subject: [Unison-hackers] version negotiation
In-Reply-To: <CABgF7-MRWgQAmkSOJHpBy4ta1xSWqMTjfFD7AUGh4-pM+d3gNg@mail.gmail.com>
References: <rmik0mc9r7k.fsf@s1.lexort.com>
 <CABgF7-MRWgQAmkSOJHpBy4ta1xSWqMTjfFD7AUGh4-pM+d3gNg@mail.gmail.com>
Message-ID: <CAALvpZT04BXNzcht6gf-Ryd8sVbg5Mj5Tbs_09Jo0a5=RDAt-g@mail.gmail.com>

On Wed, Jun 30, 2021 at 3:51 PM Benjamin Pierce <bcpierce at cis.upenn.edu> wrote:
>
> I'm not 100% clear how this will work.  If the type of some data structure changes, then how will the code be able to switch protocols without some kind of translation layer?

Yes, there must be a translation layer. Luckily, the encoding format
itself largely fills the role of translation layer, together with
conditional/versioned de-/serialization.

From gdt at lexort.com  Wed Jun 30 12:00:51 2021
From: gdt at lexort.com (Greg Troxel)
Date: Wed, 30 Jun 2021 12:00:51 -0400
Subject: [Unison-hackers] version negotiation
In-Reply-To: <CAALvpZRg04fF+5vRNysbcSxqbMkMA==BEq71fKddS1DAzhOeWA@mail.gmail.com>
 (=?utf-8?Q?=22T=C3=B5ivo_Leedj=C3=A4rv=22's?= message of "Wed, 30 Jun 2021
 14:19:19 +0200")
References: <rmik0mc9r7k.fsf@s1.lexort.com>
 <CAALvpZRg04fF+5vRNysbcSxqbMkMA==BEq71fKddS1DAzhOeWA@mail.gmail.com>
Message-ID: <rmizgv77364.fsf@s1.lexort.com>


T?ivo Leedj?rv <toivol at gmail.com> writes:

> On Wed, Jun 30, 2021 at 1:38 AM Greg Troxel <gdt at lexort.com> wrote:
>>
>> My understanding is that right now, unison simply assumes that the
>> remote side speaks the same protocol, and things deteriorate from there
>> if it is not true (because of a 2.48/2.51 mismatch).
>
> Pretty much, but there is a hello string sent by the server which the
> client verifies. If there is a 2.48/2.51 mismatch then the connection
> is terminated immediately. There is no OCaml version check (just that
> it is >= 4.02 something). That may work or fail in mysterious ways.

ok.

>> #507, as I understand it, causes the server to start off with proto
>> version negotation instead, and the server can be configured not to, so
>> that old clients can connect.
>
> Correct. Or, in case of ssh, with the latest commit the client can
> actually announce that it supports proto version negotiation and if
> not, the server will treat it as an old client.

So, the server will default to old unless
  - invoked over ssh and arg from client that it does version
    negotitation is passed, or
  - socket server and same arg is given, which flips the socket server
    from "compatible with 2.51.x" to "compatible with >= 2.51.5"

>> So, I am wondering if instead, we say:
>>
>>   2.51.5 will simply announce as 2.51.5
>
> Right now, only server announces itself. Clients don't. And, right
> now, all existing clients expect to receive a _fixed_ hello string
> (which is "Unison 2.51 with OCaml >= 4.01.2\n").

fixed but with arbitrary numbers presuambly

> Right now, the server announces itself and then immediately starts the
> OCaml-dependent protocol. This makes it very difficult for client to
> say anything. This, combined with the fixed hello string above, makes
> adding any auto-detection rather difficult. That's why the
> auto-detection trick over ssh uses command line arguments.

I am not quite willing to give up because if we can succeed here it will
make a lot of people's lives easier.  OTOH it's only socket users we are
worrying about, and I think that's unsound :-)

So how about:

  server sends vesrion string

  server sends first message of existing protocol

  if client wants to do old (because client is old, or because client
  sees a version < 2.51.5), client speaks old protocol as it used to

  if client is new and sees 2.51.5 or higher, then client knows server
  can do the negotiation protocol.  Instead of sending a well-formed
  old-protocol message, it sends "BEGIN-VERSION-NEGOTIATION\n" or
  something, perhaps shorter to be shorter than any legit old-proto
  message.  The server can read the bytes into a buffer, and check for
  1) valid old-protocol frame and 2) matching the magic token.  If magic
  token, then server sends magic token and starts the negotiation
  protocol.  If valid old-proto frame, it's processed.

  If client sent BEGIN-VERSION-NEGOTIATION, client discards bytes until it
  finds the server magic token.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210630/318dddf4/attachment.asc>

From toivol at gmail.com  Wed Jun 30 12:42:30 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Wed, 30 Jun 2021 18:42:30 +0200
Subject: [Unison-hackers] version negotiation
In-Reply-To: <rmizgv77364.fsf@s1.lexort.com>
References: <rmik0mc9r7k.fsf@s1.lexort.com>
 <CAALvpZRg04fF+5vRNysbcSxqbMkMA==BEq71fKddS1DAzhOeWA@mail.gmail.com>
 <rmizgv77364.fsf@s1.lexort.com>
Message-ID: <CAALvpZTcwXwB6rhOOXaz0eB4Ghm5xiAiUPsSq++7Oj5WjGtP4A@mail.gmail.com>

On Wed, Jun 30, 2021 at 6:00 PM Greg Troxel <gdt at lexort.com> wrote:
>
> So, the server will default to old unless
>   - invoked over ssh and arg from client that it does version
>     negotitation is passed, or

yes

>   - socket server and same arg is given, which flips the socket server
>     from "compatible with 2.51.x" to "compatible with >= 2.51.5"

the opposite, with current PR, but this could be flipped for a
transition period.

> > Right now, only server announces itself. Clients don't. And, right
> > now, all existing clients expect to receive a _fixed_ hello string
> > (which is "Unison 2.51 with OCaml >= 4.01.2\n").
>
> fixed but with arbitrary numbers presuambly

Can be changed at build time. All the servers/clients already deployed
out there verify this string byte-for-byte.

> So how about:
>
>   server sends vesrion string
>
>   server sends first message of existing protocol
>
>   if client wants to do old (because client is old, or because client
>   sees a version < 2.51.5), client speaks old protocol as it used to

so far so good.

>   if client is new and sees 2.51.5 or higher, then client knows server

Here's where it gets difficult. To keep compatibility with old
clients, it is difficult for server to announce that it is >= 2.51.5.
I don't say it is impossible...
Alternative is for the client to query it in a backwards-compatible
way. The naive approach that comes to mind is to use the old protocol
and just query it... except the old protocol is OCaml-dependent. So
this would work only during a transition period when users still have
to match OCaml versions.

If the client does not know that server is >= 2.51.5 then it can only
query/announce itself in a completely backwards-compatible way.

I'll keep thinking about it. There could be some tricky ways, abusing
the current protocol, like sending two 'write tokens' instead of
one...

From toivol at gmail.com  Wed Jun 30 13:00:10 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Wed, 30 Jun 2021 19:00:10 +0200
Subject: [Unison-hackers] version negotiation
In-Reply-To: <CAALvpZTcwXwB6rhOOXaz0eB4Ghm5xiAiUPsSq++7Oj5WjGtP4A@mail.gmail.com>
References: <rmik0mc9r7k.fsf@s1.lexort.com>
 <CAALvpZRg04fF+5vRNysbcSxqbMkMA==BEq71fKddS1DAzhOeWA@mail.gmail.com>
 <rmizgv77364.fsf@s1.lexort.com>
 <CAALvpZTcwXwB6rhOOXaz0eB4Ghm5xiAiUPsSq++7Oj5WjGtP4A@mail.gmail.com>
Message-ID: <CAALvpZR8eGToN+UnZ3vJ4K20tXK7eFDtdncbKHJ6N3v4V0OJ3Q@mail.gmail.com>

On Wed, Jun 30, 2021 at 6:42 PM T?ivo Leedj?rv <toivol at gmail.com> wrote:
>
> I'll keep thinking about it. There could be some tricky ways, abusing
> the current protocol, like sending two 'write tokens' instead of
> one...

I looked in the code. This may very well be a perfectly working
solution. Sending multiple 'write tokens' in sequence should not break
the old clients.

Let's say it works like this.

Old server:
1. Sends header string.
2. <old protocol is enabled> Sends write token.
3. Old protocol communication.

New server:
1. Sends old header string.
2. <old protocol is enabled> Sends, let's say 5 write tokens.
3a. Old client will process all 5 separately and continue old protocol
communication normally.
3b. New client will know it is a magic signal and switch to protocol
negotiation mode.

This works because the 'write token' does not depend on OCaml version
or Marshal module.

That's the theory. It is easy enough to try out, so that's what I'll do.

From toivol at gmail.com  Wed Jun 30 13:18:18 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Wed, 30 Jun 2021 19:18:18 +0200
Subject: [Unison-hackers] version negotiation
In-Reply-To: <CAALvpZR8eGToN+UnZ3vJ4K20tXK7eFDtdncbKHJ6N3v4V0OJ3Q@mail.gmail.com>
References: <rmik0mc9r7k.fsf@s1.lexort.com>
 <CAALvpZRg04fF+5vRNysbcSxqbMkMA==BEq71fKddS1DAzhOeWA@mail.gmail.com>
 <rmizgv77364.fsf@s1.lexort.com>
 <CAALvpZTcwXwB6rhOOXaz0eB4Ghm5xiAiUPsSq++7Oj5WjGtP4A@mail.gmail.com>
 <CAALvpZR8eGToN+UnZ3vJ4K20tXK7eFDtdncbKHJ6N3v4V0OJ3Q@mail.gmail.com>
Message-ID: <CAALvpZRDijZ1JSn90UmDS8Tu8QAtOgBGDBS1F9BPsjhvVixykw@mail.gmail.com>

On Wed, Jun 30, 2021 at 7:00 PM T?ivo Leedj?rv <toivol at gmail.com> wrote:
>
> I looked in the code. This may very well be a perfectly working
> solution. Sending multiple 'write tokens' in sequence should not break
> the old clients.

No luck...

> 3a. Old client will process all 5 separately and continue old protocol
> communication normally.

In fact, it will not process multiple write tokens. There is an
assertion in the code that prevents this.
All is not lost, though. I'll try with a bit more complicated version
of this 'protocol abuse'.

From toivol at gmail.com  Wed Jun 30 15:39:49 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Wed, 30 Jun 2021 21:39:49 +0200
Subject: [Unison-hackers] version negotiation
In-Reply-To: <CAALvpZRDijZ1JSn90UmDS8Tu8QAtOgBGDBS1F9BPsjhvVixykw@mail.gmail.com>
References: <rmik0mc9r7k.fsf@s1.lexort.com>
 <CAALvpZRg04fF+5vRNysbcSxqbMkMA==BEq71fKddS1DAzhOeWA@mail.gmail.com>
 <rmizgv77364.fsf@s1.lexort.com>
 <CAALvpZTcwXwB6rhOOXaz0eB4Ghm5xiAiUPsSq++7Oj5WjGtP4A@mail.gmail.com>
 <CAALvpZR8eGToN+UnZ3vJ4K20tXK7eFDtdncbKHJ6N3v4V0OJ3Q@mail.gmail.com>
 <CAALvpZRDijZ1JSn90UmDS8Tu8QAtOgBGDBS1F9BPsjhvVixykw@mail.gmail.com>
Message-ID: <CAALvpZS1n_ZU_8xHdY+wnkuAAwhd3XO3b6-2UY959TuNX1bZcA@mail.gmail.com>

Sorry for spamming, this will be the last one for today.

I believe I have a working solution to automatically detect that both
server and client are >= 2.51.5 and then proceed with protocol upgrade
and negotiation. I will clean up the code and push into the PR.

It is somewhat of an ugly solution but since it only matters with
already released versions then we can be sure about how they behave
(still, must get plenty of field testing this time). The solution
relies on aspects of the protocol that do not depend on OCaml version.

This covers the last bit of the puzzle. All scenarios will now
automatically detect >= 2.51.5 compatibility and switch to new
protocol negotiation. This means that the server default can be <
2.51.5.