From toivol at gmail.com  Wed Feb  3 06:47:33 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Wed, 3 Feb 2021 12:47:33 +0100
Subject: [Unison-hackers] More invasive code cleanup?
Message-ID: <CAALvpZSGphR2C-YyxsCF48FVq_e7YO=89GUe-LS3HgQU=vPTBQ@mail.gmail.com>

Eventually version compatibility will be broken (for example, when
switching to a more stable protocol). When version compatibility is
going to be broken, what is the appetite for other more invasive
changes? I would like to get opinions and arguments.

I am thinking of several code cleanup and maintenance changes, not all
of which break version compatibility, that can be more invasive and
need more preparation, review and testing.

Just as an example, from the top of my head:
 - Removing CTimeStamp (which has long not been used) -> changes archive format
 - system/system_win* seems to be nearly (or completely) obsolete with
newer OCaml versions and might be removed. Once that is gone, all
system/* becomes obsolete?
 - In several places in code you find FIXME and TODO markers
 - Sometimes you find comments like (*FIX: remove when Unison version > 2.40 *)
 - There is some code for functions that are now covered by Stdlib (I
assume the code is in Unison tree because it was not available in
earlier versions of OCaml)

Much of this can make the code smaller and easier to maintain in
future. Some could fix long-standing issues. None would bring in new
external dependencies.

Is there readyness to review and test? Or is the wider opinion to
leave the code alone as much as possible and only maintain it to keep
it running with newer OCaml and OS versions?

-- 
Toivo

From gdt at lexort.com  Wed Feb  3 10:01:04 2021
From: gdt at lexort.com (Greg Troxel)
Date: Wed, 03 Feb 2021 10:01:04 -0500
Subject: [Unison-hackers] More invasive code cleanup?
In-Reply-To: <CAALvpZSGphR2C-YyxsCF48FVq_e7YO=89GUe-LS3HgQU=vPTBQ@mail.gmail.com>
 (=?utf-8?Q?=22T=C3=B5ivo_Leedj=C3=A4rv=22's?= message of "Wed, 3 Feb 2021
 12:47:33 +0100")
References: <CAALvpZSGphR2C-YyxsCF48FVq_e7YO=89GUe-LS3HgQU=vPTBQ@mail.gmail.com>
Message-ID: <rmieehx19hr.fsf@s1.lexort.com>


T?ivo Leedj?rv <toivol at gmail.com> writes:

> Eventually version compatibility will be broken (for example, when
> switching to a more stable protocol). When version compatibility is
> going to be broken, what is the appetite for other more invasive
> changes? I would like to get opinions and arguments.

I think the protocol breaks (both ocmal and otherwise) are the biggest
problems unison has right now, by a wide margin.  Obviously (backwards
compat being hard probably) we're going to have another break as we move
to some stable format that is with any luck extendable.

In my view, it would take a very good reason to have protocol break
before the big break.  To put that in perspective, I don't think ACL
support is a good enough reason, based on the balance of bugtracker and
list comments about protocol breaks vs "I can't use unison because I
need ACL sync".

I know St?phane has some code for a new protocol, and I think we should
have a big picture discussion about the approach.  The world has
increasing gone to "JSON is the answer" for interchange in many places,
and there is BSON.  I realize there is also protobuf and some others.  I
would like to avoid a format used only by uniso, and prefer a way to use
something maintained by others and widely depended on.   Besides that
sort of issue, I think we need a plan for being able to extend the
format while allowing version compat.   The ACL changes -- but I could
be confused -- seem like a good trial case.  Basically, design a format
for unison w/o ACL, and then a change or ACL, and evaluate if the two
versions will interop when not doing acl sync.

> I am thinking of several code cleanup and maintenance changes, not all
> of which break version compatibility, that can be more invasive and
> need more preparation, review and testing.
>
> Just as an example, from the top of my head:
>  - Removing CTimeStamp (which has long not been used) -> changes archive format

So that would be another thought experiment for the new format.

>  - system/system_win* seems to be nearly (or completely) obsolete with
> newer OCaml versions and might be removed. Once that is gone, all
> system/* becomes obsolete?
>  - In several places in code you find FIXME and TODO markers
>  - Sometimes you find comments like (*FIX: remove when Unison version > 2.40 *)

I don't object to these.   But I would like to see others review and
test.  I wonder if we should be thinking about improved automated tests
before this sort of rototill.

>  - There is some code for functions that are now covered by Stdlib (I
> assume the code is in Unison tree because it was not available in
> earlier versions of OCaml)

I don't understand enough to have an opinion (but sounds reasonable).

> Is there readyness to review and test? Or is the wider opinion to
> leave the code alone as much as possible and only maintain it to keep
> it running with newer OCaml and OS versions?

I would love to hear from others who are willing to spend time on this.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210203/577cea95/attachment-0001.asc>

From bcpierce at cis.upenn.edu  Wed Feb  3 11:10:16 2021
From: bcpierce at cis.upenn.edu (Benjamin Pierce)
Date: Wed, 3 Feb 2021 11:10:16 -0500
Subject: [Unison-hackers] More invasive code cleanup?
In-Reply-To: <rmieehx19hr.fsf@s1.lexort.com>
References: <CAALvpZSGphR2C-YyxsCF48FVq_e7YO=89GUe-LS3HgQU=vPTBQ@mail.gmail.com>
 <rmieehx19hr.fsf@s1.lexort.com>
Message-ID: <CABgF7-NwrbSU=JZLrbRr==aOo91d=Qp=qsMvvzZCY4TxyBJkJw@mail.gmail.com>

Roughly speaking, there are three sorts of code in Unison: (1)
user-interface, (2) core update detection, propagation, and record-keeping,
and (3) the interface to the filesystem, communication with remote hosts,
and other low-level stuff.

Type (1) code is pretty easy and pretty safe to change and extend. If
there's cleanup to do there, that's also easy.

Type (3) code requires a bit more insight into how Unison works, more OCaml
fluency, and more careful thought about how the information generated at
the low level is used at higher levels. The success of recent work on the
communication protocol, ACLs, etc. shows that it's feasible to continue
making improvements at this level.

Type (2) is a different story. Even when I had my fingers on pretty much
the whole codebase on a regular basis, I was scared to touch the update
detection or reconciliation functionality, and every time I did it was a
stressful (and time consuming) experience. The logic there is very clever
(Jerome Vouillon, who wrote a lot of it, is a *very* good engineer), but
also quite intricate and easy to break. Unless there is a strong need to do
so, I would advocate leaving things alone there.

Best,

    - Benjamin

On Wed, Feb 3, 2021 at 10:01 AM Greg Troxel <gdt at lexort.com> wrote:

>
> T?ivo Leedj?rv <toivol at gmail.com> writes:
>
> > Eventually version compatibility will be broken (for example, when
> > switching to a more stable protocol). When version compatibility is
> > going to be broken, what is the appetite for other more invasive
> > changes? I would like to get opinions and arguments.
>
> I think the protocol breaks (both ocmal and otherwise) are the biggest
> problems unison has right now, by a wide margin.  Obviously (backwards
> compat being hard probably) we're going to have another break as we move
> to some stable format that is with any luck extendable.
>
> In my view, it would take a very good reason to have protocol break
> before the big break.  To put that in perspective, I don't think ACL
> support is a good enough reason, based on the balance of bugtracker and
> list comments about protocol breaks vs "I can't use unison because I
> need ACL sync".
>
> I know St?phane has some code for a new protocol, and I think we should
> have a big picture discussion about the approach.  The world has
> increasing gone to "JSON is the answer" for interchange in many places,
> and there is BSON.  I realize there is also protobuf and some others.  I
> would like to avoid a format used only by uniso, and prefer a way to use
> something maintained by others and widely depended on.   Besides that
> sort of issue, I think we need a plan for being able to extend the
> format while allowing version compat.   The ACL changes -- but I could
> be confused -- seem like a good trial case.  Basically, design a format
> for unison w/o ACL, and then a change or ACL, and evaluate if the two
> versions will interop when not doing acl sync.
>
> > I am thinking of several code cleanup and maintenance changes, not all
> > of which break version compatibility, that can be more invasive and
> > need more preparation, review and testing.
> >
> > Just as an example, from the top of my head:
> >  - Removing CTimeStamp (which has long not been used) -> changes archive
> format
>
> So that would be another thought experiment for the new format.
>
> >  - system/system_win* seems to be nearly (or completely) obsolete with
> > newer OCaml versions and might be removed. Once that is gone, all
> > system/* becomes obsolete?
> >  - In several places in code you find FIXME and TODO markers
> >  - Sometimes you find comments like (*FIX: remove when Unison version >
> 2.40 *)
>
> I don't object to these.   But I would like to see others review and
> test.  I wonder if we should be thinking about improved automated tests
> before this sort of rototill.
>
> >  - There is some code for functions that are now covered by Stdlib (I
> > assume the code is in Unison tree because it was not available in
> > earlier versions of OCaml)
>
> I don't understand enough to have an opinion (but sounds reasonable).
>
> > Is there readyness to review and test? Or is the wider opinion to
> > leave the code alone as much as possible and only maintain it to keep
> > it running with newer OCaml and OS versions?
>
> I would love to hear from others who are willing to spend time on this.
> _______________________________________________
> Unison-hackers mailing list
> Unison-hackers at LISTS.SEAS.UPENN.EDU
> https://LISTS.SEAS.UPENN.EDU/mailman/listinfo/unison-hackers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://LISTS.SEAS.UPENN.EDU/pipermail/unison-hackers/attachments/20210203/27c5f567/attachment.htm>

From toivol at gmail.com  Thu Feb  4 11:40:46 2021
From: toivol at gmail.com (=?UTF-8?B?VMO1aXZvIExlZWRqw6Rydg==?=)
Date: Thu, 4 Feb 2021 17:40:46 +0100
Subject: [Unison-hackers] More invasive code cleanup?
In-Reply-To: <CABgF7-NwrbSU=JZLrbRr==aOo91d=Qp=qsMvvzZCY4TxyBJkJw@mail.gmail.com>
References: <CAALvpZSGphR2C-YyxsCF48FVq_e7YO=89GUe-LS3HgQU=vPTBQ@mail.gmail.com>
 <rmieehx19hr.fsf@s1.lexort.com>
 <CABgF7-NwrbSU=JZLrbRr==aOo91d=Qp=qsMvvzZCY4TxyBJkJw@mail.gmail.com>
Message-ID: <CAALvpZRXNe85x6k12skZNhOMV3O5dAQoTEPYsJhpewYd3oYcZA@mail.gmail.com>

Thank you both.

Very good points, Benjamin, and definitely something to keep in mind.
The more I poke around in the code, the more obvious it becomes that
what you wrote about type (2) is absolutely true.

Given the rather low intensity of reviews and testing, better
automated testing coverage looks to be somewhat of a priority even.

On Wed, Feb 3, 2021 at 5:10 PM Benjamin Pierce <bcpierce at cis.upenn.edu> wrote:
>
> Roughly speaking, there are three sorts of code in Unison: (1) user-interface, (2) core update detection, propagation, and record-keeping, and (3) the interface to the filesystem, communication with remote hosts, and other low-level stuff.
>
> Type (1) code is pretty easy and pretty safe to change and extend. If there's cleanup to do there, that's also easy.
>
> Type (3) code requires a bit more insight into how Unison works, more OCaml fluency, and more careful thought about how the information generated at the low level is used at higher levels. The success of recent work on the communication protocol, ACLs, etc. shows that it's feasible to continue making improvements at this level.
>
> Type (2) is a different story. Even when I had my fingers on pretty much the whole codebase on a regular basis, I was scared to touch the update detection or reconciliation functionality, and every time I did it was a stressful (and time consuming) experience. The logic there is very clever (Jerome Vouillon, who wrote a lot of it, is a very good engineer), but also quite intricate and easy to break. Unless there is a strong need to do so, I would advocate leaving things alone there.
>
> Best,
>
>     - Benjamin
>
> On Wed, Feb 3, 2021 at 10:01 AM Greg Troxel <gdt at lexort.com> wrote:
>>
>>
>> T?ivo Leedj?rv <toivol at gmail.com> writes:
>>
>> > Eventually version compatibility will be broken (for example, when
>> > switching to a more stable protocol). When version compatibility is
>> > going to be broken, what is the appetite for other more invasive
>> > changes? I would like to get opinions and arguments.
>>
>> I think the protocol breaks (both ocmal and otherwise) are the biggest
>> problems unison has right now, by a wide margin.  Obviously (backwards
>> compat being hard probably) we're going to have another break as we move
>> to some stable format that is with any luck extendable.
>>
>> In my view, it would take a very good reason to have protocol break
>> before the big break.  To put that in perspective, I don't think ACL
>> support is a good enough reason, based on the balance of bugtracker and
>> list comments about protocol breaks vs "I can't use unison because I
>> need ACL sync".
>>
>> I know St?phane has some code for a new protocol, and I think we should
>> have a big picture discussion about the approach.  The world has
>> increasing gone to "JSON is the answer" for interchange in many places,
>> and there is BSON.  I realize there is also protobuf and some others.  I
>> would like to avoid a format used only by uniso, and prefer a way to use
>> something maintained by others and widely depended on.   Besides that
>> sort of issue, I think we need a plan for being able to extend the
>> format while allowing version compat.   The ACL changes -- but I could
>> be confused -- seem like a good trial case.  Basically, design a format
>> for unison w/o ACL, and then a change or ACL, and evaluate if the two
>> versions will interop when not doing acl sync.
>>
>> > I am thinking of several code cleanup and maintenance changes, not all
>> > of which break version compatibility, that can be more invasive and
>> > need more preparation, review and testing.
>> >
>> > Just as an example, from the top of my head:
>> >  - Removing CTimeStamp (which has long not been used) -> changes archive format
>>
>> So that would be another thought experiment for the new format.
>>
>> >  - system/system_win* seems to be nearly (or completely) obsolete with
>> > newer OCaml versions and might be removed. Once that is gone, all
>> > system/* becomes obsolete?
>> >  - In several places in code you find FIXME and TODO markers
>> >  - Sometimes you find comments like (*FIX: remove when Unison version > 2.40 *)
>>
>> I don't object to these.   But I would like to see others review and
>> test.  I wonder if we should be thinking about improved automated tests
>> before this sort of rototill.
>>
>> >  - There is some code for functions that are now covered by Stdlib (I
>> > assume the code is in Unison tree because it was not available in
>> > earlier versions of OCaml)
>>
>> I don't understand enough to have an opinion (but sounds reasonable).
>>
>> > Is there readyness to review and test? Or is the wider opinion to
>> > leave the code alone as much as possible and only maintain it to keep
>> > it running with newer OCaml and OS versions?
>>
>> I would love to hear from others who are willing to spend time on this.
>> _______________________________________________
>> Unison-hackers mailing list
>> Unison-hackers at LISTS.SEAS.UPENN.EDU
>> https://LISTS.SEAS.UPENN.EDU/mailman/listinfo/unison-hackers

From steph at glondu.net  Fri Feb  5 04:55:42 2021
From: steph at glondu.net (=?UTF-8?Q?St=c3=a9phane_Glondu?=)
Date: Fri, 5 Feb 2021 10:55:42 +0100
Subject: [Unison-hackers] More invasive code cleanup?
In-Reply-To: <rmieehx19hr.fsf@s1.lexort.com>
References: <CAALvpZSGphR2C-YyxsCF48FVq_e7YO=89GUe-LS3HgQU=vPTBQ@mail.gmail.com>
 <rmieehx19hr.fsf@s1.lexort.com>
Message-ID: <20ecb13f-98ce-a606-61a0-63107e6e5b67@glondu.net>

Le 03/02/2021 ? 16:01, Greg Troxel a ?crit?:
> I know St?phane has some code for a new protocol, and I think we should
> have a big picture discussion about the approach.  The world has
> increasing gone to "JSON is the answer" for interchange in many places,
> and there is BSON.  I realize there is also protobuf and some others.  I
> would like to avoid a format used only by uniso, and prefer a way to use
> something maintained by others and widely depended on.

I agree a big picture discussion is welcome.

In my branch:

  https://github.com/glondu/unison/tree/umarshal

I use a simple custom serialization format on which the protocol is
built. The core is in src/ubase/umarshal.ml. The changes may look big,
but are pretty simple and systematic. The constraints for this version
were: 1) do not depend on OCaml version and 2) do not add dependencies.
I also previously did two other trials (protobuf and bin-prot), but only
with constraint 1.

>   Besides that
> sort of issue, I think we need a plan for being able to extend the
> format while allowing version compat.

Note that my branch is not that, but I think is an interesting first
step towards it (which would need even more invasive changes).


Cheers,

-- 
St?phane