From gdt at lexort.com Sun Jan 9 09:53:21 2022 From: gdt at lexort.com (Greg Troxel) Date: Sun, 09 Jan 2022 09:53:21 -0500 Subject: [Unison-hackers] intent to merge "RFC Version Handshake" Message-ID: T?ivo has put a lot of work into "RPC Version Handshake", which is a way for two unison programs to negotiate an RPC version, allowing old versions to keep working while two new versions can choose a (not yet added) version. This is a compatible path to an ocaml-independent wire format, not requiring a flag day. We just had a release, and there's not much else ready to merge, so it feels like a good time to me to merge this and look toward removing ocaml version dependence. I have not seen review comments from other than myself and St?phane, and I don't want to wait for comments that will not come. So, if you are inclined to review: https://github.com/bcpierce00/unison/pull/507 please either comment there or for any really big-picture comments also write here. I will wait until 14Z on 15 January before hitting merge, and am likely to do so as soon after as I get to it, unless there are comments that lead me to believe I shouldn't. Thanks, Greg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 194 bytes Desc: not available URL: From gdt at lexort.com Thu Jan 27 10:06:23 2022 From: gdt at lexort.com (Greg Troxel) Date: Thu, 27 Jan 2022 10:06:23 -0500 Subject: [Unison-hackers] path forward on independence from ocaml version Message-ID: T?ivo has added RPC version negotiation that is backwards compatible, so with that feature, we can add a new RPC version and not impose a flag day on users. This makes not only adding a new wire protocol far easier, but it makes adding the next wire protocol later far less of a big deal. The next step is to decide on an approach for a new wire protocol and then to get an implementation staged as a pull request. Or perhaps, a new archive format, which is perhaps easier, and might be a good chioce in terms of sequencing, even though it has less of an impact. The obvious requirement is that it be an implemenation of a protocol spec, so that it is independent of ocaml version, as well as independent of CPU type, ILP32/LP64/ILP64/etc. and endianness. I won't insist on it working on a PDP-11, but it ought to work on Vax, sparc, sparc64, BE mips, etc., as long as ocaml does. Less obvious requirements and desirable properties: - a format that is already standardized is preferred. JSON or BSON would be a win. - efficient encoding, both in space and in CPU time to go to/from - avoid homegrown, relying on already debugged external libraries and standards - minimal external dependencies, because building unison will require those. One library that is in widespread use and already packaged in most packaging systems, and easy in the rest,would be totally ok. - avoid vendoring in external code - new code should go back to some oldish ocaml version, to offer a non-flag-day upgrade path to people. But right now people have to have flag days so this is not critical. It's an interesting question how old matters; I'll ask on -users. These requirements self-conflict, more or less. It would be ideal if there were a libary that marhsals to BSON, if it had a spec, if it were mature, if the rest of the ocaml world used it, if it supported back to say 4.02, and if it and all dependencies were widely packaged. I am inclined to give up on efficiency first, as size of metadata does not seem like unison's big problem. Back to reality, this seems to be the closest external library (3 links, one thing): https://gitlab.com/nomadic-labs/data-encoding/ https://nomadic-labs.gitlab.io/data-encoding/data-encoding/Data_encoding/index.html https://discuss.ocaml.org/t/ann-data-encoding-0-3-performances-and-streaming/7205 and issues are: no apparent spec (mimor) ocaml 4.10 required fairly large dependency footprint There is also umarshal, a work in progress by Stephane, which is similar to data_encoding's Binary option, and ocaml's Marhshal, but a fixed protocol. While it's homegrown, it should be possible to have ocaml compat going way back, and I don't expect it will need a lot of maintenance. has added RPC version negotiation that is backwards compatible, so with that feature, we can add a new RPC version and not impose a flag day on users. This makes not only adding a new wire protocol far easier, but it makes adding the next wire protocol later far less of a big deal. The next step is to decide on an approach for a new wire protocol and then to get an implementation staged as a pull request. Or perhaps, a new archive format, which is perhaps easier, and might be a good chioce in terms of sequencing, even though it has less of an impact. The obvious requirement is that it be an implemenation of a protocol spec, so that it is independent of ocaml version, as well as independent of CPU type, ILP32/LP64/ILP64/etc. and endianness. I won't insist on it working on a PDP-11, but it ought to work on Vax, sparc, sparc64, BE mips, etc., as long as ocaml does. Less obvious requirements and desirable properties: - a format that is already standardized is preferred. JSON or BSON would be a win. - efficient encoding, both in space and in CPU time to go to/from - avoid homegrown, relying on already debugged external libraries and standards - minimal external dependencies, because building unison will require those. One library that is in widespread use and already packaged in most packaging systems, and easy in the rest,would be totally ok. - avoid vendoring in external code - new code should go back to some oldish ocaml version, to offer a non-flag-day upgrade path to people. But right now people have to have flag days so this is not critical. It's an interesting question how old matters; I'll ask on -users. These requirements self-conflict, more or less. It would be ideal if there were libary that marhsals to BSON, if it had a spec, if it were mature, if the rest of the ocaml world used it, if it supported back to say 4.02, and if it were widely packaged. I am inclined to give up on efficiency first, as size of metadata does not seem like unison's big problem. Back to reality, this seems to be the closest external library (3 links, one thing): https://gitlab.com/nomadic-labs/data-encoding/ https://nomadic-labs.gitlab.io/data-encoding/data-encoding/Data_encoding/index.html https://discuss.ocaml.org/t/ann-data-encoding-0-3-performances-and-streaming/7205 and issues are: no apparent spec (mimor) ocaml 4.10 required fairly large dependency footprint The same people have https://gitlab.com/nomadic-labs/json-data-encoding which seems 1) json only 2) lighter weight but 3) still 4.10 adn 4) still a fairly heeft dependency load. There is also umarshal, a work in progress by Stephane, which is similar to data_encoding's Binary option, and ocaml's Marhshal, but a fixed protocol. While it's homegrown, it should be possible to have ocaml compat going way back, and I don't expect it will need a lot of maintenance. T?ivo and I have discussed, and we lean to umarshal because we think it will solve our probblem with the least total work, and the "avoid boutique protocol" problem shouldn't get that much weight compared to the pain of 4.10 only and large dependencies. Therefore, questions for hackers@: Does anybody know of a library that meets requirements, or close enough to discuss? Any thoughts that this path is unwise? The list has been quiet. It looks like this will be T?ivo taking the existing code forward, with some code review by me, a few people testing, and that's it. So don't hold your breath, and it would be great if more people dug in. Greg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 194 bytes Desc: not available URL: From gdt at lexort.com Mon Jan 31 18:41:53 2022 From: gdt at lexort.com (Greg Troxel) Date: Mon, 31 Jan 2022 18:41:53 -0500 Subject: [Unison-hackers] testing/review needed -- PR to remove ocaml version dependence Message-ID: See: https://github.com/bcpierce00/unison/pull/626 for a full description of what it does, authorship, etc. (mostly St?phane's work, brought forward by T?ivo). I would like, fairly soon, to: merge this PR to master release an alpha, perhaps numbered 2.51.70, on the path to a 2.52.0 release By fairly soon, I mean "later this week or this weekend" and "sometime next week, perhaps next weekend", assuming no issues are found. So far, I know of no reasons to refrain from merging, but I would like to actually run it myself, and recently I've had extra snow shoveling to do. It would be helpful if some of the people (400?) on the hackers list could: read the code and speak up if you find problems, either here, comments whitin github, or directly to T?ivo and me. test in various scenearios (see the PR for test charts) in particular, testing cross OS, cross CPU type, cross endian would be very helpful. I don't have any particular reason to suspect bugs, but this is structurally the kind of code that has that sort of bug. I haven't been hearing anything back from previous mails to hackers, so I don't intend to wait for review and testing that won't come -- which will lead to users perhaps testing the beta and almost certainly testing the release. That shouldn't be that big a deal as we have infinitely many version numbers available. I do not expect to consider the new RPC version stable/unchanged until there is an actual (non-alpha, non-beta, non-rc) release, so if you test, 1) note that your archive files will be updated, and this is a one-way trip and 2) you may have to flag-day upgrade the systems you are testing on if we find a format bug while testing, at any time before full release. But since you're a hacker, that should be no problem! Thanks, Greg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 194 bytes Desc: not available URL: