Anyone interested in "package versioning"?

Thu May 19 15:18:16 UTC 2016

 
> Inline clarifications.
>
> On Thursday, May 19, 2016, Stelian Ionescu <sionescu at cddr.org> wrote:
>> __
>>
>>> I don't mean any other language, but I do mean any other relatively
>>> modern one. Rust, JavaScript, Clojure are maybe some examples off
>>> the top of my head.
>>
>> Hans gave a good example about Clojure. Javascript, with Node.js/npm,
>> is in an even worse position.
>>
>>
>>> Either modules are somehow first class (by way of having modules-as-function-
>>> dictionaries), or the respective packaging/loading/whatever systems
>>> will at least let you know that about unsatisfied/mismatched
>>> versions.
>>>
>>> I think in light of this discussion you can boil down one's opinion
>>> on this matter to one of three alternatives:
>>>
>>> 1. Lisp+ecosystem and associated libraries are problematic in that
>>>    it's not possible to load multiple versions of a library due to
>>>    the way CL packages work. (Packages or systems aren't first-class
>>>    modules. Namespace names are global and aren't late-bound.)
>>
>> Hans described the QA problem with loading two versions of the "same"
>> library in the same process: if data structures from one version are
>> fed into the other version, all sorts of problems can happen, and
>> worse, they can happen without noticing. Think of what can happen if
>> the semantics of a string or integer slot changes.
>>
>
> See below.
>>> 2. There's something like a SocialProblem (TM) where Lispers lack
>>>    (a) versioning discipline, (b) dependency versioning discipline
>>>    which exacerbates issues like inadvertent multiple loading, or
>>>    incompatible loading of libraries.
>>
>> Semantic versioning won't help you without automatic tools to
>> determine if the API of a library changed in an incompatible way,
>> e.g. if a function removed a keyword arg, or changed the type of an
>> existing arg.
>>
>
> I would say that semvers should in principle help you detect
> incompatibilities, but as you hinted with "illusion", maybe these in
> practice can't be trusted or relied upon generally.
>
>
>
>
>>
>>> 3. There's actually no issue in principle: inconsistencies need to
>>>    "just" get fixed by the library vendor/user, or better tooling
>>>    needs to be used to track your project's direct and transitive
>>>    dependencies.
>>>
>>> It's my personal opinion, as is evident in previous messages, that I
>>> think 1 & 2 are issues.
>>
>> You didn't give an example of how exactly those two things are a
>> problem, and how adding those features would solve the problem(s).
>
> #1: Assuming that there isn't incompatible passing of data, if A
> #   depends on B1 and C depends on B2, and in isolation A-B1
> #   works/passes tests, and C-B2 works/passes tests, then A-B1 and C-
> #   B2 loaded together (assuming a non-existent Lisp) should work
> #   fine. The problem being addressed is lack of synchrony in the
> #   universe of Lisp modules.
>
> (My first-order thought is that even incompatible data wouldn't be an
> issue if those data types are properly encapsulated in classes or
> structures, by virtue of the fact that those types become distinct by
> way of different names. I envision that only untyped things could
> cause problems, like [ap]-lists, strings, etc.)
>
> #2: If there was diligence is properly versioning libraries, and
> #   diligence in specifying dependency version requirements, then
> #   detection of issues could happen. This is solving recognition of a
> #   possible issue.
>
>
>>
>>> Some additional food for thought:
>>>
>>> * Lisp is old. Some libraries go back to the 80s.
>>>
>>> * Not everything is very strictly maintained. Some useful libraries
>>>   haven't been updated for some 5-10 years.
>>
>> What useful libraries that actually need updates haven't been
>> updated ?
>
> I'll get back to this when I'm at my computer next.
>
>>
>>> * Like many other languages, Java is continuously updated, and as
>>>   are the libraries of its large standard set. While Java/etc
>>>   libraries certainly can and do depend on third party libraries,
>>>   the majority of depended-on libraries are standard ones. This is
>>>   obviously not the case for Lisp.
>>
>> Just look into Maven packages. Large parts of the Java standard
>> library are obsolete and people use third party libraries(just like
>> with CL). In some cases those libraries make it back into the standard
>> (http://www.joda.org/joda-time/), but that's rare.
>>
>>
>>> * Because library distribution wasn't a relatively solved problem
>>>   until recently in Lisp history, some library authors unfortunately
>>>   include copies of Lisp libraries as a part of their library tree.
>>
>> That's very rare, in my experience. Do you have examples ?
>>
>>
>>> * Hot patching libraries makes this all a ton more complicated in
>>>   principle.
>>
>> Where did you see this?
>>
>>
>
> For these last two questions, I don't have the examples off hand and
> unfortunately it was at a company I no longer work at.
>
> Including: ALEXANDRIA has been included directly before. I don't
> remember which library it was. I only detected it because of function
> redefinition warnings on first uncached build.  There were other
> examples but I don't recall.
>
> Hot Patching: I think (??) some linear algebra library was being hot
> patched for some specific use of that library.
>
>>> * Visible libraries are global to the entire Lisp universe. (Unlike
>>>   Python where you can, just within a file, import and late-bind a
>>>   namespace.)
>>>
>>> * Tangential: the whole name/nickname collision issue. Package-local
>>>   nicknames are a language extension which could solve this, even if
>>>   not (yet?) universally used.
>>
>> Local nicknames would make certain things more convenient, but what
>> does that have to do with the original question, versioning ?
>
> It was tangential to packages and names of things.
>
>>
>>> Making it easier to work in this large, old, and diverse ecosystem
>>> of implementations and libraries seems important to me, and
>>> versioning is one useful and tangible way to make things easier.
>>>
>>> Robert
>>>
>>> On Wednesday, May 18, 2016, Scott L. Burson <Scott at sympoiesis.com>
>>> wrote:
>>>> I don't understand when you say this is "not an issue that one runs
>>>> into writing in another language and associated ecosystem".  AFAIK,
>>>> in Java, you can have only one version of a given package loaded
>>>> into the JVM at any one time.  Am I missing something here?
>>>> -- Scott
>>>>
>>>> On Wed, May 18, 2016 at 11:41 PM, Robert Smith <quad at symbo1ics.com>
>>>> wrote:
>>>>> I really honestly felt that this rebuttal was very unhelpful. I
>>>>> basically got from it:
>>>>>  * Lisp isn't designed for this kind of use.
>>>>>  * Since Lisp wasn't designed for this, the problem is not Lisp,
>>>>>    it's your use of Lisp.
>>>>>  * To solve the problem (which is not Lisp's), fix (third party)
>>>>>    inconsistencies yourself. Because it's not a Lisp problem, two
>>>>>    systems depending on two Lisp-simultaneous-incompatible
>>>>>    versions of a system is a problem. (Even though each
>>>>>    system+dependency combination is perfectly valid in isolation.)
>>>>>  * Or, in the end, change your entire operating system to a
>>>>>    completely different one with completely different
>>>>>    philosophies.
>>>>> This kind of response is just very practically a non-starter for
>>>>> most people. Maybe that means Lisp isn't the right tool for much
>>>>> of the world, like startups, then. Which advances my point, I
>>>>> think, in the original Quora post. (But I admit this thread is
>>>>> about package or system versioning, not the original Quora point.)
>>>>>
>>>>> I wrote some stuff inline below.
>>>>>
>>>>> On Wed, May 18, 2016 at 10:14 PM, Faré <fahree at gmail.com> wrote:
>>>>>> On Wed, May 18, 2016 at 5:49 PM, Robert Smith
>>>>>> <quad at symbo1ics.com> wrote:
>>>>>>  >
>>>>>>  > On Wed, May 18, 2016 at 4:57 AM, Alessio Stalla
>>>>>>  > <alessiostalla at gmail.com> wrote:
>>>>>>  >>
>>>>>>  >> The stance on packages found in the mentioned Quora post is
>>>>>>  >> based on the old misconception about packages being modules,
>>>>>>  >> or "software packages" in the Linux distribution sense.
>>>>>>  >> They're not. They are really just namespaces, containers of
>>>>>>  >> symbols. So it does not make any sense, to me, that they have
>>>>>>  >> versions. Software has versions, not names.
>>>>>>  >
>>>>>>
>>>>>>
>>>>>> > Systems are what should get versioned but systems are not what
>>>>>> > are referred
>>>>>>  > to by source files. Systems largely coordinate the compilation
>>>>>>  > and loading of files. Source files generally have no notion of
>>>>>>  > a system, actually, which may be what is ultimately
>>>>>>  > problematic. Packages are referred to by source files,
>>>>>>  > however. When I said packages, I meant it, and it was not a
>>>>>>  > conflation with the notion of a system.
>>>>>>  >
>>>>>> Why should source files know about systems? Why should packages
>>>>>> know about systems?
>>>>>
>>>>>
>>>>> Maybe they should, maybe they shouldn't. But the crappy packages-as-
>>>>> namespaces system isn't cutting it for large programs, and modules
>>>>> usually naturally have an associated namespace, but there's no
>>>>> formal correspondence between packages and systems. But more often
>>>>> than not there is an implicit association, judging by how most
>>>>> modern Common Lisp libraries are written. So maybe there's value
>>>>> in taking advantage of that correspondence.
>>>>>
>>>>>
>>>>>> > At the companies I have worked in, we rake in 100s of systems,
>>>>>> > which consist
>>>>>>  > of n*100s of systems where n is the average number of packages
>>>>>>  > per system. These version collisions happen enough, and we are
>>>>>>  > lucky when they're not silent.
>>>>>>  >
>>>>>> Was it in CL? Was it all in the same process? CL is not designed
>>>>>> for  these kinds of things.
>>>>>
>>>>>
>>>>> CL in the same process.
>>>>>
>>>>> CL very clearly wasn't designed for a time where 100s of packages
>>>>> would be available and simultaneously usable.
>>>>>
>>>>> But I'm not ready to concede and say "well, Lisp is only good for
>>>>> five or six massive packages or systems written by a few guys,
>>>>> like the good ol' days".
>>>>>
>>>>>
>>>>>> > Often enough, it's an extremely subtle issue that system A
>>>>>> > depends
>>>>>>  > on {system B at time T1} and system C depends on {system B at
>>>>>>  > time T2}, where T1 and T2 correspond to different versions of
>>>>>>  > system B, even if not explicitly labeled as such.
>>>>>>  >
>>>>>> A and C are incompatible. You need to fix A, B and/or C. The
>>>>>> build system can record this incompatibility and issue an early
>>>>>> error  when you try to build them together. It can't fix A, B
>>>>>> and/or C for you.
>>>>>
>>>>>
>>>>> A and C are incompatible only because of this whole thing we are
>>>>> talking about. Obviously they're incompatible. That's the whole
>>>>> problem. Saying "Fix A, B, C" is, to me, declaring that this whole
>>>>> thing isn't a problem, and the problem is library writers. Why
>>>>> can't A and C depend on whatever they want at some time? They of
>>>>> course can, but it's non-sense to think people will rename their
>>>>> namespaces every iteration of their software. That's just not how
>>>>> Lisp code has been written.
>>>>>
>>>>>
>>>>>> > Let's suppose we simplify the problem however and assume that B
>>>>>> > has been
>>>>>>  > properly versioned in time, and systems A and C refer properly
>>>>>>  > to the correct version of system B as a dependency. Even then,
>>>>>>  > they will not be able to be simultaneously loaded in a
>>>>>>  > functionally correct fashion. (They *can* both be loaded, but
>>>>>>  > you'll do some clobbering of state along the way.)
>>>>>>  >
>>>>>> No. The build system can error out and tell you the easy way that
>>>>>> you have to fix your bugs before you discover it the hard way
>>>>>
>>>>>
>>>>> The build system erroring would be a step above the status quo.
>>>>> But maybe there's not actually a logical error. We are only
>>>>> considering it an error because of the way packages work in
>>>>> Lisp, no?
>>>>>
>>>>>
>>>>>> > Whether we think about modules as systems or as packages or as
>>>>>> > whatever, the
>>>>>>  > problem would need to be addressed at the package level, if we
>>>>>>  > are not assuming that we can't synchronize the universe of
>>>>>>  > people writing Common Lisp.
>>>>>>  >
>>>>>> The packages have nothing to do with it, unless you fork B into
>>>>>> B1 and B2 (or B1 and B, or B and B2).
>>>>>>
>>>>>>  > Overall, I'd summarize as this. Whether we have the technology
>>>>>>  > in the Common Lisp ecosystem to accommodate this problem or
>>>>>>  > not, it's my opinion and the opinion of many of my colleagues
>>>>>>  > that there is a problem for development teams making extensive
>>>>>>  > use of open and closed source code. There definitely is not a
>>>>>>  > discipline that you see with other languages and their
>>>>>>  > ecosystems (JS/npm, Clojure, etc.) about proper versioning of
>>>>>>  > systems and furthermore proper loading of the proper versions.
>>>>>>  > I don't think that's debatable. The closest we have to this
>>>>>>  > solution is relying on the global nature of a particular
>>>>>>  > Quicklisp distribution, but not all Lisp software is free and
>>>>>>  > open source, and even software within Quicklisp has tons of
>>>>>>  > implied version dependencies.
>>>>>>  >
>>>>>> This is not a very Lisp-specific problem. This is a problem with
>>>>>> building software in general. The correct approach is to fix
>>>>>> incompatibilities, not to try to deny their existence only to
>>>>>> find out the hard way you can't.
>>>>>>
>>>>>>  Google Bazel provides a good way to deterministically build
>>>>>>  software that follows the correct discipline. So does NixOS.
>>>>>>  There may be other such tools. Use them.
>>>>>
>>>>>
>>>>> I don't know much about Bazel, and I know a little about NixOS.
>>>>> Regardless, this seems to be moving the problem into how we
>>>>> globally synchronize our systems.
>>>>>
>>>>> I am absolutely boggled by this attitude. This is an issue that
>>>>> one runs into when writing Common Lisp code, and not an issue that
>>>>> one runs into writing in another language and associated
>>>>> ecosystem. To me, that's a Lisp problem. We can do some creative
>>>>> academic definitions, I think, but it's a problem when choosing
>>>>> Lisp as a tool.
>>>>>
>>>>> I don't know. The original question I was answering was why Lisp
>>>>> isn't what we are all using today, and I think this weird thread
>>>>> serves as additional evidence.
>>>>>
>>>>> Am I way off base here?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Robert
>>
>>
>> --
>> Stelian Ionescu a.k.a. fe[nl]ix
>> Quidquid latine dictum sit, altum videtur.
>>
 
 
--
Stelian Ionescu a.k.a. fe[nl]ix
Quidquid latine dictum sit, altum videtur.
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/pro/attachments/20160519/f5bc97fa/attachment-0001.html>