Anyone interested in "package versioning"?

[Stelian Ionescu]

> I don't mean any other language, but I do mean any other relatively
> modern one. Rust, JavaScript, Clojure are maybe some examples off the
> top of my head.
 
Hans gave a good example about Clojure. Javascript, with Node.js/npm, is
in an even worse position.
 
 
>  Either modules are somehow first class (by way of having modules-as-function-
>  dictionaries), or the respective packaging/loading/whatever systems
>  will at least let you know that about unsatisfied/mismatched
>  versions.
>
> I think in light of this discussion you can boil down one's opinion on
> this matter to one of three alternatives:
>
> 1. Lisp+ecosystem and associated libraries are problematic in that
>    it's not possible to load multiple versions of a library due to the
>    way CL packages work. (Packages or systems aren't first-class
>    modules. Namespace names are global and aren't late-bound.)
 
Hans described the QA problem with loading two versions of the "same"
library in the same process: if data structures from one version are fed
into the other version, all sorts of problems can happen, and worse,
they can happen without noticing. Think of what can happen if the
semantics of a string or integer slot changes.
 
 
> 2. There's something like a SocialProblem (TM) where Lispers lack (a)
>    versioning discipline, (b) dependency versioning discipline which
>    exacerbates issues like inadvertent multiple loading, or
>    incompatible loading of libraries.
 
Semantic versioning won't help you without automatic tools to determine
if the API of a library changed in an incompatible way, e.g. if a
function removed a keyword arg, or changed the type of an existing arg.
 
 
> 3. There's actually no issue in principle: inconsistencies need to
>    "just" get fixed by the library vendor/user, or better tooling
>    needs to be used to track your project's direct and transitive
>    dependencies.
>
> It's my personal opinion, as is evident in previous messages, that I
> think 1 & 2 are issues.
 
You didn't give an example of how exactly those two things are a
problem, and how adding those features would solve the problem(s).
 
 
> Some additional food for thought:
>
> * Lisp is old. Some libraries go back to the 80s.
>
> * Not everything is very strictly maintained. Some useful libraries
>   haven't been updated for some 5-10 years.
 
What useful libraries that actually need updates haven't been updated ?
 
 
> * Like many other languages, Java is continuously updated, and as are
>   the libraries of its large standard set. While Java/etc libraries
>   certainly can and do depend on third party libraries, the majority
>   of depended-on libraries are standard ones. This is obviously not
>   the case for Lisp.
 
Just look into Maven packages. Large parts of the Java standard library
are obsolete and people use third party libraries(just like with CL). In
some cases those libraries make it back into the standard
(http://www.joda.org/joda-time/), but that's rare.
 
 
> * Because library distribution wasn't a relatively solved problem
>   until recently in Lisp history, some library authors unfortunately
>   include copies of Lisp libraries as a part of their library tree.
 
That's very rare, in my experience. Do you have examples ?
 
 
> * Hot patching libraries makes this all a ton more complicated in
>   principle.
 
Where did you see this?
 
 
> * Visible libraries are global to the entire Lisp universe. (Unlike
>   Python where you can, just within a file, import and late-bind a
>   namespace.)
>
> * Tangential: the whole name/nickname collision issue. Package-local
>   nicknames are a language extension which could solve this, even if
>   not (yet?) universally used.
 
Local nicknames would make certain things more convenient, but what does
that have to do with the original question, versioning ?
 
 
> Making it easier to work in this large, old, and diverse ecosystem of
> implementations and libraries seems important to me, and versioning is
> one useful and tangible way to make things easier.
>
> Robert
>
> On Wednesday, May 18, 2016, Scott L. Burson
> <Scott at sympoiesis.com> wrote:
>> I don't understand when you say this is "not an issue that one runs
>> into writing in another language and associated ecosystem".  AFAIK,
>> in Java, you can have only one version of a given package loaded into
>> the JVM at any one time.  Am I missing something here?
>> -- Scott
>>
>> On Wed, May 18, 2016 at 11:41 PM, Robert Smith
>> <quad at symbo1ics.com> wrote:
>>> I really honestly felt that this rebuttal was very unhelpful. I
>>> basically got from it:
>>>  * Lisp isn't designed for this kind of use.
>>>  * Since Lisp wasn't designed for this, the problem is not Lisp,
>>>    it's your use of Lisp.
>>>  * To solve the problem (which is not Lisp's), fix (third party)
>>>    inconsistencies yourself. Because it's not a Lisp problem, two
>>>    systems depending on two Lisp-simultaneous-incompatible versions
>>>    of a system is a problem. (Even though each system+dependency
>>>    combination is perfectly valid in isolation.)
>>>  * Or, in the end, change your entire operating system to a
>>>    completely different one with completely different philosophies.
>>> This kind of response is just very practically a non-starter for
>>> most people. Maybe that means Lisp isn't the right tool for much of
>>> the world, like startups, then. Which advances my point, I think, in
>>> the original Quora post. (But I admit this thread is about package
>>> or system versioning, not the original Quora point.)
>>>
>>> I wrote some stuff inline below.
>>>
>>> On Wed, May 18, 2016 at 10:14 PM, Faré <fahree at gmail.com> wrote:
>>>> On Wed, May 18, 2016 at 5:49 PM, Robert Smith <quad at symbo1ics.com>
>>>> wrote:
>>>>  >
>>>>  > On Wed, May 18, 2016 at 4:57 AM, Alessio Stalla
>>>>  > <alessiostalla at gmail.com> wrote:
>>>>  >>
>>>>  >> The stance on packages found in the mentioned Quora post is
>>>>  >> based on the old misconception about packages being modules, or
>>>>  >> "software packages" in the Linux distribution sense. They're
>>>>  >> not. They are really just namespaces, containers of symbols. So
>>>>  >> it does not make any sense, to me, that they have versions.
>>>>  >> Software has versions, not names.
>>>>  >
>>>>
>>>>
>>>> > Systems are what should get versioned but systems are not what
>>>> > are referred
>>>>  > to by source files. Systems largely coordinate the compilation
>>>>  > and loading of files. Source files generally have no notion of a
>>>>  > system, actually, which may be what is ultimately problematic.
>>>>  > Packages are referred to by source files, however. When I said
>>>>  > packages, I meant it, and it was not a conflation with the
>>>>  > notion of a system.
>>>>  >
>>>> Why should source files know about systems? Why should packages
>>>> know about systems?
>>>
>>>
>>> Maybe they should, maybe they shouldn't. But the crappy packages-as-
>>> namespaces system isn't cutting it for large programs, and modules
>>> usually naturally have an associated namespace, but there's no
>>> formal correspondence between packages and systems. But more often
>>> than not there is an implicit association, judging by how most
>>> modern Common Lisp libraries are written. So maybe there's value in
>>> taking advantage of that correspondence.
>>>
>>>
>>>> > At the companies I have worked in, we rake in 100s of systems,
>>>> > which consist
>>>>  > of n*100s of systems where n is the average number of packages
>>>>  > per system. These version collisions happen enough, and we are
>>>>  > lucky when they're not silent.
>>>>  >
>>>> Was it in CL? Was it all in the same process? CL is not designed
>>>> for these kinds of things.
>>>
>>>
>>> CL in the same process.
>>>
>>> CL very clearly wasn't designed for a time where 100s of packages
>>> would be available and simultaneously usable.
>>>
>>> But I'm not ready to concede and say "well, Lisp is only good for
>>> five or six massive packages or systems written by a few guys, like
>>> the good ol' days".
>>>
>>>
>>>> > Often enough, it's an extremely subtle issue that system A
>>>> > depends
>>>>  > on {system B at time T1} and system C depends on {system B at
>>>>  > time T2}, where T1 and T2 correspond to different versions of
>>>>  > system B, even if not explicitly labeled as such.
>>>>  >
>>>> A and C are incompatible. You need to fix A, B and/or C. The build
>>>> system can record this incompatibility and issue an early error
>>>> when you try to build them together. It can't fix A, B and/or C
>>>> for you.
>>>
>>>
>>> A and C are incompatible only because of this whole thing we are
>>> talking about. Obviously they're incompatible. That's the whole
>>> problem. Saying "Fix A, B, C" is, to me, declaring that this whole
>>> thing isn't a problem, and the problem is library writers. Why can't
>>> A and C depend on whatever they want at some time? They of course
>>> can, but it's non-sense to think people will rename their namespaces
>>> every iteration of their software. That's just not how Lisp code has
>>> been written.
>>>
>>>
>>>> > Let's suppose we simplify the problem however and assume that B
>>>> > has been
>>>>  > properly versioned in time, and systems A and C refer properly
>>>>  > to the correct version of system B as a dependency. Even then,
>>>>  > they will not be able to be simultaneously loaded in a
>>>>  > functionally correct fashion. (They *can* both be loaded, but
>>>>  > you'll do some clobbering of state along the way.)
>>>>  >
>>>> No. The build system can error out and tell you the easy way that
>>>> you have to fix your bugs before you discover it the hard way
>>>
>>>
>>> The build system erroring would be a step above the status quo. But
>>> maybe there's not actually a logical error. We are only considering
>>> it an error because of the way packages work in Lisp, no?
>>>
>>>
>>>> > Whether we think about modules as systems or as packages or as
>>>> > whatever, the
>>>>  > problem would need to be addressed at the package level, if we
>>>>  > are not assuming that we can't synchronize the universe of
>>>>  > people writing Common Lisp.
>>>>  >
>>>> The packages have nothing to do with it, unless you fork B into B1
>>>> and B2 (or B1 and B, or B and B2).
>>>>
>>>>  > Overall, I'd summarize as this. Whether we have the technology
>>>>  > in the Common Lisp ecosystem to accommodate this problem or not,
>>>>  > it's my opinion and the opinion of many of my colleagues that
>>>>  > there is a problem for development teams making extensive use of
>>>>  > open and closed source code. There definitely is not a
>>>>  > discipline that you see with other languages and their
>>>>  > ecosystems (JS/npm, Clojure, etc.) about proper versioning of
>>>>  > systems and furthermore proper loading of the proper versions. I
>>>>  > don't think that's debatable. The closest we have to this
>>>>  > solution is relying on the global nature of a particular
>>>>  > Quicklisp distribution, but not all Lisp software is free and
>>>>  > open source, and even software within Quicklisp has tons of
>>>>  > implied version dependencies.
>>>>  >
>>>> This is not a very Lisp-specific problem. This is a problem with
>>>> building software in general. The correct approach is to fix
>>>> incompatibilities, not to try to deny their existence only to find
>>>> out the hard way you can't.
>>>>
>>>>  Google Bazel provides a good way to deterministically build
>>>>  software that follows the correct discipline. So does NixOS. There
>>>>  may be other such tools. Use them.
>>>
>>>
>>> I don't know much about Bazel, and I know a little about NixOS.
>>> Regardless, this seems to be moving the problem into how we globally
>>> synchronize our systems.
>>>
>>> I am absolutely boggled by this attitude. This is an issue that
>>> one runs into when writing Common Lisp code, and not an issue that
>>> one runs into writing in another language and associated
>>> ecosystem. To me, that's a Lisp problem. We can do some creative
>>> academic definitions, I think, but it's a problem when choosing
>>> Lisp as a tool.
>>>
>>> I don't know. The original question I was answering was why Lisp
>>> isn't what we are all using today, and I think this weird thread
>>> serves as additional evidence.
>>>
>>> Am I way off base here?
>>>
>>> Cheers,
>>>
>>> Robert
 
 
--
Stelian Ionescu a.k.a. fe[nl]ix
Quidquid latine dictum sit, altum videtur.
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/pro/attachments/20160519/fa78dd31/attachment-0001.html>