Anyone interested in "package versioning"?

[Scott L. Burson]

I don't understand when you say this is "not an issue that one runs into
writing in another language and associated ecosystem".  AFAIK, in Java, you
can have only one version of a given package loaded into the JVM at any one
time.  Am I missing something here?

-- Scott

On Wed, May 18, 2016 at 11:41 PM, Robert Smith <quad at symbo1ics.com> wrote:

> I really honestly felt that this rebuttal was very unhelpful. I basically
> got from it:
>
>    - Lisp isn't designed for this kind of use.
>    - Since Lisp wasn't designed for this, the problem is not Lisp, it's
>    your use of Lisp.
>    - To solve the problem (which is not Lisp's), fix (third party)
>    inconsistencies yourself. Because it's not a Lisp problem, two systems
>    depending on two Lisp-simultaneous-incompatible versions of a system is a
>    problem. (Even though each system+dependency combination is perfectly valid
>    in isolation.)
>    - Or, in the end, change your entire operating system to a completely
>    different one with completely different philosophies.
>
> This kind of response is just very practically a non-starter for most
> people. Maybe that means Lisp isn't the right tool for much of the world,
> like startups, then. Which advances my point, I think, in the original
> Quora post. (But I admit this thread is about package or system versioning,
> not the original Quora point.)
>
> I wrote some stuff inline below.
>
> On Wed, May 18, 2016 at 10:14 PM, Faré <fahree at gmail.com> wrote:
>
>> On Wed, May 18, 2016 at 5:49 PM, Robert Smith <quad at symbo1ics.com> wrote:
>> >
>> > On Wed, May 18, 2016 at 4:57 AM, Alessio Stalla <
>> alessiostalla at gmail.com>
>> > wrote:
>> >>
>> >> The stance on packages found in the mentioned Quora post is based on
>> the
>> >> old misconception about packages being modules, or "software packages"
>> in
>> >> the Linux distribution sense. They're not. They are really just
>> namespaces,
>> >> containers of symbols. So it does not make any sense, to me, that they
>> have
>> >> versions. Software has versions, not names.
>> >
>>
>>
>> > Systems are what should get versioned but systems are not what are
>> referred
>> > to by source files. Systems largely coordinate the compilation and
>> loading
>> > of files. Source files generally have no notion of a system, actually,
>> which
>> > may be what is ultimately problematic. Packages are referred to by
>> source
>> > files, however. When I said packages, I meant it, and it was not a
>> > conflation with the notion of a system.
>> >
>> Why should source files know about systems?
>> Why should packages know about systems?
>>
>
> Maybe they should, maybe they shouldn't. But the crappy
> packages-as-namespaces system isn't cutting it for large programs, and
> modules usually naturally have an associated namespace, but there's no
> formal correspondence between packages and systems. But more often than not
> there is an implicit association, judging by how most modern Common Lisp
> libraries are written. So maybe there's value in taking advantage of that
> correspondence.
>
>
>> > At the companies I have worked in, we rake in 100s of systems, which
>> consist
>> > of n*100s of systems where n is the average number of packages per
>> system.
>> > These version collisions happen enough, and we are lucky when they're
>> not
>> > silent.
>> >
>> Was it in CL? Was it all in the same process? CL is not designed for
>> these kinds of things.
>>
>>
> CL in the same process.
>
> CL very clearly wasn't designed for a time where 100s of packages would be
> available and simultaneously usable.
>
> But I'm not ready to concede and say "well, Lisp is only good for five or
> six massive packages or systems written by a few guys, like the good ol'
> days".
>
>
>> > Often enough, it's an extremely subtle issue that system A depends
>> > on {system B at time T1} and system C depends on {system B at time T2},
>> > where T1 and T2 correspond to different versions of system B, even if
>> not
>> > explicitly labeled as such.
>> >
>> A and C are incompatible. You need to fix A, B and/or C.
>> The build system can record this incompatibility and issue an early error
>> when you try to build them together. It can't fix A, B and/or C for you.
>>
>>
> A and C are incompatible only because of this whole thing we are talking
> about. Obviously they're incompatible. That's the whole problem. Saying
> "Fix A, B, C" is, to me, declaring that this whole thing isn't a problem,
> and the problem is library writers. Why can't A and C depend on whatever
> they want at some time? They of course can, but it's non-sense to think
> people will rename their namespaces every iteration of their software.
> That's just not how Lisp code has been written.
>
>
>> > Let's suppose we simplify the problem however and assume that B has been
>> > properly versioned in time, and systems A and C refer properly to the
>> > correct version of system B as a dependency. Even then, they will not be
>> > able to be simultaneously loaded in a functionally correct fashion.
>> (They
>> > *can* both be loaded, but you'll do some clobbering of state along the
>> way.)
>> >
>> No. The build system can error out and
>> tell you the easy way that you have to fix your bugs
>> before you discover it the hard way
>>
>
> The build system erroring would be a step above the status quo. But maybe
> there's not actually a logical error. We are only considering it an error
> because of the way packages work in Lisp, no?
>
> > Whether we think about modules as systems or as packages or as whatever,
>> the
>> > problem would need to be addressed at the package level, if we are not
>> > assuming that we can't synchronize the universe of people writing Common
>> > Lisp.
>> >
>> The packages have nothing to do with it, unless you fork B into B1 and B2
>> (or B1 and B, or B and B2).
>>
>> > Overall, I'd summarize as this. Whether we have the technology in the
>> Common
>> > Lisp ecosystem to accommodate this problem or not, it's my opinion and
>> the
>> > opinion of many of my colleagues that there is a problem for development
>> > teams making extensive use of open and closed source code. There
>> definitely
>> > is not a discipline that you see with other languages and their
>> ecosystems
>> > (JS/npm, Clojure, etc.) about proper versioning of systems and
>> furthermore
>> > proper loading of the proper versions. I don't think that's debatable.
>> The
>> > closest we have to this solution is relying on the global nature of a
>> > particular Quicklisp distribution, but not all Lisp software is free and
>> > open source, and even software within Quicklisp has tons of implied
>> version
>> > dependencies.
>> >
>> This is not a very Lisp-specific problem.
>> This is a problem with building software in general.
>> The correct approach is to fix incompatibilities,
>> not to try to deny their existence only to find out the hard way you
>> can't.
>>
>> Google Bazel provides a good way to deterministically build software
>> that follows the correct discipline. So does NixOS. There may be other
>> such tools. Use them.
>>
>>
> I don't know much about Bazel, and I know a little about NixOS.
> Regardless, this seems to be moving the problem into how we globally
> synchronize our systems.
>
> I am absolutely boggled by this attitude. This is an issue that one runs
> into when writing Common Lisp code, and not an issue that one runs into
> writing in another language and associated ecosystem. To me, that's a Lisp
> problem. We can do some creative academic definitions, I think, but it's a
> problem when choosing Lisp as a tool.
>
> I don't know. The original question I was answering was why Lisp isn't
> what we are all using today, and I think this weird thread serves as
> additional evidence.
>
> Am I way off base here?
>
> Cheers,
>
> Robert
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/pro/attachments/20160518/17bc495e/attachment.html>