From attila at lendvai.name Wed Apr 14 15:12:00 2021 From: attila at lendvai.name (Attila Lendvai) Date: Wed, 14 Apr 2021 15:12:00 +0000 Subject: only perfom when output is missing Message-ID: dear hackers, i cannot convince ASDF to only perform my custom operation if the output file doesn't exist; i.e. regardless of the modification times. it's in cffi/c2ffi: foo.h -> foo.spec -> foo.lisp when the spec is there, regardless of the mod time, i'd like to skip the spec generation task. neither reading the manual, nor grepping around helped me to come up with the required magic. my custom operation-done-p isn't even called if the file's mod time instructs ASDF to run the operation. any hints are appreciated, including pointers to examples or tests. - attila -------------- next part -------------- An HTML attachment was scrubbed... URL: From attila at lendvai.name Wed Apr 14 15:19:11 2021 From: attila at lendvai.name (Attila Lendvai) Date: Wed, 14 Apr 2021 17:19:11 +0200 Subject: only perfom when the output is missing Message-ID: dear hackers, i cannot convince ASDF to only perform my custom operation if the output file doesn't exist; i.e. regardless of the modification times. it's in cffi/c2ffi: foo.h -> foo.spec -> foo.lisp when the spec is there, regardless of the mod time, i'd like to skip the spec generation task. neither reading the manual, nor grepping around helped me to come up with the required magic. my custom operation-done-p isn't even called if the file's mod time instructs ASDF to run the operation. any hints are appreciated, including pointers to examples or tests. - attila PS: i sent the same mail from the wrong account, don't allow that through moderation please. -------------- next part -------------- An HTML attachment was scrubbed... URL: From attila.lendvai at gmail.com Wed Apr 14 23:38:34 2021 From: attila.lendvai at gmail.com (Attila Lendvai) Date: Thu, 15 Apr 2021 01:38:34 +0200 Subject: only perfom when the output is missing In-Reply-To: References: Message-ID: FTR, i have managed to solve it with this: (defclass generate-lisp-op (selfward-operation) ((selfward-operation :initform '()))) ; we will specify it in our own COMPONENT-DEPENDS-ON (defmethod component-depends-on ((op generate-lisp-op) (c c2ffi-file)) `((load-op ,(find-system "cffi/c2ffi-generator")) ;; Regenerating the spec file is a lot of headache, so we ignore ;; the file modification times, and only communicate the ;; dependency to ASDF if the spec file is missing. ,@(let ((spec-file (input-file op c))) ; TODO is it legal to call ASDF:INPUT-FILES here? (when (or (not (probe-file spec-file)) (zerop (with-input-from-file (s spec-file) (file-length s)))) `((generate-spec-op ,c)))) ,@(call-next-method))) the commit: https://github.com/cffi/cffi/commit/46975c644aeb0a832cd170e953cc8ec3cbdc78df - attila PS: sorry for the double send, i have cleaned up my mail settings. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpgoldman at sift.info Sat Apr 17 16:46:09 2021 From: rpgoldman at sift.info (Robert Goldman) Date: Sat, 17 Apr 2021 11:46:09 -0500 Subject: only perfom when the output is missing In-Reply-To: References: Message-ID: <31D65294-7572-4E41-ABBB-F6F13184629C@sift.info> On 14 Apr 2021, at 18:38, Attila Lendvai wrote: > FTR, i have managed to solve it with this: > > (defclass generate-lisp-op (selfward-operation) > ((selfward-operation :initform '()))) ; we will specify it in our > own > COMPONENT-DEPENDS-ON > > (defmethod component-depends-on ((op generate-lisp-op) (c c2ffi-file)) > `((load-op ,(find-system "cffi/c2ffi-generator")) > ;; Regenerating the spec file is a lot of headache, so we ignore > ;; the file modification times, and only communicate the > ;; dependency to ASDF if the spec file is missing. > ,@(let ((spec-file (input-file op c))) ; TODO is it legal to call > ASDF:INPUT-FILES here? > (when (or (not (probe-file spec-file)) > (zerop (with-input-from-file (s spec-file) > (file-length s)))) > `((generate-spec-op ,c)))) > ,@(call-next-method))) > > the commit: > https://github.com/cffi/cffi/commit/46975c644aeb0a832cd170e953cc8ec3cbdc78df I was wondering why it was important to avoid the build when the `.h` file is newer than the `.spec` file. This shouldn't happen very often, should it? Is it because: 1. The `.h` file is getting refreshed too often by something upstream of ASDF? E.g., is `make` being too eager about rewriting the `.h` file? 2. There is something wrong in the ASDF dependencies? Is it possible that there was an extraneous dependency on the `.spec` file that was causing it to mistakenly think it needed refreshing when some component upstream in the ASDF system was modified? E.g., was this in a `:serial` system so that other changes were causing ASDF to be too eager about re-generating the spec file? I don't have any great ideas about fixing the problem if it's caused by something outside ASDF, but if the second explanation is the case, it would be worth investigating further so that you don't need to make this change (and can regenerate the spec file when the header file is updated without causing lots of wasted work). I hate to see this kind of spoofing, because it can so easily lead to confusing and hard-to-debug issues later on when the `.h` file really *does* change and ASDF ignores the change. If it is something outside ASDF that's rewriting the header file when it's not necessary then you could do something more sophisticated and squirrel away a copy of the old `.h` file and use `diff` or `cmp` in `operation-done-p`. Best, R > > - attila > > PS: sorry for the double send, i have cleaned up my mail settings. Don't worry about the double send: you are now cleared to submit emails from either of your email accounts, so even if you send from the wrong account, it should turn up OK. -------------- next part -------------- An HTML attachment was scrubbed... URL: From attila.lendvai at gmail.com Sat Apr 17 20:43:34 2021 From: attila.lendvai at gmail.com (Attila Lendvai) Date: Sat, 17 Apr 2021 22:43:34 +0200 Subject: only perfom when the output is missing In-Reply-To: <31D65294-7572-4E41-ABBB-F6F13184629C@sift.info> References: <31D65294-7572-4E41-ABBB-F6F13184629C@sift.info> Message-ID: thanks for the ideas Robert! > I was wondering why it was important to avoid the build when the .h file > is newer than the .spec file. This shouldn't happen very often, should > it? Is it because: > the problem is much more down to earth: generating the spec file requires launching a libllvm based external tool called c2ffi. compiling and properly configuring this tool can sometimes consume a day, and the users of e.g. hu.dwim.sdl shouldn't ever need to do that. it's only needed when you want to capture newly added artifacts from the SDL.h files. one way to avoid this could be to not communicate the dependency to ASDF, and make the .h -> .spec generation manual (this used to be the case). for some reason that feels lame, but not the end of the world either. > 1. > > The .h file is getting refreshed too often by something upstream of > ASDF? E.g., is make being too eager about rewriting the .h file? > > the problem is not with too often, but with anything greater than zero, e.g. a git checkout. - attila -------------- next part -------------- An HTML attachment was scrubbed... URL: From fahree at gmail.com Sat Apr 17 22:09:23 2021 From: fahree at gmail.com (=?UTF-8?B?RmFyw6k=?=) Date: Sat, 17 Apr 2021 18:09:23 -0400 Subject: only perfom when the output is missing In-Reply-To: References: <31D65294-7572-4E41-ABBB-F6F13184629C@sift.info> Message-ID: One way to avoid unneeded regeneration in the future without lying about the dependency would be to modify the script to: 1. Remember in the output the hash of the input. 2. In the regeneration script, have a shortcut that leaves the output unchanged if the input matches. That won't help if you have to deal with branches back in time, but in those cases, you can use touch to reset the .h file to an old date. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The greatest lies always come in a thin coating of truth. On Sat, Apr 17, 2021 at 4:44 PM Attila Lendvai wrote: > > thanks for the ideas Robert! > >> >> I was wondering why it was important to avoid the build when the .h file is newer than the .spec file. This shouldn't happen very often, should it? Is it because: > > > the problem is much more down to earth: generating the spec file requires launching a libllvm based external tool called c2ffi. compiling and properly configuring this tool can sometimes consume a day, and the users of e.g. hu.dwim.sdl shouldn't ever need to do that. it's only needed when you want to capture newly added artifacts from the SDL.h files. > > one way to avoid this could be to not communicate the dependency to ASDF, and make the .h -> .spec generation manual (this used to be the case). for some reason that feels lame, but not the end of the world either. > >> The .h file is getting refreshed too often by something upstream of ASDF? E.g., is make being too eager about rewriting the .h file? > > > the problem is not with too often, but with anything greater than zero, e.g. a git checkout. > > - attila From rpgoldman at sift.info Sat Apr 17 23:29:26 2021 From: rpgoldman at sift.info (Robert Goldman) Date: Sat, 17 Apr 2021 18:29:26 -0500 Subject: only perfom when the output is missing In-Reply-To: References: <31D65294-7572-4E41-ABBB-F6F13184629C@sift.info> Message-ID: <7CD4AF8D-58E1-412B-AB05-72419641935B@sift.info> On 17 Apr 2021, at 17:09, Faré wrote: > One way to avoid unneeded regeneration in the future without lying > about the dependency would be to modify the script to: > 1. Remember in the output the hash of the input. > 2. In the regeneration script, have a shortcut that leaves the output > unchanged if the input matches. > > That won't help if you have to deal with branches back in time, but in > those cases, you can use touch to reset the .h file to an old date. That's neat. If the hash is written to the file system (and into the Git repo), then it can be checked, and that would avoid a redo even after an initial checkout! If you do this, Attila, please let us know -- it might be a useful capability to generalize and put in the ASDF contribs. > > —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• > http://fare.tunes.org > The greatest lies always come in a thin coating of truth. > > > On Sat, Apr 17, 2021 at 4:44 PM Attila Lendvai > wrote: >> >> thanks for the ideas Robert! >> >>> >>> I was wondering why it was important to avoid the build when the .h >>> file is newer than the .spec file. This shouldn't happen very often, >>> should it? Is it because: >> >> >> the problem is much more down to earth: generating the spec file >> requires launching a libllvm based external tool called c2ffi. >> compiling and properly configuring this tool can sometimes consume a >> day, and the users of e.g. hu.dwim.sdl shouldn't ever need to do >> that. it's only needed when you want to capture newly added artifacts >> from the SDL.h files. >> >> one way to avoid this could be to not communicate the dependency to >> ASDF, and make the .h -> .spec generation manual (this used to be the >> case). for some reason that feels lame, but not the end of the world >> either. >> >>> The .h file is getting refreshed too often by something upstream of >>> ASDF? E.g., is make being too eager about rewriting the .h file? >> >> >> the problem is not with too often, but with anything greater than >> zero, e.g. a git checkout. >> >> - attila From fahree at gmail.com Sun Apr 18 00:18:12 2021 From: fahree at gmail.com (=?UTF-8?B?RmFyw6k=?=) Date: Sat, 17 Apr 2021 20:18:12 -0400 Subject: only perfom when the output is missing In-Reply-To: <7CD4AF8D-58E1-412B-AB05-72419641935B@sift.info> References: <31D65294-7572-4E41-ABBB-F6F13184629C@sift.info> <7CD4AF8D-58E1-412B-AB05-72419641935B@sift.info> Message-ID: That would be a great extension to have for ASDF in general, but would have to be an extension, unless the maintainer is willing to put the implementation of a fast hash function in ASDF. If so, I nominate BLAKE3. It should take a few hundreds of lines of code—a relatively small amount by ASDF 3.3 standards. And if the entry function is not-inline, like all UIOP functions, it can be replaced by a very fast implementation when a suitable cryptographic library is loaded. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org He who says he will die for a cause will probably lie for it and may kill for it. — John McCarthy On Sat, Apr 17, 2021 at 7:29 PM Robert Goldman wrote: > > On 17 Apr 2021, at 17:09, Faré wrote: > > > One way to avoid unneeded regeneration in the future without lying > > about the dependency would be to modify the script to: > > 1. Remember in the output the hash of the input. > > 2. In the regeneration script, have a shortcut that leaves the output > > unchanged if the input matches. > > > > That won't help if you have to deal with branches back in time, but in > > those cases, you can use touch to reset the .h file to an old date. > > That's neat. If the hash is written to the file system (and into the > Git repo), then it can be checked, and that would avoid a redo even > after an initial checkout! > > If you do this, Attila, please let us know -- it might be a useful > capability to generalize and put in the ASDF contribs. > > > > —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• > > http://fare.tunes.org > > The greatest lies always come in a thin coating of truth. > > > > > > On Sat, Apr 17, 2021 at 4:44 PM Attila Lendvai > > wrote: > >> > >> thanks for the ideas Robert! > >> > >>> > >>> I was wondering why it was important to avoid the build when the .h > >>> file is newer than the .spec file. This shouldn't happen very often, > >>> should it? Is it because: > >> > >> > >> the problem is much more down to earth: generating the spec file > >> requires launching a libllvm based external tool called c2ffi. > >> compiling and properly configuring this tool can sometimes consume a > >> day, and the users of e.g. hu.dwim.sdl shouldn't ever need to do > >> that. it's only needed when you want to capture newly added artifacts > >> from the SDL.h files. > >> > >> one way to avoid this could be to not communicate the dependency to > >> ASDF, and make the .h -> .spec generation manual (this used to be the > >> case). for some reason that feels lame, but not the end of the world > >> either. > >> > >>> The .h file is getting refreshed too often by something upstream of > >>> ASDF? E.g., is make being too eager about rewriting the .h file? > >> > >> > >> the problem is not with too often, but with anything greater than > >> zero, e.g. a git checkout. > >> > >> - attila From rpgoldman at sift.info Thu Apr 29 16:12:44 2021 From: rpgoldman at sift.info (Robert Goldman) Date: Thu, 29 Apr 2021 11:12:44 -0500 Subject: ASDF components are brittle for backwards compatibility Message-ID: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> ASDF checks to make sure all of the initargs are defined when parsing a `defsystem`. This is good for catching errors, but is terrible for extensibility. This means that any attempt to add additional metadata will be backwards incompatible. I can think of two ways to fix this: 1. Add a "garbage can" slot to `component` that will be filled with a property list, and allow programmers to do whatever they want here. This doesn't seem great to me. 2. Add a new `defsystem` parsing error class that is `unknown-component-property`, raise it when an unknown property is encountered and allow the user to continue, discarding the initarg and accompanying value. The second alternative is what I favor. It isn't great, because it will only maintain extensibility going forward, but I think it's the best we can do. I suppose that we could also add an initarg to tell ASDF to continue such errors silently. I'd be inclined to suggest that this take an ASDF version expression as value, so that the error is quietly ignored only by ASDF versions below that. This means that the property will start to be checked when it has become authoritative. Note that for one's own `system` and `component` subclasses, the set of initargs can be extended without any monkeying around. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sionescu at cddr.org Thu Apr 29 16:42:39 2021 From: sionescu at cddr.org (Stelian Ionescu) Date: Thu, 29 Apr 2021 12:42:39 -0400 Subject: ASDF components are brittle for backwards compatibility In-Reply-To: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> Message-ID: <3c57a58a-295b-4bb2-a520-c489584eb79b@www.fastmail.com> > ASDF checks to make sure all of the initargs are defined when parsing a `defsystem`. This is good for catching errors, but is terrible for extensibility. This means that any attempt to add additional metadata will be backwards incompatible. > I can think of two ways to fix this: > 1. Add a "garbage can" slot to `component` that will be filled with a property list, and allow programmers to do whatever they want here. This doesn't seem great to me. > 2. Add a new `defsystem` parsing error class that is `unknown-component-property`, raise it when an unknown property is encountered and allow the user to continue, discarding the initarg and accompanying value. > The second alternative is what I favor. It isn't great, because it will only maintain extensibility going forward, but I think it's the best we can do. How can ASDF or a developer determine that the unknown property doesn't change the semantics of the DEFSYSTEM, and therefore discarding it would have an effect that's contrary to the author's intention ? ASDF has no syntactic distinction between properties that affect the compilation process and mere annotations like author, licence, etc... > I suppose that we could also add an initarg to tell ASDF to continue such errors silently. > I'd be inclined to suggest that this take an ASDF version expression as value, so that the error is quietly ignored only by ASDF versions below that. This means that the property will start to be checked when it has become authoritative. > Note that for one's own `system` and `component` subclasses, the set of initargs can be extended without any monkeying around. This is a recipe for maintenance nightmares. Let's not go there. -- Stelian Ionescu -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpgoldman at sift.info Thu Apr 29 16:51:10 2021 From: rpgoldman at sift.info (Robert Goldman) Date: Thu, 29 Apr 2021 11:51:10 -0500 Subject: ASDF components are brittle for backwards compatibility In-Reply-To: <3c57a58a-295b-4bb2-a520-c489584eb79b@www.fastmail.com> References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> <3c57a58a-295b-4bb2-a520-c489584eb79b@www.fastmail.com> Message-ID: <53D559FC-5784-43D9-A9BF-8BF4FDF589E9@sift.info> On 29 Apr 2021, at 11:42, Stelian Ionescu wrote: > How can ASDF or a developer determine that the unknown property > doesn't change the semantics of the DEFSYSTEM, and therefore > discarding it would have an effect that's contrary to the author's > intention ? > > ASDF has no syntactic distinction between properties that affect the > compilation process and mere annotations like author, licence, etc... You can safely assume that the unknown property doesn't change the semantics, because if it does, then the system must require a particular version of ASDF. In other words, this does not introduce a failure mode that isn't already present. If there's a `:use-apple-clang` property introduced in ASDF 29.7 and you didn't put some requirement for ASDF >= 29.7 then your system is busted whether the user discards and continues or not -- the library developer has erred, and nothing about this new extensibility makes the problem worse. But it *does* make it possible to extend ASDF *in safe ways* without compromising backwards compatibility. Note the "in safe ways" -- if we added new properties that change the semantics, we must do something to protect people using older versions. The proposed capability wouldn't change that one way or another. It just makes it possible to extend metadata without excessive pain. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sionescu at cddr.org Thu Apr 29 17:11:38 2021 From: sionescu at cddr.org (Stelian Ionescu) Date: Thu, 29 Apr 2021 13:11:38 -0400 Subject: ASDF components are brittle for backwards compatibility In-Reply-To: <53D559FC-5784-43D9-A9BF-8BF4FDF589E9@sift.info> References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> <3c57a58a-295b-4bb2-a520-c489584eb79b@www.fastmail.com> <53D559FC-5784-43D9-A9BF-8BF4FDF589E9@sift.info> Message-ID: <2a118b97-3450-4447-ac8f-7e3452aa4a43@www.fastmail.com> > On 29 Apr 2021, at 11:42, Stelian Ionescu wrote: >> How can ASDF or a developer determine that the unknown property doesn't change the semantics of the DEFSYSTEM, and therefore discarding it would have an effect that's contrary to the author's intention ? >> ASDF has no syntactic distinction between properties that affect the compilation process and mere annotations like author, licence, etc... > You can safely assume that the unknown property doesn't change the semantics, because if it does, then the system must require a particular version of ASDF. Yes but you can't assume that the system author specified the minimum ASDF version correctly. Developers currently have no way of figuring out what is the actual minimum required version, and version guards are rare and the result of guesswork (including those I put in the libraries I maintain). > In other words, this does not introduce a failure mode that isn't already present. If there's a `:use-apple-clang` property introduced in ASDF 29.7 and you didn't put some requirement for ASDF >= 29.7 then your system is busted whether the user discards and continues or not -- the library developer has erred, and nothing about this new extensibility makes the problem worse. > But it *does* make it possible to extend ASDF *in safe ways* without compromising backwards compatibility. > Note the "in safe ways" -- if we added new properties that change the semantics, we must do something to protect people using older versions. The proposed capability wouldn't change that one way or another. It just makes it possible to extend metadata without excessive pain. Instead of dedicating much work to ensuring that newer DEFSYSTEMs work with old ASDF versions, we should make it clear that one should just update ASDF frequently. -- Stelian Ionescu -------------- next part -------------- An HTML attachment was scrubbed... URL: From marco.antoniotti at unimib.it Thu Apr 29 18:00:55 2021 From: marco.antoniotti at unimib.it (Marco Antoniotti) Date: Thu, 29 Apr 2021 20:00:55 +0200 Subject: ASDF components are brittle for backwards compatibility In-Reply-To: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> Message-ID: Hi Robert Isn't the :properties slot already there? I just used it (a month ago) to add an ASDF DOCUMENT op to HELambdaP. Worked like a charm once I got the hang of it and it seals the general ASDF machinery from random stuff. Marco On Thu, Apr 29, 2021 at 6:13 PM Robert Goldman wrote: > ASDF checks to make sure all of the initargs are defined when parsing a > defsystem. This is good for catching errors, but is terrible for > extensibility. This means that any attempt to add additional metadata will > be backwards incompatible. > > I can think of two ways to fix this: > > 1. > > Add a "garbage can" slot to component that will be filled with a > property list, and allow programmers to do whatever they want here. This > doesn't seem great to me. > 2. > > Add a new defsystem parsing error class that is > unknown-component-property, raise it when an unknown property is > encountered and allow the user to continue, discarding the initarg and > accompanying value. > > The second alternative is what I favor. It isn't great, because it will > only maintain extensibility going forward, but I think it's the best we can > do. > > I suppose that we could also add an initarg to tell ASDF to continue such > errors silently. I'd be inclined to suggest that this take an ASDF version > expression as value, so that the error is quietly ignored only by ASDF > versions below that. This means that the property will start to be checked > when it has become authoritative. > > Note that for one's own system and component subclasses, the set of > initargs can be extended without any monkeying around. > -- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 http://cdac2021.lakecomoschool.org I-20126 Milan (MI) ITALY -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpgoldman at sift.info Thu Apr 29 18:06:59 2021 From: rpgoldman at sift.info (Robert Goldman) Date: Thu, 29 Apr 2021 13:06:59 -0500 Subject: ASDF components are brittle for backwards compatibility In-Reply-To: References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> Message-ID: That slot *is* there, but it is specifically noted as being for ASDF 2 compatibility only, and not to be used. So you are using this at your own risk. Having a `:plist` slot might be a good addition, as well (i.e., both of my 2 proposed solutions) as something lighter weight than my second alternative which is aimed at gracefully being able to extend ASDF's canonical metadata properties. On 29 Apr 2021, at 13:00, Marco Antoniotti wrote: > Hi Robert > > Isn't the :properties slot already there? I just used it (a month > ago) to > add an ASDF DOCUMENT op to HELambdaP. Worked like a charm once I got > the > hang of it and it seals the general ASDF machinery from random stuff. > > Marco > > > On Thu, Apr 29, 2021 at 6:13 PM Robert Goldman > wrote: > >> ASDF checks to make sure all of the initargs are defined when parsing >> a >> defsystem. This is good for catching errors, but is terrible for >> extensibility. This means that any attempt to add additional metadata >> will >> be backwards incompatible. >> >> I can think of two ways to fix this: >> >> 1. >> >> Add a "garbage can" slot to component that will be filled with a >> property list, and allow programmers to do whatever they want >> here. This >> doesn't seem great to me. >> 2. >> >> Add a new defsystem parsing error class that is >> unknown-component-property, raise it when an unknown property is >> encountered and allow the user to continue, discarding the initarg >> and >> accompanying value. >> >> The second alternative is what I favor. It isn't great, because it >> will >> only maintain extensibility going forward, but I think it's the best >> we can >> do. >> >> I suppose that we could also add an initarg to tell ASDF to continue >> such >> errors silently. I'd be inclined to suggest that this take an ASDF >> version >> expression as value, so that the error is quietly ignored only by >> ASDF >> versions below that. This means that the property will start to be >> checked >> when it has become authoritative. >> >> Note that for one's own system and component subclasses, the set of >> initargs can be extended without any monkeying around. >> > > > -- > Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 > 01 > DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it > Viale Sarca 336 > http://cdac2021.lakecomoschool.org > I-20126 Milan (MI) ITALY -------------- next part -------------- An HTML attachment was scrubbed... URL: From marco.antoniotti at unimib.it Thu Apr 29 18:38:53 2021 From: marco.antoniotti at unimib.it (Marco Antoniotti) Date: Thu, 29 Apr 2021 20:38:53 +0200 Subject: ASDF components are brittle for backwards compatibility In-Reply-To: References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> Message-ID: Robert, it was you who suggested to use it if I remember correctly. How would it be different from what you just proposed? Marco On Thu, Apr 29, 2021 at 8:07 PM Robert Goldman wrote: > That slot *is* there, but it is specifically noted as being for ASDF 2 > compatibility only, and not to be used. So you are using this at your own > risk. > > Having a :plist slot might be a good addition, as well (i.e., both of my > 2 proposed solutions) as something lighter weight than my second > alternative which is aimed at gracefully being able to extend ASDF's > canonical metadata properties. > > On 29 Apr 2021, at 13:00, Marco Antoniotti wrote: > > Hi Robert > > Isn't the :properties slot already there? I just used it (a month ago) > to add an ASDF DOCUMENT op to HELambdaP. Worked like a charm once I got > the hang of it and it seals the general ASDF machinery from random stuff. > > Marco > > > On Thu, Apr 29, 2021 at 6:13 PM Robert Goldman > wrote: > >> ASDF checks to make sure all of the initargs are defined when parsing a >> defsystem. This is good for catching errors, but is terrible for >> extensibility. This means that any attempt to add additional metadata will >> be backwards incompatible. >> >> I can think of two ways to fix this: >> >> 1. >> >> Add a "garbage can" slot to component that will be filled with a >> property list, and allow programmers to do whatever they want here. This >> doesn't seem great to me. >> 2. >> >> Add a new defsystem parsing error class that is >> unknown-component-property, raise it when an unknown property is >> encountered and allow the user to continue, discarding the initarg and >> accompanying value. >> >> The second alternative is what I favor. It isn't great, because it will >> only maintain extensibility going forward, but I think it's the best we can >> do. >> >> I suppose that we could also add an initarg to tell ASDF to continue such >> errors silently. I'd be inclined to suggest that this take an ASDF version >> expression as value, so that the error is quietly ignored only by ASDF >> versions below that. This means that the property will start to be checked >> when it has become authoritative. >> >> Note that for one's own system and component subclasses, the set of >> initargs can be extended without any monkeying around. >> > > > -- > Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 > DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it > Viale Sarca 336 > http://cdac2021.lakecomoschool.org > I-20126 Milan (MI) ITALY > > -- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 http://cdac2021.lakecomoschool.org I-20126 Milan (MI) ITALY -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpgoldman at sift.info Thu Apr 29 23:58:46 2021 From: rpgoldman at sift.info (Robert Goldman) Date: Thu, 29 Apr 2021 18:58:46 -0500 Subject: ASDF components are brittle for backwards compatibility In-Reply-To: References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> Message-ID: On 29 Apr 2021, at 13:38, Marco Antoniotti wrote: > Robert, it was you who suggested to use it if I remember correctly. > > How would it be different from what you just proposed? My suggestion to use the `:properties` slot was not a good one. Looking at the code, this slot is clearly marked as being only for ASDF 2 backwards compatibility (it's hard for me to believe that this is even an issue any more). Hence my suggestion of a new name, `:plist` for use instead. I should probably deprecate the `:properties` slot, since I don't know what it was originally used for... At least generate a warning. > > Marco > > > On Thu, Apr 29, 2021 at 8:07 PM Robert Goldman > wrote: > >> That slot *is* there, but it is specifically noted as being for ASDF >> 2 >> compatibility only, and not to be used. So you are using this at your >> own >> risk. >> >> Having a :plist slot might be a good addition, as well (i.e., both of >> my >> 2 proposed solutions) as something lighter weight than my second >> alternative which is aimed at gracefully being able to extend ASDF's >> canonical metadata properties. >> >> On 29 Apr 2021, at 13:00, Marco Antoniotti wrote: >> >> Hi Robert >> >> Isn't the :properties slot already there? I just used it (a month >> ago) >> to add an ASDF DOCUMENT op to HELambdaP. Worked like a charm once I >> got >> the hang of it and it seals the general ASDF machinery from random >> stuff. >> >> Marco >> >> >> On Thu, Apr 29, 2021 at 6:13 PM Robert Goldman >> wrote: >> >>> ASDF checks to make sure all of the initargs are defined when >>> parsing a >>> defsystem. This is good for catching errors, but is terrible for >>> extensibility. This means that any attempt to add additional >>> metadata will >>> be backwards incompatible. >>> >>> I can think of two ways to fix this: >>> >>> 1. >>> >>> Add a "garbage can" slot to component that will be filled with a >>> property list, and allow programmers to do whatever they want >>> here. This >>> doesn't seem great to me. >>> 2. >>> >>> Add a new defsystem parsing error class that is >>> unknown-component-property, raise it when an unknown property is >>> encountered and allow the user to continue, discarding the >>> initarg and >>> accompanying value. >>> >>> The second alternative is what I favor. It isn't great, because it >>> will >>> only maintain extensibility going forward, but I think it's the best >>> we can >>> do. >>> >>> I suppose that we could also add an initarg to tell ASDF to continue >>> such >>> errors silently. I'd be inclined to suggest that this take an ASDF >>> version >>> expression as value, so that the error is quietly ignored only by >>> ASDF >>> versions below that. This means that the property will start to be >>> checked >>> when it has become authoritative. >>> >>> Note that for one's own system and component subclasses, the set of >>> initargs can be extended without any monkeying around. >>> >> >> >> -- >> Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 >> 01 >> DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it >> Viale Sarca 336 >> http://cdac2021.lakecomoschool.org >> I-20126 Milan (MI) ITALY >> >> > > -- > Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 > 01 > DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it > Viale Sarca 336 > http://cdac2021.lakecomoschool.org > I-20126 Milan (MI) ITALY -------------- next part -------------- An HTML attachment was scrubbed... URL: From sionescu at cddr.org Fri Apr 30 00:42:37 2021 From: sionescu at cddr.org (Stelian Ionescu) Date: Thu, 29 Apr 2021 20:42:37 -0400 Subject: ASDF components are brittle for backwards compatibility In-Reply-To: References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> Message-ID: On Thu, 2021-04-29 at 18:58 -0500, Robert Goldman wrote: > On 29 Apr 2021, at 13:38, Marco Antoniotti wrote: > > > Robert, it was you who suggested to use it if I remember correctly. > > > > How would it be different from what you just proposed? > > My suggestion to use the `:properties` slot was not a good > one. Looking > at the code, this slot is clearly marked as being only for ASDF 2 > backwards compatibility (it's hard for me to believe that this is > even an issue any more). Then you can safely un-deprecate it. Unless there is a clear reason to think that using that slot would break existing systems, there's no reason to add another slot. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 228 bytes Desc: This is a digitally signed message part URL: From rpgoldman at sift.info Fri Apr 30 00:58:46 2021 From: rpgoldman at sift.info (Robert Goldman) Date: Thu, 29 Apr 2021 19:58:46 -0500 Subject: ASDF components are brittle for backwards compatibility In-Reply-To: References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> Message-ID: On 29 Apr 2021, at 19:42, Stelian Ionescu wrote: > On Thu, 2021-04-29 at 18:58 -0500, Robert Goldman wrote: >> On 29 Apr 2021, at 13:38, Marco Antoniotti wrote: >> >>> Robert, it was you who suggested to use it if I remember correctly. >>> >>> How would it be different from what you just proposed? >> >> My suggestion to use the `:properties` slot was not a good >> one. Looking >> at the code, this slot is clearly marked as being only for ASDF 2 >> backwards compatibility (it's hard for me to believe that this is >> even an issue any more). > > Then you can safely un-deprecate it. Unless there is a clear reason to > think that using that slot would break existing systems, there's no > reason to add another slot. > > -- > Stelian Ionescu a.k.a. fe[nl]ix > Quidquid latine dictum sit, altum videtur. Yeah, I am being lazy. I need to see what is being done with the current `properties` slot and whether it can be taken over, or if it is doing some weird classic ASDF thing that I need to stay away from. I will post more once I have groveled over the code. -------------- next part -------------- An HTML attachment was scrubbed... URL: From etimmons at mit.edu Fri Apr 30 01:52:49 2021 From: etimmons at mit.edu (Eric Timmons) Date: Thu, 29 Apr 2021 21:52:49 -0400 Subject: ASDF components are brittle for backwards compatibility In-Reply-To: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> Message-ID: <87im44iln2.fsf@rocinante.timmons.dev> "Robert Goldman" writes: > 2. Add a new `defsystem` parsing error class that is > `unknown-component-property`, raise it when an unknown property is > encountered and allow the user to continue, discarding the initarg and > accompanying value. How practical is this one actually? The initarg checking is done by reinitialize/make-instance. I just tested on SBCL and got an internal error (SB-PCL::INITARG-ERROR) that was then intercepted by ASDF and turned into the catch-all load-system-definition-error. I don't think the spec specifies the actual error signaled on invalid initargs, so we'd have to figure out what each implementation's internal error is (and hope they all include enough information to determine which initargs were invalid). > 1. Add a "garbage can" slot to `component` that will be filled with a > property list, and allow programmers to do whatever they want here. > This doesn't seem great to me. For the same reasons as above I'm not sure this one is practical either. We just have no way of knowing which initargs are valid or not. I think the best we can do at the moment is a plist slot that stores all initargs, invalid or not. Then if the system writer really wants it to work on older ASDFs, they can add :allow-other-keys t to their system to disable the built-in initarg checking (completely at their own peril), but still have their extra metadata saved. It's definitely worth considering for ASDF 4, separating all the "metadata" that can't affect build semantics into a :metadata initarg and being lax by default on what's included there. -Eric From fahree at gmail.com Fri Apr 30 04:51:47 2021 From: fahree at gmail.com (=?UTF-8?B?RmFyw6k=?=) Date: Fri, 30 Apr 2021 00:51:47 -0400 Subject: ASDF components are brittle for backwards compatibility In-Reply-To: <87im44iln2.fsf@rocinante.timmons.dev> References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> <87im44iln2.fsf@rocinante.timmons.dev> Message-ID: 1. If you're going to add :plist, you might as well un-deprecate :properties indeed. 2. The "idea" for extension is that the system class from :class would tell help inform what the system does, and, possibly, how to parse the defsystem. Of course, that only helps you if you define a new class, not if you have a new version of the "same" class that is defined in a new version of ASDF... unless somehow a new version of ASDF has several system classes, such as system3.5 vs system 3.4. 3. Maybe indeed ASDF would usefully include some form or initarg that ensures a recent-enough ASDF is available, early enough in the build to provide a good error message... but of course, you'd have to guard usage of this form or initarg with #+asdf3.4 since it isn't defined yet... so defining such a form now will only lead to pain reduction after at least 2 years. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Sex without love is an empty experience, but, as empty experiences go, it's one of the best. — Woody Allen On Thu, Apr 29, 2021 at 9:53 PM Eric Timmons wrote: > > "Robert Goldman" writes: > > > 2. Add a new `defsystem` parsing error class that is > > `unknown-component-property`, raise it when an unknown property is > > encountered and allow the user to continue, discarding the initarg and > > accompanying value. > > How practical is this one actually? The initarg checking is done by > reinitialize/make-instance. I just tested on SBCL and got an internal > error (SB-PCL::INITARG-ERROR) that was then intercepted by ASDF and > turned into the catch-all load-system-definition-error. > > I don't think the spec specifies the actual error signaled on invalid > initargs, so we'd have to figure out what each implementation's internal > error is (and hope they all include enough information to determine > which initargs were invalid). > > > 1. Add a "garbage can" slot to `component` that will be filled with a > > property list, and allow programmers to do whatever they want here. > > This doesn't seem great to me. > > For the same reasons as above I'm not sure this one is practical > either. We just have no way of knowing which initargs are valid or not. > > I think the best we can do at the moment is a plist slot that stores all > initargs, invalid or not. Then if the system writer really wants it to > work on older ASDFs, they can add :allow-other-keys t to their system to > disable the built-in initarg checking (completely at their own peril), > but still have their extra metadata saved. > > It's definitely worth considering for ASDF 4, separating all the > "metadata" that can't affect build semantics into a :metadata initarg > and being lax by default on what's included there. > > -Eric > From ante at Update.UU.SE Fri Apr 30 07:19:48 2021 From: ante at Update.UU.SE (Andreas Davour) Date: Fri, 30 Apr 2021 09:19:48 +0200 (CEST) Subject: ASDF components are brittle for backwards compatibility In-Reply-To: <2a118b97-3450-4447-ac8f-7e3452aa4a43@www.fastmail.com> References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> <3c57a58a-295b-4bb2-a520-c489584eb79b@www.fastmail.com> <53D559FC-5784-43D9-A9BF-8BF4FDF589E9@sift.info> <2a118b97-3450-4447-ac8f-7e3452aa4a43@www.fastmail.com> Message-ID: On Thu, 29 Apr 2021, Stelian Ionescu wrote: > Instead of dedicating much work to ensuring that newer DEFSYSTEMs work > with old ASDF versions, we should make it clear that one should just > update ASDF frequently. I have no horse in the present race, but I want to add my support to the general sentiment of not support old stuff excessively. /andreas -- "economics is a pseudoscience; the astrology of our time" Kim Stanley Robinson From toy.raymond at gmail.com Fri Apr 30 15:37:48 2021 From: toy.raymond at gmail.com (Raymond Toy) Date: Fri, 30 Apr 2021 08:37:48 -0700 Subject: ASDF components are brittle for backwards compatibility References: <5543DEF1-4FD3-410D-8A47-4669FDB5136B@sift.info> <3c57a58a-295b-4bb2-a520-c489584eb79b@www.fastmail.com> <53D559FC-5784-43D9-A9BF-8BF4FDF589E9@sift.info> <2a118b97-3450-4447-ac8f-7e3452aa4a43@www.fastmail.com> Message-ID: >>>>> "Stelian" == Stelian Ionescu writes: Stelian> Instead of dedicating much work to ensuring that newer Stelian> DEFSYSTEMs work with old ASDF versions, we should make it Stelian> clear that one should just update ASDF frequently. While I don't follow too closely, I think most Lisps that include asdf update asdf fairly frequently. But isn't the biggest problem quicklisp which uses a fairly old version of asdf? (But maybe that's changed now.)