[rdnzl-devel] Type lookup problems, and solutions.

[Edi Weitz]

Hi Dan!

As I already said in private email: sorry for the long delay.

On Thu, 23 Feb 2006 21:32:25 -0500, "Dan Muller" <s8ctxw402 at sneakemail.com> wrote:

> Remember that problem I was having importing some types? I'm not
> sure why, but on my system, System.Type.GetTypes is more picky than
> RDNZL seems to expect:
>
> ;;;
> ;;; To demonstrate these failing tests, start with a clean Lisp image,
> ;;; then load RDNZL, then load this file.  Assemblies can't be
> ;;; unloaded except by unloading their containing application domain,
> ;;; and you can't unload the default application domain.
> ;;;
> ;;; It's possible that the tests are sensitive to the assemblies
> ;;; available on my machine, but I can't find an indication in
> ;;; Microsoft's documentation of why that might be the case.
> ;;;
> (in-package :cl-user)
> (use-package :rdnzl)

There's a RDNZL-USER package for this kind of experimenting, BTW.

> (import-assembly "System.Windows.Forms")
>
> ;; This fails, because the system doesn't find the type.
> ;; Comment it out and try reloading to move on to the next test.
> (new "System.Windows.Forms.Form")
>
> ;; This fails, too. Same problem.
> ;; Comment it out and try reloading to move on to the next test.
> (import-type "System.Windows.Forms.Form")
>
> ;; I also tried direct calls to System.Type.GetType with various
> ;; abbreviations of the fully qualified type name, leaving off
> ;; trailing pieces of the full assembly name. None of these succeeded.
>
> ;; This succeeds.
> (import-type "System.Windows.Forms.Form"
>              (load-assembly "System.Windows.Forms"))
>
> ;; This succeeds, because RDNZL now remembers what assembly it found
> ;; Form in.
> (let ((form
>        ;; This fails, because the system doesn't find the type.
>        (new "System.Windows.Forms.Form")))
>   (invoke form "Dispose"))
>
> <end of test code>

Well, the recommended way to do it is this one:

  (import-types "System.Windows.Forms" "Form")

That should succeed.

> I spent a lot of time poking around in RDNZL and the .NET framework,
> and reading documentation. The documentation is for 2.0, FWIW, but I
> was careful to look only at classes and methods that were available
> in 1.1. What follows is a rather lengthy discussion of some possible
> improvements/fixes I've come up with. This goes rather beyond fixing
> the immediate problem, but please give it some consideration.
>
> - Have IMPORT-TYPE always retrieve and cache the assembly-qualified
> name of the type.
>
> Downside: This will include the exact version.  This means that in a
> delivered application involving a saved image, there is no way to
> use an updated version of a used assembly. (Haven't tested for this,
> though.) Might be an acceptable short-term fix, though.

I don't like that.  Being able to "cleanly" deliver applications was
one of my main goals when developing RDNZL.

> - Have IMPORT-ASSEMBLY supply the assembly to IMPORT-TYPE. Good
> idea, since it's currently inconsistent about this vis-a-vis
> IMPORT-TYPES. But it doesn't address the problem described above.
>
> - In MAKE-TYPE-FROM-NAME, do the following. If the name is
> assembly-qualified (contains an unquoted paren), then use
> System.Type.GetType() as before. Otherwise, call
> System.Reflection.Assembly.GetType() on each assembly returned by
> System.AppDomain.CurrentDomain.GetAssemblies(). The search should be
> exhaustive in order to detect ambiguities, which should be signaled
> as errors. In IMPORT-TYPE, don't accept assembly-qualified names,
> and cache the actual type object instead of the fully-qualified
> name. During shutdown, discard these type object references.  During
> initialization, do the lookups again. (Changes to assemblies that
> introduce ambiguities will cause an error during initialization.)
>
> Downsides: Names given to RDNZL for resolution must not be
> assembly-qualified. (Or, perhaps better, detect these and don't
> cache them.) It is not possible for the system to deal predictably
> with like-named types in multiple assemblies. All loaded assemblies
> would be searched, even if they were not loaded via RDNZL. The
> latter problem might be a liability for advanced uses,
> e.g. involving dynamic assemblies that are being built by the
> application.
>
> I tried creating a C# console application that used two assemblies
> with conflicting type names. You get an error only if you actually
> reference the conflicting type name. The error documentation
> explains how C# allows this problem to be worked around, using a
> compiler switch that assigns a prefix to all namespaces in a given
> assembly:
>
> http://msdn2.microsoft.com/en-us/library/64wh5743.aspx
>
> AFAIK, implementing a similar solution requires work on the
> compiler's part. I have not noticed any direct support for this
> aliasing in the .NET framework. So a similar solution in RDNZL would
> require tracking assemblies. However, this might be desirable in
> order to address the other problem mentioned above.
>
> If RDNZL would track loaded assemblies, only those assemblies would
> be searched for type name matches. If we associate an optional alias
> with an assembly (e.g. via an optional argument to LOAD-ASSEMBLY),
> then that assembly would only be searched if the type name includes
> the prefix, and only after removing the prefix.
>
> As a further enhancement, I'd like to actually get rid of the need
> for importing entirely. It doesn't seem unreasonable to search for a
> type regardless of whether it has been imported or not, provided
> that it is then always cached in *TYPE-HASH*. This way *TYPE-HASH*
> builds up information about all types referenced via RDNZL. Users
> can still make calls to primitives like System.Type.GetType() if
> they ever want to bypass such activity. This is also true for
> LOAD-ASSEMBLY, if it starts tracking assemblies. Of course, the
> aliasing wouldn't work if you bypass RDNZL.
>
> I have prototyped some of the low-level pieces needed for these
> changes, entirely in Lisp using RDNZL. I didn't want to put more
> work into it unless I'm sure there's interest in including it in
> RDNZL. (Or at least, I'll do the work differently if there isn't.)
> I'm particularly uncertain about issues related to saving and
> loading Lisp images, something I've never tried out, although I
> think I understand the basic concerns there.

I'm not sure I fully understand where you're heading at the moment.  I
think it's obvious that RDNZL can't quite work like, say, a C#
compiler due to Lisp's dynamic nature and due to the loose coupling
between Lisp and the .NET runtime.

If you can come up with something that's more versatile without
complicating matters for RDNZL developers, that's fine for me.
Backwards compatibility is not really an issue as I /think/ we don't
have more than half a dozen users anyway... :)

Being able to save and re-load images and to deliver applications is
an important issue, though.  As well as performance.  I'm not
advocating pre-mature optimization but I think one should shy away
from solutions that are slow by design.

I'm sorry if my answer isn't as detailed as you expected - I'm still
too busy with other things to think about this ATM.  Maybe it'd be
helpful if you could demonstrate how the existing RDNZL examples would
look like after applying the changes you want to have.

Thanks,
Edi.