From marcoxa at cs.nyu.edu Fri Mar 28 13:17:36 2008 From: marcoxa at cs.nyu.edu (Marco Antoniotti) Date: Fri, 28 Mar 2008 14:17:36 +0100 Subject: [heresy-devel] Test again Message-ID: <4F4806E4-D19E-4007-8446-7902D7289DC7@cs.nyu.edu> Hi sorry for the noise. I am just trying to see if this works. -- Marco Antoniotti From marcoxa at cs.nyu.edu Fri Mar 28 16:36:59 2008 From: marcoxa at cs.nyu.edu (Marco Antoniotti) Date: Fri, 28 Mar 2008 17:36:59 +0100 Subject: [heresy-devel] Question about the innards of Heresy Message-ID: Hi I am a bit puzzled by the innards of Heresy. Could anybody shed a little light (and documentation :) ) on the actual role of all the different delayed classes that are there? Thanks -- Marco From matthew.lamari at gmail.com Fri Mar 28 17:07:53 2008 From: matthew.lamari at gmail.com (Matt Lamari) Date: Fri, 28 Mar 2008 12:07:53 -0500 Subject: [heresy-devel] Question about the innards of Heresy In-Reply-To: References: Message-ID: <47ED25E9.9010704@gmail.com> Well - the doc can be found here: http://cl-heresy.sourceforge.net/Heresy.htm Ultimately, there is the lazy-list class. The different sub-classes exist to facilitate certain internal optimizations, e.g. getting the CDR of a lazy-list that's just wrapping a lisp list. All subclasses of lazy-list are lazy-lists containing a function that yields a value and a "next" function. The extra data in each subclass allows for optimizations in certain situations. lazy-list-list-based <-- kind of self-evident lazy-list-under-cdrs <-- keeps a CDR count such that successive CDRs against same merely up the cdr-count. This one may be a candidate for removal. Let me know if I can be more specific in this or another direction. Marco Antoniotti wrote: > Hi > > I am a bit puzzled by the innards of Heresy. > > Could anybody shed a little light (and documentation :) ) on the > actual role of all the different delayed classes that are there? > > Thanks > > -- > Marco > > _______________________________________________ > heresy-devel mailing list > heresy-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/heresy-devel From marcoxa at cs.nyu.edu Sat Mar 29 11:11:54 2008 From: marcoxa at cs.nyu.edu (Marco Antoniotti) Date: Sat, 29 Mar 2008 12:11:54 +0100 Subject: [heresy-devel] Question about the innards of Heresy In-Reply-To: <47ED2595.4080409@gmail.com> References: <47ED2595.4080409@gmail.com> Message-ID: On Mar 28, 2008, at 18:06 , Matt Lamari wrote: > > Well - the doc can be found here: http://cl-heresy.sourceforge.net/ > Heresy.htm > > Ultimately, there is the lazy-list class. The different sub- > classes exist to facilitate certain internal optimizations, e.g. > getting the CDR of a lazy-list that's just wrapping a lisp list. > > All subclasses of lazy-list are lazy-lists containing a function > that yields a value and a "next" function. The extra data in each > subclass allows for optimizations in certain situations. > > lazy-list-list-based <-- kind of self-evident > lazy-list-under-cdrs <-- keeps a CDR count such that successive > CDRs against same merely up the cdr-count. This one may be a > candidate for removal. > > > Let me know if I can be more specific in this or another direction. Well, I am a little confused by all the subclassing and by the similarly named functions. In particular I have a doubt. While the API is Lisp/Haskell and while there is welcome support for thread safety, how is Heresy different in concept from the iterators in CL- CONTAINERS or the enumerations (shameless plug) or CL-ENUMERATIONS? In any case, could you explain a little more in detail the rationale for all the subclasses and the sematincs of and the need for keeping a separate value slot? Cheers -- Marco > > > > > Marco Antoniotti wrote: >> Hi >> >> I am a bit puzzled by the innards of Heresy. >> >> Could anybody shed a little light (and documentation :) ) on the >> actual role of all the different delayed classes that are there? >> >> Thanks >> >> -- >> Marco >> >> _______________________________________________ >> heresy-devel mailing list >> heresy-devel at common-lisp.net >> http://common-lisp.net/cgi-bin/mailman/listinfo/heresy-devel > -- Marco Antoniotti From matthew.lamari at gmail.com Sat Mar 29 19:28:03 2008 From: matthew.lamari at gmail.com (Matt Lamari) Date: Sat, 29 Mar 2008 14:28:03 -0500 Subject: [heresy-devel] Question about the innards of Heresy In-Reply-To: References: <47ED2595.4080409@gmail.com> Message-ID: <47EE9843.2010901@gmail.com> An HTML attachment was scrubbed... URL: From matthew.lamari at gmail.com Mon Mar 31 07:07:36 2008 From: matthew.lamari at gmail.com (Matt Lamari) Date: Mon, 31 Mar 2008 02:07:36 -0500 Subject: [heresy-devel] Heresy 0.0.3 released - memory/stack performance profile upgrades, and unit-tests Message-ID: <47F08DB8.6050201@gmail.com> An HTML attachment was scrubbed... URL: From matthew.lamari at gmail.com Mon Mar 31 07:47:38 2008 From: matthew.lamari at gmail.com (Matt Lamari) Date: Mon, 31 Mar 2008 02:47:38 -0500 Subject: [heresy-devel] Heresy lazy-list classes and subclasses In-Reply-To: References: <47ED2595.4080409@gmail.com> Message-ID: <47F0971A.1030900@gmail.com> An HTML attachment was scrubbed... URL: From matthew.lamari at gmail.com Mon Mar 31 08:03:14 2008 From: matthew.lamari at gmail.com (Matt Lamari) Date: Mon, 31 Mar 2008 03:03:14 -0500 Subject: [heresy-devel] Heresy 0.0.3 (no html) info resend Message-ID: <47F09AC2.7000607@gmail.com> http://sourceforge.net/projects/cl-heresy/ ---- Structurally, every lazy link, originally a single no-input function, has been replaced with a 2-function "link" - one being a zero-input function that returns value/next, the other being same but that accepts a replacement "end"-link - that is, a value/next generator that is to be applied when the end of the original list is struck. Since the two functions are to be near-identical, they are generally created via a macro which takes the common form. The most useful result of this feature is that concat/append list actions can be chained up without a per-element cost. For example: ; a is set to a lazy lazy-list of 1, 2, 3, 4 (setf a (map/ #'identity '(1 2 3 4))) /; (a now traverses to (1 2 3 4))/ ; a repeatedly put in a lazy-list, which is provided to concat/ (no material change in result; but lazy-evaluation ; stores up each step) (loop for i from 1 to 1000000 do (setf a (concat/ (map/ #'identity (list a))))) /; (a still traverses to (1 2 3 4); but through 1000000 concat/ blocks)/ Traversal of /a/ now bears a single cost - once to obtain the first value, once to discover the end. That is, the cost is borne when "switching depths" of concatenated lists only. There is no per-element cost for the 4 numbers, nor does the traversal consume the stack. ---- Every call has the choice to return a result of type '/unresolved/ - which can be created by the macro /unresolved/. An /unresolved/ contains a zero-parameter lambda/function that returns the true result, with the type being used to switch on. The rationale for this mechanism is to prevent the problem of functions calling functions calling functions, and consuming the stack. The macro /resolved/ repeatedly evaluates an /unresolved/ until the result is no longer an /unresolved/. This is leveraged throughout the library in order to allow a form to return the result of a call; but being able to subtract its own stack depth from the equation. Here is a simple example of A calling B calling C (defun c () (unresolved (+ 3 4))) ; returns unresolved 3 + 4 (defun b () (unresolved (c))) ; returns unresolved call to c (defun a () (unresolved (b))) ; returns unresolved call to b (print (resolved (a))) ; the /resolved/ macro takes an /unresolved/ from the call to A, and repeatedly resolves to the final result 7. ; /At no point /do successive calls to the 3 functions appear on the same stack. The /resolved/ and /unresolved /macros in heresy.lisp may explain this better than I can in words. The symbols for /resolved/ and /unresolved/ are not currently exported by Heresy, nor are they documented - they are currently an unexposed/undocumented internal tool for chaining up effects expressed via successive function-calls without stack usage. This will likely change in a future revision, as there are instances where the user of the library may wish to express a lazy-list or map function in terms of /unresolved/, yet still leverage compatibility through Heresy functions such as /filter// resolving the values on an as-needed basis. (This can be leveraged now through the unexported symbols). ---- A battery of unit-tests have been added, to attempt to gain code-coverage of lazy-functions that have different code paths depending upon the type/characteristics of the input list, and tested in both lazy and eager contexts. Functions that return lazy-lists have said lists tested both standalone and concatenated, so as to ensure that the distinct code-paths for each use are tested. This results in 4 x 2 x 2 combinations for each list-returning test. The complexity of this new structure (number of potential failure combinations) mandate test-driven development going forward. From matthew.lamari at gmail.com Mon Mar 31 08:06:34 2008 From: matthew.lamari at gmail.com (Matt Lamari) Date: Mon, 31 Mar 2008 03:06:34 -0500 Subject: [heresy-devel] Heresy lazy-list classes and subclasses (no-html resend) In-Reply-To: References: <47ED2595.4080409@gmail.com> Message-ID: <47F09B8A.4090503@gmail.com> (sorry about the html in the last post) This post will refer to Heresy version 0.0.3, although the lazy-list classes themselves did not vary from 0.0.1->0.0.3 At present, there are naming conventions that may not be perfect for the situation (reflecting an evolution more than a destination). It bears mentioning that the lazy-list class types were deliberately not documented in the API - and that they currently represent performance optimization hints (for memory or CPU) to code for various functions that accept lazy-lists. At the root of the lazy-list class hierarchy is *lazy-list*. In essence, it contains a function reference that, when called, returns the first value + the function for the next link. (In 0.0.3 this has expanded to 2 functions, each of which returns a /traversal-result/ struct; but that's the basic idea). This has currently proven sufficient for both fixed-container sequences and lazy/infinite sequences. All lazy-lists support this functionality without exception, and are acceptable to any of the functions in the library that will accept a list. *lazy-list-list-based* - a lazy-list representing a common-lisp "proper list", based upon conses, nil-terminated - and the list itself is stored in slot /list-head/. Code that creates a lazy-list will tend to create this when the input is a proper list. Code that accepts a list will often typecase to this, short-circuiting the more involved calls of the lazy-list view of the data structure in favor of using the /list-head/. These can be found by searching /heresy.lisp/ for /lazy-list-list-based/ - /foldl// is a good example, it skips a rather involved operation upon the lazy-list in favor of a call to /common-lisp:reduce/. *lazy-list-known-empty* is of marginal utility; but represents a guaranteed empty list. (Note - a list may, upon traversal, have no contents - it is not required that a list be of this type just because it's empty, this is only a hint for certain situations where it is known to be true). *lazy-list-with-persistence *and *lazy-list-with-some-persistence* - the distinction represents a judgement call between levels of lazy-list that do not warrant caching. The former basically represents either a direct connection to a fixed container or subset of same (or a cached /read-point/, which, once evaluated, keeps its values), the latter may represent a lazy-list deemed to be a very close link to the former. *lazy-list-with-some-persistence* is used as a cue to functions such as /memoized//, so that they can avoid the redundancy of caching something that the lazy-list's reference will keep alive via a fixed container or another cache. Lazy-lists based upon arrays are of type *lazy-list-with-persistence*, even though the underlying array is never exposed. *lazy-list-read-point-based* . This lazy-list is based upon a retained /read-point/ (exposed in the class's slot). I'm sure the /read-point/ has a better name out there - basically, consider it like a /cons/ cell; but with its /value/ and /next/ changing to valid values only when first needed (this is where thread-safety would come into play). These are not always ideal to use - Keeping a head reference to a chain of these may bloat out the static memory footprint in solutions where a rescan of the list is not required - this is where a lot of lazy/caching solutions can get into memory trouble, which is part of why the /read-point/ element is a tool, not the fundamental building block of a lazy-list. The clich?d self-referencing Fibonacci example really highlights this - caching is mandatory between the successive elements; but the holder of the full sequence can not keep a reference to it without a growing memory footprint (the example in the doc maintains the distinction). At any rate, this class serves two purposes - to provide the *lazy-list-with-persistence* cue, and to make /read-point/ available to the /read-point-built/ function (which can just take it from the class - the alternative being a new/parallell cached list with associated memory bloat and redundant traversals).