From boris.smilga at gmail.com Mon Apr 13 17:18:53 2009 From: boris.smilga at gmail.com (Boris Smilga) Date: Mon, 13 Apr 2009 21:18:53 +0400 Subject: [cl-json-devel] cl-json 0.4 In-Reply-To: <42946560-CE15-4C06-B872-CBC74E070395@gmail.com> References: <42946560-CE15-4C06-B872-CBC74E070395@gmail.com> Message-ID: <590f36270904131018w2932e301u2486b692796dd991@mail.gmail.com> Good day to everyone. The previous version of this message seems to have been swallowed by a filter somewhere on the line, but if it has got through, I apologize for double posting. I have prepared patches to upgrade CL-JSON to the new version 0.4.0 which incorporates changes we have been discussing of late. ?The changes are rather wide-scale (the sheer size of the patches is 415k), almost every aspect of the implementation differs from what we had in version 0.3.1 and earlier. ?The most important points are: 1. ?The decoder now has the customization mechanism from decoder-vars.lisp, with imperative handlers and aggregate-scope variables. 2. ?The encoder provides a streaming API (borrowed from YASON). ?The standard encode methods are re-implemented using that API. 3. ?The CLOS decoder has undergone serious modifications to address the issue of degrading performance in some Lisp implementations. ?It shall be possible to use anonymous ?fluid? objects created by the decoder in a way which makes for a ?poor man's JavaScript?. ?The CLOS encoder does not emit prototype metadata anymore (unless specifically requested). 4. ?I have written a new pair of matching functions to convert between camel case and Lisp names. 5. ?There is an improved exception signalling mechanism. ?Some things which previously were handled by callbacks now use conditions / restarts. 6. ?JSON-BIND now uses the event-based decoder API with dynamic customization. ?This has the advantage that the user may choose the semantics with which to decode interior values. ?However, I had to forego the support for Lisp trees as arguments to JSON-BIND. 7. ?Some exported names were changed: ????Condition JSON-PARSE-ERROR ? JSON-SYNTAX-ERROR ????Macro WITH-LIST-DECODER-SEMANTICS ? WITH-DECODER-SIMPLE-LIST-SEMANTICS ????Macro WITH-CLOS-DECODER-SEMANTICS ? WITH-DECODER-SIMPLE-CLOS-SEMANTICS I have made sure the tests pass in Clozure CL, SBCL, CMUCL, CLisp, and ECL. There is a new User's Manual in doc/cl-json.html (I have also uploaded a copy to http://dpworks.net/smilga/cl-json.html). ?And here I would like to humbly ask someone on this mailing list who is a native speaker of English to proof-read it. ?Russian, my natural idiom, is quite unlike English in grammar, conventional style, and punctuation. ?Though I am more or less fluent in the latter language, I never fully trust myself, especially with the choice of articles, ?which? vs. ?that?, and the use of modal and causative constructions. ?Also, if you think some passages confused or inarticulate, by all means do not hesitate to point me to them. ?Thank you in advance. As I suppose that it was the size of the attachment which has led to the previous message being blocked, I'm going to try to send it in a separate message, and if that fails the file can be downloaded from http://dpworks.net/smilga/cl-json-version-0.4.dpatch.gz. Sincerely, ?- B. Smilga. From boris.smilga at gmail.com Mon Apr 6 10:19:15 2009 From: boris.smilga at gmail.com (Boris Smilga) Date: Mon, 6 Apr 2009 14:19:15 +0400 Subject: [cl-json-devel] cl-json 0.4 Message-ID: <42946560-CE15-4C06-B872-CBC74E070395@gmail.com> Good day to everyone. The bundle attached herewith contains patches to upgrade CL-JSON to the new version 0.4.0 which incorporates changes we have been discussing of late. The changes are rather wide-scale (the sheer size of the patches is 415k), almost every aspect of the implementation differs from what we had in version 0.3.1 and earlier. The most important points are: 1. The decoder now has the customization mechanism from decoder- vars.lisp, with imperative handlers and aggregate-scope variables. 2. The encoder provides a streaming API (borrowed from YASON). The standard encode methods are re-implemented using that API. 3. The CLOS decoder has undergone serious modifications to address the issue of degrading performance in some Lisp implementations. It shall be possible to use anonymous ?fluid? objects created by the decoder in a way which makes for a ?poor man's JavaScript?. The CLOS encoder does not emit prototype metadata anymore (unless specifically requested). 4. I have written a new pair of matching functions to convert between camel case and Lisp names. 5. There is an improved exception signalling mechanism. Some things which previously were handled by callbacks now use conditions / restarts. 6. JSON-BIND now uses the event-based decoder API with dynamic customization. This has the advantage that the user may choose the semantics with which to decode interior values. However, I had to forego the support for Lisp trees as arguments to JSON-BIND. 7. Some exported names were changed: Condition JSON-PARSE-ERROR ? JSON-SYNTAX-ERROR Macro WITH-LIST-DECODER-SEMANTICS ? WITH-DECODER-SIMPLE- LIST-SEMANTICS Macro WITH-CLOS-DECODER-SEMANTICS ? WITH-DECODER-SIMPLE- CLOS-SEMANTICS I have made sure the tests pass in Clozure CL, SBCL, CMUCL, CLisp, and ECL. There is a new User's Manual in doc/cl-json.html (I have also uploaded a copy to http://dpworks.net/smilga/cl-json.html). And here I would like to humbly ask someone on this mailing list who is a native speaker of English to proof-read it. Russian, my natural idiom, is quite unlike English in grammar, conventional style, and punctuation. Though I am more or less fluent in the latter, I never fully trust myself, especially with the choice of articles, ?which? / ?that?, and modal and causative constructions. Also, if you think some passages confused or inarticulate, by all means do not hesitate to point me to them. Thank you in advance. Sincerely, - B. Smilga. -------------- next part -------------- A non-text attachment was scrubbed... Name: version-0.4.dpatch.gz Type: application/x-gzip Size: 89158 bytes Desc: not available URL: From boris.smilga at gmail.com Mon Apr 13 17:23:04 2009 From: boris.smilga at gmail.com (Boris Smilga) Date: Mon, 13 Apr 2009 21:23:04 +0400 Subject: [cl-json-devel] cl-json 0.4, the patches Message-ID: <590f36270904131023x3ff35c7ep2e3370cd41e23d86@mail.gmail.com> Trying to send the patch bundle again. Please check the attachment. - B. Sm. -------------- next part -------------- A non-text attachment was scrubbed... Name: cl-json-version-0.4.dpatch.gz Type: application/x-gzip Size: 89158 bytes Desc: not available URL: From henrik at evahjelte.com Tue Apr 14 09:53:26 2009 From: henrik at evahjelte.com (Henrik Hjelte) Date: Tue, 14 Apr 2009 11:53:26 +0200 Subject: [cl-json-devel] cl-json 0.4 In-Reply-To: <42946560-CE15-4C06-B872-CBC74E070395@gmail.com> References: <42946560-CE15-4C06-B872-CBC74E070395@gmail.com> Message-ID: <50e8e4f60904140253q6d10df55g9785e598cfea3664@mail.gmail.com> Thanks Boris and Hans for all this, I have just started to look at the new patches and it looks very impressive. I have pushed all pathes to the darcs repo, and added a tag "Pre-0.4-bundle" before and a tag "0.4-bundle" after. One thing you might want to know before upgrading, when comparing the performance testcases the new version seems a bit slower on my sbcl (see below). I'll look more into this. Best wishes, Henrik Hjelte SBCL 1.0.21.34 Encoder: Pre-0.4-bundle: CL-USER> (5am:run! 'json-test::encoder-performance) Encoding 652 varying chars from memory 2000 times. Evaluation took: 0.160 seconds of real time 0.152009 seconds of total run time (0.144009 user, 0.008000 system) [ Run times consist of 0.008 seconds GC time, and 0.145 seconds non-GC time. ] 95.00% CPU 346,830,716 processor cycles 14,245,920 bytes consed 0.4-bundle: CL-USER> (5am:run! 'json-test::encoder-performance) Encoding 652 varying chars from memory 2000 times. Evaluation took: 0.143 seconds of real time 0.144009 seconds of total run time (0.136008 user, 0.008001 system) [ Run times consist of 0.020 seconds GC time, and 0.125 seconds non-GC time. ] 100.70% CPU 309,824,182 processor cycles 26,578,824 bytes consed Decoder: Pre-0.4-bundle: CL-USER> (5am:run! 'json-test::decoder-performance) Decoding 1387 varying chars from memory 1000 times. Evaluation took: 0.667 seconds of real time 0.652041 seconds of total run time (0.632039 user, 0.020002 system) [ Run times consist of 0.024 seconds GC time, and 0.629 seconds non-GC time. ] 97.75% CPU 1,440,436,365 processor cycles 80,807,408 bytes consed 0.4-bundle: CL-USER> (5am:run! 'json-test::decoder-performance) Decoding 1387 varying chars from memory 1000 times. Evaluation took: 2.139 seconds of real time 2.108132 seconds of total run time (1.628102 user, 0.480030 system) [ Run times consist of 0.032 seconds GC time, and 2.077 seconds non-GC time. ] 98.55% CPU 4,624,819,100 processor cycles 105,295,912 bytes consed From henrik at evahjelte.com Tue Apr 14 22:13:52 2009 From: henrik at evahjelte.com (Henrik Hjelte) Date: Wed, 15 Apr 2009 00:13:52 +0200 Subject: [cl-json-devel] Problem with latest sbcl Message-ID: <50e8e4f60904141513r52d8738u590057de41091a35@mail.gmail.com> Trying the latest patches on todays cvs version of sbcl, 1.0.27.9, totally breaks the testcases. Whereas the old version works. So be a bit careful to use the darcs version for a while in a stable environment. Otherwise, It seems there are lots of cool new features, and the documentation is great. I've put it online here: http://common-lisp.net/project/cl-json/cl-json.html Boris, one thing I am not sure about, the security implications of the clos decoder. How do you make sure that an evil user doesn't create a lispClass ticking-bomb in the lispPackage not-so-secret? Or do I read the docs to bad? Regards, Henrik From boris.smilga at gmail.com Wed Apr 15 16:55:07 2009 From: boris.smilga at gmail.com (Boris Smilga) Date: Wed, 15 Apr 2009 20:55:07 +0400 Subject: [cl-json-devel] cl-json 0.4 In-Reply-To: <50e8e4f60904140253q6d10df55g9785e598cfea3664@mail.gmail.com> References: <42946560-CE15-4C06-B872-CBC74E070395@gmail.com> <50e8e4f60904140253q6d10df55g9785e598cfea3664@mail.gmail.com> Message-ID: <590f36270904150955m4f68c676gaa1131b26836c45c@mail.gmail.com> On Tue, Apr 14, 2009 at 1:53 PM, Henrik Hjelte wrote: > One thing you might want to know before upgrading, when comparing the > performance testcases the new version seems a bit slower on my sbcl > (see below). To be sure, that's 200% deceleration, and it stays that way if the COUNT in the test is increased 10, 100, etc. times. ?A bit? seems like an understatement here. I'm afraid that's the price we pay for dynamic customization, as most of the accrued run time is used up (prima facie) by the handler invocation machinery?which is there for every little char! Methinks there is an evident way to optimize this, and I'm going to try it out (tomorrow if I have time). ? - B. Sm. From boris.smilga at gmail.com Wed Apr 15 17:40:44 2009 From: boris.smilga at gmail.com (Boris Smilga) Date: Wed, 15 Apr 2009 21:40:44 +0400 Subject: [cl-json-devel] Problem with latest sbcl In-Reply-To: <50e8e4f60904141513r52d8738u590057de41091a35@mail.gmail.com> References: <50e8e4f60904141513r52d8738u590057de41091a35@mail.gmail.com> Message-ID: <590f36270904151040v6be474c5v1750047f2dfa3bc1@mail.gmail.com> On Wed, Apr 15, 2009 at 2:13 AM, Henrik Hjelte wrote: > Trying the latest patches on todays cvs version of sbcl, 1.0.27.9, > totally breaks the testcases. Whereas the old version works. So be a > bit careful to use the darcs version for a while in a stable > environment. Uh... That could be SBCL bugs, couldn't it? To be earnest, I only use Lisp implementations from ports collections as a general rule, but I'll look into that. I would not be surprised at seeing ?s?o?m?e? tests fail, but if the breakdown is as massive as you imply, this is definitely an emergency case. > Boris, one thing I am not sure about, the security implications of the > clos decoder. How do you make sure that an evil user doesn't create a > lispClass ticking-bomb in the lispPackage not-so-secret? Or do I read > the docs to bad? No, your concerns are perfectly justified, the manual doesn't really address this issue. I should write that up. The simplest way to prevent undesired objects from being created, as far as I understand, is to define a prohibitive MAKE-OBJECT method specialized for the (name of) every dangerous class?or, if that better suits your policy, a general prohibitive method plus a permissive method for every class guaranteed safe. - B. Sm. From boris.smilga at gmail.com Thu Apr 16 14:31:57 2009 From: boris.smilga at gmail.com (Boris Smilga) Date: Thu, 16 Apr 2009 18:31:57 +0400 Subject: [cl-json-devel] Problem with latest sbcl In-Reply-To: <590f36270904151040v6be474c5v1750047f2dfa3bc1@mail.gmail.com> References: <50e8e4f60904141513r52d8738u590057de41091a35@mail.gmail.com> <590f36270904151040v6be474c5v1750047f2dfa3bc1@mail.gmail.com> Message-ID: <590f36270904160731o3761de0cv5604c12be5f51986@mail.gmail.com> On Wed, Apr 15, 2009 at 9:40 PM, Boris Smilga wrote: > On Wed, Apr 15, 2009 at 2:13 AM, Henrik Hjelte wrote: >> Trying the latest patches on todays cvs version of sbcl, 1.0.27.9, >> totally breaks the testcases. Whereas the old version works. So be a >> bit careful to use the darcs version for a while in a stable >> environment. > > Uh... That could be SBCL bugs, couldn't it? Not a bug, of course. SBCL just began to enforce the ANSI provision that the EXTENSION argument to VECTOR-PUSH-EXTEND must be a positive integer. All test failures but one were happening when VECTOR-PUSH-EXTEND was being passed 0 as that argument. That's fixed now, I'll post patches some time later. - B. Sm. From boris.smilga at gmail.com Tue Apr 21 09:29:14 2009 From: boris.smilga at gmail.com (Boris Smilga) Date: Tue, 21 Apr 2009 13:29:14 +0400 Subject: [cl-json-devel] cl-json 0.4 In-Reply-To: <590f36270904150955m4f68c676gaa1131b26836c45c@mail.gmail.com> References: <42946560-CE15-4C06-B872-CBC74E070395@gmail.com> <50e8e4f60904140253q6d10df55g9785e598cfea3664@mail.gmail.com> <590f36270904150955m4f68c676gaa1131b26836c45c@mail.gmail.com> Message-ID: <590f36270904210229u3f80eb3m1b82cb4ec15202e2@mail.gmail.com> On Wed, Apr 15, 2009 at 8:55 PM, Boris Smilga wrote: > On Tue, Apr 14, 2009 at 1:53 PM, Henrik Hjelte wrote: > >> One thing you might want to know before upgrading, when comparing the >> performance testcases the new version seems a bit slower on my sbcl >> (see below). > > To be sure, that's 200% deceleration, and it stays that way if the > COUNT in the test is increased 10, 100, etc. times. ??A bit? seems > like an understatement here. ?I'm afraid that's the price we pay for > dynamic customization, as most of the accrued run time is used up > (prima facie) by the handler invocation machinery?which is there for > every little char! ?Methinks there is an evident way to optimize this, > and I'm going to try it out (tomorrow if I have time). Guess who is out in the left field again... Bypassing the handler mechanism does not, per se, gain much (my first impression was evidently wrong); moreover, checking whether to use the standard or the handler-free track actually impairs the performance. However, I was able to pinpoint one factor which is responsible for about half of the performance regression (in the DECODER-PERFORMANCE test, the deceleration is down to ?80% from ?160% on Darwin/PPC, and to ?120% from ?190% on FreeBSD/i386). This factor was the use of vector accumulators in member handlers as opposed to list accumulators?the same thing that has caused test failures in SBCL 1.0.27. Ironically, it was intended as an optimization. (CLisp is the only implementation where it does work, and even in it the improvement is slight; in others, it is negative). The attached patch altogether discards vector accumulators. I'll look if more can be done to enhance the performance. - B. Sm. -------------- next part -------------- A non-text attachment was scrubbed... Name: vaccum.dpatch Type: application/octet-stream Size: 20852 bytes Desc: not available URL: From henrik at evahjelte.com Mon Apr 27 21:48:59 2009 From: henrik at evahjelte.com (Henrik Hjelte) Date: Mon, 27 Apr 2009 23:48:59 +0200 Subject: [cl-json-devel] cl-json 0.4 In-Reply-To: <590f36270904210229u3f80eb3m1b82cb4ec15202e2@mail.gmail.com> References: <42946560-CE15-4C06-B872-CBC74E070395@gmail.com> <50e8e4f60904140253q6d10df55g9785e598cfea3664@mail.gmail.com> <590f36270904150955m4f68c676gaa1131b26836c45c@mail.gmail.com> <590f36270904210229u3f80eb3m1b82cb4ec15202e2@mail.gmail.com> Message-ID: <50e8e4f60904271448q6b9d1fcdjdd8ec75296ba63f4@mail.gmail.com> Hello, I have now tried Boris vaccum-patch on sbcl 1.0.27.9, and it works (almost) perfectly. There was one failing testcase on the test json-number, but I believe it is because of a previous bug in sbcl that is now fixed.bugfix in sbcl. Boris, I have pushed a patch, you might want to take a look at it to see if I understood it right. Now all tests run OK. And the performance is much improved, but still three times slower than before on decoding. But maybe that is a price worth paying? What do people think, should the current darcs version be released as 0.4, or should we wait on something? Cheers, Henrik Encoding 652 varying chars from memory 2000 times. Evaluation took: 0.163 seconds of real time 0.160010 seconds of total run time (0.124007 user, 0.036003 system) [ Run times consist of 0.032 seconds GC time, and 0.129 seconds non-GC time. ] 98.16% CPU 352,256,740 processor cycles 26,355,248 bytes consed ....................................................Decoding 1387 varying chars from memory 1000 times. Evaluation took: 1.875 seconds of real time 1.860117 seconds of total run time (1.320083 user, 0.540034 system) [ Run times consist of 0.104 seconds GC time, and 1.757 seconds non-GC time. ] 99.20% CPU 4,051,873,881 processor cycles 70,342,416 bytes consed From boris.smilga at gmail.com Wed Apr 29 01:28:23 2009 From: boris.smilga at gmail.com (Boris Smilga) Date: Wed, 29 Apr 2009 05:28:23 +0400 Subject: [cl-json-devel] cl-json 0.4 In-Reply-To: <50e8e4f60904271448q6b9d1fcdjdd8ec75296ba63f4@mail.gmail.com> References: <42946560-CE15-4C06-B872-CBC74E070395@gmail.com> <50e8e4f60904140253q6d10df55g9785e598cfea3664@mail.gmail.com> <590f36270904150955m4f68c676gaa1131b26836c45c@mail.gmail.com> <590f36270904210229u3f80eb3m1b82cb4ec15202e2@mail.gmail.com> <50e8e4f60904271448q6b9d1fcdjdd8ec75296ba63f4@mail.gmail.com> Message-ID: <609F04D1-9D2B-4419-8900-848B70BA21CB@gmail.com> On 28 Apr 2009, at 01:48, Henrik Hjelte wrote: > Hello, I have now tried Boris vaccum-patch on sbcl 1.0.27.9, and it > works (almost) perfectly. There was one failing testcase on the test > json-number, but I believe it is because of a previous bug in sbcl > that is now fixed.bugfix in sbcl. Boris, I have pushed a patch, you > might want to take a look at it to see if I understood it right. [...] It is more complex than just an old bug having been fixed: if I am not mistaken, that particular piece of behaviour in SBCL showed a difference between Darwin/PPC and FreeBSD/Intel, rather than a difference between versions. I am now trying to upgrade SBCL from 1.0.18 to 1.0.23 on the former platform, and, as soon as it is up, I will post a more informed follow-up on the issue. > And the performance is much improved, but still three times slower > than before on decoding. But maybe that is a price worth paying? Well, I have worked on the two versions with a profiler, and I think there is still much room for improvement. If you look into the table attached (which I have handcrafted from the reports of the deterministic profilers on 0.3 and on 0.4 ? the design had changed a lot, so I had to juxtapose ?blocks? of roughly comparable functionality), you would be able to see two most outstanding foci of regression: the string reader and the camel-case converter. The latter is actually fuller and more complex in 0.4, and probably deserves its 1 sec. extra runtime (I will be more than happy to see someone devise a lighter analogue though). The regression in the former, however, is completely gratuitious. I cannot quite see, at this moment, what the cause is, but, by all means, this thing should be brought back into the bounds of decency. - B. Smilga. -------------- next part -------------- A non-text attachment was scrubbed... Name: decoder-regression.table Type: application/octet-stream Size: 2170 bytes Desc: not available URL: