From sky at viridian-project.de Tue Jun 16 17:38:57 2009 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 16 Jun 2009 19:38:57 +0200 (CEST) Subject: [cl-json-devel] (encode-json-to-string "") => "\"<\\/div>\"" ? Message-ID: <3cedd816b771f14de8968131f9480ce1.squirrel@mail.stardawn.org> Where do those two extra backslashes in front of the slash come from? Is this intended? Thanks, Leslie -- http://www.linkedin.com/in/polzer From henrik at evahjelte.com Tue Jun 16 18:41:12 2009 From: henrik at evahjelte.com (Henrik Hjelte) Date: Tue, 16 Jun 2009 20:41:12 +0200 Subject: [cl-json-devel] (encode-json-to-string "") => "\"<\\/div>\"" ? In-Reply-To: <3cedd816b771f14de8968131f9480ce1.squirrel@mail.stardawn.org> References: <3cedd816b771f14de8968131f9480ce1.squirrel@mail.stardawn.org> Message-ID: <50e8e4f60906161141v1a3d72eeg6b6abb6346ec671e@mail.gmail.com> On Tue, Jun 16, 2009 at 7:38 PM, Leslie P. Polzer wrote: > > Where do those two extra backslashes in front of the slash > come from? Is this intended? > > ?Thanks, > > ? ?Leslie Json allows / in strings, but if I am reading the diagram on json.org correctly prefers that it is prefixed by a backslash like this \/ . Probably prefered, because why would it be in the list of backslashed characters otherwise? See the mentioning about solidus in the string diagram. I don't really know why. And the first backslash is the lisp backslash for the backslash character. So it is valid JSON, and probably prefered to /. /Henrik From sky at viridian-project.de Tue Jun 16 19:12:52 2009 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 16 Jun 2009 21:12:52 +0200 (CEST) Subject: [cl-json-devel] (encode-json-to-string "") => "\"<\\/div>\"" ? In-Reply-To: <50e8e4f60906161141v1a3d72eeg6b6abb6346ec671e@mail.gmail.com> References: <3cedd816b771f14de8968131f9480ce1.squirrel@mail.stardawn.org> <50e8e4f60906161141v1a3d72eeg6b6abb6346ec671e@mail.gmail.com> Message-ID: Henrik Hjelte wrote: > Json allows / in strings, but if I am reading the diagram on json.org > correctly prefers that it is prefixed by a backslash > like this \/ . Probably prefered, because why would it be in the list > of backslashed characters otherwise? See the mentioning about solidus > in the string diagram. I don't really know why. Curious. Thanks for explaining, I will change the Weblocks tests to match this new output then. Leslie -- http://www.linkedin.com/in/polzer From boris.smilga at gmail.com Wed Jun 17 19:17:10 2009 From: boris.smilga at gmail.com (Boris Smilga) Date: Wed, 17 Jun 2009 23:17:10 +0400 Subject: [cl-json-devel] (encode-json-to-string "") => "\"<\\/div>\"" ? In-Reply-To: <50e8e4f60906161141v1a3d72eeg6b6abb6346ec671e@mail.gmail.com> References: <3cedd816b771f14de8968131f9480ce1.squirrel@mail.stardawn.org> <50e8e4f60906161141v1a3d72eeg6b6abb6346ec671e@mail.gmail.com> Message-ID: <590f36270906171217t4929d1ftc2df80ee00c3f869@mail.gmail.com> On Tue, Jun 16, 2009 at 10:41 PM, Henrik Hjelte wrote: > On Tue, Jun 16, 2009 at 7:38 PM, Leslie P. > Polzer wrote: >> >> Where do those two extra backslashes in front of the slash >> come from? Is this intended? > > Json allows / in strings, but if I am reading the diagram on json.org > correctly prefers that it is prefixed by a backslash > like this \/ . Probably prefered, because why would it be in the list > of backslashed characters otherwise? See the mentioning about solidus > in the string diagram. I don't really know why. The most plausible explanation is that a JavaScript embedded in an HTML page can not include the sequence of characters LESS-THAN SIGN, SOLIDUS: per HTML 4.01 ? 18.2.1, the contents of a SCRIPT element is CDATA which means (? 6.2) that in the following text no markup is interpreted?up to the first occurence of ? Hello all, CL-JSON does not allow the user to customize the means used to decode the keys for object literals. It may be important to avoid interning in a web setting, for example, since interns of many unique symbols could potentially use a lot of memory. An attack could exploit this by submitting something that is passed through cl-json that has many very large, unique symbols. There used to be a way to get around this with the factory method customization, but the current library does not include a means of changing the decoding behavior for a key to avoid interning it. Unless I am missing something, could this functionality be added? Red -------------- next part -------------- An HTML attachment was scrubbed... URL: From boris.smilga at gmail.com Thu Jun 25 20:29:58 2009 From: boris.smilga at gmail.com (Boris Smilga) Date: Fri, 26 Jun 2009 00:29:58 +0400 Subject: [cl-json-devel] Not interning object keys In-Reply-To: References: Message-ID: <68315BE4-8174-47F3-AC85-AEF92609DB7E@gmail.com> On 25 Jun 2009, at 22:27, Red Daly wrote: > CL-JSON does not allow the user to customize the means used to > decode the keys for object literals. It may be important to avoid > interning in a web setting, for example, since interns of many > unique symbols could potentially use a lot of memory. An attack > could exploit this by submitting something that is passed through > cl-json that has many very large, unique symbols. Indeed, thank you for pointing this out. > There used to be a way to get around this with the factory method > customization, but the current library does not include a means of > changing the decoding behavior for a key to avoid interning it. [...] It is the same thing in the new version, except that customization works in a somewhat different way. Broadly speaking, you have to redefine the way a level of JSON Object structure is accumulated to form the corresponding Lisp structure. E. g., in the following example new (KEY . VALUE) pairs are clipped onto the end of a list accumulator, to form an alist: (defvar *accumulator* nil) (defvar *accumulator-last* nil) (defun init-accumulator () (setq *accumulator* (cons nil nil) *accumulator-last* *accumulator*)) (defun collect-key (key) (setq *accumulator-last* (setf (cdr *accumulator-last*) (cons (cons key nil) nil)))) (defun collect-value (value) (setf (cdar *accumulator-last*) value)) (defun accumulator-get-value () (cdr *accumulator*)) (json:bind-custom-vars (:beginning-of-object #'init-accumulator :object-key #'collect-key :object-value #'collect-value :end-of-object #'accumulator-get-value :object-scope '(*accumulator* *accumulator-last*)) (json:decode-json-from-string "{\"foo\": [{\"bar\": \"xyzzy\"}, {\"baz\": true}], \"quux\": 123}")) => (("foo" (("bar" . "xyzzy")) (("baz" . T))) ("quux" . 123)) > [...] Unless I am missing something, could this functionality be > added? No problem, but could you maybe provide a sample of phantasy code which showed how this kind of customization interface should look from the user's side? With that, we'll be more able to meet your expectations. (I cannot, of course, guarantee that your interface shall be reproduced 100% exactly, rather it'll serve as a guideline.) Yours, - B. Smilga. From reddaly at gmail.com Thu Jun 25 21:09:21 2009 From: reddaly at gmail.com (Red Daly) Date: Thu, 25 Jun 2009 14:09:21 -0700 Subject: [cl-json-devel] Not interning object keys In-Reply-To: <68315BE4-8174-47F3-AC85-AEF92609DB7E@gmail.com> References: <68315BE4-8174-47F3-AC85-AEF92609DB7E@gmail.com> Message-ID: I created this patch before I got your email. What my usage is now the following: (let ((json:*json-identifier-name-to-lisp* 'identity) (json:*json-symbolize-lisp-key* 'identity)) (json:decode-json-from-string json)) The patch is attached: I have not taken the time to read through how the accumulator works. This approach modifies the default behavior in a more fine-grained way. On Thu, Jun 25, 2009 at 1:29 PM, Boris Smilga wrote: > On 25 Jun 2009, at 22:27, Red Daly wrote: > > CL-JSON does not allow the user to customize the means used to decode the >> keys for object literals. It may be important to avoid interning in a web >> setting, for example, since interns of many unique symbols could potentially >> use a lot of memory. An attack could exploit this by submitting something >> that is passed through cl-json that has many very large, unique symbols. >> > > Indeed, thank you for pointing this out. > > There used to be a way to get around this with the factory method >> customization, but the current library does not include a means of changing >> the decoding behavior for a key to avoid interning it. [...] >> > > It is the same thing in the new version, except that customization works in > a somewhat different way. Broadly speaking, you have to redefine the way a > level of JSON Object structure is accumulated to form the corresponding Lisp > structure. E. g., in the following example new (KEY . VALUE) pairs are > clipped onto the end of a list accumulator, to form an alist: > > (defvar *accumulator* nil) > (defvar *accumulator-last* nil) > > (defun init-accumulator () > (setq *accumulator* (cons nil nil) > *accumulator-last* *accumulator*)) > > (defun collect-key (key) > (setq *accumulator-last* > (setf (cdr *accumulator-last*) > (cons (cons key nil) nil)))) > > (defun collect-value (value) > (setf (cdar *accumulator-last*) value)) > > (defun accumulator-get-value () > (cdr *accumulator*)) > > (json:bind-custom-vars > (:beginning-of-object #'init-accumulator > :object-key #'collect-key > :object-value #'collect-value > :end-of-object #'accumulator-get-value > :object-scope '(*accumulator* *accumulator-last*)) > (json:decode-json-from-string > "{\"foo\": [{\"bar\": \"xyzzy\"}, {\"baz\": true}], > \"quux\": 123}")) > > => (("foo" (("bar" . "xyzzy")) (("baz" . T))) ("quux" . 123)) Thanks for an example of a custom accumulator modification. Just out of curiosity, have you seen the accumulator paradigm crop up in other contexts? > > > [...] Unless I am missing something, could this functionality be added? >> > > > No problem, but could you maybe provide a sample of phantasy code which > showed how this kind of customization interface should look from the user's > side? With that, we'll be more able to meet your expectations. (I cannot, > of course, guarantee that your interface shall be reproduced 100% exactly, > rather it'll serve as a guideline.) It would be nice to have something like: (let ((json:*json-symbolize-lisp-key* 'identity)) ...) This is still a little confusing unless you are familiar with the *json-identifier-name-to-lisp* variable, in which case it makes a little more sense. > > > Yours, > - B. Smilga. > > -Red patch: diff -rN old-cl-json/src/common.lisp new-cl-json/src/common.lisp 94a95,98 > > (defvar *json-symbolize-lisp-key* 'json-intern > "Designator for a function which, during decoding, maps the *json-identifier-name-to-lisp* > -transformed key to the value it will have in the result object.") \ No newline at end of file diff -rN old-cl-json/src/decoder.lisp new-cl-json/src/decoder.lisp 474c474 < (let ((key (json-intern (funcall *json-identifier-name-to-lisp* key)))) --- > (let ((key (funcall *json-identifier-name-to-lisp* key))) 604,605c604,605 < (string (json-intern < (funcall *json-identifier-name-to-lisp* value))) --- > (string (funcall *json-symbolize-lisp-key* > (funcall *json-identifier-name-to-lisp* value))) 609c609 < collect (cons (json-intern key) value)))) --- > collect (cons (funcall *json-symbolize-lisp-key* key) value)))) diff -rN old-cl-json/src/package.lisp new-cl-json/src/package.lisp 16a17 > #:*json-symbolize-lisp-key* -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cl-json-keycust-patch Type: application/octet-stream Size: 1039 bytes Desc: not available URL: From henrik at evahjelte.com Fri Jun 26 21:39:18 2009 From: henrik at evahjelte.com (Henrik Hjelte) Date: Fri, 26 Jun 2009 23:39:18 +0200 Subject: [cl-json-devel] Not interning object keys In-Reply-To: References: <68315BE4-8174-47F3-AC85-AEF92609DB7E@gmail.com> Message-ID: <50e8e4f60906261439j683c3491m1d59e42922415eca@mail.gmail.com> On Thu, Jun 25, 2009 at 11:09 PM, Red Daly wrote: > I created this patch before I got your email.? What my usage is now the > following: > > ??? (let ((json:*json-identifier-name-to-lisp* 'identity) > ????????? (json:*json-symbolize-lisp-key* 'identity)) > ????? (json:decode-json-from-string json)) > > The patch is attached: I agree with the problem and solution. Thanks for the patch. I just want all code for features to have at least one testcase, so I'll apply it later when I have done one. Thanks, Henrik From boris.smilga at gmail.com Fri Jun 26 22:20:10 2009 From: boris.smilga at gmail.com (Boris Smilga) Date: Sat, 27 Jun 2009 02:20:10 +0400 Subject: [cl-json-devel] Not interning object keys In-Reply-To: <50e8e4f60906261439j683c3491m1d59e42922415eca@mail.gmail.com> References: <68315BE4-8174-47F3-AC85-AEF92609DB7E@gmail.com> <50e8e4f60906261439j683c3491m1d59e42922415eca@mail.gmail.com> Message-ID: <6BFE3EB4-9AF1-4324-B77E-861828D5B3DA@gmail.com> Just a minor cavil: does not *STRING-TO-KEY* (or maybe *IDENTIFIER- NAME-TO-KEY*) seem like a better name for this variable? Rationale: I would say that the -JSON- and -LISP- parts of the original *JSON-SYMBOLIZE-LISP-KEY* are not in place here because, unlike *JSON-IDENTIFIER-NAME-TO-LISP*, the function does not translate between JSON and Lisp conventions. Further, -SYMBOLIZE- is a very imperative form, and imperatives should, in my humble opinion, be limited to functions with side effects. What do you think? - B. Sm. From reddaly at gmail.com Tue Jun 30 15:41:42 2009 From: reddaly at gmail.com (Red Daly) Date: Tue, 30 Jun 2009 08:41:42 -0700 Subject: [cl-json-devel] Not interning object keys In-Reply-To: <6BFE3EB4-9AF1-4324-B77E-861828D5B3DA@gmail.com> References: <68315BE4-8174-47F3-AC85-AEF92609DB7E@gmail.com> <50e8e4f60906261439j683c3491m1d59e42922415eca@mail.gmail.com> <6BFE3EB4-9AF1-4324-B77E-861828D5B3DA@gmail.com> Message-ID: On Fri, Jun 26, 2009 at 3:20 PM, Boris Smilga wrote: > Just a minor cavil: does not *STRING-TO-KEY* (or maybe > *IDENTIFIER-NAME-TO-KEY*) seem like a better name for this variable? I would prefer *IDENTIFIER-NAME-TO-KEY* since it gives more context about what the function is used for. *STRING-TO-KEY* is confusing since it's unclear where the "string" is coming from. > > > Rationale: I would say that the -JSON- and -LISP- parts of the original > *JSON-SYMBOLIZE-LISP-KEY* are not in place here because, unlike > *JSON-IDENTIFIER-NAME-TO-LISP*, the function does not translate between JSON > and Lisp conventions. Further, -SYMBOLIZE- is a very imperative form, and > imperatives should, in my humble opinion, be limited to functions with side > effects. > > What do you think? > > - B. Sm. > -Red -------------- next part -------------- An HTML attachment was scrubbed... URL: