From henrik at evahjelte.com Thu Dec 3 21:31:48 2009 From: henrik at evahjelte.com (Henrik Hjelte) Date: Thu, 3 Dec 2009 22:31:48 +0100 Subject: [cl-json-devel] New encoder Message-ID: <50e8e4f60912031331u4079c261j3ddd5a80c9bd1c07@mail.gmail.com> I have added a new alternative encoder to the cl-json darcs repo. I call it the "explicit" (sexp) encoder and now call the old/other (sexp) decoder the "guessing" encoder. The old "guessing" encoder is still the default, so this should not break any code. Also there is a third "streaming" encoder. In other words, flexibility. The "streaming" encoder is a way to format json to a stream. This gives you detailed control, but some of us prefer an intermediate s-exp format. The old "guessing" encoder tries to make a reasonable json representation of a lisp s-expression. For example a list becomes a json array. An a-list becomes a json object. An empty list becomes an empty json array. But some might want it to be json null. Or json false. Or an empty json literal object. With the "explicit" encoder, you give directives in the s-expression explaining exactly what json you want. This might give you more control. Also new is that you can use p-lists to output objects. Also, you can include a pre-formated json string if you wish. I have not updated the documentation, because it was autogenerated by Boris Smilga, and I don't know how to do it. Boris, can you help with this or the code to your doc-maker? There are some testcases and doc-strings explaining the use in the code, I selected one of each and put below in the mail. Also, I have found a bug with json-bind (not fixed, but I have added a testcase for it in case someone want to give it a try). Note that there is no corresponding decoder yet, simply because I have not had any egoistic reasons to add it. But it should be easy if someone want an exercise. Hope someone has a use for this, Henrik "Write the JSON representation of the list S to STREAM (or to *JSON-OUTPUT*), using one of the two rules specified by first calling USE-GUESSING-ENCODER or USE-EXPLICIT-ENCODER. The guessing encoder: If S is a list encode S as a JSON Array, if S is a dotted list encode it as an Object (per ENCODE-JSON-ALIST). The explicit decoder: If S is a list, the first symbol defines the encoding: If (car S) is 'TRUE return a JSON true value. If (car S) is 'FALSE return a JSON false value. If (car S) is 'NULL return a JSON null value. If (car S) is 'JSON princ the strings in (cdr s) to stream If (car S) is 'LIST or 'ARRAY encode (cdr S) as a a JSON Array. If (car S) is 'OBJECT encode (cdr S) as A JSON Object, interpreting (cdr S) either as an A-LIST or a P-LIST." (test explicit-encoder-complex-objects (let ((sample-1-alists `(:object (:method . some-function) (:id . 1) (:params . (:list (:object (:id . bar-id) (:name . bar)) (:true) (:list (:object (:name . a) (:id . b))) (:list (:object (:name . foo) (:id . foo-id))) )))) (sample-2-plists `(:object :method some-function :id 1 :params (:list (:json "{\"id\":\"barId\",\"name\":\"bar\"}") (:true) (:list (:object "name" a :id b)) (:list (:object :name foo :id foo-id)) ))) (correct-json " {\"method\":\"someFunction\", \"id\":1,\"params\": [{\"id\":\"barId\",\"name\":\"bar\"}, true, [{\"name\":\"a\",\"id\":\"b\"}], [{\"name\":\"foo\",\"id\":\"fooId\"}]]}") exact-json sample-1-json sample-2-json) (setf sample-1-json (with-explicit-encoder (encode-json-to-string sample-1-alists))) (setf sample-2-json (with-explicit-encoder (encode-json-to-string sample-2-plists))) (is (string= sample-1-json sample-2-json)) (setf exact-json (remove #\Newline (remove #\Space correct-json :test #'char=) :test #'char=)) (is (string= sample-1-json exact-json)))) From rpgoldman at sift.info Wed Dec 30 15:37:30 2009 From: rpgoldman at sift.info (Robert Goldman) Date: Wed, 30 Dec 2009 09:37:30 -0600 Subject: [cl-json-devel] Question about JSON-RPC Message-ID: <4B3B73BA.3080108@sift.info> I am setting up a JSON-RPC server over sockets, rather than HTTP, and have a couple of questions about json-rpc.lisp: 1. Am I correct in thinking that INVOKE-JSON does not handle notifications? It looks like it will always try to return a result. Question: would it be reasonable to modify INVOKE-JSON to handle notifications by just executing the function and /not/ returning a result? 2. My socket server must parse JSON messages and decide what type they are before it can call invoke-json. This suggests refactoring invoke-json into two stages: a. the part that calls the decoder and pulls out the three slots b. an inside function that accepts method, params, id, and does the actual invoking. I was thinking of submitting a patch along those lines. Please let me know if that seems wrong. thanks, r From henrik at evahjelte.com Wed Dec 30 18:50:48 2009 From: henrik at evahjelte.com (Henrik Hjelte) Date: Wed, 30 Dec 2009 19:50:48 +0100 Subject: [cl-json-devel] Question about JSON-RPC In-Reply-To: <4B3B73BA.3080108@sift.info> References: <4B3B73BA.3080108@sift.info> Message-ID: <50e8e4f60912301050pb85693cy28aff844c5c634d8@mail.gmail.com> On Wed, Dec 30, 2009 at 4:37 PM, Robert Goldman wrote: > I am setting up a JSON-RPC server over sockets, rather than HTTP, and > have a couple of questions about json-rpc.lisp: > > 1. ?Am I correct in thinking that INVOKE-JSON does not handle > notifications? ?It looks like it will always try to return a result. > Question: ?would it be reasonable to modify INVOKE-JSON to handle > notifications by just executing the function and /not/ returning a result? Yes, but how would the API change. An idea is to let the caller decide if he wants return values, maybe by exposing two functions. > > 2. ?My socket server must parse JSON messages and decide what type they > are before it can call invoke-json. ?This suggests refactoring > invoke-json into two stages: > > ?a. ?the part that calls the decoder and pulls out the three slots > > ?b. ?an inside function that accepts method, params, id, and does the > actual invoking. > > I was thinking of submitting a patch along those lines. ?Please let me > know if that seems wrong. I agree with the problem. Also, I think a problem with the json-rpc implementation is that the decoder-encoder is harded to be the "guessing" encoder (and decoder). I have had an idea for how to make an new type of parser that could fix this.?It is quite easy to do, but I don't have enough egoistic motives for coding it myself yet. I wrote it down below however, maybe you or someone else wants to go this way? But if you have another solution for the json-rpc problem you are facing I'll accept patches too, best against the latest darcs version. /Henrik Idea for a new json-parser: Rationale: --------- The current parser only parses a whole json document (json-object) completely. If the json-document is big and you are only interested in a part of it this is very inefficient. Also, maybe you want to use one type of parser for a part of the big json-document, another parser for another part and no parser at all for another part. Sketch: ------ A new function parses the json-document into a DOM(document object model) like object. Only the minimal info is parsed in each step, you might want to say that the dom-nodes are read lazily. Lets call this function "scan" (just an idea), and it takes a stream as input. Assume the stream now contains a json-object. The minimal API we might want is a way to query the json-object with a key and get the value (as an unparsed string). We could start reading from the stream, pass the opening curly bracket and some whitespace, read the first key (parse it to a string). If it matches the key we expect, we can pass the trailing colon and then return the stream to the caller. Now the caller could parse the value in any way he wants from the stream. If the first key didn't match what we looked for, we now need to advance the stream past the value. But we don't really need to "parse it". Assume the value is a complex nested JSON object. Since JSON is so nicely uniform designed, even in this case we only need to read the stream as fast as we can and just count the numbers of opening and closing brackets. We don't need to parse the actual contents. Almost at least. We will need to keep track of backslashed characters and also ignore curly brackets inside strings. Fortunately skipping strings is easy since they start and end with doublequoutes. Advancing past arrays is done in the same way, just by counting brackets and skipping strings. Advancing past strings is even easier. Advancing past numbers or the keywords "true", "false" or "null" is done by skipping whitespace, then reading anything up to whitespace, a comma or closing curly bracket. I think that while we are scanning the stream we should collect the characters in a string for later access. Because in most cases we want to get the first level of the whole json-document, then do more processing with some of the values. A minimal API could be like this: scan-opening-bracket(stream): Skip and opening bracket so we can start using scan-key. scan-key(stream): returns a string value for the key part of a JSON-object being read from stream. Also advances the stream past the trailing whitespace, colon, whitespace. Returns nil when we are at the closing bracket (no more keys). An option could be :get-whitespace and the function would then also return the leading and trailing whitespace as second and third return values. (If you want to build up an "exact" copy of the input for some reason). scan-value(stream): returns a string value for the value. scan-to-next-key(stream). To be called after scan-value. passes whitespace, comma and whitespace to the next key. Returns t. If there is no next key(we are at the closing curly-bracket), returns nil. Now, this small API should be sufficient to build a little dom-like parser on top of. An easy version could be a function that reads all keys and unparsed values and preserves them in hash-table like structure. From rpgoldman at sift.info Wed Dec 30 21:15:03 2009 From: rpgoldman at sift.info (Robert Goldman) Date: Wed, 30 Dec 2009 15:15:03 -0600 Subject: [cl-json-devel] Question about JSON-RPC In-Reply-To: <50e8e4f60912301050pb85693cy28aff844c5c634d8@mail.gmail.com> References: <4B3B73BA.3080108@sift.info> <50e8e4f60912301050pb85693cy28aff844c5c634d8@mail.gmail.com> Message-ID: <4B3BC2D7.7050704@sift.info> On 12/30/09 Dec 30 -12:50 PM, Henrik Hjelte wrote: > On Wed, Dec 30, 2009 at 4:37 PM, Robert Goldman wrote: >> I am setting up a JSON-RPC server over sockets, rather than HTTP, and >> have a couple of questions about json-rpc.lisp: >> >> 1. Am I correct in thinking that INVOKE-JSON does not handle >> notifications? It looks like it will always try to return a result. >> Question: would it be reasonable to modify INVOKE-JSON to handle >> notifications by just executing the function and /not/ returning a result? > > Yes, but how would the API change. > An idea is to let the caller decide if he wants return values, maybe > by exposing two functions. If I understand the CL-JSON spec correctly, the caller decides that s/he doesn't want return values by setting the ID field of the request to null. My modified invoke-json function checks for a null id field and returns nothing if it finds that. > >> >> 2. My socket server must parse JSON messages and decide what type they >> are before it can call invoke-json. This suggests refactoring >> invoke-json into two stages: >> >> a. the part that calls the decoder and pulls out the three slots >> >> b. an inside function that accepts method, params, id, and does the >> actual invoking. >> >> I was thinking of submitting a patch along those lines. Please let me >> know if that seems wrong. > > I agree with the problem. OK, if this seems like a reasonable refactoring, I will grapple with darcs and try to get it to yield me a patch to send you. Best, R From rpgoldman at sift.info Wed Dec 30 21:26:02 2009 From: rpgoldman at sift.info (Robert Goldman) Date: Wed, 30 Dec 2009 15:26:02 -0600 Subject: [cl-json-devel] Question about JSON-RPC In-Reply-To: <4B3BC2D7.7050704@sift.info> References: <4B3B73BA.3080108@sift.info> <50e8e4f60912301050pb85693cy28aff844c5c634d8@mail.gmail.com> <4B3BC2D7.7050704@sift.info> Message-ID: <4B3BC56A.8000708@sift.info> Here is the patch (if I've successfully interacted with darcs) that gives the refactoring of invoke-json. I have used this to build a CL JSON-RPC system that is able to interact in a peer-to-peer way. That is, you can open CL connections from which you can either respond to JSON-RPC requests, or issue them. Unfortunately, in its current form this isn't really suitable for distribution because I have hacked it together using ACL-specific multiprocessing and socket extensions. I do have some emulation code for SBCL, and if I have cause (or if there's any interest), I might be able to bash together something that would at least be portable to SBCL. Best, r -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: notifications-patch-bundle URL: From rpgoldman at sift.info Wed Dec 30 21:28:15 2009 From: rpgoldman at sift.info (Robert Goldman) Date: Wed, 30 Dec 2009 15:28:15 -0600 Subject: [cl-json-devel] Another json-rpc question Message-ID: <4B3BC5EF.2020005@sift.info> I was testing my error handling code and had some trouble with the restarts in invoke-rpc. A number of these restarts seem to take an error-object as an argument. But when I tested my own CL-JSON methods I couldn't figure out how to get these error-objects into the restart. How is that to be done? thanks, r From rpgoldman at sift.info Thu Dec 31 22:17:08 2009 From: rpgoldman at sift.info (Robert Goldman) Date: Thu, 31 Dec 2009 16:17:08 -0600 Subject: [cl-json-devel] Question about NIL translation Message-ID: <4B3D22E4.4020107@sift.info> Continuing to work on my CL-JSON based JSON-RPC server, I am concerned about the fact that we translate NIL -> null. If a CL function returns NIL, and the return value is simply encoded into the JSON return package, then we get a message that /should/ indicate an error, according to the JSON-RPC server. In fact, if our CL function is invoked by JSON-RPC, then we really should have NIL either encoded as [] or as false, but /never/ as null. Any ideas about how we should modify invoke-rpc-parsed to fix this? What is the right way to tell CL-JSON to encode a false? Thanks, R