From psilord at cs.wisc.edu Thu Apr 14 05:59:44 2011 From: psilord at cs.wisc.edu (Peter Keller) Date: Thu, 14 Apr 2011 00:59:44 -0500 Subject: [xcvb-devel] run-program/process-output-stream Message-ID: <20110414055944.GA8298@cs.wisc.edu> Hello, Suppose I call this function like this: (run-program/process-output-stream '("echo" "string") 'slurp-stream-lines) I get back this list: ("string") ;; notice no newline. Now suppose I call it like this: (run-program/process-output-stream '("echo" "string") 'slurp-stream-string) I would then get back: "string " ^ Notice the newline Is this the expected behavior for this function wrt these output-processors? Thank you. -pete From fahree at gmail.com Thu Apr 14 14:30:53 2011 From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=) Date: Thu, 14 Apr 2011 10:30:53 -0400 Subject: [xcvb-devel] run-program/process-output-stream In-Reply-To: <20110414055944.GA8298@cs.wisc.edu> References: <20110414055944.GA8298@cs.wisc.edu> Message-ID: On 14 April 2011 01:59, Peter Keller wrote: > (run-program/process-output-stream '("echo" "string") 'slurp-stream-lines) > > I get back this list: > ("string") ;; notice no newline. > > Now suppose I call it like this: > > (run-program/process-output-stream '("echo" "string") 'slurp-stream-string) > > I would then get back: > > "string > " > ? ? ? ^ Notice the newline > > Is this the expected behavior for this function wrt these output-processors? > Yes. In one case, you get the entire output in a single string that you may process as you wish, in the other you use read-line, which strips newlines for you. [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Always design a thing by considering it in its next larger context ? a chair in a room, a room in a house, a house in an environment, an environment in a city plan. ? Eliel Saarinen From psilord at cs.wisc.edu Thu Apr 14 15:14:08 2011 From: psilord at cs.wisc.edu (Peter Keller) Date: Thu, 14 Apr 2011 10:14:08 -0500 Subject: [xcvb-devel] run-program/process-output-stream In-Reply-To: References: <20110414055944.GA8298@cs.wisc.edu> Message-ID: <20110414151408.GB5108@cs.wisc.edu> On Thu, Apr 14, 2011 at 10:30:53AM -0400, Far? wrote: > On 14 April 2011 01:59, Peter Keller wrote: > > (run-program/process-output-stream '("echo" "string") 'slurp-stream-lines) > > > > I get back this list: > > ("string") ;; notice no newline. > > > > Now suppose I call it like this: > > > > (run-program/process-output-stream '("echo" "string") 'slurp-stream-string) > > > > I would then get back: > > > > "string > > " > > ? ? ? ^ Notice the newline > > > > Is this the expected behavior for this function wrt these output-processors? > > > Yes. In one case, you get the entire output in a single string that > you may process as you wish, in the other you use read-line, which > strips newlines for you. Ah, I was just curious, because in perl, this code: @foo = `cat file`; (which is similar in concept to run-program/process-output-lines with the 'slurp-stream-lines output processor) preserves the newline for each line. I'm not advocating change, but I was surprised by the newline difference between the two stream processors. Also, would you prefer that I added docstrings for how to use these functions? Or just commented above them their use/edge cases and whatnot? BTW, I got hu.dwim.stefil working (and the required externals installed in the same manner as the others) and have started writing a basic unit-test suite for run-program/*. I need to figure out how to test for signaled conditions and other things too, but I'm making progress. Later, -pete From fahree at gmail.com Thu Apr 14 15:31:08 2011 From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=) Date: Thu, 14 Apr 2011 11:31:08 -0400 Subject: [xcvb-devel] run-program/process-output-stream In-Reply-To: <20110414151408.GB5108@cs.wisc.edu> References: <20110414055944.GA8298@cs.wisc.edu> <20110414151408.GB5108@cs.wisc.edu> Message-ID: > Ah, I was just curious, because in perl, this code: > > @foo = `cat file`; > > (which is similar in concept to run-program/process-output-lines with the > 'slurp-stream-lines output processor) preserves the newline for each line. > I am aware of that. I wanted some small code use trivial Lisp functionality, so I use read-line that behaves like this. > I'm not advocating change, but I was surprised by the newline difference > between the two stream processors. > > Also, would you prefer that I added docstrings for how to use these > functions? Or just commented above them their use/edge cases and whatnot? > This indeed deserves to be documented, in a comment if not in a docstring. I have no preference. As I realize it has to deal with portability, I'm considering having master either depend on driver, or be merged into driver so that it remains a single file. What do you think? > BTW, I got hu.dwim.stefil working (and the required externals installed in > the same manner as the others) and have started writing a basic unit-test > suite for run-program/*. I need to figure out how to test for signaled > conditions and other things too, but I'm making progress. > Excellent. Note that under windows, things are subtly different. Sigh. Having to deal with PATHs under Windows is the promise of HELL. [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Long live consumership, which is individual and responsible choice! Down with citizenship, which is collective and irresponsible choice! From psilord at cs.wisc.edu Thu Apr 14 16:31:36 2011 From: psilord at cs.wisc.edu (Peter Keller) Date: Thu, 14 Apr 2011 11:31:36 -0500 Subject: [xcvb-devel] run-program/process-output-stream In-Reply-To: References: <20110414055944.GA8298@cs.wisc.edu> <20110414151408.GB5108@cs.wisc.edu> Message-ID: <20110414163136.GA6952@cs.wisc.edu> On Thu, Apr 14, 2011 at 11:31:08AM -0400, Far? wrote: > > Ah, I was just curious, because in perl, this code: > > > > @foo = `cat file`; > > > > (which is similar in concept to run-program/process-output-lines with the > > 'slurp-stream-lines output processor) preserves the newline for each line. > > > I am aware of that. I wanted some small code use trivial Lisp functionality, > so I use read-line that behaves like this. Ok. I figured as much. The behavior really isn't that surprising from a lisp-point of view. > > I'm not advocating change, but I was surprised by the newline difference > > between the two stream processors. > > > > Also, would you prefer that I added docstrings for how to use these > > functions? Or just commented above them their use/edge cases and whatnot? > > > This indeed deserves to be documented, in a comment if not in a > docstring. I have no preference. Ok. I'll do whatever feels right. > As I realize it has to deal with portability, I'm considering having > master either depend on driver, or be merged into driver so that it > remains a single file. What do you think? IMHO, I personally would make a subdirectory called xcvb/vendor and break up both driver.lisp and master.lisp into individual files representing their little areas. So: xcvb/vendor/run-program.lisp [holds run-program/*, slurp-*, other I/O stuff] xcvb/vendor/environment.lisp [holds getenv, setenv, do-find-symbol, ] xcvb/vendor/debugging.lisp [hold debugging, with-profiling, etc] xcvb/vendor/strings-and-escapes.lisp [escape-windows-command, escape-token, etc] xcvb/vendor/specials.lisp [all special variables] Then each portability thing can be thought about and abstracted properly without being cluttered. > > BTW, I got hu.dwim.stefil working (and the required externals installed in > > the same manner as the others) and have started writing a basic unit-test > > suite for run-program/*. I need to figure out how to test for signaled > > conditions and other things too, but I'm making progress. > > > Excellent. Note that under windows, things are subtly different. Sigh. Yeah. In dealing with the newline issue, I did this: ;; add a "newline" to a string (defmacro nl (str) `(format nil "~A~%" ,str)) And then here the skeleton of how I'd use it: Suppose there is a file containing the single string 'test\n': ;; a basic test, minus the deftest stuff. (let ((the-string )) (is (string= the-string (nl "test"))) Because then if a lisp implementation uses \n\r versus \n or some other horrible thing, it should "just work" in the string= comparison. Am I doing the correct thing in the above? Also, is #+unix and #+win32 available as conditional compilation features for all implementations we'd like to support? > Having to deal with PATHs under Windows is the promise of HELL. Maybe you can lift this interface: http://perldoc.perl.org/File/Spec.html It would be verbose, but you already have a virtual path representation anyway so maybe it wouldn't be so horrible. Later, -pete From fahree at gmail.com Fri Apr 15 04:24:11 2011 From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=) Date: Fri, 15 Apr 2011 00:24:11 -0400 Subject: [xcvb-devel] run-program/process-output-stream In-Reply-To: <20110414163136.GA6952@cs.wisc.edu> References: <20110414055944.GA8298@cs.wisc.edu> <20110414151408.GB5108@cs.wisc.edu> <20110414163136.GA6952@cs.wisc.edu> Message-ID: >> As I realize it has to deal with portability, I'm considering having >> master either depend on driver, or be merged into driver so that it >> remains a single file. What do you think? > > IMHO, I personally would make a subdirectory called xcvb/vendor and break > up both driver.lisp and master.lisp into individual files representing > their little areas. > Yes and no. Once again, for the driver as well as the master or asdf, the ability to load the whole shebang in one command is appreciable; and since they are the kind of things you load before anything else, that command must be load (or require, once it becomes standard), and not some fancy thing. OR, you could assume that ASDF will always be present (a somewhat depressing thought, since the xcvb master is meant to replace ASDF, but possibly realistic), and indeed define a system that you may subdivide as you want. > Yeah. In dealing with the newline issue, I did this: > > ;; add a "newline" to a string > (defmacro nl (str) `(format nil "~A~%" ,str)) > Sounds good. > And then here the skeleton of how I'd use it: > > Suppose there is a file containing the single string 'test\n': > > ;; a basic test, minus the deftest stuff. > (let ((the-string )) > ?(is (string= the-string (nl "test"))) > > Because then if a lisp implementation uses \n\r versus \n or some other > horrible thing, it should "just work" in the string= comparison. > > Am I doing the correct thing in the above? > Sounds right. > Also, is #+unix and #+win32 available as conditional compilation features for > all implementations we'd like to support? > As of late, ASDF does magic to achieve features :asdf-windows and :asdf-unix. We could do similarly in say xcvb-driver. >> Having to deal with PATHs under Windows is the promise of HELL. > Maybe you can lift this interface: > > http://perldoc.perl.org/File/Spec.html > I don't think it's relevant for xcvb-master. No, the problem would rather beto either detect some essential programs and use their full name. [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] "Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is a well-armed lamb contesting the vote." From psilord at cs.wisc.edu Fri Apr 15 05:43:58 2011 From: psilord at cs.wisc.edu (Peter Keller) Date: Fri, 15 Apr 2011 00:43:58 -0500 Subject: [xcvb-devel] run-program/process-output-stream In-Reply-To: References: <20110414055944.GA8298@cs.wisc.edu> <20110414151408.GB5108@cs.wisc.edu> <20110414163136.GA6952@cs.wisc.edu> Message-ID: <20110415054358.GA23411@cs.wisc.edu> On Fri, Apr 15, 2011 at 12:24:11AM -0400, Far? wrote: > >> As I realize it has to deal with portability, I'm considering having > >> master either depend on driver, or be merged into driver so that it > >> remains a single file. What do you think? > > > > IMHO, I personally would make a subdirectory called xcvb/vendor and break > > up both driver.lisp and master.lisp into individual files representing > > their little areas. > > > Yes and no. Once again, for the driver as well as the master or asdf, > the ability to load the whole shebang in one command is appreciable; > and since they are the kind of things you load before anything else, > that command must be load (or require, once it becomes standard), > and not some fancy thing. OR, you could assume that ASDF will always > be present (a somewhat depressing thought, since the xcvb master > is meant to replace ASDF, but possibly realistic), > and indeed define a system that you may subdivide as you want. Ah, so there is some other reason for a particular source layout instead of categorization for human understanding purposes. :) Given the above, I'd merge driver.lisp and master.lisp together into one file. It would be around 1100 lines (and likely more after time), but most of the functionality in those filea are pretty distinct from each other. I'd just separate the sections with comment blocks. > > Yeah. In dealing with the newline issue, I did this: > > > > ;; add a "newline" to a string > > (defmacro nl (str) `(format nil "~A~%" ,str)) > > > Sounds good. Come to think of it, the above defmacro can just be a defun, no need to do macro expansion... > As of late, ASDF does magic to achieve features > :asdf-windows and :asdf-unix. We could do similarly in say xcvb-driver. I was just curious because then certain unit tests could be made to be more unix or windows specific, for example: (defvar *cat* #+unix "/bin/cat" #+win32 "type") > >> Having to deal with PATHs under Windows is the promise of HELL. > > Maybe you can lift this interface: > > > > http://perldoc.perl.org/File/Spec.html > > > I don't think it's relevant for xcvb-master. > No, the problem would rather beto either detect > some essential programs and use their full name. Ok, I see what you mean. I barely know anything about windows, so I can't give much better information on that topic. Later, -pete From psilord at cs.wisc.edu Tue Apr 26 05:48:39 2011 From: psilord at cs.wisc.edu (Peter Keller) Date: Tue, 26 Apr 2011 00:48:39 -0500 Subject: [xcvb-devel] run-program unit test failures Message-ID: <20110426054839.GA14981@cs.wisc.edu> Hello, What is the intended usage of run-program/process-output-stream? What kind of arguments do you give it and how do they relate to your argument escape codes? Suppose unix, would these be valid programs: '("cp" "foo" "bar") '("echo" "foo bar") '("cat" "/etc/passwd" "|" "grep" "root") '("cat /etc/passwd | grep root") What if I had a file named '|'? '("cat" "|") ;; doesn't work. how do I escape that? Must I write the escape by hand? '("cat" "\|") ;; Like this? So far, I think I've found some bad edge cases with run-program/process-output-stream concerning special shell characters. But, I need to know what you intend before reporting (or fixing) the bugs or not. Later, -pete From fahree at gmail.com Tue Apr 26 18:17:11 2011 From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=) Date: Tue, 26 Apr 2011 14:17:11 -0400 Subject: [xcvb-devel] run-program unit test failures In-Reply-To: <20110426054839.GA14981@cs.wisc.edu> References: <20110426054839.GA14981@cs.wisc.edu> Message-ID: > What is the intended usage of run-program/process-output-stream? > If the command given to run-program/* is a list, then it directly encodes the values for executed program's argv, as passed to execve or spawn or equivalent. If it's a string, then it's a command passed to the shell or command interpreter, as in "/bin/sh" "-c" "command..." or (under Windows) "CMD /C command...". > What kind of arguments do you give it and how do they relate to your > argument escape codes? Suppose unix, would these be valid programs: > > '("cp" "foo" "bar") > '("echo" "foo bar") Should work fine. > '("cat" "/etc/passwd" "|" "grep" "root") Won't work, unless you have files named "|" grep and root. Maybe you want "cat /etc/passwd | grep root" which will be passed to the interpreter. > '("cat /etc/passwd | grep root") Even worse. Lose the list. > What if I had a file named '|'? > > '("cat" "|") ;; doesn't work. > Yes it should. > how do I escape that? Must I write the escape by hand? > > '("cat" "\|") ;; Like this? > Should be the same. > So far, I think I've found some bad edge cases with > run-program/process-output-stream concerning special shell > characters. But, I need to know what you intend before reporting (or > fixing) the bugs or not. > I think it's working fine, at least under SBCL. Definitely requires more documentation, though. Test cases also make for great documentation. [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The early bird gets the worm, but the second mouse gets the cheese. ? Stephen Pinker From psilord at cs.wisc.edu Tue Apr 26 18:45:27 2011 From: psilord at cs.wisc.edu (Peter Keller) Date: Tue, 26 Apr 2011 13:45:27 -0500 Subject: [xcvb-devel] run-program unit test failures In-Reply-To: References: <20110426054839.GA14981@cs.wisc.edu> Message-ID: <20110426184527.GA3154@cs.wisc.edu> On Tue, Apr 26, 2011 at 02:17:11PM -0400, Far? wrote: > > What is the intended usage of run-program/process-output-stream? > > > If the command given to run-program/* is a list, > then it directly encodes the values for executed program's argv, > as passed to execve or spawn or equivalent. > > If it's a string, then it's a command passed to the shell > or command interpreter, as in "/bin/sh" "-c" "command..." > or (under Windows) "CMD /C command...". > Definitely requires more documentation, though. > > Test cases also make for great documentation. Ok, I'll write some more test cases and document the function with some doctrings or something. Thanks for the pointer with the intention of the interface! That better clears up how I need to test it. Later, -pete. From psilord at cs.wisc.edu Thu Apr 28 04:03:17 2011 From: psilord at cs.wisc.edu (Peter Keller) Date: Wed, 27 Apr 2011 23:03:17 -0500 Subject: [xcvb-devel] race with default object directory Message-ID: <20110428040317.GA24633@cs.wisc.edu> Hello, While testing out the default object directory changes we made, I ran into a race condition which exposed vague semantics. Suppose we have this in a Makefile and we run it to produce the executable. Notice I'm using the shiny new feature of the default directory and the show-settings command. XCVB_OBJECT_DIRECTORY=`xcvb show object-directory` XCVB_IMAGE=${XCVB_OBJECT_DIRECTORY}/example-1.image # Required for xcvb to find the source registry export CL_SOURCE_REGISTRY := ${XCVB_ROOT}//:${CL_SOURCE_REGISTRY} # Build by having xcvb run the simple-build steps itself. build: xcvb simple-build \ --build "example-1" \ --lisp-implementation ${LISP} \ --verbosity 10 cl-launch --lisp ${LISP} \ --image ${XCVB_IMAGE} \ --no-include \ --dump '!' \ --output example-1 \ --init '(xcvb-example-1::main)' In this example, example-1 in the cwd is the produced executable. Now, we do this: $ cd `xcvb show object-directory` $ ls psilord at meta:~/.cache/xcvb/common-lisp/sbcl-1.0.47-linux-x86$ ls total 56528 -rwxr-xr-x 1 psilord psilord 28897308 Apr 28 03:42 _.image -rw-r--r-- 1 psilord psilord 375 Apr 28 03:41 ___initial.manifest drwxr-xr-x 2 psilord psilord 4096 Apr 28 03:42 example-1 -rwxr-xr-x 1 psilord psilord 28897308 Apr 28 03:42 example-1.image -rw-r--r-- 1 psilord psilord 393 Apr 28 03:42 example-1.manifest drwxr-xr-x 2 psilord psilord 4096 Apr 28 03:41 xcvb If you notice there is stuff is not contained in the example-1 directory. If I had two makes going at the same time, each building different things, but happening to choose the same executable name, a collision happens in the toplevel object-directory. A collision could always happen with _.image depending upon which make wrote it first and which make needs it next! Same with ___initial.manifest. There is also some notion about the driver codes too, should they be rebuilt for each project or shared among all projects? It must be that they are always in a consistent state if shared. Either way, we need to fix the above race condition otherwise people can be in for a nasty surprise. Maybe show-settings can take a --build which will show the setting in relation to that build, as opposed to the xcvb toplevel. For example: xcvb show object-directory => ~/.cache/xcvb/common-lisp/sbcl-1.0.47-linux-x86 xcvb show --build "example-1" object-directory => ~/.cache/xcvb/common-lisp/sbcl-1.0.47-linux-x86/example-1 We need a more clearly defined place, semantics, and full path retrieval method for any images produced by an arbitrary run of xcvb. Later, -pete