From ch-lisp at bobobeach.com  Sun Apr  8 02:04:43 2012
From: ch-lisp at bobobeach.com (Cyrus Harmon)
Date: Sat, 7 Apr 2012 19:04:43 -0700
Subject: [cl-ppcre-devel] scan/property behavior?
Message-ID: <3F5E1D38-804F-4960-B506-53F9CCDDCF18@bobobeach.com>


I was noticing some strange behavior with cxml-rng and David Lichteblau pointed out that the problem might be due to cl-ppcre and provided this example:

(let ((regex1 '(:SEQUENCE :start-anchor 
                (:greedy-repetition 0 42 (:property digit-char-p))
                #\-))
      (regex2 '(:SEQUENCE :start-anchor
                (:property digit-char-p)
                #\-))
      (str "8-"))
  (values (multiple-value-list (ppcre:scan regex1 str))
          (multiple-value-list (ppcre:scan regex2 str))))


Regex1 works as expected (it matches 8-) but the second one doesn't. It's possible I don't understand (well more than possible) how :property works, but I would have expected either both or neither of these to fail. Could this be due to a bug in cl-ppcre?

thanks,

Cyrus




From edi at agharta.de  Sun Apr  8 11:42:55 2012
From: edi at agharta.de (Edi Weitz)
Date: Sun, 8 Apr 2012 13:42:55 +0200
Subject: [cl-ppcre-devel] scan/property behavior?
In-Reply-To: <3F5E1D38-804F-4960-B506-53F9CCDDCF18@bobobeach.com>
References: <3F5E1D38-804F-4960-B506-53F9CCDDCF18@bobobeach.com>
Message-ID: <CAD3YvRWqV+u5LLL3tMvp_uh909HAK_-sKTktjjtE77YBuzAqJw@mail.gmail.com>

Yep, that seems to be a bug.  I looked into it for a couple of minutes
and I /think/ the patch below fixes it, but it's been a long time
since I've looked into the ugly parts of CL-PPCRE, so take this with a
grain of salt.

Cheers,
Edi.


--- convert.lisp        (revision 4677)
+++ convert.lisp        (working copy)
@@ -701,11 +701,15 @@
 (defmethod convert-compound-parse-tree ((token (eql :property))
parse-tree &key)
   "The case for \(:PROPERTY <name>) where <name> is a string."
   (declare #.*standard-optimize-settings*)
+  (declare (special accumulate-start-p))
+  (setq accumulate-start-p nil)
   (make-instance 'char-class :test-function (resolve-property (second
parse-tree))))

 (defmethod convert-compound-parse-tree ((token (eql
:inverted-property)) parse-tree &key)
   "The case for \(:INVERTED-PROPERTY <name>) where <name> is a string."
   (declare #.*standard-optimize-settings*)
+  (declare (special accumulate-start-p))
+  (setq accumulate-start-p nil)
   (make-instance 'char-class :test-function (complement*
(resolve-property (second parse-tree)))))

 (defmethod convert-compound-parse-tree ((token (eql :flags)) parse-tree &key)

On Sun, Apr 8, 2012 at 4:04 AM, Cyrus Harmon <ch-lisp at bobobeach.com> wrote:
>
> I was noticing some strange behavior with cxml-rng and David Lichteblau pointed out that the problem might be due to cl-ppcre and provided this example:
>
> (let ((regex1 '(:SEQUENCE :start-anchor
> ? ? ? ? ? ? ? ?(:greedy-repetition 0 42 (:property digit-char-p))
> ? ? ? ? ? ? ? ?#\-))
> ? ? ?(regex2 '(:SEQUENCE :start-anchor
> ? ? ? ? ? ? ? ?(:property digit-char-p)
> ? ? ? ? ? ? ? ?#\-))
> ? ? ?(str "8-"))
> ?(values (multiple-value-list (ppcre:scan regex1 str))
> ? ? ? ? ?(multiple-value-list (ppcre:scan regex2 str))))
>
>
> Regex1 works as expected (it matches 8-) but the second one doesn't. It's possible I don't understand (well more than possible) how :property works, but I would have expected either both or neither of these to fail. Could this be due to a bug in cl-ppcre?
>
> thanks,
>
> Cyrus
>
>
> _______________________________________________
> cl-ppcre-devel site list
> cl-ppcre-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/cl-ppcre-devel
>



From ch-lisp at bobobeach.com  Sun Apr  8 15:14:59 2012
From: ch-lisp at bobobeach.com (Cyrus Harmon)
Date: Sun, 8 Apr 2012 08:14:59 -0700
Subject: [cl-ppcre-devel] scan/property behavior?
In-Reply-To: <CAD3YvRWqV+u5LLL3tMvp_uh909HAK_-sKTktjjtE77YBuzAqJw@mail.gmail.com>
References: <3F5E1D38-804F-4960-B506-53F9CCDDCF18@bobobeach.com>
	<CAD3YvRWqV+u5LLL3tMvp_uh909HAK_-sKTktjjtE77YBuzAqJw@mail.gmail.com>
Message-ID: <15D25221-A683-49A2-A212-93DB053C0F27@bobobeach.com>


Yes, that seems to fix things. Thanks for the prompt fix!

I've committed the changes to my cl-pppcre github repo:

https://github.com/slyrus/cl-ppcre/tree/property-bugfix

thanks again for the prompt attention to this,

Cyrus

On Apr 8, 2012, at 4:42 AM, Edi Weitz wrote:

> Yep, that seems to be a bug.  I looked into it for a couple of minutes
> and I /think/ the patch below fixes it, but it's been a long time
> since I've looked into the ugly parts of CL-PPCRE, so take this with a
> grain of salt.
> 
> Cheers,
> Edi.
> 
> 
> --- convert.lisp        (revision 4677)
> +++ convert.lisp        (working copy)
> @@ -701,11 +701,15 @@
> (defmethod convert-compound-parse-tree ((token (eql :property))
> parse-tree &key)
>   "The case for \(:PROPERTY <name>) where <name> is a string."
>   (declare #.*standard-optimize-settings*)
> +  (declare (special accumulate-start-p))
> +  (setq accumulate-start-p nil)
>   (make-instance 'char-class :test-function (resolve-property (second
> parse-tree))))
> 
> (defmethod convert-compound-parse-tree ((token (eql
> :inverted-property)) parse-tree &key)
>   "The case for \(:INVERTED-PROPERTY <name>) where <name> is a string."
>   (declare #.*standard-optimize-settings*)
> +  (declare (special accumulate-start-p))
> +  (setq accumulate-start-p nil)
>   (make-instance 'char-class :test-function (complement*
> (resolve-property (second parse-tree)))))
> 
> (defmethod convert-compound-parse-tree ((token (eql :flags)) parse-tree &key)
> 
> On Sun, Apr 8, 2012 at 4:04 AM, Cyrus Harmon <ch-lisp at bobobeach.com> wrote:
>> 
>> I was noticing some strange behavior with cxml-rng and David Lichteblau pointed out that the problem might be due to cl-ppcre and provided this example:
>> 
>> (let ((regex1 '(:SEQUENCE :start-anchor
>>                (:greedy-repetition 0 42 (:property digit-char-p))
>>                #\-))
>>      (regex2 '(:SEQUENCE :start-anchor
>>                (:property digit-char-p)
>>                #\-))
>>      (str "8-"))
>>  (values (multiple-value-list (ppcre:scan regex1 str))
>>          (multiple-value-list (ppcre:scan regex2 str))))
>> 
>> 
>> Regex1 works as expected (it matches 8-) but the second one doesn't. It's possible I don't understand (well more than possible) how :property works, but I would have expected either both or neither of these to fail. Could this be due to a bug in cl-ppcre?
>> 
>> thanks,
>> 
>> Cyrus
>> 
>> 
>> _______________________________________________
>> cl-ppcre-devel site list
>> cl-ppcre-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/cl-ppcre-devel
>> 
> 
> _______________________________________________
> cl-ppcre-devel site list
> cl-ppcre-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/cl-ppcre-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/cl-ppcre-devel/attachments/20120408/27bb0e32/attachment.html>

From hans.huebner at gmail.com  Sun Apr  8 16:52:29 2012
From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Sun, 8 Apr 2012 18:52:29 +0200
Subject: [cl-ppcre-devel] scan/property behavior?
In-Reply-To: <15D25221-A683-49A2-A212-93DB053C0F27@bobobeach.com>
References: <3F5E1D38-804F-4960-B506-53F9CCDDCF18@bobobeach.com>
	<CAD3YvRWqV+u5LLL3tMvp_uh909HAK_-sKTktjjtE77YBuzAqJw@mail.gmail.com>
	<15D25221-A683-49A2-A212-93DB053C0F27@bobobeach.com>
Message-ID: <CABj0bwLVhKz4wAbaq3ncM-GbP=Ad82rYei=PQZyGv-fgzV_DFA@mail.gmail.com>

On Sun, Apr 8, 2012 at 5:14 PM, Cyrus Harmon <ch-lisp at bobobeach.com> wrote:
>
> Yes, that seems to fix things. Thanks for the prompt fix!
>
> I've committed the changes to my cl-pppcre github repo:
>
> https://github.com/slyrus/cl-ppcre/tree/property-bugfix

I have pulled your change, thanks!

-Hans



From mario.maio at libero.it  Fri Apr 13 10:03:24 2012
From: mario.maio at libero.it (Mario Maio)
Date: Fri, 13 Apr 2012 12:03:24 +0200
Subject: [cl-ppcre-devel] omitting initial void string in split returns
Message-ID: <4F87F9EC.8070705@libero.it>

I want to parse strings of space-separated integers, and split is 
perfect except when I have vertical aligned integers. In such cases I've 
got one or more spaces at beginning of string, and the resulting list 
has a void string as 1st element:

CL-USER> (cl-ppcre:split "\\s+" "   43  76  87  33   89  ")

("" "43" "76" "87" "33" "89")

Is there an option to avoid such initial void string?

Thanks.

Mario




From ch-lisp at bobobeach.com  Fri Apr 13 11:55:24 2012
From: ch-lisp at bobobeach.com (Cyrus Harmon)
Date: Fri, 13 Apr 2012 04:55:24 -0700
Subject: [cl-ppcre-devel] differing results between cl-ppcre and perl --
	operator precedence?
Message-ID: <B95A02F7-D1BC-4B62-956E-8EEDEC303D04@bobobeach.com>

In trying to track down a problem matching a cxml-rng pattern in soiree, I came across the following somewhat odd behavior in cl-pppcre.

In perl, I see:

(sly at barbaresco):~/projects/soiree$ echo "Ta" | perl -ne 'print if /^T(a)|(b)$/'
Ta

(sly at barbaresco):~/projects/soiree$ echo "Tb" | perl -ne 'print if /^T(a)|(b)$/'
Tb

Whereas with cl-ppcre, I see:

SOIREE-ICALENDAR> (cl-ppcre:scan-to-strings "^T(a)|(b)$" "Ta")
"Ta"
#("a" NIL)

SOIREE-ICALENDAR> (cl-ppcre:scan-to-strings "^T(a)|(b)$" "Tb")
"b"
#(NIL "b")

Parsing the string suggests that the order of operations may be different in cl-ppcre than in perl. Is this intentional?

SOIREE-ICALENDAR> (cl-ppcre:parse-string "^T(a)|(b)$")
(:ALTERNATION (:SEQUENCE :START-ANCHOR #\T (:REGISTER #\a))
 (:SEQUENCE (:REGISTER #\b) :END-ANCHOR))

thanks,

Cyrus




From edi at agharta.de  Fri Apr 13 11:58:27 2012
From: edi at agharta.de (Edi Weitz)
Date: Fri, 13 Apr 2012 13:58:27 +0200
Subject: [cl-ppcre-devel] omitting initial void string in split returns
In-Reply-To: <4F87F9EC.8070705@libero.it>
References: <4F87F9EC.8070705@libero.it>
Message-ID: <CAD3YvRW6=SzneUpKKNfT+XRSPnKuK3QOrtXv=LMac8O4C=QyDQ@mail.gmail.com>

I don't think there is (unless you want the initial space to be a part
of the first string like "  43").  The easiest solution would be to
call something like REMOVE-IF or DELETE-IF directly afterwards.

Cheers,
Edi.

On Fri, Apr 13, 2012 at 12:03 PM, Mario Maio <mario.maio at libero.it> wrote:
> I want to parse strings of space-separated integers, and split is perfect
> except when I have vertical aligned integers. In such cases I've got one or
> more spaces at beginning of string, and the resulting list has a void string
> as 1st element:
>
> CL-USER> (cl-ppcre:split "\\s+" " ? 43 ?76 ?87 ?33 ? 89 ?")
>
> ("" "43" "76" "87" "33" "89")
>
> Is there an option to avoid such initial void string?
>
> Thanks.
>
> Mario
>
>
> _______________________________________________
> cl-ppcre-devel site list
> cl-ppcre-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/cl-ppcre-devel
>



From n_reijnders at hotmail.com  Fri Apr 13 12:03:53 2012
From: n_reijnders at hotmail.com (Noldus Reijnders)
Date: Fri, 13 Apr 2012 14:03:53 +0200
Subject: [cl-ppcre-devel] omitting initial void string in split returns
In-Reply-To: <CAD3YvRW6=SzneUpKKNfT+XRSPnKuK3QOrtXv=LMac8O4C=QyDQ@mail.gmail.com>
References: <4F87F9EC.8070705@libero.it>,
	<CAD3YvRW6=SzneUpKKNfT+XRSPnKuK3QOrtXv=LMac8O4C=QyDQ@mail.gmail.com>
Message-ID: <BAY164-W3577A64E605F109BFC436A843B0@phx.gbl>


How about using (string-trim " " " 43 76 87 33 89 ") first and then doing split?
See http://cl-cookbook.sourceforge.net/strings.html#trim 
Noldus
> From: edi at agharta.de
> Date: Fri, 13 Apr 2012 13:58:27 +0200
> To: cl-ppcre-devel at common-lisp.net
> Subject: Re: [cl-ppcre-devel] omitting initial void string in split returns
> 
> I don't think there is (unless you want the initial space to be a part
> of the first string like "  43").  The easiest solution would be to
> call something like REMOVE-IF or DELETE-IF directly afterwards.
> 
> Cheers,
> Edi.
> 
> On Fri, Apr 13, 2012 at 12:03 PM, Mario Maio <mario.maio at libero.it> wrote:
> > I want to parse strings of space-separated integers, and split is perfect
> > except when I have vertical aligned integers. In such cases I've got one or
> > more spaces at beginning of string, and the resulting list has a void string
> > as 1st element:
> >
> > CL-USER> (cl-ppcre:split "\\s+" "   43  76  87  33   89  ")
> >
> > ("" "43" "76" "87" "33" "89")
> >
> > Is there an option to avoid such initial void string?
> >
> > Thanks.
> >
> > Mario
> >
> >
> > _______________________________________________
> > cl-ppcre-devel site list
> > cl-ppcre-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/cl-ppcre-devel
> >
> 
> _______________________________________________
> cl-ppcre-devel site list
> cl-ppcre-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/cl-ppcre-devel
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/cl-ppcre-devel/attachments/20120413/363752ae/attachment.html>

From edi at agharta.de  Fri Apr 13 12:12:05 2012
From: edi at agharta.de (Edi Weitz)
Date: Fri, 13 Apr 2012 14:12:05 +0200
Subject: [cl-ppcre-devel] differing results between cl-ppcre and perl --
 operator precedence?
In-Reply-To: <B95A02F7-D1BC-4B62-956E-8EEDEC303D04@bobobeach.com>
References: <B95A02F7-D1BC-4B62-956E-8EEDEC303D04@bobobeach.com>
Message-ID: <CAD3YvRUW=Hv-xqae=c5GWqONP510+EXu9k-OzNnXdtb7b-63=A@mail.gmail.com>

My Perl behaves differently.  See below:

edi at miles:/tmp$ cat test.pl
#!/usr/bin/perl

$a = "Ta";
$b = "Tb";
print "a: '$&'\n" if ($a =~ /^T(a)|(b)/);
print "b: '$&'\n" if ($b =~ /^T(a)|(b)/);

edi at miles:/tmp$ perl test.pl
a: 'Ta'
b: 'b'
edi at miles:/tmp$ perl -v

This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi

Copyright 1987-2006, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

On Fri, Apr 13, 2012 at 1:55 PM, Cyrus Harmon <ch-lisp at bobobeach.com> wrote:
> In trying to track down a problem matching a cxml-rng pattern in soiree, I came across the following somewhat odd behavior in cl-pppcre.
>
> In perl, I see:
>
> (sly at barbaresco):~/projects/soiree$ echo "Ta" | perl -ne 'print if /^T(a)|(b)$/'
> Ta
>
> (sly at barbaresco):~/projects/soiree$ echo "Tb" | perl -ne 'print if /^T(a)|(b)$/'
> Tb
>
> Whereas with cl-ppcre, I see:
>
> SOIREE-ICALENDAR> (cl-ppcre:scan-to-strings "^T(a)|(b)$" "Ta")
> "Ta"
> #("a" NIL)
>
> SOIREE-ICALENDAR> (cl-ppcre:scan-to-strings "^T(a)|(b)$" "Tb")
> "b"
> #(NIL "b")
>
> Parsing the string suggests that the order of operations may be different in cl-ppcre than in perl. Is this intentional?
>
> SOIREE-ICALENDAR> (cl-ppcre:parse-string "^T(a)|(b)$")
> (:ALTERNATION (:SEQUENCE :START-ANCHOR #\T (:REGISTER #\a))
> ?(:SEQUENCE (:REGISTER #\b) :END-ANCHOR))
>
> thanks,
>
> Cyrus
>
>
> _______________________________________________
> cl-ppcre-devel site list
> cl-ppcre-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/cl-ppcre-devel
>



From ch-lisp at bobobeach.com  Fri Apr 13 12:23:31 2012
From: ch-lisp at bobobeach.com (Cyrus Harmon)
Date: Fri, 13 Apr 2012 05:23:31 -0700
Subject: [cl-ppcre-devel] differing results between cl-ppcre and perl --
	operator precedence?
In-Reply-To: <CAD3YvRUW=Hv-xqae=c5GWqONP510+EXu9k-OzNnXdtb7b-63=A@mail.gmail.com>
References: <B95A02F7-D1BC-4B62-956E-8EEDEC303D04@bobobeach.com>
	<CAD3YvRUW=Hv-xqae=c5GWqONP510+EXu9k-OzNnXdtb7b-63=A@mail.gmail.com>
Message-ID: <F6A7C320-2E70-45E9-B97C-F87713163061@bobobeach.com>


Our perls behave the same. I was seeing a match (to b only) and then my one-liner was printing the original string which matched, not the matched portion.

Thanks and sorry for the false alarm.

Cyrus

On Apr 13, 2012, at 5:12 AM, Edi Weitz wrote:

> My Perl behaves differently.  See below:
> 
> edi at miles:/tmp$ cat test.pl
> #!/usr/bin/perl
> 
> $a = "Ta";
> $b = "Tb";
> print "a: '$&'\n" if ($a =~ /^T(a)|(b)/);
> print "b: '$&'\n" if ($b =~ /^T(a)|(b)/);
> 
> edi at miles:/tmp$ perl test.pl
> a: 'Ta'
> b: 'b'
> edi at miles:/tmp$ perl -v
> 
> This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi
> 
> Copyright 1987-2006, Larry Wall
> 
> Perl may be copied only under the terms of either the Artistic License or the
> GNU General Public License, which may be found in the Perl 5 source kit.
> 
> Complete documentation for Perl, including FAQ lists, should be found on
> this system using "man perl" or "perldoc perl".  If you have access to the
> Internet, point your browser at http://www.perl.org/, the Perl Home Page.
> 
> On Fri, Apr 13, 2012 at 1:55 PM, Cyrus Harmon <ch-lisp at bobobeach.com> wrote:
>> In trying to track down a problem matching a cxml-rng pattern in soiree, I came across the following somewhat odd behavior in cl-pppcre.
>> 
>> In perl, I see:
>> 
>> (sly at barbaresco):~/projects/soiree$ echo "Ta" | perl -ne 'print if /^T(a)|(b)$/'
>> Ta
>> 
>> (sly at barbaresco):~/projects/soiree$ echo "Tb" | perl -ne 'print if /^T(a)|(b)$/'
>> Tb
>> 
>> Whereas with cl-ppcre, I see:
>> 
>> SOIREE-ICALENDAR> (cl-ppcre:scan-to-strings "^T(a)|(b)$" "Ta")
>> "Ta"
>> #("a" NIL)
>> 
>> SOIREE-ICALENDAR> (cl-ppcre:scan-to-strings "^T(a)|(b)$" "Tb")
>> "b"
>> #(NIL "b")
>> 
>> Parsing the string suggests that the order of operations may be different in cl-ppcre than in perl. Is this intentional?
>> 
>> SOIREE-ICALENDAR> (cl-ppcre:parse-string "^T(a)|(b)$")
>> (:ALTERNATION (:SEQUENCE :START-ANCHOR #\T (:REGISTER #\a))
>>  (:SEQUENCE (:REGISTER #\b) :END-ANCHOR))
>> 
>> thanks,
>> 
>> Cyrus
>> 
>> 
>> _______________________________________________
>> cl-ppcre-devel site list
>> cl-ppcre-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/cl-ppcre-devel
>> 
> 
> _______________________________________________
> cl-ppcre-devel site list
> cl-ppcre-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/cl-ppcre-devel




From philipp at marek.priv.at  Thu Apr 19 20:20:35 2012
From: philipp at marek.priv.at (Philipp Marek)
Date: Thu, 19 Apr 2012 22:20:35 +0200
Subject: [cl-ppcre-devel] sbcl unhandled exception
Message-ID: <c77eae1f6cf0b8ab0a7cfcc71b18bf45.squirrel@webmail.hitco.org>

Hello everyone,

I'm trying to teach cl-ppcre (taken from github, updated) to use (unsigned-byte 8) data
as well, to see how much faster it gets for ASCII and binary files.

Attached is a diff that's my current state. It's not nice, but - it's not cleaned up yet.


The problem is that with sbcl 1.0.55.0-debian (amd64) I get

    Unhandled memory fault at #x0.

and that's not good.


I'm doing this:
	(asdf:clear-system :cl-ppcre)
	(asdf:oos 'asdf:compile-op :cl-ppcre :force T)
	(asdf:oos 'asdf:load-source-op :cl-ppcre)

then, optionally,
	(trace "CL-PPCRE")

and then crash SBCL with
	(scan-to-strings "B(A+)(.)" "AAAAAABAAAAAAAAAA")


Any ideas? Thanks a lot.


Regards,

Phil
-------------- next part --------------
A non-text attachment was scrubbed...
Name: unhandled-exception-at-0_cl-ppcre.patch
Type: text/x-diff
Size: 25536 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/cl-ppcre-devel/attachments/20120419/f11b6959/attachment.patch>

From philipp at marek.priv.at  Thu Apr 26 19:29:28 2012
From: philipp at marek.priv.at (Philipp Marek)
Date: Thu, 26 Apr 2012 21:29:28 +0200
Subject: [cl-ppcre-devel] Using (unsigned-byte 8) instead of a (string) as
	TARGET-STRING
Message-ID: <066aff3470fe07380311e1ea78ea337d.squirrel@webmail.hitco.org>

Hello everybody,

I've got a first (unoptimized) patch that allows to use (unsigned-byte 8)
instead of a string as TARGET-STRING. My motivation is to use that for
searching in big, binary files, which might not fit into RAM with byte =>
character conversion (which would be 1:4, as the binaries would have to be read
as latin1 or similar).


Using that patch on a ~3MB file with the string match at about 80% of the file
size shows a nice speedup: only half to a third cpu time used, and much less
memory usage (for the string).


Details:
-rw-r--r-- 1 root root 3176746 Mai  3  2010 /usr/share/doc/gcc-4.4-doc/gccint.html

string, case-sensitive:
  0.360022 seconds of total run time (0.360022 user, 0.000000 system)
  907,556,608 processor cycles
string, case-insensitive:
  0.492031 seconds of total run time (0.492031 user, 0.000000 system)
  1,239,853,076 processor cycles

(unsigned-byte 8), case-sensitive:
  0.108006 seconds of total run time (0.108006 user, 0.000000 system)
  274,043,748 processor cycles
(unsigned-byte 8), case-insensitive:
  0.220013 seconds of total run time (0.220013 user, 0.000000 system)
  553,027,836 processor cycles


The small "problem" is this (one long line):

  $ time perl -e '$/=undef; $_=<>; print $1,$2,"\n" if
    /acr([o0] i)s not defined,\s+the default(\Dvalue,\s*\d+, i)s used/'
    < /usr/share/doc/gcc-4.4-doc/gccint.html
  o i value, 1, i
  real    0m0.016s
  user    0m0.004s
  sys     0m0.008s

  $ perl -v
  This is perl 5, version 14, subversion 2 (v5.14.2)
  built for x86_64-linux-gnu-thread-multi

ie. (this) perl5 is still ~8 times faster, including file reading etc. (what
the lisp code didn't take into the measurement).
(With the /i modifier it's 0.020s.)


I've not yet tried to run the whole test suite against that. There are quite
a few warnings (unused variable UB8-MODE etc.) - but with higher SAFETY the
original CL-PPCRE gave a lot of them, too.


I'd like to ask for a quick look at the patch, to get some feedback; with the
many duplications I don't really like the result, but the duplicated accesses
("schar" etc.) are too deeply integrated in cl-ppcre, I couldn't easily get
them out into a single macro or something like that.


Cyrus, Edi, could you help me clean up the changes so that they
could be taken upstream?


The other file is the one I'm using for testing.


Regards,

Phil

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ub8-test.lisp
Type: application/octet-stream
Size: 1139 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/cl-ppcre-devel/attachments/20120426/f0c7142c/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cl-ppcre-ub8-mode.patch.gz
Type: application/x-gzip
Size: 10048 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/cl-ppcre-devel/attachments/20120426/f0c7142c/attachment.bin>

From edi at weitz.de  Thu Apr 26 19:40:25 2012
From: edi at weitz.de (Edi Weitz)
Date: Thu, 26 Apr 2012 21:40:25 +0200
Subject: [cl-ppcre-devel] Using (unsigned-byte 8) instead of a (string)
	as TARGET-STRING
In-Reply-To: <066aff3470fe07380311e1ea78ea337d.squirrel@webmail.hitco.org>
References: <066aff3470fe07380311e1ea78ea337d.squirrel@webmail.hitco.org>
Message-ID: <CAD3YvRWSrDvoF7ud0-P92Yn80+swqPMJNpviMNBNfE+O0GM1nw@mail.gmail.com>

Phil,

Thanks for your work on this.  I'm afraid I won't be able to help as
I'm too busy with my new job and don't expect this to change soon.
Maybe someone else will.

I also have to admit I'm reluctant to add this to the main CL-PPCRE
code for fear of a maintenance nightmare.  CL-PPCRE already is far
from clean and clear and I'd rather not add more stuff.  Unless
someone else wants to take over maintenance completely, of course.

Cheers,
Edi.


On Thu, Apr 26, 2012 at 9:29 PM, Philipp Marek <philipp at marek.priv.at> wrote:
> Hello everybody,
>
> I've got a first (unoptimized) patch that allows to use (unsigned-byte 8)
> instead of a string as TARGET-STRING. My motivation is to use that for
> searching in big, binary files, which might not fit into RAM with byte =>
> character conversion (which would be 1:4, as the binaries would have to be read
> as latin1 or similar).
>
>
> Using that patch on a ~3MB file with the string match at about 80% of the file
> size shows a nice speedup: only half to a third cpu time used, and much less
> memory usage (for the string).
>
>
> Details:
> -rw-r--r-- 1 root root 3176746 Mai ?3 ?2010 /usr/share/doc/gcc-4.4-doc/gccint.html
>
> string, case-sensitive:
> ?0.360022 seconds of total run time (0.360022 user, 0.000000 system)
> ?907,556,608 processor cycles
> string, case-insensitive:
> ?0.492031 seconds of total run time (0.492031 user, 0.000000 system)
> ?1,239,853,076 processor cycles
>
> (unsigned-byte 8), case-sensitive:
> ?0.108006 seconds of total run time (0.108006 user, 0.000000 system)
> ?274,043,748 processor cycles
> (unsigned-byte 8), case-insensitive:
> ?0.220013 seconds of total run time (0.220013 user, 0.000000 system)
> ?553,027,836 processor cycles
>
>
> The small "problem" is this (one long line):
>
> ?$ time perl -e '$/=undef; $_=<>; print $1,$2,"\n" if
> ? ?/acr([o0] i)s not defined,\s+the default(\Dvalue,\s*\d+, i)s used/'
> ? ?< /usr/share/doc/gcc-4.4-doc/gccint.html
> ?o i value, 1, i
> ?real ? ?0m0.016s
> ?user ? ?0m0.004s
> ?sys ? ? 0m0.008s
>
> ?$ perl -v
> ?This is perl 5, version 14, subversion 2 (v5.14.2)
> ?built for x86_64-linux-gnu-thread-multi
>
> ie. (this) perl5 is still ~8 times faster, including file reading etc. (what
> the lisp code didn't take into the measurement).
> (With the /i modifier it's 0.020s.)
>
>
> I've not yet tried to run the whole test suite against that. There are quite
> a few warnings (unused variable UB8-MODE etc.) - but with higher SAFETY the
> original CL-PPCRE gave a lot of them, too.
>
>
> I'd like to ask for a quick look at the patch, to get some feedback; with the
> many duplications I don't really like the result, but the duplicated accesses
> ("schar" etc.) are too deeply integrated in cl-ppcre, I couldn't easily get
> them out into a single macro or something like that.
>
>
> Cyrus, Edi, could you help me clean up the changes so that they
> could be taken upstream?
>
>
> The other file is the one I'm using for testing.
>
>
> Regards,
>
> Phil
>
>
> _______________________________________________
> cl-ppcre-devel site list
> cl-ppcre-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/cl-ppcre-devel