[cl-ppcre-devel] Matching on very long strings.

[Edi Weitz]

On Sun, 28 Sep 2008 15:28:07 -0500, "Matthew D. Swank" <akopa.gmane.poster at gmail.com> wrote:

> Well the regexes are defined in the lexers in this file:
> http://common-lisp.net/~mswank/apache-ppcre.lisp
>
> The lexer api is in this file:
> http://common-lisp.net/~mswank/cl-ppcre-lexer.lisp
>
> Finally, the log file I'm lexing:
> http://lcpug.asternix.com/pub/Main/ApacheLogProject/access.log
>
> Compare 
> (with-open-file  (in "access.log") 
>   (let ((foo (stream-gen *apache-pcrelex-line* in))) 
>     (time (loop :for x := (funcall foo)
>                 :unless x :return nil))))
>
> with
>
> (with-open-file  (in "access.log") 
>   (let ((foo (stream-gen *apache-pcrelex* in))) 
>     (time (loop :for x := (funcall foo)
>                 :unless x :return nil))))
>
> When I slurp the entire file into a string the matches seem to be
> taking about a tenth of a second for each token.

Sorry, I don't have the time to read the entire application right now.
Can you boil this down to a single application of PPCRE:SCAN which is
too slow?

Thanks,
Edi.