[hunchentoot-devel] gzip-encoding

[Edi Weitz]

On Sat, 26 May 2007 03:24:31 -0700, "Mac Chan" <emailmac at gmail.com> wrote:

> Using the network monitor in FireBug , my initial testing (with a
> 790 KB javascript file) shows that the response time is slower with
> gzip turned on (I'm guessing that it might be related to
> flexi-stream.)

I suppose gzip itself will also take some time, won't it?  Why do you
think flexi-streams is the culprit in this case?

> Should we add this to hunchentoot or make it an extension module?
> (this will introduce two new dependencies to hunchentoot)

I'm not against it per se, but I wouldn't want to add it just for
static files.  I think static files aren't something we should be
particularly concerned with.  If you have lots of static files and
Hunchentoot can't serve them fast enough for your purposes, let Apache
serve them, or put them behind mod_lisp and use mod_gzip.  Besides,
clients will cache Javascript files, so they'll have to read your big
Ajax library only once.

If, OTOH, you also want to allow gzip encoding for dynamic content
generated with Hunchentoot, it gets a lot more complicated,
particularly because gzip-streams are binary only.  I thought about
this for some time, and I don't see an easy and elegant way around
this - unless you want to add yet another flexi-streams layer which
would result in four (!) layers of Gray streams atop each other.

> If we add this to hunchentoot, should we add a flag to turn on/off
> gzip encoding? (probably yes)

Yes, definitely.

> And again, since there's flexi-stream overhead (my guess), should we
> add a flag to control what content-type we should disable gzip
> encoding? (probably wasting cpu cycle to gzip jpeg, mp3 and png,
> etc).

Yes.  This has nothing to do with whether flexi-streams adds overhead
or not.  It is generally a bad idea to compress formats that are
already compressed... :)

>         (accept-gzip-p (search "gzip" (header-in :accept-encoding)))

I think that's a bit too simplistic.  The header could be one of

  Accept-Encoding: x-frogzipper
  Accept-Encoding: gzip; q=0
  Accept-Encoding: *

Something like this maybe (untested):

  (defun parse-qvalue (string)
    "Checks if the string STRING is a `qvalue' according to RFC 2616 and
  returns the corresponding numeric value.  Signals an error otherwise."
    (unless (ppcre:scan "^(?:[0](?:\\.[0-9]{0,3})?|[1](?:\\.0{0,3})?)$" string)
      (error "Not a qvalue: ~S" string))
    (case (char string 0)
      (#\1 1)
      (#\0 (let ((length (length string)))
             (case length
               ((1 2) 0)
               (otherwise
                (/ (parse-integer string :start 2)
                   (case length
                     (3 10)
                     (4 100)
                     (5 1000)))))))))

  (defun accepts-gzip-p ()
    "Checks whether the user agent accepts gzip encoding according to
  the incoming headers."
    (when-let (accept-encoding-header (header-in :accept-encoding))
      (dolist (part (ppcre:split "," accept-encoding-header))
        (with-input-from-string (stream part)
          (let* ((*current-error-message* "Corrupted Accept-Encoding header:")
                 (coding (read-token stream))
                 (parameters (read-name-value-pairs stream))
                 (quality (cdr (assoc "q" parameters))))
            (when (and (member coding '("gzip" "*") :test #'string=)
                       (or (null quality)
                           (plusp (handler-case (parse-qvalue quality)
                                    (error (condition)
                                      (log-message :warn "While parsing Accept-Encoding header: ~A"
                                                   condition)
                                      0)))))
              (return-from accepts-gzip-p t)))))))

Cheers,
Edi.