[mel-base-devel] maildir multipart

[Jochen Schmidt]

Am 15.05.2009 um 17:13 schrieb Timothy Ritchey:

> If I do something like:
>
> (defun check-email ()
>   (let* ((folder (mel:make-imaps-folder :host
> "imap.gmail.com" :username "XXXXX" :password "XXXXX"))
> 	 (messages (mel:messages folder)))
>     (dolist (m messages)
> 	(format t "~a~%" (find-plain-text m))) folder))
>
> I am able to find the plain text part of the messages using:
>
> (defun find-plain-text (message)
>   (let ((parts (mel:parts message))
>         (body "Unable to process email contents"))
>     (dolist (p parts)
>       (multiple-value-bind (a b) (mel:content-type p)
>         (if (and (equal a :TEXT) (equal b :PLAIN))
>             (let ((b (mel:part-body-string p))
> 		  (encoding (mel:content-transfer-encoding p)))
> 	      (if (string-equal encoding "quoted-printable")
>                   (setf body (mel:decode-quoted-printable b))
>                   (setf body b)))
> 	    (progn
> 	      (when (and (equal a :MULTIPART) (equal b :ALTERNATIVE))
> 		(find-plain-text p))))))
>     body))
>
> but if I try to move things over to a maildir, and run from there, my
> FIND-PLAIN-TEXT function doesn't work. Is the maildir message format
> different than a message in an imap folder?

No the message format is the same (octet wise). One of the design  
principles of mel-base was to never modify the content of a message.  
(Trivia: I started mel-base because the "standard" python mail library  
modified my mails while fetching them). You can  compared the file  
stored in the maildir folder with one saved using e.g. Apple Mail.app  
from the IMAP server - they ought to be identical.

The problem here is that perhaps the message is interpreted  
differently when accessing its parts. IMAP is a cool thing - you get  
many things from it without much work. RFC2060 (IMAP) defines a thing  
called "Body Structure" which describes the structure of the message- 
body. If you use IMAP, mel-base can just query the server for the body  
structure - if not it will have to compute it by itself. It seems that  
the body structure of your message parsed by your server is different  
from the one parsed by mel-base.

The folder protocol of mel-base has a GF called COMPUTE-BODYSTRUCTURE- 
USING-FOLDER. Which has two implemented methods: One for IMAP-Foldes  
and one for all others. The user-level function is called COMPUTE- 
BODYSTRUCTURE and can be called on a message object to get the body  
structure of it.

I've tried it using my current snapshot here on some multipart message  
in my mail account.

The IMAP gives:

(((:TEXT :PLAIN (:CHARSET "us-ascii") NIL NIL :QUOTED-PRINTABLE 1638  
45 NIL NIL NIL)
   (:APPLICATION :PGP-SIGNATURE (:NAME "signature.asc") NIL NIL :7BIT  
205 NIL NIL NIL NIL)
   :SIGNED
   (:MICALG "pgp-sha1" :PROTOCOL "application/pgp-signature" :BOUNDARY  
"=-N3IPeK0F6Rq/H013D1bn")
   NIL
   NIL)
  (:TEXT :PLAIN (:CHARSET "us-ascii") NIL NIL :7BIT 165 4 NIL NIL NIL)
  :MIXED
  (:BOUNDARY "===============1035646608==")
  NIL
  NIL)

And the copied message in the Maildir-folder:

(((:TEXT :PLAIN NIL NIL NIL :QUOTED-PRINTABLE 1638 45 NIL NIL NIL)
   (:APPLICATION :PGP-SIGNATURE (:NAME "signature.asc") NIL NIL NIL  
205 7 NIL NIL NIL)
   :SIGNED
   (:BOUNDARY "=-N3IPeK0F6Rq/H013D1bn" :PROTOCOL "application/pgp- 
signature" :MICALG "pgp-sha1")
   NIL
   NIL)
  (:TEXT :PLAIN (:CHARSET "us-ascii") NIL NIL :7BIT 165 4 NIL NIL NIL)
  :MIXED
  (:BOUNDARY "===============1035646608==")
  NIL
  NIL)

There seem to be some differences here too. As of now - mel-base  
doesn't get the charset "us-ascii" of the first text/plain part. This  
is ok since charset us-ascii is implicitely us-ascii. The mail-file  
doesn't specify us-ascii on that part - its the IMAP server that adds  
it to the bodystructure to make it explicit. I could do so too - but  
it is not against the spec to let it implicit. I also fixed a bug   
with computation of octet and line-counts of the parts. I always  
counted the last empty line before the closing boundary - this is  
actually not part of the part (pardon the pun ;-) ).

I've built a new release of mel-base (0.9-0) uploaded it to my server  
and registered it at CLiki (so that things like ASDF-INSTALL work).  
You can just try it - I've reimplemented the computation of the  
bodystructure so that it now gives the exact same result as my IMAP  
server. I've also reworked the character coding infrastructure (now  
using bivalent streams and flexi-streams were appropriate).

To see whats wrong in your case, I would need a bit more information.  
Can you send the result of COMPUTE-BODYSTRUCTURE for IMAP and Maildir?

ciao,
Jochen

--
Jochen Schmidt
js at codeartist.org
http://www.codeartist.org




>
> (defun check-email ()
>   (let* ((folder (mel:make-imaps-folder :host
> "imap.gmail.com" :username "XXXXX" :password "XXXXX"))
> 	 (tmp-folder (mel:make-maildir-folder "/tmp/test-maildir/" :if-does-
> not-exist :create)))
>     (mel:move-folder folder tmp-folder)
>     (mel:map-messages
>      (lambda (m)
>        (let ((body (find-plain-text m)))
> 	 (format t "~a~%" body)))
>      tmp-folder)))
>
>
>
> _______________________________________________
> mel-base-devel mailing list
> mel-base-devel at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/mel-base-devel

Jochen Schmidt
js at codeartist.org
http://www.codeartist.org