[stefil-devel] First impressions after porting a small test suite from RT to Stefil

[Luís Oliveira]

Hello,

I've ported Babel's small test suite from RT to Stefil and
consequently I have some fixes to contribute, some questions and some
suggestions.

I've pushed the small, trivial fixes to
<http://common-lisp.net/~loliveira/darcs/stefil/>. I suppose having
*test-progress-print-right-margin* default to 80 might be
controversial with you guys. I'd argue that that's the default width
for a lot of applications (xterm, emacs, etc) but the truth is that
I'm an 80-column freak, really. Feel free to ignore the patches you
disagree with.

Here are a few suggestions for Stefil. If you think they're
worthwhile, I'll prepare proper patches. However, I suspect there
might be better, more idiomatic solutions for some of these problems
within the existing framework that I'm unaware of.

  ;; example: (returns (string-size-in-octets "abc" :encoding :ascii) 3 3)
  (defmacro returns (form &rest values)
    "Asserts, through EQUALP, that FORM returns VALUES."
    `(is (equalp (multiple-value-list ,form) (list , at values))))

  (defun fail (control-string &rest arguments)
    (stefil::record-failure 'stefil::failed-assertion
                            :format-control control-string
                            :format-arguments arguments))

  (defun expected (expected &key got)
    (fail "expected ~A, got ~A instead" expected got))

Here's an example of how I use EXPECTED/FAIL:

  (deftest dec.ascii.2 ()
    (handler-case
        (octets-to-string (ub8v 97 128 99) :encoding :ascii :errorp t)
      (character-decoding-error (c)
        (is (equalp #(128) (character-decoding-error-octets c)))
        (is (eql 1 (character-coding-error-position c)))
        (is (eq :ascii (character-coding-error-encoding c))))
      (:no-error (result)
        (expected 'character-decoding-error :got result))))

Another idea would be to use SIGNALS:

   (let ((c (signals (octets-to-string ...))))
     ;; test C
     )

However, it seems to me that this would introduce bogus failures
and/or errors when SIGNALS returns NIL that would make debugging more
difficult. Any suggestions?

Since babel-tests was using RT, it had lots of small tests like this:
(deftest foo form retval1 retval2 ...) so I found myself doing a lot
of (deftest foo () (is (equalp form retval1))) and worse for multiple
return values. Before writing the RETURNS macro I wrote a DEFSTEST
macro that works like RT's:

  (defmacro defstest (name form &body return-values)
    "Similar to RT's DEFTEST."
    `(deftest ,name ()
       (returns ,form ,@(mapcar (lambda (x) `',x) return-values))))

I like it because I'm used to RT and it made porting most of these
small tests a simple matter of replacing DEFTEST with DEFSTEST but I'm
not sure how useful it'd be in general.

There a few itches I'd like to scratch before porting other test
suites I maintain. Here's my TODO list, let me know if you have any
objections or better alternatives to these features:

  * make FUNCALL-TEST-WITH-FEEDBACK-MESSAGE print a list of failed
tests (perhaps along with descriptions of the failed assertions). This
could be useful for bug reports, buildbots, and other non-interactive
use cases.

  * implement a functional interface for marking expected failures.
E.g.: #+lispworks (expected-failure
'some-test-lispworks-does-not-like).

  * implement a way of running tests through EVAL, instead of
compiling. We use this in cffi-tests to test whether removing
compiler-macros introduces bugs and stuff like that.

  * minor thing: export STEFIL::ROOT-SUITE so that one can write
(deftest (foo :in root-suite)).

Finally, reducing the dependencies to external libraries would help
with lobbying for porting cffi-tests (and others) to Stefil. Would
that be in any way acceptable for you guys?

Thanks for Stefil and sorry for the long email.

-- 
Luís Oliveira
http://student.dei.uc.pt/~lmoliv/