Fwd: [stefil-devel] First impressions after porting a small test suite from RT to Stefil

[Attila Lendvai]

> Here are a few suggestions for Stefil. If you think they're
> worthwhile, I'll prepare proper patches. However, I suspect there
> might be better, more idiomatic solutions for some of these problems
> within the existing framework that I'm unaware of.
>
>  ;; example: (returns (string-size-in-octets "abc" :encoding :ascii) 3 3)
>  (defmacro returns (form &rest values)
>    "Asserts, through EQUALP, that FORM returns VALUES."
>    `(is (equalp (multiple-value-list ,form) (list , at values))))


although we've never needed it, this looks generally useful.


>  (defun fail (control-string &rest arguments)
>    (stefil::record-failure 'stefil::failed-assertion
>                            :format-control control-string
>                            :format-arguments arguments))
>
>  (defun expected (expected &key got)
>    (fail "expected ~A, got ~A instead" expected got))


i'm not sure about these. they look useful, but i'm a bit reluctant in
growing the api too much. but...


> Here's an example of how I use EXPECTED/FAIL:
>
>  (deftest dec.ascii.2 ()
>    (handler-case
>        (octets-to-string (ub8v 97 128 99) :encoding :ascii :errorp t)
>      (character-decoding-error (c)
>        (is (equalp #(128) (character-decoding-error-octets c)))
>        (is (eql 1 (character-coding-error-position c)))
>        (is (eq :ascii (character-coding-error-encoding c))))
>      (:no-error (result)
>        (expected 'character-decoding-error :got result))))


OTOH, this looks like a good reason for them and it would be hard to
do this using IS.


> Another idea would be to use SIGNALS:
>
>   (let ((c (signals (octets-to-string ...))))
>     ;; test C
>     )
>
> However, it seems to me that this would introduce bogus failures
> and/or errors when SIGNALS returns NIL that would make debugging more
> difficult. Any suggestions?


(let ((c (signals (octets-to-string ...))))
 (when c
   ;; test c
 ))

which doesn't look too good.


> Since babel-tests was using RT, it had lots of small tests like this:
> (deftest foo form retval1 retval2 ...) so I found myself doing a lot
> of (deftest foo () (is (equalp form retval1))) and worse for multiple
> return values. Before writing the RETURNS macro I wrote a DEFSTEST
> macro that works like RT's:
>
>  (defmacro defstest (name form &body return-values)
>    "Similar to RT's DEFTEST."
>    `(deftest ,name ()
>       (returns ,form ,@(mapcar (lambda (x) `',x) return-values))))
>
> I like it because I'm used to RT and it made porting most of these
> small tests a simple matter of replacing DEFTEST with DEFSTEST but I'm
> not sure how useful it'd be in general.


i think it's trivial enough to let the users do it for themselves. we
have several different deftest variants in the various libs anyway.


> There a few itches I'd like to scratch before porting other test
> suites I maintain. Here's my TODO list, let me know if you have any
> objections or better alternatives to these features:
>
>  * make FUNCALL-TEST-WITH-FEEDBACK-MESSAGE print a list of failed
> tests (perhaps along with descriptions of the failed assertions). This
> could be useful for bug reports, buildbots, and other non-interactive
> use cases.


sounds good! the original reason for that function was that asdf does
not return anything from the operations, so you can't communicate back
the results normally when running an 'asdf:test-op.


>  * implement a functional interface for marking expected failures.
> E.g.: #+lispworks (expected-failure
> 'some-test-lispworks-does-not-like).


i think this would be non-trivial (iow, fragile) to implement in the
current setup and as speed does not matter here, maybe i'd simply add
a

(defmacro with-expected-failures* (expression &body body)
 "Any failure inside the dynamic extent of this block is registered
as an expected failure when the expression evaluates to T."
 `(bind ((*failures-and-errors-are-expected* (if ,expression
                                                 #t

*failures-and-errors-are-expected*)))
    , at body))

(with-expected-failures* (featurep :lispworks)
 ...)


>  * implement a way of running tests through EVAL, instead of
> compiling. We use this in cffi-tests to test whether removing
> compiler-macros introduces bugs and stuff like that.


i'm open for this.


>  * minor thing: export STEFIL::ROOT-SUITE so that one can write
> (deftest (foo :in root-suite)).


done.


> Finally, reducing the dependencies to external libraries would help
> with lobbying for porting cffi-tests (and others) to Stefil. Would
> that be in any way acceptable for you guys?


i got rid of :defclass-star and i'm open for getting rid of :iterate
(although i always regret that later... :)

but alexandria and bind are pretty much by-default dependencies for us nowadays.


> Thanks for Stefil and sorry for the long email.


thanks for being the first serious external user with actual feedback! :)

--
 attila