<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Yes, this kind of thing is essential if you want to<br>
design a system that stays up.<br>
<br>
I have another fun thing along these lines. When<br>
you start a request that comes into the server,<br>
you assign it an integer. As the handling of<br>
the request reaches transaction boundaries<br>
(transactions on an underlying database<br>
management system), you decrease the<br>
counter by one and keep passing it along<br>
the chain of handling. (The chain might<br>
include sending messages from one<br>
component to another, and getting replies,<br>
and so on. For now I'm assuming that<br>
request handling only has one thread.)<br>
<br>
Whenever the count reaches zero, the<br>
component that is currently handling<br>
the request is killed. The idea is<br>
to make sure that you test "all the<br>
places" where a crash might happen;<br>
that is, "all" with respect to the database<br>
state.<br>
<br>
This tool is testing the system under a<br>
certain set of assumptions. It's assuming<br>
that the DBMS does what it's told to do.<br>
It's testing "stop" failures. It works best<br>
when the only side-effects are to the<br>
database system. If there are other<br>
side effects, e.g. the components<br>
get things into their cache that stay<br>
there and are used in subsequent<br>
requests, then you are less sure that<br>
you are testing out all possible paths.<br>
<br>
I like to call these things "failure injection"<br>
(I didn't invent that term). We have other<br>
failure injection stuff all over our system.<br>
The "transaction ticking time bomb" one<br>
is just my favorite.<br>
<br>
Using a random tool might not find all of<br>
these states. Random tools are great,<br>
but there are other useful test tools, too.<br>
In general, trying to "test all possible<br>
circumstances" is very, very hard; you<br>
never know what particular input<br>
values might be the ones that cause<br>
a problem, and then you have to<br>
worry about variable A having<br>
certain values while variable B<br>
has certain other values, leading<br>
to a combinatorial explosion of<br>
circumstances to test.<br>
<br>
In my opinion, one of the best ways<br>
to deal with this problem is by<br>
having experienced Q/A people<br>
who have a knack for guessing<br>
what cases ought to be tested.<br>
<br>
There's a lot I can say about this,<br>
but the main thing I'll say is<br>
that this is one of the reasons I have<br>
doubts about the wisdom of the<br>
methodology used at Facebook,<br>
in which there is no Q/A department,<br>
and programmers are expected to do<br>
their own Q/A. That's just one of<br>
the reasons.<br>
<br>
It helps that at Facebook, you can roll<br>
out a new feature to a very small subset<br>
of users, and un-install it quickly if<br>
it's causing a problem, and usually if<br>
it doesn't work, that's not very important<br>
when there's only a very small number<br>
of early adopters. Things generally don't work that<br>
way with an airline reservation system.<br>
The methodologies that are suited<br>
for one situation are not necessarily<br>
those suitable for another.<br>
<br>
-- Dan<br>
<br>
<br>
<br>
<br>
Peter Seibel wrote:
<blockquote
cite="mid:AANLkTimTcNvnWB-9HXNWT4cDoMbxc2ADpW_u0Bm8zPdM@mail.gmail.com"
type="cite">
<pre wrap="">Presumably this kind of thing is the reason for the Chaos Monkey:
<a class="moz-txt-link-freetext" href="http://www.readwriteweb.com/cloud/2010/12/chaos-monkey-how-netflix-uses.php">http://www.readwriteweb.com/cloud/2010/12/chaos-monkey-how-netflix-uses.php</a>
-Peter
On Mon, Dec 20, 2010 at 6:37 PM, Scott L. Burson <a class="moz-txt-link-rfc2396E" href="mailto:Scott@sympoiesis.com"><Scott@sympoiesis.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On Fri, Dec 17, 2010 at 11:16 AM, Ryan Davis <a class="moz-txt-link-rfc2396E" href="mailto:ryan@acceleration.net"><ryan@acceleration.net></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">We do something like this. For lisp websites my company makes, we have
a password-protected admin section with some light UI to help us manage
the site (turn logging levels up/down, clear caches, etc), and one of
those tools is a "evaluate this code in the running lisp" textarea, with
a dropdown to select what package it runs in. This is very rarely used
to patch the site in emergency situations or for trivial changes where
we don't want to bring down the site. This has bit us a few times,
where we fixed a small bug directly in the running lisp and then forgot
to publish the new code and had mystery regressions when the lisp
process was restarted.
</pre>
</blockquote>
<pre wrap="">A really funny cautionary tale about this sort of thing:
<a class="moz-txt-link-freetext" href="http://thedailywtf.com/Articles/Designed-For-Reliability.aspx">http://thedailywtf.com/Articles/Designed-For-Reliability.aspx</a>
-- Scott
_______________________________________________
pro mailing list
<a class="moz-txt-link-abbreviated" href="mailto:pro@common-lisp.net">pro@common-lisp.net</a>
<a class="moz-txt-link-freetext" href="http://common-lisp.net/cgi-bin/mailman/listinfo/pro">http://common-lisp.net/cgi-bin/mailman/listinfo/pro</a>
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
</body>
</html>