Merge the 'rules' branch.

Give the first alpha a code name.

This branch mostly gets rid of all the approval oriented handlers in favor of
a chain-of-rules based approach.  This will be much more powerful and
extensible, allowing rule definition by plugin and chain creation via web
page.

When a message is processed by the incoming queue, it gets sent through a
chain of rules.  The starting chain is defined on the mailing list object, and
there is a built-in default starting chain, called 'built-in'.  Each chain is
made up of links, which describe a rule and an action, along with possibly
some other information.  Actions allow processing to take a detour through
another chain, jump to another chain, stop processing, run a function, etc.

The built-in chain essentially implements the original early part of the
handler pipeline.  If a message makes it through the built-in chain, it gets
sent to the prep queue, where the message is decorated and such before sending
out to the list membership.  The 'accept' chain is what moves the message into
the prep queue.

There are also 'hold', 'discard', and 'reject' chains, which do what you would
expect them to.  There are lots of built-in rules, implementing everything
from the old emergency handler to new handlers such as one not allowing empty
subject headers.

IMember grows an is_moderated attribute.

The 'adminapproved' metadata key is renamed 'moderator_approved'.

Fix some bogus uses of noreply_address to no_reply_address.

Stash an 'original_size' attribute on the message after parsing its plain
text.  This can be used later to ensure the original message does not exceed a
specified size without have to flatten the message again.

The KNOWN_SPAMMERS global variable is replaced with HEADER_MATCHES.  The
mailing list's header_filter_rules variable is replaced with header_matches
which has the same semantics as HEADER_MATCHES, but is list-specific.

DEFAULT_MAIL_COMMANDS_MAX_LINES -> EMAIL_COMMANDS_MAX_LINES.

Update smtplistener.py to be much better, to use maildir format instead of
mbox format, to respond to RSET commands by clearing the maildir, and by
silencing annoying asyncore error messages.

Extend the doctest runner so that it will run .txt files in any docs
subdirectory in the code tree.

Add plugable keys 'mailman.mta' and 'mailman.rules'.  The latter may have only
one setting while the former is extensible.

There are lots of doctests which should give all the gory details.

Mailman/Post.py -> Mailman/inject.py and the command line usage of this module
is removed.

SQLALCHEMY_ECHO, which was unused, is removed.

Backport the ability to specify additional footer interpolation variables by
the message metadata 'decoration-data' key.

can_acknowledge() defines whether a message can be responded to by the email
robot.

Simplify the implementation of _reset() based on Storm fixes.  Be able to
handle lists in Storm values.

Do some reorganization.

14 files changed, 1255 insertions, 0 deletions

diff --git a/Mailman/rules/docs/administrivia.txt b/Mailman/rules/docs/administrivia.txt
new file mode 100644
index 000000000..de802fa85
--- /dev/null
+++ b/Mailman/rules/docs/administrivia.txt
@@ -0,0 +1,100 @@
+Administrivia
+=============
+
+The 'administrivia' rule matches when the message contains some common email
+commands in the Subject header or first few lines of the payload.  This is
+used to catch messages posted to the list which should have been sent to the
+-request robot address.
+
+    >>> from Mailman.configuration import config
+    >>> mlist = config.db.list_manager.create(u'_xtest@example.com')
+    >>> mlist.administrivia = True
+    >>> rule = config.rules['administrivia']
+    >>> rule.name
+    'administrivia'
+
+For example, if the Subject header contains the word 'unsubscribe', the rule
+matches.
+
+    >>> msg_1 = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... Subject: unsubscribe
+    ...
+    ... """)
+    >>> rule.check(mlist, msg_1, {})
+    True
+
+Similarly, if the body of the message contains the word 'subscribe' in the
+first few lines of text, the rule matches.
+
+    >>> msg_2 = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... Subject: I wish to join your list
+    ...
+    ... subscribe
+    ... """)
+    >>> rule.check(mlist, msg_2, {})
+    True
+
+In both cases, administrivia checking can be disabled.
+
+    >>> mlist.administrivia = False
+    >>> rule.check(mlist, msg_1, {})
+    False
+    >>> rule.check(mlist, msg_2, {})
+    False
+
+To make the administrivia heuristics a little more robust, the rule actually
+looks for a minimum and maximum number of arguments, so that it really does
+seem like a mis-addressed email command.  In this case, the 'confirm' command
+requires at least one argument.  We don't give that here so the rule will not
+match.
+
+    >>> mlist.administrivia = True
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... Subject: confirm
+    ...
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
+
+But a real 'confirm' message will match.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... Subject: confirm 12345
+    ...
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    True
+
+We don't show all the other possible email commands, but you get the idea.
+
+
+Non-administrivia
+-----------------
+
+Of course, messages that don't contain administrivia, don't match the rule.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... Subject: examine
+    ...
+    ... persuade
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
+
+Also, only text/plain parts are checked for administrivia, so any email
+commands in other content type subparts are ignored.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... Subject: some administrivia
+    ... Content-Type: text/x-special
+    ...
+    ... subscribe
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
diff --git a/Mailman/rules/docs/approve.txt b/Mailman/rules/docs/approve.txt
new file mode 100644
index 000000000..32367a76b
--- /dev/null
+++ b/Mailman/rules/docs/approve.txt
@@ -0,0 +1,473 @@
+Pre-approved postings
+=====================
+
+Messages can contain a pre-approval, which is used to bypass the message
+approval queue.  This has several use cases:
+
+- A list administrator can send an emergency message to the mailing list from
+  an unregistered address, say if they are away from their normal email.
+
+- An automated script can be programmed to send a message to an otherwise
+  moderated list.
+
+In order to support this, a mailing list can be given a 'moderator password'
+which is shared among all the administrators.
+
+    >>> from Mailman.configuration import config
+    >>> mlist = config.db.list_manager.create(u'_xtest@example.com')
+    >>> mlist.moderator_password = u'abcxyz'
+
+The 'approved' rule determines whether the message contains the proper
+approval or not.
+
+    >>> rule = config.rules['approved']
+    >>> rule.name
+    'approved'
+
+
+No approval
+-----------
+
+If the message has no Approve or Approved header, then the rule does not
+match.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ...
+    ... An important message.
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
+
+If the message has an Approve or Approved header with a value that does not
+match the moderator password, then the rule does not match.  However, the
+header is still removed.
+
+    >>> msg['Approve'] = u'12345'
+    >>> rule.check(mlist, msg, {})
+    False
+    >>> print msg['approve']
+    None
+
+    >>> del msg['approve']
+    >>> msg['Approved'] = u'12345'
+    >>> rule.check(mlist, msg, {})
+    False
+    >>> print msg['approved']
+    None
+
+    >>> del msg['approved']
+
+
+Using an approval header
+------------------------
+
+If the moderator password is given in an Approve header, then the rule
+matches, and the Approve header is stripped.
+
+    >>> msg['Approve'] = u'abcxyz'
+    >>> rule.check(mlist, msg, {})
+    True
+    >>> print msg['approve']
+    None
+
+Similarly, for the Approved header.
+
+    >>> msg['Approved'] = u'abcxyz'
+    >>> rule.check(mlist, msg, {})
+    True
+    >>> print msg['approved']
+    None
+
+
+Using a pseudo-header
+---------------------
+
+Different mail user agents have varying degrees to which they support custom
+headers like Approve and Approved.  For this reason, Mailman also supports
+using a 'pseudo-header', which is really just the first non-whitespace line in
+the payload of the message.  If this pseudo-header looks like a matching
+Approve or Approved header, the message is similarly allowed to pass.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ...
+    ... Approve: abcxyz
+    ... An important message.
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    True
+
+The pseudo-header is removed.
+
+    >>> print msg.as_string()
+    From: aperson@example.com
+    Content-Transfer-Encoding: 7bit
+    MIME-Version: 1.0
+    Content-Type: text/plain; charset="us-ascii"
+    <BLANKLINE>
+    An important message.
+    <BLANKLINE>
+
+Similarly for the Approved header.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ...
+    ... Approved: abcxyz
+    ... An important message.
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    True
+
+    >>> print msg.as_string()
+    From: aperson@example.com
+    Content-Transfer-Encoding: 7bit
+    MIME-Version: 1.0
+    Content-Type: text/plain; charset="us-ascii"
+    <BLANKLINE>
+    An important message.
+    <BLANKLINE>
+
+As before, a mismatch in the pseudo-header does not approve the message, but
+the pseudo-header line is still removed.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ...
+    ... Approve: 123456
+    ... An important message.
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
+
+    >>> print msg.as_string()
+    From: aperson@example.com
+    Content-Transfer-Encoding: 7bit
+    MIME-Version: 1.0
+    Content-Type: text/plain; charset="us-ascii"
+    <BLANKLINE>
+    An important message.
+    <BLANKLINE>
+
+Similarly for the Approved header.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ...
+    ... Approved: 123456
+    ... An important message.
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
+
+    >>> print msg.as_string()
+    From: aperson@example.com
+    Content-Transfer-Encoding: 7bit
+    MIME-Version: 1.0
+    Content-Type: text/plain; charset="us-ascii"
+    <BLANKLINE>
+    An important message.
+    <BLANKLINE>
+
+
+MIME multipart support
+----------------------
+
+Mailman searches for the pseudo-header as the first non-whitespace line in the
+first text/plain message part of the message.  This allows the feature to be
+used with MIME documents.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... MIME-Version: 1.0
+    ... Content-Type: multipart/mixed; boundary="AAA"
+    ...
+    ... --AAA
+    ... Content-Type: application/x-ignore
+    ...
+    ... Approve: 123456
+    ... The above line will be ignored.
+    ...
+    ... --AAA
+    ... Content-Type: text/plain
+    ...
+    ... Approve: abcxyz
+    ... An important message.
+    ... --AAA--
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    True
+
+Like before, the pseudo-header is removed, but only from the text parts.
+
+    >>> print msg.as_string()
+    From: aperson@example.com
+    MIME-Version: 1.0
+    Content-Type: multipart/mixed; boundary="AAA"
+    <BLANKLINE>
+    --AAA
+    Content-Type: application/x-ignore
+    <BLANKLINE>
+    Approve: 123456
+    The above line will be ignored.
+    <BLANKLINE>
+    --AAA
+    Content-Transfer-Encoding: 7bit
+    MIME-Version: 1.0
+    Content-Type: text/plain; charset="us-ascii"
+    <BLANKLINE>
+    An important message.
+    --AAA--
+    <BLANKLINE>
+
+The same goes for the Approved message.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... MIME-Version: 1.0
+    ... Content-Type: multipart/mixed; boundary="AAA"
+    ...
+    ... --AAA
+    ... Content-Type: application/x-ignore
+    ...
+    ... Approved: 123456
+    ... The above line will be ignored.
+    ...
+    ... --AAA
+    ... Content-Type: text/plain
+    ...
+    ... Approved: abcxyz
+    ... An important message.
+    ... --AAA--
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    True
+
+And the header is removed.
+
+    >>> print msg.as_string()
+    From: aperson@example.com
+    MIME-Version: 1.0
+    Content-Type: multipart/mixed; boundary="AAA"
+    <BLANKLINE>
+    --AAA
+    Content-Type: application/x-ignore
+    <BLANKLINE>
+    Approved: 123456
+    The above line will be ignored.
+    <BLANKLINE>
+    --AAA
+    Content-Transfer-Encoding: 7bit
+    MIME-Version: 1.0
+    Content-Type: text/plain; charset="us-ascii"
+    <BLANKLINE>
+    An important message.
+    --AAA--
+    <BLANKLINE>
+
+Here, the correct password is in the non-text/plain part, so it is ignored.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... MIME-Version: 1.0
+    ... Content-Type: multipart/mixed; boundary="AAA"
+    ...
+    ... --AAA
+    ... Content-Type: application/x-ignore
+    ...
+    ... Approve: abcxyz
+    ... The above line will be ignored.
+    ...
+    ... --AAA
+    ... Content-Type: text/plain
+    ...
+    ... Approve: 123456
+    ... An important message.
+    ... --AAA--
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
+
+And yet the pseudo-header is still stripped.
+
+    >>> print msg.as_string()
+    From: aperson@example.com
+    MIME-Version: 1.0
+    Content-Type: multipart/mixed; boundary="AAA"
+    <BLANKLINE>
+    --AAA
+    Content-Type: application/x-ignore
+    <BLANKLINE>
+    Approve: abcxyz
+    The above line will be ignored.
+    <BLANKLINE>
+    --AAA
+    Content-Transfer-Encoding: 7bit
+    MIME-Version: 1.0
+    Content-Type: text/plain; charset="us-ascii"
+    <BLANKLINE>
+    An important message.
+    --AAA--
+
+As before, the same goes for the Approved header.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... MIME-Version: 1.0
+    ... Content-Type: multipart/mixed; boundary="AAA"
+    ...
+    ... --AAA
+    ... Content-Type: application/x-ignore
+    ...
+    ... Approved: abcxyz
+    ... The above line will be ignored.
+    ...
+    ... --AAA
+    ... Content-Type: text/plain
+    ...
+    ... Approved: 123456
+    ... An important message.
+    ... --AAA--
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
+
+And the pseudo-header is removed.
+
+    >>> print msg.as_string()
+    From: aperson@example.com
+    MIME-Version: 1.0
+    Content-Type: multipart/mixed; boundary="AAA"
+    <BLANKLINE>
+    --AAA
+    Content-Type: application/x-ignore
+    <BLANKLINE>
+    Approved: abcxyz
+    The above line will be ignored.
+    <BLANKLINE>
+    --AAA
+    Content-Transfer-Encoding: 7bit
+    MIME-Version: 1.0
+    Content-Type: text/plain; charset="us-ascii"
+    <BLANKLINE>
+    An important message.
+    --AAA--
+
+
+Stripping text/html parts
+-------------------------
+
+Because some mail readers will include both a text/plain part and a text/html
+alternative, the 'approved' rule has to search the alternatives and strip
+anything that looks like an Approve or Approved headers.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... MIME-Version: 1.0
+    ... Content-Type: multipart/mixed; boundary="AAA"
+    ...
+    ... --AAA
+    ... Content-Type: text/html
+    ...
+    ... <html>
+    ... <head></head>
+    ... <body>
+    ... <b>Approved: abcxyz</b>
+    ... <p>The above line will be ignored.
+    ... </body>
+    ... </html>
+    ...
+    ... --AAA
+    ... Content-Type: text/plain
+    ...
+    ... Approved: abcxyz
+    ... An important message.
+    ... --AAA--
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    True
+
+And the header-like text in the text/html part was stripped.
+
+    >>> print msg.as_string()
+    From: aperson@example.com
+    MIME-Version: 1.0
+    Content-Type: multipart/mixed; boundary="AAA"
+    <BLANKLINE>
+    --AAA
+    Content-Transfer-Encoding: 7bit
+    MIME-Version: 1.0
+    Content-Type: text/html; charset="us-ascii"
+    <BLANKLINE>
+    <html>
+    <head></head>
+    <body>
+    <b></b>
+    <p>The above line will be ignored.
+    </body>
+    </html>
+    <BLANKLINE>
+    --AAA
+    Content-Transfer-Encoding: 7bit
+    MIME-Version: 1.0
+    Content-Type: text/plain; charset="us-ascii"
+    <BLANKLINE>
+    An important message.
+    --AAA--
+    <BLANKLINE>
+
+This is true even if the rule does not match.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... MIME-Version: 1.0
+    ... Content-Type: multipart/mixed; boundary="AAA"
+    ...
+    ... --AAA
+    ... Content-Type: text/html
+    ...
+    ... <html>
+    ... <head></head>
+    ... <body>
+    ... <b>Approve: 123456</b>
+    ... <p>The above line will be ignored.
+    ... </body>
+    ... </html>
+    ...
+    ... --AAA
+    ... Content-Type: text/plain
+    ...
+    ... Approve: 123456
+    ... An important message.
+    ... --AAA--
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
+
+    >>> print msg.as_string()
+    From: aperson@example.com
+    MIME-Version: 1.0
+    Content-Type: multipart/mixed; boundary="AAA"
+    <BLANKLINE>
+    --AAA
+    Content-Transfer-Encoding: 7bit
+    MIME-Version: 1.0
+    Content-Type: text/html; charset="us-ascii"
+    <BLANKLINE>
+    <html>
+    <head></head>
+    <body>
+    <b></b>
+    <p>The above line will be ignored.
+    </body>
+    </html>
+    <BLANKLINE>
+    --AAA
+    Content-Transfer-Encoding: 7bit
+    MIME-Version: 1.0
+    Content-Type: text/plain; charset="us-ascii"
+    <BLANKLINE>
+    An important message.
+    --AAA--
+    <BLANKLINE>
diff --git a/Mailman/rules/docs/emergency.txt b/Mailman/rules/docs/emergency.txt
new file mode 100644
index 000000000..aecbcb90d
--- /dev/null
+++ b/Mailman/rules/docs/emergency.txt
@@ -0,0 +1,74 @@
+Emergency
+=========
+
+When the mailing list has its emergency flag set, all messages posted to the
+list are held for moderator approval.
+
+    >>> from Mailman.app.lifecycle import create_list
+    >>> mlist = create_list(u'_xtest@example.com')
+    >>> mlist.web_page_url = u'http://www.example.com/'
+    >>> msg = message_from_string("""\
+    ... From: aperson@example.com
+    ... To: _xtest@example.com
+    ... Subject: My first post
+    ... Message-ID: <first>
+    ...
+    ... An important message.
+    ... """)
+
+The emergency rule is matched as part of the built-in chain.  The emergency
+rule matches if the flag is set on the mailing list.
+
+    >>> from Mailman.app.chains import process
+    >>> mlist.emergency = True
+    >>> process(mlist, msg, {}, 'built-in')
+
+There are two messages in the virgin queue.  The one addressed to the original
+sender will contain a token we can use to grab the held message out of the
+pending requests.
+
+    >>> from Mailman.queue import Switchboard
+    >>> virginq = Switchboard(config.VIRGINQUEUE_DIR)
+
+    >>> def get_held_message():
+    ...     import re
+    ...     qfiles = []
+    ...     for filebase in virginq.files:
+    ...         qmsg, qdata = virginq.dequeue(filebase)
+    ...         virginq.finish(filebase)
+    ...         qfiles.append(qmsg)
+    ...     from operator import itemgetter
+    ...     qfiles.sort(key=itemgetter('to'))
+    ...     cookie = None
+    ...     for line in qfiles[1].get_payload().splitlines():
+    ...         mo = re.search('confirm/[^/]+/(?P<cookie>.*)$', line)
+    ...         if mo:
+    ...             cookie = mo.group('cookie')
+    ...             break
+    ...     assert cookie is not None, 'No confirmation token found'
+    ...     data = config.db.pendings.confirm(cookie)
+    ...     requestdb = config.db.requests.get_list_requests(mlist)
+    ...     rkey, rdata = requestdb.get_request(data['id'])
+    ...     return config.db.message_store.get_message_by_id(
+    ...         rdata['_mod_message_id'])
+
+    >>> msg = get_held_message()
+    >>> print msg.as_string()
+    From: aperson@example.com
+    To: _xtest@example.com
+    Subject: My first post
+    Message-ID: <first>
+    X-Mailman-Rule-Hits: emergency
+    X-Mailman-Rule-Misses: approved
+    X-Message-ID-Hash: RXJU4JL6N2OUN3OYMXXPPSCR7P7JE2BW
+    <BLANKLINE>
+    An important message.
+    <BLANKLINE>
+
+However, if the message metadata has a 'moderator_approved' key set, then even
+if the mailing list has its emergency flag set, the message still goes through
+to the membership.
+
+    >>> process(mlist, msg, dict(moderator_approved=True), 'built-in')
+    >>> len(virginq.files)
+    0
diff --git a/Mailman/rules/docs/header-matching.txt b/Mailman/rules/docs/header-matching.txt
new file mode 100644
index 000000000..fbd0ff65f
--- /dev/null
+++ b/Mailman/rules/docs/header-matching.txt
@@ -0,0 +1,145 @@
+Header matching
+===============
+
+Mailman can do pattern based header matching during its normal rule
+processing.  There is a set of site-wide default header matchines specified in
+the configuaration file under the HEADER_MATCHES variable.
+
+    >>> from Mailman.app.lifecycle import create_list
+    >>> mlist = create_list(u'_xtest@example.com')
+
+Because the default HEADER_MATCHES variable is empty when the configuration
+file is read, we'll just extend the current header matching chain with a
+pattern that matches 4 or more stars, discarding the message if it hits.
+
+    >>> from Mailman.configuration import config
+    >>> chain = config.chains['header-match']
+    >>> chain.extend('x-spam-score', '[*]{4,}', 'discard')
+
+First, if the message has no X-Spam-Score header, the message passes through
+the chain untouched (i.e. no disposition).
+
+    >>> msg = message_from_string("""\
+    ... From: aperson@example.com
+    ... To: _xtest@example.com
+    ... Subject: Not spam
+    ... Message-ID: <one>
+    ...
+    ... This is a message.
+    ... """)
+
+    >>> from Mailman.app.chains import process
+
+Pass through is seen as nothing being in the log file after processing.
+
+    # XXX This checks the vette log file because there is no other evidence
+    # that this chain has done anything.
+    >>> import os
+    >>> fp = open(os.path.join(config.LOG_DIR, 'vette'))
+    >>> fp.seek(0, 2)
+    >>> file_pos = fp.tell()
+    >>> process(mlist, msg, {}, 'header-match')
+    >>> fp.seek(file_pos)
+    >>> print 'LOG:', fp.read()
+    LOG:
+    <BLANKLINE>
+
+Now, if the header exists but does not match, then it also passes through
+untouched.
+
+    >>> msg['X-Spam-Score'] = '***'
+    >>> del msg['subject']
+    >>> msg['Subject'] = 'This is almost spam'
+    >>> del msg['message-id']
+    >>> msg['Message-ID'] = '<two>'
+    >>> file_pos = fp.tell()
+    >>> process(mlist, msg, {}, 'header-match')
+    >>> fp.seek(file_pos)
+    >>> print 'LOG:', fp.read()
+    LOG:
+    <BLANKLINE>
+
+But now if the header matches, then the message gets discarded.
+
+    >>> del msg['x-spam-score']
+    >>> msg['X-Spam-Score'] = '****'
+    >>> del msg['subject']
+    >>> msg['Subject'] = 'This is spam, but barely'
+    >>> del msg['message-id']
+    >>> msg['Message-ID'] = '<three>'
+    >>> file_pos = fp.tell()
+    >>> process(mlist, msg, {}, 'header-match')
+    >>> fp.seek(file_pos)
+    >>> print 'LOG:', fp.read()
+    LOG: ... DISCARD: <three>
+    <BLANKLINE>
+
+For kicks, let's show a message that's really spammy.
+
+    >>> del msg['x-spam-score']
+    >>> msg['X-Spam-Score'] = '**********'
+    >>> del msg['subject']
+    >>> msg['Subject'] = 'This is really spammy'
+    >>> del msg['message-id']
+    >>> msg['Message-ID'] = '<four>'
+    >>> file_pos = fp.tell()
+    >>> process(mlist, msg, {}, 'header-match')
+    >>> fp.seek(file_pos)
+    >>> print 'LOG:', fp.read()
+    LOG: ... DISCARD: <four>
+    <BLANKLINE>
+
+Flush out the extended header matching rules.
+
+    >>> chain.flush()
+
+
+List-specific header matching
+-----------------------------
+
+Each mailing list can also be configured with a set of header matching regular
+expression rules.  These are used to impose list-specific header filtering
+with the same semantics as the global `HEADER_MATCHES` variable.
+
+The list administrator wants to match not on four stars, but on three plus
+signs, but only for the current mailing list.
+
+    >>> mlist.header_matches = [('x-spam-score', '[+]{3,}', 'discard')]
+
+A message with a spam score of two pluses does not match.
+
+    >>> del msg['x-spam-score']
+    >>> msg['X-Spam-Score'] = '++'
+    >>> del msg['message-id']
+    >>> msg['Message-ID'] = '<five>'
+    >>> file_pos = fp.tell()
+    >>> process(mlist, msg, {}, 'header-match')
+    >>> fp.seek(file_pos)
+    >>> print 'LOG:', fp.read()
+    LOG:
+
+A message with a spam score of three pluses does match.
+
+    >>> del msg['x-spam-score']
+    >>> msg['X-Spam-Score'] = '+++'
+    >>> del msg['message-id']
+    >>> msg['Message-ID'] = '<six>'
+    >>> file_pos = fp.tell()
+    >>> process(mlist, msg, {}, 'header-match')
+    >>> fp.seek(file_pos)
+    >>> print 'LOG:', fp.read()
+    LOG: ... DISCARD: <six>
+    <BLANKLINE>
+
+As does a message with a spam score of four pluses.
+
+    >>> del msg['x-spam-score']
+    >>> msg['X-Spam-Score'] = '+++'
+    >>> del msg['message-id']
+    >>> msg['Message-ID'] = '<seven>'
+    >>> file_pos = fp.tell()
+    >>> process(mlist, msg, {}, 'header-match')
+    >>> fp.seek(file_pos)
+    >>> print 'LOG:', fp.read()
+    LOG: ... DISCARD: <seven>
+    <BLANKLINE>
diff --git a/Mailman/rules/docs/implicit-dest.txt b/Mailman/rules/docs/implicit-dest.txt
new file mode 100644
index 000000000..5a7f06c0c
--- /dev/null
+++ b/Mailman/rules/docs/implicit-dest.txt
@@ -0,0 +1,76 @@
+Implicit destination
+====================
+
+The 'implicit-dest' rule matches when the mailing list's posting address is
+not explicitly mentioned in the set of message recipients.
+
+    >>> from Mailman.configuration import config
+    >>> mlist = config.db.list_manager.create(u'_xtest@example.com')
+    >>> rule = config.rules['implicit-dest']
+    >>> rule.name
+    'implicit-dest'
+
+This rule matches messages that have implicit destination, meaning that the
+mailing list's posting address isn't included in the explicit recipients.
+
+    >>> mlist.require_explicit_destination = True
+    >>> mlist.acceptable_aliases = u''
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.org
+    ... Subject: An implicit message
+    ...
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    True
+
+You can disable implicit destination checks for the mailing list.
+
+    >>> mlist.require_explicit_destination = False
+    >>> rule.check(mlist, msg, {})
+    False
+
+Even with some recipients, if the posting address is not included, the rule
+will match.
+
+    >>> mlist.require_explicit_destination = True
+    >>> msg['To'] = 'myfriend@example.com'
+    >>> rule.check(mlist, msg, {})
+    True
+
+Add the posting address as a recipient and the rule will no longer match.
+
+    >>> msg['Cc'] = '_xtest@example.com'
+    >>> rule.check(mlist, msg, {})
+    False
+
+Alternatively, if one of the acceptable aliases is in the recipients list,
+then the rule will not match.
+
+    >>> del msg['cc']
+    >>> rule.check(mlist, msg, {})
+    True
+    >>> mlist.acceptable_aliases = u'myfriend@example.com'
+    >>> rule.check(mlist, msg, {})
+    False
+
+A message gated from NNTP will obviously have an implicit destination.  Such
+gated messages will not be held for implicit destination because it's assumed
+that Mailman pulled it from the appropriate news group.
+
+    >>> rule.check(mlist, msg, dict(fromusenet=True))
+    False
+
+
+Alias patterns
+--------------
+
+It's also possible to specify an alias pattern, i.e. a regular expression to
+match against the recipients.  For example, we can say that if there is a
+recipient in the example.net domain, then the rule does not match.
+
+    >>> mlist.acceptable_aliases = u'^.*@example.net'
+    >>> rule.check(mlist, msg, {})
+    True
+    >>> msg['To'] = 'you@example.net'
+    >>> rule.check(mlist, msg, {})
+    False
diff --git a/Mailman/rules/docs/loop.txt b/Mailman/rules/docs/loop.txt
new file mode 100644
index 000000000..8fe86cf45
--- /dev/null
+++ b/Mailman/rules/docs/loop.txt
@@ -0,0 +1,49 @@
+Posting loops
+=============
+
+To avoid a posting loop, Mailman has a rule to check for the existence of an
+X-BeenThere header with the value of the list's posting address.
+
+    >>> from Mailman.configuration import config
+    >>> mlist = config.db.list_manager.create(u'_xtest@example.com')
+    >>> rule = config.rules['loop']
+    >>> rule.name
+    'loop'
+
+The header could be missing, in which case the rule does not match.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ...
+    ... An important message.
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
+
+The header could be present, but not match the list's posting address.
+
+    >>> msg['X-BeenThere'] = u'not-this-list@example.com'
+    >>> rule.check(mlist, msg, {})
+    False
+
+If the header is present and does match the posting address, the rule
+matches.
+
+    >>> del msg['x-beenthere']
+    >>> msg['X-BeenThere'] = mlist.posting_address
+    >>> rule.check(mlist, msg, {})
+    True
+
+Even if there are multiple X-BeenThere headers, as long as one with the
+posting address exists, the rule matches.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... X-BeenThere: not-this-list@example.com
+    ... X-BeenThere: _xtest@example.com
+    ... X-BeenThere: foo@example.com
+    ...
+    ... An important message.
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    True
diff --git a/Mailman/rules/docs/max-size.txt b/Mailman/rules/docs/max-size.txt
new file mode 100644
index 000000000..0d64b0cf7
--- /dev/null
+++ b/Mailman/rules/docs/max-size.txt
@@ -0,0 +1,40 @@
+Message size
+============
+
+The 'message-size' rule matches when the posted message is bigger than a
+specified maximum.  Generally this is used to prevent huge attachments from
+getting posted to the list.  This value is calculated in terms of KB (1024
+bytes).
+
+    >>> from Mailman.configuration import config
+    >>> mlist = config.db.list_manager.create(u'_xtest@example.com')
+    >>> rule = config.rules['max-size']
+    >>> rule.name
+    'max-size'
+
+For example, setting the maximum message size to 1 means that any message
+bigger than that will match the rule.
+
+    >>> mlist.max_message_size = 1 # 1024 bytes
+    >>> one_line = u'x' * 79
+    >>> big_body = u'\n'.join([one_line] * 15)
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... To: _xtest@example.com
+    ...
+    ... """ + big_body)
+    >>> rule.check(mlist, msg, {})
+    True
+
+Setting the maximum message size to zero means no size check is performed.
+
+    >>> mlist.max_message_size = 0
+    >>> rule.check(mlist, msg, {})
+    False
+
+Of course, if the maximum size is larger than the message's size, then it's
+still okay.
+
+    >>> mlist.max_message_size = msg.original_size/1024.0 + 1
+    >>> rule.check(mlist, msg, {})
+    False
diff --git a/Mailman/rules/docs/moderation.txt b/Mailman/rules/docs/moderation.txt
new file mode 100644
index 000000000..cab8f20d3
--- /dev/null
+++ b/Mailman/rules/docs/moderation.txt
@@ -0,0 +1,70 @@
+Member moderation
+=================
+
+Each user has a moderation flag.  When set, and the list is set to moderate
+postings, then only members with a cleared moderation flag will be able to
+email the list without having those messages be held for approval.  The
+'moderation' rule determines whether the message should be moderated or not.
+
+    >>> from Mailman.configuration import config
+    >>> mlist = config.db.list_manager.create(u'_xtest@example.com')
+    >>> rule = config.rules['moderation']
+    >>> rule.name
+    'moderation'
+
+In the simplest case, the sender is not a member of the mailing list, so the
+moderation rule can't match.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.org
+    ... To: _xtest@example.com
+    ... Subject: A posted message
+    ...
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
+
+Let's add the message author as a non-moderated member.
+
+    >>> user = config.db.user_manager.create_user(
+    ...     u'aperson@example.org', u'Anne Person')
+    >>> address = list(user.addresses)[0]
+    >>> from Mailman.interfaces import MemberRole
+    >>> member = address.subscribe(mlist, MemberRole.member)
+    >>> member.is_moderated
+    False
+    >>> rule.check(mlist, msg, {})
+    False
+
+Once the member's moderation flag is set though, the rule matches.
+
+    >>> member.is_moderated = True
+    >>> rule.check(mlist, msg, {})
+    True
+
+
+Non-members
+-----------
+
+There is another, related rule for matching non-members, which simply matches
+if the sender is /not/ a member of the mailing list.
+
+    >>> rule = config.rules['non-member']
+    >>> rule.name
+    'non-member'
+
+If the sender is a member of this mailing list, the rule does not match.
+
+    >>> rule.check(mlist, msg, {})
+    False
+
+But if the sender is not a member of this mailing list, the rule matches.
+
+    >>> msg = message_from_string(u"""\
+    ... From: bperson@example.org
+    ... To: _xtest@example.com
+    ... Subject: A posted message
+    ...
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    True
diff --git a/Mailman/rules/docs/news-moderation.txt b/Mailman/rules/docs/news-moderation.txt
new file mode 100644
index 000000000..f32919ce5
--- /dev/null
+++ b/Mailman/rules/docs/news-moderation.txt
@@ -0,0 +1,37 @@
+Newsgroup moderation
+====================
+
+The 'news-moderation' rule matches all messages posted to mailing lists that
+gateway to a moderated newsgroup.  The reason for this is that such messages
+must get forwarded on to the newsgroup moderator.  From there it will get
+posted to the newsgroup, and from there, gated to the mailing list.  It's a
+circuitous route, but it works nonetheless by holding all messages posted
+directly to the mailing list.
+
+    >>> from Mailman.configuration import config
+    >>> mlist = config.db.list_manager.create(u'_xtest@example.com')
+    >>> rule = config.rules['news-moderation']
+    >>> rule.name
+    'news-moderation'
+
+Set the list configuraiton variable to enable newsgroup moderation.
+
+    >>> from Mailman.interfaces import NewsModeration
+    >>> mlist.news_moderation = NewsModeration.moderated
+
+And now all messages will match the rule.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.org
+    ... Subject: An announcment
+    ...
+    ... Great things are happening.
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    True
+
+When moderation is turned off, the rule does not match.
+
+    >>> mlist.news_moderation = NewsModeration.none
+    >>> rule.check(mlist, msg, {})
+    False
diff --git a/Mailman/rules/docs/no-subject.txt b/Mailman/rules/docs/no-subject.txt
new file mode 100644
index 000000000..3627ac03f
--- /dev/null
+++ b/Mailman/rules/docs/no-subject.txt
@@ -0,0 +1,34 @@
+No Subject header
+=================
+
+This rule matches if the message has no Subject header, or if the header is
+the empty string when stripped.
+
+    >>> from Mailman.configuration import config
+    >>> mlist = config.db.list_manager.create(u'_xtest@example.com')
+    >>> rule = config.rules['no-subject']
+    >>> rule.name
+    'no-subject'
+
+A message with a non-empty subject does not match the rule.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.org
+    ... To: _xtest@example.com
+    ... Subject: A posted message
+    ...
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
+
+Delete the Subject header and the rule matches.
+
+    >>> del msg['subject']
+    >>> rule.check(mlist, msg, {})
+    True
+
+Even a Subject header with only whitespace still matches the rule.
+
+    >>> msg['Subject'] = u'    '
+    >>> rule.check(mlist, msg, {})
+    True
diff --git a/Mailman/rules/docs/recipients.txt b/Mailman/rules/docs/recipients.txt
new file mode 100644
index 000000000..21d04b8ae
--- /dev/null
+++ b/Mailman/rules/docs/recipients.txt
@@ -0,0 +1,41 @@
+Maximum number of recipients
+============================
+
+The 'max-recipients' rule matches when there are more than the maximum allowed
+number of explicit recipients addressed by the message.
+
+    >>> from Mailman.configuration import config
+    >>> mlist = config.db.list_manager.create(u'_xtest@example.com')
+    >>> rule = config.rules['max-recipients']
+    >>> rule.name
+    'max-recipients'
+
+In this case, we'll create a message with 5 recipients.  These include all
+addresses in the To and CC headers.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... To: _xtest@example.com, bperson@example.com
+    ... Cc: cperson@example.com
+    ... Cc: dperson@example.com (Dan Person)
+    ... To: Elly Q. Person <eperson@example.com>
+    ...
+    ... Hey folks!
+    ... """)
+
+For backward compatibility, the message must have fewer than the maximum
+number of explicit recipients.
+
+    >>> mlist.max_num_recipients = 5
+    >>> rule.check(mlist, msg, {})
+    True
+
+    >>> mlist.max_num_recipients = 6
+    >>> rule.check(mlist, msg, {})
+    False
+
+Zero means any number of recipients are allowed.
+
+    >>> mlist.max_num_recipients = 0
+    >>> rule.check(mlist, msg, {})
+    False
diff --git a/Mailman/rules/docs/rules.txt b/Mailman/rules/docs/rules.txt
new file mode 100644
index 000000000..d2d291331
--- /dev/null
+++ b/Mailman/rules/docs/rules.txt
@@ -0,0 +1,70 @@
+Rules
+=====
+
+Rules are applied to each message as part of a rule chain.  Individual rules
+simply return a boolean specifying whether the rule matches or not.  Chain
+links determine what happens when a rule matches.
+
+
+All rules
+---------
+
+Rules are maintained in the configuration object as a dictionary mapping rule
+names to rule objects.
+
+    >>> from zope.interface.verify import verifyObject
+    >>> from Mailman.configuration import config
+    >>> from Mailman.interfaces import IRule
+    >>> for rule_name in sorted(config.rules):
+    ...     rule = config.rules[rule_name]
+    ...     print rule_name, verifyObject(IRule, rule)
+    administrivia True
+    any True
+    approved True
+    emergency True
+    implicit-dest True
+    loop True
+    max-recipients True
+    max-size True
+    moderation True
+    news-moderation True
+    no-subject True
+    non-member True
+    suspicious-header True
+    truth True
+
+You can get a rule by name.
+
+    >>> rule = config.rules['emergency']
+    >>> verifyObject(IRule, rule)
+    True
+
+
+Rule checks
+-----------
+
+Individual rules can be checked to see if they match, by running the rule's
+`check()` method.  This returns a boolean indicating whether the rule was
+matched or not.
+
+    >>> mlist = config.db.list_manager.create(u'_xtest@example.com')
+    >>> msg = message_from_string("""\
+    ... From: aperson@example.com
+    ...
+    ... An important message.
+    ... """)
+
+For example, the emergency rule just checks to see if the emergency flag is
+set on the mailing list, and the message has not been pre-approved by the list
+administrator.
+
+    >>> rule.name
+    'emergency'
+    >>> mlist.emergency = False
+    >>> rule.check(mlist, msg, {})
+    False
+    >>> mlist.emergency = True
+    >>> rule.check(mlist, msg, {})
+    True
+    >>> rule.check(mlist, msg, dict(moderator_approved=True))
+    False
diff --git a/Mailman/rules/docs/suspicious.txt b/Mailman/rules/docs/suspicious.txt
new file mode 100644
index 000000000..6b0eeda35
--- /dev/null
+++ b/Mailman/rules/docs/suspicious.txt
@@ -0,0 +1,36 @@
+Suspicious headers
+==================
+
+Suspicious headers are a way for Mailman to hold messages that match a
+particular regular expression.  This mostly historical feature is fairly
+confusing to users, and the list attribute that controls this is misnamed.
+
+    >>> from Mailman.configuration import config
+    >>> mlist = config.db.list_manager.create(u'_xtest@example.com')
+    >>> rule = config.rules['suspicious-header']
+    >>> rule.name
+    'suspicious-header'
+
+Set the so-called suspicious header configuration variable.
+
+    >>> mlist.bounce_matching_headers = u'From: .*person@(blah.)?example.com'
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.com
+    ... To: _xtest@example.com
+    ... Subject: An implicit message
+    ... 
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    True
+
+But if the header doesn't match the regular expression, the rule won't match.
+This one comes from a .org address.
+
+    >>> msg = message_from_string(u"""\
+    ... From: aperson@example.org
+    ... To: _xtest@example.com
+    ... Subject: An implicit message
+    ... 
+    ... """)
+    >>> rule.check(mlist, msg, {})
+    False
diff --git a/Mailman/rules/docs/truth.txt b/Mailman/rules/docs/truth.txt
new file mode 100644
index 000000000..baa40772a
--- /dev/null
+++ b/Mailman/rules/docs/truth.txt
@@ -0,0 +1,10 @@
+Truth
+=====
+
+The 'truth' rule always matches.  This makes it useful as a terminus rule for
+unconditionally jumping to another chain.
+
+    >>> from Mailman.configuration import config
+    >>> rule = config.rules['truth']
+    >>> rule.check(False, False, False)
+    True