Split member and nonmember moderation.

 * member-moderation happens at the same place in the built-in chain that the
   previously named moderation rule happens.  nonmember-moderation happens
   after all the other normal moderation rules.
 * Handle unsubscribed nonmember posts.

Other changes:

 * Message.senders now filters out Nones and empty strings.
 * Various test cleanups and simplifications.
 * More `address` -> `email` fixes.
 * Give Link class a useful repr.
 * Fix a potential UnboundLocalError.
 * Various other small changes.

4 files changed, 122 insertions, 95 deletions

diff --git a/src/mailman/rules/docs/emergency.txt b/src/mailman/rules/docs/emergency.txt
index 2a94234c6..f28f9eed9 100644
--- a/src/mailman/rules/docs/emergency.txt
+++ b/src/mailman/rules/docs/emergency.txt
@@ -5,75 +5,33 @@ Emergency
 When the mailing list has its emergency flag set, all messages posted to the
 list are held for moderator approval.
 
-    >>> mlist = create_list('_xtest@example.com')
+    >>> mlist = create_list('test@example.com')
+    >>> rule = config.rules['emergency']
     >>> msg = message_from_string("""\
     ... From: aperson@example.com
-    ... To: _xtest@example.com
+    ... To: test@example.com
     ... Subject: My first post
     ... Message-ID: <first>
     ...
     ... An important message.
     ... """)
 
-The emergency rule is matched as part of the built-in chain.  The emergency
-rule matches if the flag is set on the mailing list.
+By default, the mailing list does not have its emergency flag set.
 
-    >>> from mailman.core.chains import process
-    >>> mlist.emergency = True
-    >>> process(mlist, msg, {}, 'built-in')
-
-There are two messages in the virgin queue.  The one addressed to the original
-sender will contain a token we can use to grab the held message out of the
-pending requests.
-::
-
-    >>> virginq = config.switchboards['virgin']
+    >>> mlist.emergency
+    False
+    >>> rule.check(mlist, msg, {})
+    False
 
-    >>> from mailman.interfaces.messages import IMessageStore
-    >>> from mailman.interfaces.pending import IPendings
-    >>> from mailman.interfaces.requests import IRequests
-    >>> from zope.component import getUtility
-    >>> message_store = getUtility(IMessageStore)
+The emergency rule matches if the flag is set on the mailing list.
 
-    >>> def get_held_message():
-    ...     import re
-    ...     qfiles = []
-    ...     for filebase in virginq.files:
-    ...         qmsg, qdata = virginq.dequeue(filebase)
-    ...         virginq.finish(filebase)
-    ...         qfiles.append(qmsg)
-    ...     from operator import itemgetter
-    ...     qfiles.sort(key=itemgetter('to'))
-    ...     cookie = None
-    ...     for line in qfiles[1].get_payload().splitlines():
-    ...         mo = re.search('confirm/[^/]+/(?P<cookie>.*)$', line)
-    ...         if mo:
-    ...             cookie = mo.group('cookie')
-    ...             break
-    ...     assert cookie is not None, 'No confirmation token found'
-    ...     data = getUtility(IPendings).confirm(cookie)
-    ...     requestdb = getUtility(IRequests).get_list_requests(mlist)
-    ...     rkey, rdata = requestdb.get_request(data['id'])
-    ...     return message_store.get_message_by_id(
-    ...         rdata['_mod_message_id'])
-
-    >>> msg = get_held_message()
-    >>> print msg.as_string()
-    From: aperson@example.com
-    To: _xtest@example.com
-    Subject: My first post
-    Message-ID: <first>
-    X-Mailman-Rule-Hits: emergency
-    X-Mailman-Rule-Misses: approved
-    X-Message-ID-Hash: RXJU4JL6N2OUN3OYMXXPPSCR7P7JE2BW
-    <BLANKLINE>
-    An important message.
-    <BLANKLINE>
+    >>> mlist.emergency = True
+    >>> rule.check(mlist, msg, {})
+    True
 
 However, if the message metadata has a ``moderator_approved`` key set, then
 even if the mailing list has its emergency flag set, the message still goes
 through to the membership.
 
-    >>> process(mlist, msg, dict(moderator_approved=True), 'built-in')
-    >>> len(virginq.files)
-    0
+    >>> rule.check(mlist, msg, dict(moderator_approved=True))
+    False
diff --git a/src/mailman/rules/docs/header-matching.txt b/src/mailman/rules/docs/header-matching.txt
index 663fcbc8a..b07118e11 100644
--- a/src/mailman/rules/docs/header-matching.txt
+++ b/src/mailman/rules/docs/header-matching.txt
@@ -6,7 +6,7 @@ Mailman can do pattern based header matching during its normal rule
 processing.  There is a set of site-wide default header matches specified in
 the configuration file under the ``[spam.headers]`` section.
 
-    >>> mlist = create_list('_xtest@example.com')
+    >>> mlist = create_list('test@example.com')
 
 Because the default ``[spam.headers]`` section is empty, we'll just extend the
 current header matching chain with a pattern that matches 4 or more stars,
@@ -21,7 +21,7 @@ through the chain untouched (i.e. no disposition).
 
     >>> msg = message_from_string("""\
     ... From: aperson@example.com
-    ... To: _xtest@example.com
+    ... To: test@example.com
     ... Subject: Not spam
     ... Message-ID: <one>
     ...
diff --git a/src/mailman/rules/docs/moderation.txt b/src/mailman/rules/docs/moderation.txt
index ce88c8576..fdca04599 100644
--- a/src/mailman/rules/docs/moderation.txt
+++ b/src/mailman/rules/docs/moderation.txt
@@ -1,41 +1,44 @@
-=================
-Member moderation
-=================
+==========
+Moderation
+==========
 
 All members and nonmembers have a moderation action.  When the action is not
-`defer`, the `moderation` rule flags the message as needing a moderation
-shortcut.  This might be to automatically accept, discard, reject, or hold the
-message.
+`defer`, the `moderation` rule flags the message as needing moderation.  This
+might be to automatically accept, discard, reject, or hold the message.
+
+Two separate rules check for member and nonmember moderation.  Member
+moderation happens early in the built-in chain, while nonmember moderation
+happens later in the chain, after normal moderation checks.
 
     >>> mlist = create_list('test@example.com')
-    >>> rule = config.rules['moderation']
-    >>> print rule.name
-    moderation
 
-Let's add the message author as a non-moderated member.
-::
 
-    >>> from mailman.interfaces.member import MemberRole
-    >>> from mailman.interfaces.usermanager import IUserManager
+Member moderation
+=================
 
-    >>> from zope.component import getUtility
-    >>> user = getUtility(IUserManager).create_user(
-    ...     'aperson@example.org', 'Anne Person')
+    >>> member_rule = config.rules['member-moderation']
+    >>> print member_rule.name
+    member-moderation
 
-    >>> address = list(user.addresses)[0]
-    >>> member = address.subscribe(mlist, MemberRole.member)
+Anne, a mailing list member, sends a message to the mailing list.  Her
+postings are not moderated.
+::
+
+    >>> from mailman.testing.helpers import subscribe
+    >>> subscribe(mlist, 'Anne')
+    >>> member = mlist.members.get_member('aperson@example.com')
     >>> print member.moderation_action
     Action.defer
 
-Because the member is not moderated, the rule does not match.
+Because Anne is not moderated, the member moderation rule does not match.
 
-    >>> msg = message_from_string("""\
-    ... From: aperson@example.org
+    >>> member_msg = message_from_string("""\
+    ... From: aperson@example.com
     ... To: test@example.com
     ... Subject: A posted message
     ...
     ... """)
-    >>> rule.check(mlist, msg, {})
+    >>> member_rule.check(mlist, member_msg, {})
     False
 
 Once the member's moderation action is set to something other than `defer`,
@@ -45,11 +48,11 @@ information for the eventual moderation chain.
     >>> from mailman.interfaces.action import Action
     >>> member.moderation_action = Action.hold
     >>> msgdata = {}
-    >>> rule.check(mlist, msg, msgdata)
+    >>> member_rule.check(mlist, member_msg, msgdata)
     True
     >>> dump_msgdata(msgdata)
     moderation_action: hold
-    moderation_sender: aperson@example.org
+    moderation_sender: aperson@example.com
 
 
 Nonmembers
@@ -58,39 +61,104 @@ Nonmembers
 Nonmembers are handled in a similar way, although by default, nonmember
 postings are held for moderator approval.
 
-    >>> user = getUtility(IUserManager).create_user(
-    ...     'bperson@example.org', 'Bart Person')
+    >>> nonmember_rule = config.rules['nonmember-moderation']
+    >>> print nonmember_rule.name
+    nonmember-moderation
 
-    >>> address = list(user.addresses)[0]
-    >>> nonmember = address.subscribe(mlist, MemberRole.nonmember)
+Bart, who is not a member of the mailing list, sends a message to the list.
+
+    >>> from mailman.interfaces.member import MemberRole
+    >>> subscribe(mlist, 'Bart', MemberRole.nonmember)
+    >>> nonmember = mlist.nonmembers.get_member('bperson@example.com')
     >>> print nonmember.moderation_action
     Action.hold
 
-Because the sender's moderation action is to hold by default, the rule
-matches.  Again, the message metadata carries some useful information.
+When Bart is registered as a nonmember of the list, his moderation action is
+set to hold by default.  Thus the rule matches and the message metadata again
+carries some useful information.
 
-    >>> msg = message_from_string("""\
-    ... From: bperson@example.org
+    >>> nonmember_msg = message_from_string("""\
+    ... From: bperson@example.com
     ... To: test@example.com
     ... Subject: A posted message
     ...
     ... """)
     >>> msgdata = {}
-    >>> rule.check(mlist, msg, msgdata)
+    >>> nonmember_rule.check(mlist, nonmember_msg, msgdata)
     True
     >>> dump_msgdata(msgdata)
     moderation_action: hold
-    moderation_sender: bperson@example.org
+    moderation_sender: bperson@example.com
 
 Of course, the nonmember action can be set to defer the decision, in which
 case the rule does not match.
 
     >>> nonmember.moderation_action = Action.defer
-    >>> rule.check(mlist, msg, {})
+    >>> nonmember_rule.check(mlist, nonmember_msg, {})
     False
 
 
 Unregistered nonmembers
 =======================
 
-XXX
+The incoming queue runner ensures that all sender addresses are registered in
+the system, but it is the moderation rule that subscribes nonmember addresses
+to the mailing list if they are not already subscribed.
+::
+
+    >>> from mailman.interfaces.usermanager import IUserManager
+    >>> from zope.component import getUtility
+    >>> address = getUtility(IUserManager).create_address(
+    ...     'cperson@example.com')
+    >>> address
+    <Address: cperson@example.com [not verified] at ...>
+
+    >>> msg = message_from_string("""\
+    ... From: cperson@example.com
+    ... To: test@example.com
+    ... Subject: A posted message
+    ...
+    ... """)
+
+cperson is neither a member, nor a nonmember of the mailing list.
+::
+
+    >>> def memberkey(member):
+    ...     return member.mailing_list, member.address.email, int(member.role)
+
+    >>> dump_list(mlist.members.members, key=memberkey)
+    <Member: Anne Person <aperson@example.com>
+             on test@example.com as MemberRole.member>
+    >>> dump_list(mlist.nonmembers.members, key=memberkey)
+    <Member: Bart Person <bperson@example.com>
+             on test@example.com as MemberRole.nonmember>
+    
+However, when the nonmember moderation rule runs, it adds the cperson as a
+nonmember of the list.  The rule also matches.
+
+    >>> msgdata = {}
+    >>> nonmember_rule.check(mlist, msg, msgdata)
+    True
+    >>> dump_msgdata(msgdata)
+    moderation_action: hold
+    moderation_sender: cperson@example.com
+
+    >>> dump_list(mlist.members.members, key=memberkey)
+    <Member: Anne Person <aperson@example.com>
+             on test@example.com as MemberRole.member>
+    >>> dump_list(mlist.nonmembers.members, key=memberkey)
+    <Member: Bart Person <bperson@example.com>
+             on test@example.com as MemberRole.nonmember>
+    <Member: cperson@example.com
+             on test@example.com as MemberRole.nonmember>
+
+
+Cross-membership checks
+=======================
+
+Of course, the member moderation rule does not match for nonmembers...
+
+    >>> member_rule.check(mlist, nonmember_msg, {})
+    False
+    >>> nonmember_rule.check(mlist, member_msg, {})
+    False
diff --git a/src/mailman/rules/docs/rules.txt b/src/mailman/rules/docs/rules.txt
index 321f1b277..3c2eab04d 100644
--- a/src/mailman/rules/docs/rules.txt
+++ b/src/mailman/rules/docs/rules.txt
@@ -26,9 +26,10 @@ names to rule objects.
     loop True
     max-recipients True
     max-size True
-    moderation True
+    member-moderation True
     news-moderation True
     no-subject True
+    nonmember-moderation True
     suspicious-header True
     truth True