diff options
| author | Barry Warsaw | 2008-02-02 23:03:19 -0500 |
|---|---|---|
| committer | Barry Warsaw | 2008-02-02 23:03:19 -0500 |
| commit | f03c31acb800d79c606ee3e206868aef8a08bfda (patch) | |
| tree | 15e0b72f129b6ee5f4515647c8c25e0c970a80d9 | |
| parent | 7c5b4d64df6532548742460d405a8a64e35b22c2 (diff) | |
| parent | 4823801716b1bf1711d63b649b0fafd6acd30821 (diff) | |
| download | mailman-f03c31acb800d79c606ee3e206868aef8a08bfda.tar.gz mailman-f03c31acb800d79c606ee3e206868aef8a08bfda.tar.zst mailman-f03c31acb800d79c606ee3e206868aef8a08bfda.zip | |
Merge the 'rules' branch.
Give the first alpha a code name.
This branch mostly gets rid of all the approval oriented handlers in favor of
a chain-of-rules based approach. This will be much more powerful and
extensible, allowing rule definition by plugin and chain creation via web
page.
When a message is processed by the incoming queue, it gets sent through a
chain of rules. The starting chain is defined on the mailing list object, and
there is a built-in default starting chain, called 'built-in'. Each chain is
made up of links, which describe a rule and an action, along with possibly
some other information. Actions allow processing to take a detour through
another chain, jump to another chain, stop processing, run a function, etc.
The built-in chain essentially implements the original early part of the
handler pipeline. If a message makes it through the built-in chain, it gets
sent to the prep queue, where the message is decorated and such before sending
out to the list membership. The 'accept' chain is what moves the message into
the prep queue.
There are also 'hold', 'discard', and 'reject' chains, which do what you would
expect them to. There are lots of built-in rules, implementing everything
from the old emergency handler to new handlers such as one not allowing empty
subject headers.
IMember grows an is_moderated attribute.
The 'adminapproved' metadata key is renamed 'moderator_approved'.
Fix some bogus uses of noreply_address to no_reply_address.
Stash an 'original_size' attribute on the message after parsing its plain
text. This can be used later to ensure the original message does not exceed a
specified size without have to flatten the message again.
The KNOWN_SPAMMERS global variable is replaced with HEADER_MATCHES. The
mailing list's header_filter_rules variable is replaced with header_matches
which has the same semantics as HEADER_MATCHES, but is list-specific.
DEFAULT_MAIL_COMMANDS_MAX_LINES -> EMAIL_COMMANDS_MAX_LINES.
Update smtplistener.py to be much better, to use maildir format instead of
mbox format, to respond to RSET commands by clearing the maildir, and by
silencing annoying asyncore error messages.
Extend the doctest runner so that it will run .txt files in any docs
subdirectory in the code tree.
Add plugable keys 'mailman.mta' and 'mailman.rules'. The latter may have only
one setting while the former is extensible.
There are lots of doctests which should give all the gory details.
Mailman/Post.py -> Mailman/inject.py and the command line usage of this module
is removed.
SQLALCHEMY_ECHO, which was unused, is removed.
Backport the ability to specify additional footer interpolation variables by
the message metadata 'decoration-data' key.
can_acknowledge() defines whether a message can be responded to by the email
robot.
Simplify the implementation of _reset() based on Storm fixes. Be able to
handle lists in Storm values.
Do some reorganization.
87 files changed, 3792 insertions, 1606 deletions
diff --git a/Mailman/Defaults.py b/Mailman/Defaults.py index c4d9d7dbc..e37545e52 100644 --- a/Mailman/Defaults.py +++ b/Mailman/Defaults.py @@ -121,9 +121,6 @@ DEFAULT_VAR_DIRECTORY = '/var/mailman' # any variable in the Configuration object. DEFAULT_DATABASE_URL = 'sqlite:///$DATA_DIR/mailman.db' -# For debugging purposes -SQLALCHEMY_ECHO = False - ##### @@ -164,16 +161,21 @@ DEFAULT_URL_HOST = '@URLHOST@' # Spam avoidance defaults ##### -# This variable contains a list of 2-tuple of the format (header, regex) which -# the Mailman/Handlers/SpamDetect.py module uses to match against the current -# message. If the regex matches the given header in the current message, then -# it is flagged as spam. header is case-insensitive and should not include -# the trailing colon. regex is always matched with re.IGNORECASE. +# This variable contains a list of tuple of the format: +# +# (header, pattern[, chain]) +# +# which is used to match against the current message's headers. If the +# pattern matches the given header in the current message, then the named +# chain is jumped to. header is case-insensitive and should not include the +# trailing colon. pattern is always matched with re.IGNORECASE. chain is +# optional; if not given the 'hold' chain is used, but if given it may be any +# existing chain, such as 'discard', 'reject', or 'accept'. # -# Note that the more searching done, the slower the whole process gets. Spam -# detection is run against all messages coming to either the list, or the -# -owners address, unless the message is explicitly approved. -KNOWN_SPAMMERS = [] +# Note that the more searching done, the slower the whole process gets. +# Header matching is run against all messages coming to either the list, or +# the -owners address, unless the message is explicitly approved. +HEADER_MATCHES = [] @@ -922,7 +924,7 @@ ${web_page_url}listinfo${cgiext}/${list_name} DEFAULT_SCRUB_NONDIGEST = False # Mail command processor will ignore mail command lines after designated max. -DEFAULT_MAIL_COMMANDS_MAX_LINES = 25 +EMAIL_COMMANDS_MAX_LINES = 10 # Is the list owner notified of admin requests immediately by mail, as well as # by daily pending-request reminder? diff --git a/Mailman/Handlers/Approve.py b/Mailman/Handlers/Approve.py deleted file mode 100644 index 1198e6ea7..000000000 --- a/Mailman/Handlers/Approve.py +++ /dev/null @@ -1,116 +0,0 @@ -# Copyright (C) 1998-2007 by the Free Software Foundation, Inc. -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, -# USA. - -"""Determine whether the message is pre-approved for delivery.""" - -import re - -from email.Iterators import typed_subpart_iterator - -from Mailman import Errors -from Mailman.configuration import config - -EMPTYSTRING = '' - - - -def process(mlist, msg, msgdata): - # Short circuits - if msgdata.get('approved'): - # Digests, Usenet postings, and some other messages come pre-approved. - # XXX we may want to further filter Usenet messages, so the test above - # may not be entirely correct. - return - # See if the message has an Approved or Approve header with a valid - # moderator password. Also look at the first non-whitespace line in the - # file to see if it looks like an Approved header. - missing = object() - password = msg.get('approved', msg.get('approve', missing)) - if password is missing: - # Find the first text/plain part in the message - part = None - stripped = False - for part in typed_subpart_iterator(msg, 'text', 'plain'): - break - # XXX I'm not entirely sure why, but it is possible for the payload of - # the part to be None, and you can't splitlines() on None. - if part and part.get_payload() is not None: - lines = part.get_payload(decode=True).splitlines(True) - for lineno, line in enumerate(lines): - if line.strip(): - break - if ':' in line: - header, value = line.split(':', 1) - if header.lower() in ('approved', 'approve'): - password = value.strip() - # Now strip the first line from the payload so the - # password doesn't leak. - del lines[lineno] - reset_payload(part, EMPTYSTRING.join(lines)) - stripped = True - if stripped: - # MAS: Bug 1181161 - Now try all the text parts in case it's - # multipart/alternative with the approved line in HTML or other - # text part. We make a pattern from the Approved line and delete - # it from all text/* parts in which we find it. It would be - # better to just iterate forward, but email compatability for pre - # Python 2.2 returns a list, not a true iterator. - # - # This will process all the multipart/alternative parts in the - # message as well as all other text parts. We shouldn't find the - # pattern outside the multipart/alternative parts, but if we do, - # it is probably best to delete it anyway as it does contain the - # password. - # - # Make a pattern to delete. We can't just delete a line because - # line of HTML or other fancy text may include additional message - # text. This pattern works with HTML. It may not work with rtf - # or whatever else is possible. - pattern = header + ':(\s| )*' + re.escape(password) - for part in typed_subpart_iterator(msg, 'text'): - if part is not None and part.get_payload() is not None: - lines = part.get_payload(decode=True) - if re.search(pattern, lines): - reset_payload(part, re.sub(pattern, '', lines)) - if password is not missing and password == mlist.moderator_password: - # BAW: should we definitely deny if the password exists but does not - # match? For now we'll let it percolate up for further determination. - msgdata['approved'] = True - # Used by the Emergency module - msgdata['adminapproved'] = True - # Has this message already been posted to this list? - beentheres = [s.strip().lower() for s in msg.get_all('x-beenthere', [])] - if mlist.posting_address in beentheres: - raise Errors.LoopError - - -def reset_payload(part, payload): - # Set decoded payload maintaining content-type, format and delsp. - # TK: Messages with 'charset=' cause trouble. So, instead of - # part.get_content_charset('us-ascii') ... - cset = part.get_content_charset() or 'us-ascii' - ctype = part.get_content_type() - format = part.get_param('format') - delsp = part.get_param('delsp') - del part['content-transfer-encoding'] - del part['content-type'] - part.set_payload(payload, cset) - part.set_type(ctype) - if format: - part.set_param('Format', format) - if delsp: - part.set_param('DelSp', delsp) diff --git a/Mailman/Handlers/Decorate.py b/Mailman/Handlers/Decorate.py index 984cd9670..6891fed8d 100644 --- a/Mailman/Handlers/Decorate.py +++ b/Mailman/Handlers/Decorate.py @@ -57,6 +57,7 @@ def process(mlist, msg, msgdata): except Errors.NotAMemberError: pass # These strings are descriptive for the log file and shouldn't be i18n'd + d.update(msgdata.get('decoration-data', {})) header = decorate(mlist, mlist.msg_header, d) footer = decorate(mlist, mlist.msg_footer, d) # Escape hatch if both the footer and header are empty diff --git a/Mailman/Handlers/Emergency.py b/Mailman/Handlers/Emergency.py deleted file mode 100644 index a282da621..000000000 --- a/Mailman/Handlers/Emergency.py +++ /dev/null @@ -1,37 +0,0 @@ -# Copyright (C) 2002-2007 by the Free Software Foundation, Inc. -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -"""Put an emergency hold on all messages otherwise approved. - -No notices are sent to either the sender or the list owner for emergency -holds. I think they'd be too obnoxious. -""" - -from Mailman import Errors -from Mailman.i18n import _ - - - -class EmergencyHold(Errors.HoldMessage): - reason = _('Emergency hold on all list traffic is in effect') - rejection = _('Your message was deemed inappropriate by the moderator.') - - - -def process(mlist, msg, msgdata): - if mlist.emergency and not msgdata.get('adminapproved'): - mlist.HoldMessage(msg, _(EmergencyHold.reason), msgdata) - raise EmergencyHold diff --git a/Mailman/Handlers/Hold.py b/Mailman/Handlers/Hold.py deleted file mode 100644 index f53c168c5..000000000 --- a/Mailman/Handlers/Hold.py +++ /dev/null @@ -1,315 +0,0 @@ -# Copyright (C) 1998-2007 by the Free Software Foundation, Inc. -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, -# USA. - -"""Determine whether this message should be held for approval. - -This modules tests only for hold situations, such as messages that are too -large, messages that have potential administrivia, etc. Definitive approvals -or denials are handled by a different module. - -If no determination can be made (i.e. none of the hold criteria matches), then -we do nothing. If the message must be held for approval, then the hold -database is updated and any administrator notification messages are sent. -Finally an exception is raised to let the pipeline machinery know that further -message handling should stop. -""" - -from __future__ import with_statement - -import email -import logging -import email.utils - -from email.mime.message import MIMEMessage -from email.mime.text import MIMEText -from types import ClassType -from zope.interface import implements - -from Mailman import Errors -from Mailman import Message -from Mailman import Utils -from Mailman import i18n -from Mailman.app.bounces import ( - has_explicit_destination, has_matching_bounce_header) -from Mailman.app.moderator import hold_message -from Mailman.app.replybot import autorespond_to_sender -from Mailman.configuration import config -from Mailman.interfaces import IPendable - -log = logging.getLogger('mailman.vette') - -# Play footsie with _ so that the following are marked as translated, but -# aren't actually translated until we need the text later on. -def _(s): - return s - -__i18n_templates__ = True - - - -class ForbiddenPoster(Errors.HoldMessage): - reason = _('Sender is explicitly forbidden') - rejection = _('You are forbidden from posting messages to this list.') - -class ModeratedPost(Errors.HoldMessage): - reason = _('Post to moderated list') - rejection = _('Your message was deemed inappropriate by the moderator.') - -class NonMemberPost(Errors.HoldMessage): - reason = _('Post by non-member to a members-only list') - rejection = _('Non-members are not allowed to post messages to this list.') - -class NotExplicitlyAllowed(Errors.HoldMessage): - reason = _('Posting to a restricted list by sender requires approval') - rejection = _('This list is restricted; your message was not approved.') - -class TooManyRecipients(Errors.HoldMessage): - reason = _('Too many recipients to the message') - rejection = _('Please trim the recipient list; it is too long.') - -class ImplicitDestination(Errors.HoldMessage): - reason = _('Message has implicit destination') - rejection = _('''Blind carbon copies or other implicit destinations are -not allowed. Try reposting your message by explicitly including the list -address in the To: or Cc: fields.''') - -class Administrivia(Errors.HoldMessage): - reason = _('Message may contain administrivia') - - def rejection_notice(self, mlist): - listurl = mlist.script_url('listinfo') - request = mlist.request_address - return _("""Please do *not* post administrative requests to the mailing -list. If you wish to subscribe, visit $listurl or send a message with the -word `help' in it to the request address, $request, for further -instructions.""") - -class SuspiciousHeaders(Errors.HoldMessage): - reason = _('Message has a suspicious header') - rejection = _('Your message had a suspicious header.') - -class MessageTooBig(Errors.HoldMessage): - def __init__(self, msgsize, limit): - Errors.HoldMessage.__init__(self) - self.__msgsize = msgsize - self.__limit = limit - - def reason_notice(self): - size = self.__msgsize - limit = self.__limit - return _('''Message body is too big: $size bytes with a limit of -$limit KB''') - - def rejection_notice(self, mlist): - kb = self.__limit - return _('''Your message was too big; please trim it to less than -$kb KB in size.''') - -class ModeratedNewsgroup(ModeratedPost): - reason = _('Posting to a moderated newsgroup') - - - -# And reset the translator -_ = i18n._ - - - -def ackp(msg): - ack = msg.get('x-ack', '').lower() - precedence = msg.get('precedence', '').lower() - if ack <> 'yes' and precedence in ('bulk', 'junk', 'list'): - return 0 - return 1 - - - -class HeldMessagePendable(dict): - implements(IPendable) - PEND_KEY = 'held message' - - - -def process(mlist, msg, msgdata): - if msgdata.get('approved'): - return - # Get the sender of the message - listname = mlist.list_name - adminaddr = listname + '-admin' - sender = msg.get_sender() - # Special case an ugly sendmail feature: If there exists an alias of the - # form "owner-foo: bar" and sendmail receives mail for address "foo", - # sendmail will change the envelope sender of the message to "bar" before - # delivering. This feature does not appear to be configurable. *Boggle*. - if not sender or sender[:len(listname)+6] == adminaddr: - sender = msg.get_sender(use_envelope=0) - # - # Possible administrivia? - if mlist.administrivia and Utils.is_administrivia(msg): - hold_for_approval(mlist, msg, msgdata, Administrivia) - # no return - # - # Are there too many recipients to the message? - if mlist.max_num_recipients > 0: - # figure out how many recipients there are - recips = email.utils.getaddresses(msg.get_all('to', []) + - msg.get_all('cc', [])) - if len(recips) >= mlist.max_num_recipients: - hold_for_approval(mlist, msg, msgdata, TooManyRecipients) - # no return - # - # Implicit destination? Note that message originating from the Usenet - # side of the world should never be checked for implicit destination. - if mlist.require_explicit_destination and \ - not has_explicit_destination(mlist, msg) and \ - not msgdata.get('fromusenet'): - # then - hold_for_approval(mlist, msg, msgdata, ImplicitDestination) - # no return - # - # Suspicious headers? - if mlist.bounce_matching_headers: - triggered = has_matching_bounce_header(mlist, msg) - if triggered: - # TBD: Darn - can't include the matching line for the admin - # message because the info would also go to the sender - hold_for_approval(mlist, msg, msgdata, SuspiciousHeaders) - # no return - # - # Is the message too big? - if mlist.max_message_size > 0: - bodylen = 0 - for line in email.Iterators.body_line_iterator(msg): - bodylen += len(line) - for part in msg.walk(): - if part.preamble: - bodylen += len(part.preamble) - if part.epilogue: - bodylen += len(part.epilogue) - if bodylen/1024.0 > mlist.max_message_size: - hold_for_approval(mlist, msg, msgdata, - MessageTooBig(bodylen, mlist.max_message_size)) - # no return - # - # Are we gatewaying to a moderated newsgroup and is this list the - # moderator's address for the group? - if mlist.news_moderation == 2: - hold_for_approval(mlist, msg, msgdata, ModeratedNewsgroup) - - - -def hold_for_approval(mlist, msg, msgdata, exc): - # BAW: This should really be tied into the email confirmation system so - # that the message can be approved or denied via email as well as the - # web. - if isinstance(exc, ClassType) or isinstance(exc, type): - # Go ahead and instantiate it now. - exc = exc() - listname = mlist.list_name - sender = msgdata.get('sender', msg.get_sender()) - usersubject = msg.get('subject') - charset = Utils.GetCharSet(mlist.preferred_language) - if usersubject: - usersubject = Utils.oneline(usersubject, charset) - else: - usersubject = _('(no subject)') - message_id = msg.get('message-id', 'n/a') - owneraddr = mlist.owner_address - adminaddr = mlist.bounces_address - requestaddr = mlist.request_address - # We need to send both the reason and the rejection notice through the - # translator again, because of the games we play above - reason = Utils.wrap(exc.reason_notice()) - msgdata['rejection_notice'] = Utils.wrap(exc.rejection_notice(mlist)) - id = hold_message(mlist, msg, msgdata, reason) - # Now we need to craft and send a message to the list admin so they can - # deal with the held message. - d = {'listname' : listname, - 'hostname' : mlist.host_name, - 'reason' : _(reason), - 'sender' : sender, - 'subject' : usersubject, - 'admindb_url': mlist.script_url('admindb'), - } - # We may want to send a notification to the original sender too - fromusenet = msgdata.get('fromusenet') - # Since we're sending two messages, which may potentially be in different - # languages (the user's preferred and the list's preferred for the admin), - # we need to play some i18n games here. Since the current language - # context ought to be set up for the user, let's craft his message first. - # - # This message should appear to come from <list>-admin so as to handle any - # bounce processing that might be needed. - pendable = HeldMessagePendable(type=HeldMessagePendable.PEND_KEY, id=id) - token = config.db.pendings.add(pendable) - # Get the language to send the response in. If the sender is a member, - # then send it in the member's language, otherwise send it in the mailing - # list's preferred language. - member = mlist.members.get_member(sender) - lang = (member.preferred_language if member else mlist.preferred_language) - if not fromusenet and ackp(msg) and mlist.respond_to_post_requests and \ - autorespond_to_sender(mlist, sender, lang): - # Get a confirmation token - d['confirmurl'] = '%s/%s' % ( - mlist.script_url('confirm'), token) - lang = msgdata.get('lang', lang) - subject = _('Your message to $listname awaits moderator approval') - text = Utils.maketext('postheld.txt', d, lang=lang, mlist=mlist) - nmsg = Message.UserNotification(sender, adminaddr, subject, text, lang) - nmsg.send(mlist) - # Now the message for the list owners. Be sure to include the list - # moderators in this message. This one should appear to come from - # <list>-owner since we really don't need to do bounce processing on it. - if mlist.admin_immed_notify: - # Now let's temporarily set the language context to that which the - # admin is expecting. - with i18n.using_language(mlist.preferred_language): - lang = mlist.preferred_language - charset = Utils.GetCharSet(lang) - # We need to regenerate or re-translate a few values in d - d['reason'] = _(reason) - d['subject'] = usersubject - # craft the admin notification message and deliver it - subject = _('$listname post from $sender requires approval') - nmsg = Message.UserNotification(owneraddr, owneraddr, subject, - lang=lang) - nmsg.set_type('multipart/mixed') - text = MIMEText( - Utils.maketext('postauth.txt', d, raw=1, mlist=mlist), - _charset=charset) - dmsg = MIMEText(Utils.wrap(_("""\ -If you reply to this message, keeping the Subject: header intact, Mailman will -discard the held message. Do this if the message is spam. If you reply to -this message and include an Approved: header with the list password in it, the -message will be approved for posting to the list. The Approved: header can -also appear in the first line of the body of the reply.""")), - _charset=Utils.GetCharSet(lang)) - dmsg['Subject'] = 'confirm ' + token - dmsg['Sender'] = requestaddr - dmsg['From'] = requestaddr - dmsg['Date'] = email.utils.formatdate(localtime=True) - dmsg['Message-ID'] = email.utils.make_msgid() - nmsg.attach(text) - nmsg.attach(MIMEMessage(msg)) - nmsg.attach(MIMEMessage(dmsg)) - nmsg.send(mlist, **{'tomoderators': 1}) - # Log the held message - log.info('%s post from %s held, message-id=%s: %s', - listname, sender, message_id, reason) - # raise the specific MessageHeld exception to exit out of the message - # delivery pipeline - raise exc diff --git a/Mailman/Handlers/SpamDetect.py b/Mailman/Handlers/SpamDetect.py deleted file mode 100644 index b7e687284..000000000 --- a/Mailman/Handlers/SpamDetect.py +++ /dev/null @@ -1,138 +0,0 @@ -# Copyright (C) 1998-2007 by the Free Software Foundation, Inc. -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, -# USA. - -"""Do more detailed spam detection. - -This module hard codes site wide spam detection. By hacking the -KNOWN_SPAMMERS variable, you can set up more regular expression matches -against message headers. If spam is detected the message is discarded -immediately. - -TBD: This needs to be made more configurable and robust. -""" - -import re - -from cStringIO import StringIO -from email.Generator import Generator - -from Mailman import Errors -from Mailman import i18n -from Mailman.Handlers.Hold import hold_for_approval -from Mailman.configuration import config - -# First, play footsie with _ so that the following are marked as translated, -# but aren't actually translated until we need the text later on. -def _(s): - return s - - - -class SpamDetected(Errors.DiscardMessage): - """The message contains known spam""" - -class HeaderMatchHold(Errors.HoldMessage): - reason = _('The message headers matched a filter rule') - - -# And reset the translator -_ = i18n._ - - - -class Tee: - def __init__(self, outfp_a, outfp_b): - self._outfp_a = outfp_a - self._outfp_b = outfp_b - - def write(self, s): - self._outfp_a.write(s) - self._outfp_b.write(s) - - -# Class to capture the headers separate from the message body -class HeaderGenerator(Generator): - def __init__(self, outfp, mangle_from_=True, maxheaderlen=78): - Generator.__init__(self, outfp, mangle_from_, maxheaderlen) - self._headertxt = '' - - def _write_headers(self, msg): - sfp = StringIO() - oldfp = self._fp - self._fp = Tee(oldfp, sfp) - try: - Generator._write_headers(self, msg) - finally: - self._fp = oldfp - self._headertxt = sfp.getvalue() - - def header_text(self): - return self._headertxt - - - -def process(mlist, msg, msgdata): - if msgdata.get('approved'): - return - # First do site hard coded header spam checks - for header, regex in config.KNOWN_SPAMMERS: - cre = re.compile(regex, re.IGNORECASE) - for value in msg.get_all(header, []): - mo = cre.search(value) - if mo: - # we've detected spam, so throw the message away - raise SpamDetected - # Now do header_filter_rules - # TK: Collect headers in sub-parts because attachment filename - # extension may be a clue to possible virus/spam. - headers = '' - for p in msg.walk(): - g = HeaderGenerator(StringIO()) - g.flatten(p) - headers += g.header_text() - # Now reshape headers (remove extra CR and connect multiline). - headers = re.sub('\n+', '\n', headers) - headers = re.sub('\n\s', ' ', headers) - for patterns, action, empty in mlist.header_filter_rules: - if action == config.DEFER: - continue - for pattern in patterns.splitlines(): - if pattern.startswith('#'): - continue - # ignore 'empty' patterns - if not pattern.strip(): - continue - if re.search(pattern, headers, re.IGNORECASE|re.MULTILINE): - if action == config.DISCARD: - raise Errors.DiscardMessage - if action == config.REJECT: - if msgdata.get('toowner'): - # Don't send rejection notice if addressed to '-owner' - # because it may trigger a loop of notices if the - # sender address is forged. We just discard it here. - raise Errors.DiscardMessage - raise Errors.RejectMessage( - _('Message rejected by filter rule match')) - if action == config.HOLD: - if msgdata.get('toowner'): - # Don't hold '-owner' addressed message. We just - # pass it here but list-owner can set this to be - # discarded on the GUI if he wants. - return - hold_for_approval(mlist, msg, msgdata, HeaderMatchHold) - if action == config.ACCEPT: - return diff --git a/Mailman/Message.py b/Mailman/Message.py index 89bc42798..a9256dc90 100644 --- a/Mailman/Message.py +++ b/Mailman/Message.py @@ -58,7 +58,10 @@ class Message(email.message.Message): return value def get_all(self, name, failobj=None): - all_values = email.message.Message.get_all(self, name, failobj) + missing = object() + all_values = email.message.Message.get_all(self, name, missing) + if all_values is missing: + return failobj return [(unicode(value, 'ascii') if isinstance(value, str) else value) for value in all_values] diff --git a/Mailman/Post.py b/Mailman/Post.py deleted file mode 100644 index 50b9628a0..000000000 --- a/Mailman/Post.py +++ /dev/null @@ -1,62 +0,0 @@ -#! /usr/bin/env python -# -# Copyright (C) 2001-2007 by the Free Software Foundation, Inc. -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, -# USA. - -import sys - -from Mailman.configuration import config -from Mailman.queue import Switchboard - - - -def inject(listname, msg, recips=None, qdir=None): - if qdir is None: - qdir = config.INQUEUE_DIR - queue = Switchboard(qdir) - kws = {'listname' : listname, - 'tolist' : 1, - '_plaintext': 1, - } - if recips: - kws['recips'] = recips - queue.enqueue(msg, **kws) - - - -if __name__ == '__main__': - # When called as a command line script, standard input is read to get the - # list that this message is destined to, the list of explicit recipients, - # and the message to send (in its entirety). stdin must have the - # following format: - # - # line 1: the internal name of the mailing list - # line 2: the number of explicit recipients to follow. 0 means to use the - # list's membership to calculate recipients. - # line 3 - 3+recipnum: explicit recipients, one per line - # line 4+recipnum - end of file: the message in RFC 822 format (may - # include an initial Unix-from header) - listname = sys.stdin.readline().strip() - numrecips = int(sys.stdin.readline()) - if numrecips == 0: - recips = None - else: - recips = [] - for i in range(numrecips): - recips.append(sys.stdin.readline().strip()) - # If the message isn't parsable, we won't get an error here - inject(listname, sys.stdin.read(), recips) diff --git a/Mailman/Utils.py b/Mailman/Utils.py index 0bd8fa2a6..1be02e87a 100644 --- a/Mailman/Utils.py +++ b/Mailman/Utils.py @@ -537,63 +537,6 @@ def maketext(templatefile, dict=None, raw=False, lang=None, mlist=None): -ADMINDATA = { - # admin keyword: (minimum #args, maximum #args) - 'confirm': (1, 1), - 'help': (0, 0), - 'info': (0, 0), - 'lists': (0, 0), - 'options': (0, 0), - 'password': (2, 2), - 'remove': (0, 0), - 'set': (3, 3), - 'subscribe': (0, 3), - 'unsubscribe': (0, 1), - 'who': (0, 2), - } - -# Given a Message.Message object, test for administrivia (eg subscribe, -# unsubscribe, etc). The test must be a good guess -- messages that return -# true get sent to the list admin instead of the entire list. -def is_administrivia(msg): - linecnt = 0 - lines = [] - for line in email.Iterators.body_line_iterator(msg): - # Strip out any signatures - if line == '-- ': - break - if line.strip(): - linecnt += 1 - if linecnt > config.DEFAULT_MAIL_COMMANDS_MAX_LINES: - return False - lines.append(line) - bodytext = NL.join(lines) - # See if the body text has only one word, and that word is administrivia - if ADMINDATA.has_key(bodytext.strip().lower()): - return True - # Look at the first N lines and see if there is any administrivia on the - # line. BAW: N is currently hardcoded to 5. str-ify the Subject: header - # because it may be an email.Header.Header instance rather than a string. - bodylines = lines[:5] - subject = str(msg.get('subject', '')) - bodylines.append(subject) - for line in bodylines: - if not line.strip(): - continue - words = [word.lower() for word in line.split()] - minargs, maxargs = ADMINDATA.get(words[0], (None, None)) - if minargs is None and maxargs is None: - continue - if minargs <= len(words[1:]) <= maxargs: - # Special case the `set' keyword. BAW: I don't know why this is - # here. - if words[0] == 'set' and words[2] not in ('on', 'off'): - continue - return True - return False - - - def GetRequestURI(fallback=None, escape=True): """Return the full virtual path this CGI script was invoked with. diff --git a/Mailman/app/bounces.py b/Mailman/app/bounces.py index 6df5c8aa6..3f38cbf67 100644 --- a/Mailman/app/bounces.py +++ b/Mailman/app/bounces.py @@ -19,8 +19,6 @@ __all__ = [ 'bounce_message', - 'has_explicit_destination', - 'has_matching_bounce_header', ] import re @@ -62,102 +60,3 @@ def bounce_message(mlist, msg, e=None): bmsg.attach(txt) bmsg.attach(MIMEMessage(msg)) bmsg.send(mlist) - - - -# Helper function used to match a pattern against an address. -def _domatch(pattern, addr): - try: - if re.match(pattern, addr, re.IGNORECASE): - return True - except re.error: - # The pattern is a malformed regexp -- try matching safely, - # with all non-alphanumerics backslashed: - if re.match(re.escape(pattern), addr, re.IGNORECASE): - return True - return False - - -def has_explicit_destination(mlist, msg): - """Does the list's name or an acceptable alias appear in the recipients? - - :param mlist: The mailing list the message is destined for. - :param msg: The email message object. - :return: True if the message is explicitly destined for the mailing list, - otherwise False. - """ - # Check all recipient addresses against the list's explicit addresses, - # specifically To: Cc: and Resent-to: - recipients = [] - to = [] - for header in ('to', 'cc', 'resent-to', 'resent-cc'): - to.extend(getaddresses(msg.get_all(header, []))) - for fullname, address in to: - # It's possible that if the header doesn't have a valid RFC 2822 - # value, we'll get None for the address. So skip it. - if address is None or '@' not in address: - continue - address = address.lower() - if address == mlist.posting_address: - return True - recipients.append(address) - # Match the set of recipients against the list's acceptable aliases. - aliases = mlist.acceptable_aliases.splitlines() - for address in recipients: - for alias in aliases: - stripped = alias.strip() - if not stripped: - # Ignore blank or empty lines - continue - if domatch(stripped, address): - return True - return False - - - -def _parse_matching_header_opt(mlist): - """Return a list of triples [(field name, regex, line), ...].""" - # - Blank lines and lines with '#' as first char are skipped. - # - Leading whitespace in the matchexp is trimmed - you can defeat - # that by, eg, containing it in gratuitous square brackets. - all = [] - for line in mlist.bounce_matching_headers.splitlines(): - line = line.strip() - # Skip blank lines and lines *starting* with a '#'. - if not line or line.startswith('#'): - continue - i = line.find(':') - if i < 0: - # This didn't look like a header line. BAW: should do a - # better job of informing the list admin. - log.error('bad bounce_matching_header line: %s\n%s', - mlist.real_name, line) - else: - header = line[:i] - value = line[i+1:].lstrip() - try: - cre = re.compile(value, re.IGNORECASE) - except re.error, e: - # The regexp was malformed. BAW: should do a better - # job of informing the list admin. - log.error("""\ -bad regexp in bounce_matching_header line: %s -\n%s (cause: %s)""", mlist.real_name, value, e) - else: - all.append((header, cre, line)) - return all - - -def has_matching_bounce_header(mlist, msg): - """Does the message have a matching bounce header? - - :param mlist: The mailing list the message is destined for. - :param msg: The email message object. - :return: True if a header field matches a regexp in the - bounce_matching_header mailing list variable. - """ - for header, cre, line in _parse_matching_header_opt(mlist): - for value in msg.get_all(header, []): - if cre.search(value): - return True - return False diff --git a/Mailman/app/chains.py b/Mailman/app/chains.py new file mode 100644 index 000000000..2bf0b9ae1 --- /dev/null +++ b/Mailman/app/chains.py @@ -0,0 +1,115 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Application support for chain processing.""" + +__all__ = [ + 'initialize', + 'process', + ] +__metaclass__ = type + + +from Mailman.chains.accept import AcceptChain +from Mailman.chains.discard import DiscardChain +from Mailman.chains.headers import HeaderMatchChain +from Mailman.chains.hold import HoldChain +from Mailman.chains.reject import RejectChain +from Mailman.chains.builtin import BuiltInChain +from Mailman.configuration import config +from Mailman.interfaces import LinkAction + + + +def process(mlist, msg, msgdata, start_chain='built-in'): + """Process the message through a chain. + + :param start_chain: The name of the chain to start the processing with. + :param mlist: the IMailingList for this message. + :param msg: The Message object. + :param msgdata: The message metadata dictionary. + """ + # Set up some bookkeeping. + chain_stack = [] + msgdata['rule_hits'] = hits = [] + msgdata['rule_misses'] = misses = [] + # Find the starting chain and begin iterating through its links. + chain = config.chains[start_chain] + chain_iter = chain.get_links(mlist, msg, msgdata) + # Loop until we've reached the end of all processing chains. + while chain: + # Iterate over all links in the chain. Do this outside a for-loop so + # we can capture a chain's link iterator in mid-flight. This supports + # the 'detour' link action + try: + link = chain_iter.next() + except StopIteration: + # This chain is exhausted. Pop the last chain on the stack and + # continue iterating through it. If there's nothing left on the + # chain stack then we're completely finished processing. + if len(chain_stack) == 0: + return + chain, chain_iter = chain_stack.pop() + continue + # Process this link. + if link.rule.check(mlist, msg, msgdata): + if link.rule.record: + hits.append(link.rule.name) + # The rule matched so run its action. + if link.action is LinkAction.jump: + chain = link.chain + chain_iter = chain.get_links(mlist, msg, msgdata) + continue + elif link.action is LinkAction.detour: + # Push the current chain so that we can return to it when + # the next chain is finished. + chain_stack.append((chain, chain_iter)) + chain = link.chain + chain_iter = chain.get_links(mlist, msg, msgdata) + continue + elif link.action is LinkAction.stop: + # Stop all processing. + return + elif link.action is LinkAction.defer: + # Just process the next link in the chain. + pass + elif link.action is LinkAction.run: + link.function(mlist, msg, msgdata) + else: + raise AssertionError('Bad link action: %s' % link.action) + else: + # The rule did not match; keep going. + if link.rule.record: + misses.append(link.rule.name) + + + +def initialize(): + """Set up chains, both built-in and from the database.""" + for chain_class in (DiscardChain, HoldChain, RejectChain, AcceptChain): + chain = chain_class() + assert chain.name not in config.chains, ( + 'Duplicate chain name: %s' % chain.name) + config.chains[chain.name] = chain + # Set up a couple of other default chains. + chain = BuiltInChain() + config.chains[chain.name] = chain + # Create and initialize the header matching chain. + chain = HeaderMatchChain() + config.chains[chain.name] = chain + # XXX Read chains from the database and initialize them. + pass diff --git a/Mailman/app/moderator.py b/Mailman/app/moderator.py index 691268153..29577a130 100644 --- a/Mailman/app/moderator.py +++ b/Mailman/app/moderator.py @@ -51,6 +51,18 @@ slog = logging.getLogger('mailman.subscribe') def hold_message(mlist, msg, msgdata=None, reason=None): + """Hold a message for moderator approval. + + The message is added to the mailing list's request database. + + :param mlist: The mailing list to hold the message on. + :param msg: The message to hold. + :param msgdata: Optional message metadata to hold. If not given, a new + metadata dictionary is created and held with the message. + :param reason: Optional string reason why the message is being held. If + not given, the empty string is used. + :return: An id used to handle the held message later. + """ if msgdata is None: msgdata = {} else: @@ -115,10 +127,8 @@ def handle_message(mlist, id, action, if key.startswith('_mod_'): del msgdata[key] # Add some metadata to indicate this message has now been approved. - # XXX 'adminapproved' is used for backward compatibility, but it - # should really be called 'moderator_approved'. msgdata['approved'] = True - msgdata['adminapproved'] = True + msgdata['moderator_approved'] = True # Calculate a new filebase for the approved message, otherwise # delivery errors will cause duplicates. if 'filebase' in msgdata: diff --git a/Mailman/app/replybot.py b/Mailman/app/replybot.py index c46931770..62d442e82 100644 --- a/Mailman/app/replybot.py +++ b/Mailman/app/replybot.py @@ -25,6 +25,7 @@ from __future__ import with_statement __all__ = [ 'autorespond_to_sender', + 'can_acknowledge', ] import logging @@ -87,3 +88,36 @@ def autorespond_to_sender(mlist, sender, lang=None): mlist.hold_and_cmd_autoresponses[sender] = (today, count + 1) return True + + +def can_acknowledge(msg): + """A boolean specifying whether this message can be acknowledged. + + There are several reasons why a message should not be acknowledged, mostly + related to competing standards or common practices. These include: + + * The message has a X-No-Ack header with any value + * The message has an X-Ack header with a 'no' value + * The message has a Precedence header + * The message has an Auto-Submitted header and that header does not have a + value of 'no' + * The message has an empty Return-Path header, e.g. <> + * The message has any RFC 2369 headers (i.e. List-* headers) + + :param msg: a Message object. + :return: Boolean specifying whether the message can be acknowledged or not + (which is different from whether it will be acknowledged). + """ + # I wrote it this way for clarity and consistency with the docstring. + for header in msg.keys(): + if header in ('x-no-ack', 'precedence'): + return False + if header.lower().startswith('list-'): + return False + if msg.get('x-ack', '').lower() == 'no': + return False + if msg.get('auto-submitted', 'no').lower() <> 'no': + return False + if msg.get('return-path') == '<>': + return False + return True diff --git a/Mailman/app/rules.py b/Mailman/app/rules.py new file mode 100644 index 000000000..50bb8d281 --- /dev/null +++ b/Mailman/app/rules.py @@ -0,0 +1,42 @@ +# Copyright (C) 2007 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Various rule helpers""" + +__all__ = ['initialize'] +__metaclass__ = type + + +from zope.interface import implements +from zope.interface.verify import verifyObject + +from Mailman.app.plugins import get_plugins +from Mailman.configuration import config +from Mailman.interfaces import IRule + + + +def initialize(): + """Find and register all rules in all plugins.""" + # Find rules in plugins. + for rule_finder in get_plugins('mailman.rules'): + for rule_class in rule_finder(): + rule = rule_class() + verifyObject(IRule, rule) + assert rule.name not in config.rules, ( + 'Duplicate rule "%s" found in %s' % (rule.name, rule_finder)) + config.rules[rule.name] = rule diff --git a/Mailman/app/styles.py b/Mailman/app/styles.py index 4ca3f1b01..1c978363e 100644 --- a/Mailman/app/styles.py +++ b/Mailman/app/styles.py @@ -70,7 +70,7 @@ class DefaultStyle: mlist.send_goodbye_msg = config.DEFAULT_SEND_GOODBYE_MSG mlist.bounce_matching_headers = ( config.DEFAULT_BOUNCE_MATCHING_HEADERS) - mlist.header_filter_rules = [] + mlist.header_matches = [] mlist.anonymous_list = config.DEFAULT_ANONYMOUS_LIST mlist.description = u'' mlist.info = u'' @@ -225,6 +225,9 @@ class DefaultStyle: # is that they will get all messages, and they will not have an entry # in this dictionary. mlist.topics_userinterest = {} + # The processing chain that messages coming into this list get + # processed by. + mlist.start_chain = u'built-in' def match(self, mailing_list, styles): # If no other styles have matched, then the default style matches. diff --git a/Mailman/bin/inject.py b/Mailman/bin/inject.py index 60185289f..729038118 100644 --- a/Mailman/bin/inject.py +++ b/Mailman/bin/inject.py @@ -19,11 +19,11 @@ import os import sys import optparse -from Mailman import Post from Mailman import Utils from Mailman import Version from Mailman.configuration import config from Mailman.i18n import _ +from Mailman.inject import inject __i18n_templates__ = True @@ -88,7 +88,7 @@ def main(): else: msgtext = sys.stdin.read() - Post.inject(opts.listname, msgtext, qdir=qdir) + inject(opts.listname, msgtext, qdir=qdir) diff --git a/Mailman/queue/tests/__init__.py b/Mailman/chains/__init__.py index e69de29bb..e69de29bb 100644 --- a/Mailman/queue/tests/__init__.py +++ b/Mailman/chains/__init__.py diff --git a/Mailman/chains/accept.py b/Mailman/chains/accept.py new file mode 100644 index 000000000..a6a9a45eb --- /dev/null +++ b/Mailman/chains/accept.py @@ -0,0 +1,55 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The terminal 'accept' chain.""" + +__all__ = ['AcceptChain'] +__metaclass__ = type + +import logging + +from Mailman.chains.base import TerminalChainBase +from Mailman.configuration import config +from Mailman.i18n import _ +from Mailman.queue import Switchboard + + +log = logging.getLogger('mailman.vette') +SEMISPACE = '; ' + + + +class AcceptChain(TerminalChainBase): + """Accept the message for posting.""" + + name = 'accept' + description = _('Accept a message.') + + def _process(self, mlist, msg, msgdata): + """See `TerminalChainBase`.""" + # Start by decorating the message with a header that contains a list + # of all the rules that matched. These metadata could be None or an + # empty list. + rule_hits = msgdata.get('rule_hits') + if rule_hits: + msg['X-Mailman-Rule-Hits'] = SEMISPACE.join(rule_hits) + rule_misses = msgdata.get('rule_misses') + if rule_misses: + msg['X-Mailman-Rule-Misses'] = SEMISPACE.join(rule_misses) + accept_queue = Switchboard(config.PREPQUEUE_DIR) + accept_queue.enqueue(msg, msgdata) + log.info('ACCEPT: %s', msg.get('message-id', 'n/a')) diff --git a/Mailman/chains/base.py b/Mailman/chains/base.py new file mode 100644 index 000000000..16aa63176 --- /dev/null +++ b/Mailman/chains/base.py @@ -0,0 +1,118 @@ +# Copyright (C) 2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Base class for terminal chains.""" + +__all__ = [ + 'Chain', + 'Link', + 'TerminalChainBase', + ] +__metaclass__ = type + +from zope.interface import implements + +from Mailman.configuration import config +from Mailman.interfaces import ( + IChain, IChainIterator, IChainLink, IMutableChain, LinkAction) + + + +class Link: + """A chain link.""" + implements(IChainLink) + + def __init__(self, rule, action=None, chain=None, function=None): + self.rule = rule + self.action = (LinkAction.defer if action is None else action) + self.chain = chain + self.function = function + + + +class TerminalChainBase: + """A base chain that always matches and executes a method. + + The method is called 'process' and must be provided by the subclass. + """ + implements(IChain, IChainIterator) + + def _process(self, mlist, msg, msgdata): + """Process the message for the given mailing list. + + This must be overridden by subclasses. + """ + raise NotImplementedError + + def get_links(self, mlist, msg, msgdata): + """See `IChain`.""" + return iter(self) + + def __iter__(self): + """See `IChainIterator`.""" + truth = config.rules['truth'] + # First, yield a link that always runs the process method. + yield Link(truth, LinkAction.run, function=self._process) + # Now yield a rule that stops all processing. + yield Link(truth, LinkAction.stop) + + + +class Chain: + """Generic chain base class.""" + implements(IMutableChain) + + def __init__(self, name, description): + assert name not in config.chains, 'Duplicate chain name: %s' % name + self.name = name + self.description = description + self._links = [] + # Register the chain. + config.chains[name] = self + + def append_link(self, link): + """See `IMutableChain`.""" + self._links.append(link) + + def flush(self): + """See `IMutableChain`.""" + self._links = [] + + def get_links(self, mlist, msg, msgdata): + """See `IChain`.""" + return iter(ChainIterator(self)) + + def get_iterator(self): + """Return an iterator over the links.""" + # We do it this way in order to preserve a separation of interfaces, + # and allows .get_links() to be overridden. + for link in self._links: + yield link + + + +class ChainIterator: + """Generic chain iterator.""" + + implements(IChainIterator) + + def __init__(self, chain): + self._chain = chain + + def __iter__(self): + """See `IChainIterator`.""" + return self._chain.get_iterator() diff --git a/Mailman/chains/builtin.py b/Mailman/chains/builtin.py new file mode 100644 index 000000000..d702b48b9 --- /dev/null +++ b/Mailman/chains/builtin.py @@ -0,0 +1,84 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The default built-in starting chain.""" + +__all__ = ['BuiltInChain'] +__metaclass__ = type + + +import logging + +from zope.interface import implements + +from Mailman.chains.base import Link +from Mailman.configuration import config +from Mailman.i18n import _ +from Mailman.interfaces import IChain, LinkAction + + +log = logging.getLogger('mailman.vette') + + + +class BuiltInChain: + """Default built-in chain.""" + + implements(IChain) + + name = 'built-in' + description = _('The built-in moderation chain.') + + _link_descriptions = ( + ('approved', LinkAction.jump, 'accept'), + ('emergency', LinkAction.jump, 'hold'), + ('loop', LinkAction.jump, 'discard'), + # Do all of the following before deciding whether to hold the message + # for moderation. + ('administrivia', LinkAction.defer, None), + ('implicit-dest', LinkAction.defer, None), + ('max-recipients', LinkAction.defer, None), + ('max-size', LinkAction.defer, None), + ('news-moderation', LinkAction.defer, None), + ('no-subject', LinkAction.defer, None), + ('suspicious-header', LinkAction.defer, None), + # Now if any of the above hit, jump to the hold chain. + ('any', LinkAction.jump, 'hold'), + # Take a detour through the self header matching chain, which we'll + # create later. + ('truth', LinkAction.detour, 'header-match'), + # Finally, the builtin chain selfs to acceptance. + ('truth', LinkAction.jump, 'accept'), + ) + + def __init__(self): + self._cached_links = None + + def get_links(self, mlist, msg, msgdata): + """See `IChain`.""" + if self._cached_links is None: + self._cached_links = links = [] + for rule_name, action, chain_name in self._link_descriptions: + # Get the named rule. + rule = config.rules[rule_name] + # Get the chain, if one is defined. + if chain_name is None: + chain = None + else: + chain = config.chains[chain_name] + links.append(Link(rule, action, chain)) + return iter(self._cached_links) diff --git a/Mailman/chains/discard.py b/Mailman/chains/discard.py new file mode 100644 index 000000000..d4640f260 --- /dev/null +++ b/Mailman/chains/discard.py @@ -0,0 +1,43 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The terminal 'discard' chain.""" + +__all__ = ['DiscardChain'] +__metaclass__ = type + + +import logging + +from Mailman.chains.base import TerminalChainBase +from Mailman.i18n import _ + + +log = logging.getLogger('mailman.vette') + + + +class DiscardChain(TerminalChainBase): + """Discard a message.""" + + name = 'discard' + description = _('Discard a message and stop processing.') + + def _process(self, mlist, msg, msgdata): + """See `TerminalChainBase`.""" + log.info('DISCARD: %s', msg.get('message-id', 'n/a')) + # Nothing more needs to happen. diff --git a/Mailman/chains/headers.py b/Mailman/chains/headers.py new file mode 100644 index 000000000..06dfafeda --- /dev/null +++ b/Mailman/chains/headers.py @@ -0,0 +1,151 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The header-matching chain.""" + +__all__ = ['HeaderMatchChain'] +__metaclass__ = type + + +import re +import logging +import itertools + +from zope.interface import implements + +from Mailman.interfaces import IChainIterator, IRule, LinkAction +from Mailman.chains.base import Chain, Link +from Mailman.i18n import _ +from Mailman.configuration import config + + +log = logging.getLogger('mailman.vette') + + + +def make_link(entry): + """Create a Link object. + + :param entry: a 2- or 3-tuple describing a link. If a 2-tuple, it is a + header and a pattern, and a default chain of 'hold' will be used. If + a 3-tuple, the third item is the chain name to use. + :return: an ILink. + """ + if len(entry) == 2: + header, pattern = entry + chain_name = 'hold' + elif len(entry) == 3: + header, pattern, chain_name = entry + # We don't assert that the chain exists here because the jump + # chain may not yet have been created. + else: + raise AssertionError('Bad link description: %s' % entry) + rule = HeaderMatchRule(header, pattern) + chain = config.chains[chain_name] + return Link(rule, LinkAction.jump, chain) + + + +class HeaderMatchRule: + """Header matching rule used by header-match chain.""" + implements(IRule) + + # Sequential rule counter. + _count = 1 + + def __init__(self, header, pattern): + self._header = header + self._pattern = pattern + self.name = 'header-match-%002d' % HeaderMatchRule._count + HeaderMatchRule._count += 1 + self.description = u'%s: %s' % (header, pattern) + # XXX I think we should do better here, somehow recording that a + # particular header matched a particular pattern, but that gets ugly + # with RFC 2822 headers. It also doesn't match well with the rule + # name concept. For now, we just record the rather useless numeric + # rule name. I suppose we could do the better hit recording in the + # check() method, and set self.record = False. + self.record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + for value in msg.get_all(self._header, []): + if re.search(self._pattern, value, re.IGNORECASE): + return True + return False + + + +class HeaderMatchChain(Chain): + """Default header matching chain. + + This could be extended by header match rules in the database. + """ + + def __init__(self): + super(HeaderMatchChain, self).__init__( + 'header-match', _('The built-in header matching chain')) + # The header match rules are not global, so don't register them. + # These are the only rules that the header match chain can execute. + self._links = [] + # Initialize header check rules with those from the global + # HEADER_MATCHES variable. + for entry in config.HEADER_MATCHES: + self._links.append(make_link(entry)) + # Keep track of how many global header matching rules we've seen. + # This is so the flush() method will only delete those that were added + # via extend() or append_link(). + self._permanent_link_count = len(self._links) + + def extend(self, header, pattern, chain_name='hold'): + """Extend the existing header matches. + + :param header: The case-insensitive header field name. + :param pattern: The pattern to match the header's value again. The + match is not anchored and is done case-insensitively. + :param chain: Option chain to jump to if the pattern matches any of + the named header values. If not given, the 'hold' chain is used. + """ + self._links.append(make_link((header, pattern, chain_name))) + + def flush(self): + """See `IMutableChain`.""" + del self._links[self._permanent_link_count:] + + def get_links(self, mlist, msg, msgdata): + """See `IChain`.""" + list_iterator = HeaderMatchIterator(mlist) + return itertools.chain(iter(self._links), iter(list_iterator)) + + def __iter__(self): + for link in self._links: + yield link + + + +class HeaderMatchIterator: + """An iterator of both the global and list-specific chain links.""" + + implements(IChainIterator) + + def __init__(self, mlist): + self._mlist = mlist + + def __iter__(self): + """See `IChainIterator`.""" + for entry in self._mlist.header_matches: + yield make_link(entry) diff --git a/Mailman/chains/hold.py b/Mailman/chains/hold.py new file mode 100644 index 000000000..7d4c64d45 --- /dev/null +++ b/Mailman/chains/hold.py @@ -0,0 +1,177 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The terminal 'hold' chain.""" + +from __future__ import with_statement + +__all__ = ['HoldChain'] +__metaclass__ = type +__i18n_templates__ = True + + +import logging + +from email.mime.message import MIMEMessage +from email.mime.text import MIMEText +from email.utils import formatdate, make_msgid +from zope.interface import implements + +from Mailman import i18n +from Mailman.Message import UserNotification +from Mailman.Utils import maketext, oneline, wrap, GetCharSet +from Mailman.app.moderator import hold_message +from Mailman.app.replybot import autorespond_to_sender, can_acknowledge +from Mailman.chains.base import TerminalChainBase +from Mailman.configuration import config +from Mailman.interfaces import IPendable + + +log = logging.getLogger('mailman.vette') +SEMISPACE = '; ' +_ = i18n._ + + + +class HeldMessagePendable(dict): + implements(IPendable) + PEND_KEY = 'held message' + + + +class HoldChain(TerminalChainBase): + """Hold a message.""" + + name = 'hold' + description = _('Hold a message and stop processing.') + + def _process(self, mlist, msg, msgdata): + """See `TerminalChainBase`.""" + # Start by decorating the message with a header that contains a list + # of all the rules that matched. These metadata could be None or an + # empty list. + rule_hits = msgdata.get('rule_hits') + if rule_hits: + msg['X-Mailman-Rule-Hits'] = SEMISPACE.join(rule_hits) + rule_misses = msgdata.get('rule_misses') + if rule_misses: + msg['X-Mailman-Rule-Misses'] = SEMISPACE.join(rule_misses) + # Hold the message by adding it to the list's request database. + # XXX How to calculate the reason? + request_id = hold_message(mlist, msg, msgdata, None) + # Calculate a confirmation token to send to the author of the + # message. + pendable = HeldMessagePendable(type=HeldMessagePendable.PEND_KEY, + id=request_id) + token = config.db.pendings.add(pendable) + # Get the language to send the response in. If the sender is a + # member, then send it in the member's language, otherwise send it in + # the mailing list's preferred language. + sender = msg.get_sender() + member = mlist.members.get_member(sender) + language = (member.preferred_language + if member else mlist.preferred_language) + # A substitution dictionary for the email templates. + charset = GetCharSet(mlist.preferred_language) + original_subject = msg.get('subject') + if original_subject is None: + original_subject = _('(no subject)') + else: + original_subject = oneline(original_subject, charset) + substitutions = { + 'listname' : mlist.fqdn_listname, + 'subject' : original_subject, + 'sender' : sender, + 'reason' : 'XXX', #reason, + 'confirmurl' : '%s/%s' % (mlist.script_url('confirm'), token), + 'admindb_url': mlist.script_url('admindb'), + } + # At this point the message is held, but now we have to craft at least + # two responses. The first will go to the original author of the + # message and it will contain the token allowing them to approve or + # discard the message. The second one will go to the moderators of + # the mailing list, if the list is so configured. + # + # Start by possibly sending a response to the message author. There + # are several reasons why we might not go through with this. If the + # message was gated from NNTP, the author may not even know about this + # list, so don't spam them. If the author specifically requested that + # acknowledgments not be sent, or if the message was bulk email, then + # we do not send the response. It's also possible that either the + # mailing list, or the author (if they are a member) have been + # configured to not send such responses. + if (not msgdata.get('fromusenet') and + can_acknowledge(msg) and + mlist.respond_to_post_requests and + autorespond_to_sender(mlist, sender, language)): + # We can respond to the sender with a message indicating their + # posting was held. + subject = _( + 'Your message to $mlist.fqdn_listname awaits moderator approval') + send_language = msgdata.get('lang', language) + text = maketext('postheld.txt', substitutions, + lang=send_language, mlist=mlist) + adminaddr = mlist.bounces_address + nmsg = UserNotification(sender, adminaddr, subject, text, + send_language) + nmsg.send(mlist) + # Now the message for the list moderators. This one should appear to + # come from <list>-owner since we really don't need to do bounce + # processing on it. + if mlist.admin_immed_notify: + # Now let's temporarily set the language context to that which the + # administrators are expecting. + with i18n.using_language(mlist.preferred_language): + language = mlist.preferred_language + charset = GetCharSet(language) + # We need to regenerate or re-translate a few values in the + # substitution dictionary. + #d['reason'] = _(reason) # XXX reason + substitutions['subject'] = original_subject + # craft the admin notification message and deliver it + subject = _( + '$mlist.fqdn_listname post from $sender requires approval') + nmsg = UserNotification(mlist.owner_address, + mlist.owner_address, + subject, lang=language) + nmsg.set_type('multipart/mixed') + text = MIMEText( + maketext('postauth.txt', substitutions, + raw=True, mlist=mlist), + _charset=charset) + dmsg = MIMEText(wrap(_("""\ +If you reply to this message, keeping the Subject: header intact, Mailman will +discard the held message. Do this if the message is spam. If you reply to +this message and include an Approved: header with the list password in it, the +message will be approved for posting to the list. The Approved: header can +also appear in the first line of the body of the reply.""")), + _charset=GetCharSet(language)) + dmsg['Subject'] = 'confirm ' + token + dmsg['Sender'] = mlist.request_address + dmsg['From'] = mlist.request_address + dmsg['Date'] = formatdate(localtime=True) + dmsg['Message-ID'] = make_msgid() + nmsg.attach(text) + nmsg.attach(MIMEMessage(msg)) + nmsg.attach(MIMEMessage(dmsg)) + nmsg.send(mlist, **{'tomoderators': 1}) + # Log the held message + # XXX reason + reason = 'n/a' + log.info('HOLD: %s post from %s held, message-id=%s: %s', + mlist.fqdn_listname, sender, + msg.get('message-id', 'n/a'), reason) diff --git a/Mailman/chains/reject.py b/Mailman/chains/reject.py new file mode 100644 index 000000000..ffec2bdc9 --- /dev/null +++ b/Mailman/chains/reject.py @@ -0,0 +1,55 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The terminal 'reject' chain.""" + +__all__ = ['RejectChain'] +__metaclass__ = type + + +import logging + +from Mailman.app.bounces import bounce_message +from Mailman.chains.base import TerminalChainBase +from Mailman.i18n import _ + + +log = logging.getLogger('mailman.vette') +SEMISPACE = '; ' + + + +class RejectChain(TerminalChainBase): + """Reject/bounce a message.""" + + name = 'reject' + description = _('Reject/bounce a message and stop processing.') + + def _process(self, mlist, msg, msgdata): + """See `TerminalChainBase`.""" + # Start by decorating the message with a header that contains a list + # of all the rules that matched. These metadata could be None or an + # empty list. + rule_hits = msgdata.get('rule_hits') + if rule_hits: + msg['X-Mailman-Rule-Hits'] = SEMISPACE.join(rule_hits) + rule_misses = msgdata.get('rule_misses') + if rule_misses: + msg['X-Mailman-Rule-Misses'] = SEMISPACE.join(rule_misses) + # XXX Exception/reason + bounce_message(mlist, msg) + log.info('REJECT: %s', msg.get('message-id', 'n/a')) diff --git a/Mailman/configuration.py b/Mailman/configuration.py index c2e88affd..5fc359578 100644 --- a/Mailman/configuration.py +++ b/Mailman/configuration.py @@ -128,17 +128,18 @@ class Configuration(object): self.PRIVATE_ARCHIVE_FILE_DIR = join(VAR_DIR, 'archives', 'private') # Directories used by the qrunner subsystem self.QUEUE_DIR = qdir = join(VAR_DIR, 'qfiles') - self.INQUEUE_DIR = join(qdir, 'in') - self.OUTQUEUE_DIR = join(qdir, 'out') - self.CMDQUEUE_DIR = join(qdir, 'commands') + self.ARCHQUEUE_DIR = join(qdir, 'archive') + self.BADQUEUE_DIR = join(qdir, 'bad') self.BOUNCEQUEUE_DIR = join(qdir, 'bounces') + self.CMDQUEUE_DIR = join(qdir, 'commands') + self.INQUEUE_DIR = join(qdir, 'in') + self.MAILDIR_DIR = join(qdir, 'maildir') self.NEWSQUEUE_DIR = join(qdir, 'news') - self.ARCHQUEUE_DIR = join(qdir, 'archive') + self.OUTQUEUE_DIR = join(qdir, 'out') + self.PREPQUEUE_DIR = join(qdir, 'prepare') + self.RETRYQUEUE_DIR = join(qdir, 'retry') self.SHUNTQUEUE_DIR = join(qdir, 'shunt') self.VIRGINQUEUE_DIR = join(qdir, 'virgin') - self.BADQUEUE_DIR = join(qdir, 'bad') - self.RETRYQUEUE_DIR = join(qdir, 'retry') - self.MAILDIR_DIR = join(qdir, 'maildir') self.MESSAGES_DIR = join(VAR_DIR, 'messages') # Other useful files self.PIDFILE = join(datadir, 'master-qrunner.pid') @@ -174,6 +175,9 @@ class Configuration(object): # Always add and enable the default server language. code = self.DEFAULT_SERVER_LANGUAGE self.languages.enable_language(code) + # Create the registry of rules and chains. + self.chains = {} + self.rules = {} def add_domain(self, email_host, url_host=None): """Add a virtual domain. diff --git a/Mailman/database/mailinglist.py b/Mailman/database/mailinglist.py index 04c872aab..b3eb56003 100644 --- a/Mailman/database/mailinglist.py +++ b/Mailman/database/mailinglist.py @@ -113,7 +113,7 @@ class MailingList(Model): gateway_to_news = Bool() generic_nonmember_action = Int() goodbye_msg = Unicode() - header_filter_rules = Pickle() + header_matches = Pickle() hold_these_nonmembers = Pickle() include_list_post_header = Bool() include_rfc2369_headers = Bool() @@ -151,6 +151,7 @@ class MailingList(Model): send_goodbye_msg = Bool() send_reminders = Bool() send_welcome_msg = Bool() + start_chain = Unicode() subject_prefix = Unicode() subscribe_auto_approval = Pickle() subscribe_policy = Int() @@ -215,7 +216,7 @@ class MailingList(Model): return self.fqdn_listname @property - def noreply_address(self): + def no_reply_address(self): return '%s@%s' % (config.NO_REPLY_ADDRESS, self.host_name) @property diff --git a/Mailman/database/mailman.sql b/Mailman/database/mailman.sql index cff4daba0..7c53f25be 100644 --- a/Mailman/database/mailman.sql +++ b/Mailman/database/mailman.sql @@ -93,7 +93,7 @@ CREATE TABLE mailinglist ( gateway_to_news BOOLEAN, generic_nonmember_action INTEGER, goodbye_msg TEXT, - header_filter_rules BLOB, + header_matches BLOB, hold_these_nonmembers BLOB, include_list_post_header BOOLEAN, include_rfc2369_headers BOOLEAN, @@ -131,6 +131,7 @@ CREATE TABLE mailinglist ( send_goodbye_msg BOOLEAN, send_reminders BOOLEAN, send_welcome_msg BOOLEAN, + start_chain TEXT, subject_prefix TEXT, subscribe_auto_approval BLOB, subscribe_policy INTEGER, @@ -145,6 +146,7 @@ CREATE TABLE member ( id INTEGER NOT NULL, role TEXT, mailing_list TEXT, + is_moderated BOOLEAN, address_id INTEGER, preferences_id INTEGER, PRIMARY KEY (id), diff --git a/Mailman/database/member.py b/Mailman/database/member.py index f77b8c7c3..b24688423 100644 --- a/Mailman/database/member.py +++ b/Mailman/database/member.py @@ -33,6 +33,7 @@ class Member(Model): id = Int(primary=True) role = Enum() mailing_list = Unicode() + is_moderated = Bool() address_id = Int() address = Reference(address_id, 'Address.id') @@ -43,6 +44,7 @@ class Member(Model): self.role = role self.mailing_list = mailing_list self.address = address + self.is_moderated = False def __repr__(self): return '<Member: %s on %s as %s>' % ( diff --git a/Mailman/database/model.py b/Mailman/database/model.py index 3b41af123..d0cc8b6e8 100644 --- a/Mailman/database/model.py +++ b/Mailman/database/model.py @@ -45,8 +45,7 @@ class ModelMeta(PropertyPublisherMeta): @staticmethod def _reset(store): for model_class in ModelMeta._class_registry: - for row in store.find(model_class): - store.remove(row) + store.find(model_class).remove() diff --git a/Mailman/database/pending.py b/Mailman/database/pending.py index 1a272391d..c3f54c814 100644 --- a/Mailman/database/pending.py +++ b/Mailman/database/pending.py @@ -112,6 +112,10 @@ class Pendings(object): value = u'__builtin__.float\1%s' % value elif type(value) is bool: value = u'__builtin__.bool\1%s' % value + elif type(value) is list: + # We expect this to be a list of strings. + value = u'Mailman.database.pending.unpack_list\1%s' % ( + '\2'.join(value)) keyval = PendedKeyValue(key=key, value=value) pending.key_values.add(keyval) config.db.store.add(pending) @@ -156,3 +160,8 @@ class Pendings(object): for keyvalue in q: store.remove(keyvalue) store.remove(pending) + + + +def unpack_list(value): + return value.split('\2') diff --git a/Mailman/docs/antispam.txt b/Mailman/docs/antispam.txt deleted file mode 100644 index 3ad5e982e..000000000 --- a/Mailman/docs/antispam.txt +++ /dev/null @@ -1,75 +0,0 @@ -Anti-spam defences -================== - -By design, Mailman does not have very sophisticated anti-spam measures because -this type of filtering is done much more efficiently at the MTA level. For -example, if Mailman were to do spam detection, it could not reject the message -at SMTP time. - -Still, Mailman does employ a small number of rather ham-handed anti-spam -measures. - - >>> from Mailman.Handlers.SpamDetect import process - >>> from Mailman.queue import Switchboard - >>> from Mailman.configuration import config - >>> mlist = config.db.list_manager.create(u'_xtest@example.com') - - -Short circuiting ----------------- - -If a message is pre-approved, this handler does nothing. - - >>> msg = message_from_string("""\ - ... From: aperson@example.com - ... - ... An important message. - ... """) - >>> msgdata = {'approved': True} - >>> process(mlist, msg, msgdata) - >>> print msg.as_string() - From: aperson@example.com - <BLANKLINE> - An important message. - <BLANKLINE> - >>> msgdata - {'approved': True} - - -Header matching ---------------- - -There is a global configuration variable that can be set to a list of header -matches. Each item in that list is a 2-tuple of the header to match and a -regular expression. For example, if we wanted to block all message that come -from 'aperson' regardless of the domain, we'd do something like the following -in our mailman.cfg file: - - >>> old_value = config.KNOWN_SPAMMERS[:] - >>> config.KNOWN_SPAMMERS.append(('from', 'aperson')) - -Now if the same message is posted to the mailing list, and that message is not -pre-approved. The handler will throw an exception that signals the message is -spam. - - >>> msgdata = {} - >>> process(mlist, msg, msgdata) - Traceback (most recent call last): - ... - SpamDetected - >>> print msg.as_string() - From: aperson@example.com - <BLANKLINE> - An important message. - <BLANKLINE> - >>> msgdata - {} - - # Restore global state - config.KNOWN_SPAMMERS = old_value - - -Header filter rules -------------------- - -XXX Need tests. diff --git a/Mailman/docs/chains.txt b/Mailman/docs/chains.txt new file mode 100644 index 000000000..433ee8e8e --- /dev/null +++ b/Mailman/docs/chains.txt @@ -0,0 +1,340 @@ +Chains +====== + +When a new message comes into the system, Mailman uses a set of rule chains to +decide whether the message gets posted to the list, rejected, discarded, or +held for moderator approval. + +There are a number of built-in chains available that act as end-points in the +processing of messages. + + +The Discard chain +----------------- + +The Discard chain simply throws the message away. + + >>> from zope.interface.verify import verifyObject + >>> from Mailman.configuration import config + >>> from Mailman.interfaces import IChain + >>> chain = config.chains['discard'] + >>> verifyObject(IChain, chain) + True + >>> chain.name + 'discard' + >>> chain.description + u'Discard a message and stop processing.' + + >>> from Mailman.app.lifecycle import create_list + >>> mlist = create_list(u'_xtest@example.com') + >>> msg = message_from_string("""\ + ... From: aperson@example.com + ... To: _xtest@example.com + ... Subject: My first post + ... Message-ID: <first> + ... + ... An important message. + ... """) + + >>> from Mailman.app.chains import process + + # XXX This checks the vette log file because there is no other evidence + # that this chain has done anything. + >>> import os + >>> fp = open(os.path.join(config.LOG_DIR, 'vette')) + >>> file_pos = fp.tell() + >>> process(mlist, msg, {}, 'discard') + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: ... DISCARD: <first> + <BLANKLINE> + + +The Reject chain +---------------- + +The Reject chain bounces the message back to the original sender, and logs +this action. + + >>> chain = config.chains['reject'] + >>> verifyObject(IChain, chain) + True + >>> chain.name + 'reject' + >>> chain.description + u'Reject/bounce a message and stop processing.' + >>> file_pos = fp.tell() + >>> process(mlist, msg, {}, 'reject') + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: ... REJECT: <first> + +The bounce message is now sitting in the Virgin queue. + + >>> from Mailman.queue import Switchboard + >>> virginq = Switchboard(config.VIRGINQUEUE_DIR) + >>> len(virginq.files) + 1 + >>> qmsg, qdata = virginq.dequeue(virginq.files[0]) + >>> print qmsg.as_string() + Subject: My first post + From: _xtest-owner@example.com + To: aperson@example.com + ... + [No bounce details are available] + ... + Content-Type: message/rfc822 + MIME-Version: 1.0 + <BLANKLINE> + From: aperson@example.com + To: _xtest@example.com + Subject: My first post + Message-ID: <first> + <BLANKLINE> + An important message. + <BLANKLINE> + ... + + +The Hold Chain +-------------- + +The Hold chain places the message into the admin request database and +depending on the list's settings, sends a notification to both the original +sender and the list moderators. + + >>> mlist.web_page_url = u'http://www.example.com/' + >>> chain = config.chains['hold'] + >>> verifyObject(IChain, chain) + True + >>> chain.name + 'hold' + >>> chain.description + u'Hold a message and stop processing.' + + >>> file_pos = fp.tell() + >>> process(mlist, msg, {}, 'hold') + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: ... HOLD: _xtest@example.com post from aperson@example.com held, + message-id=<first>: n/a + <BLANKLINE> + +There are now two messages in the Virgin queue, one to the list moderators and +one to the original author. + + >>> len(virginq.files) + 2 + >>> qfiles = [] + >>> for filebase in virginq.files: + ... qmsg, qdata = virginq.dequeue(filebase) + ... virginq.finish(filebase) + ... qfiles.append(qmsg) + >>> from operator import itemgetter + >>> qfiles.sort(key=itemgetter('to')) + +This message is addressed to the mailing list moderators. + + >>> print qfiles[0].as_string() + Subject: _xtest@example.com post from aperson@example.com requires approval + From: _xtest-owner@example.com + To: _xtest-owner@example.com + MIME-Version: 1.0 + ... + As list administrator, your authorization is requested for the + following mailing list posting: + <BLANKLINE> + List: _xtest@example.com + From: aperson@example.com + Subject: My first post + Reason: XXX + <BLANKLINE> + At your convenience, visit: + <BLANKLINE> + http://www.example.com/admindb/_xtest@example.com + <BLANKLINE> + to approve or deny the request. + <BLANKLINE> + ... + Content-Type: message/rfc822 + MIME-Version: 1.0 + <BLANKLINE> + From: aperson@example.com + To: _xtest@example.com + Subject: My first post + Message-ID: <first> + X-Message-ID-Hash: RXJU4JL6N2OUN3OYMXXPPSCR7P7JE2BW + <BLANKLINE> + An important message. + <BLANKLINE> + ... + Content-Type: message/rfc822 + MIME-Version: 1.0 + <BLANKLINE> + Content-Type: text/plain; charset="us-ascii" + MIME-Version: 1.0 + Content-Transfer-Encoding: 7bit + Subject: confirm ... + Sender: _xtest-request@example.com + From: _xtest-request@example.com + ... + <BLANKLINE> + If you reply to this message, keeping the Subject: header intact, + Mailman will discard the held message. Do this if the message is + spam. If you reply to this message and include an Approved: header + with the list password in it, the message will be approved for posting + to the list. The Approved: header can also appear in the first line + of the body of the reply. + ... + +This message is addressed to the sender of the message. + + >>> print qfiles[1].as_string() + MIME-Version: 1.0 + Content-Type: text/plain; charset="us-ascii" + Content-Transfer-Encoding: 7bit + Subject: Your message to _xtest@example.com awaits moderator approval + From: _xtest-bounces@example.com + To: aperson@example.com + ... + Your mail to '_xtest@example.com' with the subject + <BLANKLINE> + My first post + <BLANKLINE> + Is being held until the list moderator can review it for approval. + <BLANKLINE> + The reason it is being held: + <BLANKLINE> + XXX + <BLANKLINE> + Either the message will get posted to the list, or you will receive + notification of the moderator's decision. If you would like to cancel + this posting, please visit the following URL: + <BLANKLINE> + http://www.example.com/confirm/_xtest@example.com/... + <BLANKLINE> + <BLANKLINE> + +In addition, the pending database is holding the original messages, waiting +for them to be disposed of by the original author or the list moderators. The +database is essentially a dictionary, with the keys being the randomly +selected tokens included in the urls and the values being a 2-tuple where the +first item is a type code and the second item is a message id. + + >>> import re + >>> cookie = None + >>> for line in qfiles[1].get_payload().splitlines(): + ... mo = re.search('confirm/[^/]+/(?P<cookie>.*)$', line) + ... if mo: + ... cookie = mo.group('cookie') + ... break + >>> assert cookie is not None, 'No confirmation token found' + >>> data = config.db.pendings.confirm(cookie) + >>> sorted(data.items()) + [(u'id', ...), (u'type', u'held message')] + +The message itself is held in the message store. + + >>> rkey, rdata = config.db.requests.get_list_requests(mlist).get_request( + ... data['id']) + >>> msg = config.db.message_store.get_message_by_id( + ... rdata['_mod_message_id']) + >>> print msg.as_string() + From: aperson@example.com + To: _xtest@example.com + Subject: My first post + Message-ID: <first> + X-Message-ID-Hash: RXJU4JL6N2OUN3OYMXXPPSCR7P7JE2BW + <BLANKLINE> + An important message. + <BLANKLINE> + + +The Accept chain +---------------- + +The Accept chain sends the message on the 'prep' queue, where it will be +processed and sent on to the list membership. + + >>> chain = config.chains['accept'] + >>> verifyObject(IChain, chain) + True + >>> chain.name + 'accept' + >>> chain.description + u'Accept a message.' + >>> file_pos = fp.tell() + >>> process(mlist, msg, {}, 'accept') + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: ... ACCEPT: <first> + + >>> prepq = Switchboard(config.PREPQUEUE_DIR) + >>> len(prepq.files) + 1 + >>> qmsg, qdata = prepq.dequeue(prepq.files[0]) + >>> print qmsg.as_string() + From: aperson@example.com + To: _xtest@example.com + Subject: My first post + Message-ID: <first> + X-Message-ID-Hash: RXJU4JL6N2OUN3OYMXXPPSCR7P7JE2BW + <BLANKLINE> + An important message. + <BLANKLINE> + + +Run-time chains +--------------- + +We can also define chains at run time, and these chains can be mutated. +Run-time chains are made up of links where each link associates both a rule +and a 'jump'. The rule is really a rule name, which is looked up when +needed. The jump names a chain which is jumped to if the rule matches. + +There is one built-in run-time chain, called appropriately 'built-in'. This +is the default chain to use when no other input chain is defined for a mailing +list. It runs through the default rules, providing functionality similar to +the Hold handler from previous versions of Mailman. + + >>> chain = config.chains['built-in'] + >>> verifyObject(IChain, chain) + True + >>> chain.name + 'built-in' + >>> chain.description + u'The built-in moderation chain.' + +The previously created message is innocuous enough that it should pass through +all default rules. This message will end up in the prep queue. + + >>> file_pos = fp.tell() + >>> from Mailman.app.chains import process + >>> process(mlist, msg, {}) + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: ... ACCEPT: <first> + + >>> qmsg, qdata = prepq.dequeue(prepq.files[0]) + >>> print qmsg.as_string() + From: aperson@example.com + To: _xtest@example.com + Subject: My first post + Message-ID: <first> + X-Message-ID-Hash: RXJU4JL6N2OUN3OYMXXPPSCR7P7JE2BW + X-Mailman-Rule-Misses: approved; emergency; loop; administrivia; implicit-dest; + max-recipients; max-size; news-moderation; no-subject; + suspicious-header + <BLANKLINE> + An important message. + <BLANKLINE> + +In addition, the message metadata now contains lists of all rules that have +hit and all rules that have missed. + + >>> sorted(qdata['rule_hits']) + [] + >>> sorted(qdata['rule_misses']) + ['administrivia', 'approved', 'emergency', 'implicit-dest', 'loop', + 'max-recipients', 'max-size', 'news-moderation', 'no-subject', + 'suspicious-header'] diff --git a/Mailman/docs/hold.txt b/Mailman/docs/hold.txt deleted file mode 100644 index a93953435..000000000 --- a/Mailman/docs/hold.txt +++ /dev/null @@ -1,362 +0,0 @@ -Holding messages -================ - -One of the most important functions of Mailman is to moderate messages by -holding some for approval before they will post to the mailing list. Messages -are held when they meet any of a number of criteria. - - >>> import os - >>> import errno - >>> from Mailman.Handlers.Hold import process - >>> from Mailman.queue import Switchboard - >>> from Mailman.configuration import config - >>> mlist = config.db.list_manager.create(u'_xtest@example.com') - >>> mlist.preferred_language = u'en' - >>> mlist.real_name = u'_XTest' - >>> # XXX This will almost certainly change once we've worked out the web - >>> # space layout for mailing lists now. - >>> mlist.web_page_url = u'http://lists.example.com/' - -Here's a helper function used when we don't care about what's in the virgin -queue or in the pending database. - - >>> switchboard = Switchboard(config.VIRGINQUEUE_DIR) - >>> def clear(): - ... for filebase in switchboard.files: - ... msg, msgdata = switchboard.dequeue(filebase) - ... switchboard.finish(filebase) - ... for holdfile in os.listdir(config.DATA_DIR): - ... if holdfile.startswith('heldmsg-'): - ... os.unlink(os.path.join(config.DATA_DIR, holdfile)) - ... try: - ... os.unlink(os.path.join(config.DATA_DIR, 'pending.db')) - ... except OSError, e: - ... if e.errno <> errno.ENOENT: - ... raise - - -Short circuiting ----------------- - -If the message metadata indicates that the message is pre-approved, then the -handler returns immediately. - - >>> msg = message_from_string("""\ - ... From: aperson@example.com - ... - ... An important message. - ... """) - >>> msgdata = {'approved': True} - >>> process(mlist, msg, msgdata) - >>> print msg.as_string() - From: aperson@example.com - <BLANKLINE> - An important message. - <BLANKLINE> - >>> msgdata - {'approved': True} - - -Administrivia -------------- - -Mailman scans parts of the message for administrivia, meaning text that looks -like an email command. This is to prevent people sending 'help' or -'subscribe' message, etc. to the list members. First, we enable the scanning -of administrivia for the list. - - >>> mlist.administrivia = True - >>> msg = message_from_string("""\ - ... From: aperson@example.com - ... Subject: unsubscribe - ... - ... """) - >>> process(mlist, msg, {}) - Traceback (most recent call last): - ... - Administrivia - >>> clear() - - -Maximum number of recipients ----------------------------- - -Mailman will hold messages that have more than a specified number of explicit -recipients. - - >>> mlist.max_num_recipients = 5 - >>> msg = message_from_string("""\ - ... From: aperson@example.com - ... To: _xtest@example.com, bperson@example.com - ... Cc: cperson@example.com - ... Cc: dperson@example.com (Dan Person) - ... To: Elly Q. Person <eperson@example.com> - ... - ... Hey folks! - ... """) - >>> process(mlist, msg, {}) - Traceback (most recent call last): - ... - TooManyRecipients - >>> clear() - - -Implicit destination --------------------- - -Mailman will hold messages that have implicit destination, meaning that the -mailing list's posting address isn't included in the explicit recipients. - - >>> mlist.require_explicit_destination = True - >>> mlist.acceptable_aliases = u'' - >>> msg = message_from_string("""\ - ... From: aperson@example.org - ... Subject: An implicit message - ... - ... """) - >>> process(mlist, msg, {}) - Traceback (most recent call last): - ... - ImplicitDestination - >>> clear() - -A message gated from NNTP will obviously have an implicit destination. Such -gated messages will not be held for implicit destination because it's assumed -that Mailman pulled it from the appropriate news group. - - >>> msgdata = {'fromusenet': True} - >>> process(mlist, msg, msgdata) - >>> print msg.as_string() - From: aperson@example.org - Subject: An implicit message - Message-ID: ... - X-Message-ID-Hash: ... - <BLANKLINE> - <BLANKLINE> - >>> print msgdata - {'fromusenet': True} - - -Suspicious headers ------------------- - -Suspicious headers are a way for Mailman to hold messages that match a -particular regular expression. This mostly historical feature is fairly -confusing to users, and the list attribute that controls this is misnamed. - - >>> mlist.bounce_matching_headers = u'From: .*person@(blah.)?example.com' - >>> msg = message_from_string("""\ - ... From: aperson@example.com - ... To: _xtest@example.com - ... Subject: An implicit message - ... - ... """) - >>> process(mlist, msg, {}) - Traceback (most recent call last): - ... - SuspiciousHeaders - >>> clear() - -But if the header doesn't match the regular expression, it'll get posted just -fine. This one comes from a .org address. - - >>> msg = message_from_string("""\ - ... From: aperson@example.org - ... To: _xtest@example.com - ... Subject: An implicit message - ... - ... """) - >>> msgdata = {} - >>> process(mlist, msg, msgdata) - >>> print msgdata - {} - -Just a bit of clean up. - - >>> mlist.bounce_matching_headers = None - - -Message size ------------- - -Mailman can hold messages that are bigger than a given size. Generally this -is used to prevent huge attachments from getting posted to the list. This -value is calculated in terms of KB (1024 bytes). - - >>> mlist.max_message_size = 1 - >>> one_line = 'x' * 79 - >>> big_body = '\n'.join([one_line] * 15) - >>> msg = message_from_string("""\ - ... From: aperson@example.com - ... To: _xtest@example.com - ... - ... """ + big_body) - >>> process(mlist, msg, {}) - Traceback (most recent call last): - ... - MessageTooBig - >>> clear() - - -Hold Notifications ------------------- - -Whenever Mailman holds a message, it sends notifications both to the list -owner and to the original sender, as long as it is configured to do so. We -can show this by first holding a message. - - >>> mlist.respond_to_post_requests = True - >>> mlist.admin_immed_notify = True - >>> msg = message_from_string("""\ - ... From: aperson@example.com - ... - ... """) - >>> process(mlist, msg, {}) - Traceback (most recent call last): - ... - ImplicitDestination - -There should be two messages in the virgin queue, one to the list owner and -one to the original author. - - >>> len(switchboard.files) - 2 - >>> qfiles = {} - >>> for filebase in switchboard.files: - ... qmsg, qdata = switchboard.dequeue(filebase) - ... switchboard.finish(filebase) - ... qfiles[qmsg['to']] = qmsg, qdata - >>> qmsg, qdata = qfiles['_xtest-owner@example.com'] - >>> print qmsg.as_string() - Subject: _xtest post from aperson@example.com requires approval - From: _xtest-owner@example.com - To: _xtest-owner@example.com - MIME-Version: 1.0 - Content-Type: multipart/mixed; boundary="..." - Message-ID: ... - Date: ... - Precedence: bulk - <BLANKLINE> - --... - Content-Type: text/plain; charset="us-ascii" - MIME-Version: 1.0 - Content-Transfer-Encoding: 7bit - <BLANKLINE> - As list administrator, your authorization is requested for the - following mailing list posting: - <BLANKLINE> - List: _xtest@example.com - From: aperson@example.com - Subject: (no subject) - Reason: Message has implicit destination - <BLANKLINE> - At your convenience, visit: - <BLANKLINE> - http://lists.example.com/admindb/_xtest@example.com - <BLANKLINE> - to approve or deny the request. - <BLANKLINE> - --... - Content-Type: message/rfc822 - MIME-Version: 1.0 - <BLANKLINE> - From: aperson@example.com - Message-ID: ... - X-Message-ID-Hash: ... - <BLANKLINE> - <BLANKLINE> - --... - Content-Type: message/rfc822 - MIME-Version: 1.0 - <BLANKLINE> - Content-Type: text/plain; charset="us-ascii" - MIME-Version: 1.0 - Content-Transfer-Encoding: 7bit - Subject: confirm ... - Sender: _xtest-request@example.com - From: _xtest-request@example.com - Date: ... - Message-ID: ... - <BLANKLINE> - If you reply to this message, keeping the Subject: header intact, - Mailman will discard the held message. Do this if the message is - spam. If you reply to this message and include an Approved: header - with the list password in it, the message will be approved for posting - to the list. The Approved: header can also appear in the first line - of the body of the reply. - --... - >>> sorted(qdata.items()) - [('_parsemsg', False), ('listname', u'_xtest@example.com'), - ('nodecorate', True), ('received_time', ...), - ('recips', [u'_xtest-owner@example.com']), - ('reduced_list_headers', True), - ('tomoderators', 1), ('version', 3)] - >>> qmsg, qdata = qfiles['aperson@example.com'] - >>> print qmsg.as_string() - MIME-Version: 1.0 - Content-Type: text/plain; charset="us-ascii" - Content-Transfer-Encoding: 7bit - Subject: Your message to _xtest awaits moderator approval - From: _xtest-bounces@example.com - To: aperson@example.com - Message-ID: ... - Date: ... - Precedence: bulk - <BLANKLINE> - Your mail to '_xtest' with the subject - <BLANKLINE> - (no subject) - <BLANKLINE> - Is being held until the list moderator can review it for approval. - <BLANKLINE> - The reason it is being held: - <BLANKLINE> - Message has implicit destination - <BLANKLINE> - Either the message will get posted to the list, or you will receive - notification of the moderator's decision. If you would like to cancel - this posting, please visit the following URL: - <BLANKLINE> - http://lists.example.com/confirm/_xtest@example.com/... - <BLANKLINE> - <BLANKLINE> - >>> sorted(qdata.items()) - [('_parsemsg', False), ('listname', u'_xtest@example.com'), - ('nodecorate', True), ('received_time', ...), - ('recips', [u'aperson@example.com']), - ('reduced_list_headers', True), ('version', 3)] - -In addition, the pending database is holding the original messages, waiting -for them to be disposed of by the original author or the list moderators. The -database is essentially a dictionary, with the keys being the randomly -selected tokens included in the urls and the values being a 2-tuple where the -first item is a type code and the second item is a message id. - - >>> import re - >>> cookie = None - >>> qmsg, qdata = qfiles['aperson@example.com'] - >>> for line in qmsg.get_payload().splitlines(): - ... mo = re.search('confirm/[^/]+/(?P<cookie>.*)$', line) - ... if mo: - ... cookie = mo.group('cookie') - ... break - >>> data = config.db.pendings.confirm(cookie) - >>> sorted(data.items()) - [(u'id', ...), (u'type', u'held message')] - -The message itself is held in the message store. - - >>> rkey, rdata = config.db.requests.get_list_requests(mlist).get_request( - ... data['id']) - >>> msg = config.db.message_store.get_message_by_id( - ... rdata['_mod_message_id']) - >>> print msg.as_string() - From: aperson@example.com - Message-ID: ... - X-Message-ID-Hash: ... - <BLANKLINE> - <BLANKLINE> - -Clean up. - - >>> clear() diff --git a/Mailman/docs/mlist-addresses.txt b/Mailman/docs/mlist-addresses.txt index dc2184175..4685a6eea 100644 --- a/Mailman/docs/mlist-addresses.txt +++ b/Mailman/docs/mlist-addresses.txt @@ -18,7 +18,7 @@ list. This is exactly the same as the fully qualified list name. Messages to the mailing list's 'no reply' address always get discarded without prejudice. - >>> mlist.noreply_address + >>> mlist.no_reply_address u'noreply@example.com' The mailing list's owner address reaches the human moderators. diff --git a/Mailman/docs/requests.txt b/Mailman/docs/requests.txt index ea4dcc75d..6f7dd2f14 100644 --- a/Mailman/docs/requests.txt +++ b/Mailman/docs/requests.txt @@ -318,7 +318,7 @@ indicates that the message has been approved. <BLANKLINE> >>> sorted(qdata.items()) [('_parsemsg', False), - ('adminapproved', True), (u'approved', True), + (u'approved', True), ('moderator_approved', True), (u'received_time', 123.45), (u'sender', u'aperson@example.com'), ('version', 3)] diff --git a/Mailman/initialize.py b/Mailman/initialize.py index dff2677f3..3577ebad6 100644 --- a/Mailman/initialize.py +++ b/Mailman/initialize.py @@ -63,6 +63,12 @@ def initialize_2(debug=False): verifyObject(IDatabase, database) database.initialize(debug) Mailman.configuration.config.db = database + # Initialize the rules and chains. Do the imports here so as to avoid + # circular imports. + from Mailman.app.chains import initialize as initialize_chains + from Mailman.app.rules import initialize as initialize_rules + initialize_rules() + initialize_chains() def initialize(config_path=None, propagate_logs=False): diff --git a/Mailman/inject.py b/Mailman/inject.py new file mode 100644 index 000000000..a17f8c7d1 --- /dev/null +++ b/Mailman/inject.py @@ -0,0 +1,34 @@ +# Copyright (C) 2001-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +from Mailman.configuration import config +from Mailman.queue import Switchboard + + + +def inject(listname, msg, recips=None, qdir=None): + if qdir is None: + qdir = config.INQUEUE_DIR + queue = Switchboard(qdir) + kws = dict( + listname=listname, + tolist=True, + _plaintext=True, + ) + if recips is not None: + kws['recips'] = recips + queue.enqueue(msg, **kws) diff --git a/Mailman/interfaces/__init__.py b/Mailman/interfaces/__init__.py index c4094e365..20dff7fdf 100644 --- a/Mailman/interfaces/__init__.py +++ b/Mailman/interfaces/__init__.py @@ -46,7 +46,12 @@ def _populate(): is_enum = issubclass(obj, Enum) except TypeError: is_enum = False - if IInterface.providedBy(obj) or is_enum: + is_interface = IInterface.providedBy(obj) + try: + is_exception = issubclass(obj, Exception) + except TypeError: + is_exception = False + if is_interface or is_exception or is_enum: setattr(iface_mod, name, obj) __all__.append(name) diff --git a/Mailman/interfaces/chain.py b/Mailman/interfaces/chain.py new file mode 100644 index 000000000..04c0260c2 --- /dev/null +++ b/Mailman/interfaces/chain.py @@ -0,0 +1,98 @@ +# Copyright (C) 2007 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Interface describing the basics of chains and links.""" + +from munepy import Enum +from zope.interface import Interface, Attribute + + + +class LinkAction(Enum): + # Jump to another chain. + jump = 0 + # Take a detour to another chain, returning to the original chain when + # completed (if no other jump occurs). + detour = 1 + # Stop processing all chains. + stop = 2 + # Continue processing the next link in the chain. + defer = 3 + # Run a function and continue processing. + run = 4 + + + +class IChainLink(Interface): + """A link in the chain.""" + + rule = Attribute('The rule to run for this link.') + + action = Attribute('The LinkAction to take if this rule matches.') + + chain = Attribute('The chain to jump or detour to.') + + function = Attribute( + """The function to execute. + + The function takes three arguments and returns nothing. + :param mlist: the IMailingList object + :param msg: the message being processed + :param msgdata: the message metadata dictionary + """) + + + +class IChain(Interface): + """A chain of rules.""" + + name = Attribute('Chain name; must be unique.') + description = Attribute('A brief description of the chain.') + + def get_links(mlist, msg, msgdata): + """Get an `IChainIterator` for processing. + + :param mlist: the IMailingList object + :param msg: the message being processed + :param msgdata: the message metadata dictionary + :return: An `IChainIterator`. + """ + + + +class IChainIterator(Interface): + """An iterator over chain rules.""" + + def __iter__(): + """Iterate over all the IChainLinks in this chain. + + :return: an IChainLink. + """ + + + +class IMutableChain(IChain): + """Like `IChain` but can be mutated.""" + + def append_link(link): + """Add a new chain link to the end of this chain. + + :param link: The chain link to add. + """ + + def flush(): + """Delete all links in this chain.""" diff --git a/Mailman/interfaces/mailinglist.py b/Mailman/interfaces/mailinglist.py index 2d3811785..2c828a43c 100644 --- a/Mailman/interfaces/mailinglist.py +++ b/Mailman/interfaces/mailinglist.py @@ -82,7 +82,7 @@ class IMailingList(Interface): delivery is currently enabled. """) - noreply_address = Attribute( + no_reply_address = Attribute( """The address to which all messages will be immediately discarded, without prejudice or record. This address is specific to the ddomain, even though it's available on the IMailingListAddresses interface. diff --git a/Mailman/interfaces/member.py b/Mailman/interfaces/member.py index 18f0b034e..8fc410a77 100644 --- a/Mailman/interfaces/member.py +++ b/Mailman/interfaces/member.py @@ -78,6 +78,9 @@ class IMember(Interface): role = Attribute( """The role of this membership.""") + is_moderated = Attribute( + """True if the membership is moderated, otherwise False.""") + def unsubscribe(): """Unsubscribe (and delete) this member from the mailing list.""" diff --git a/Mailman/interfaces/rules.py b/Mailman/interfaces/rules.py new file mode 100644 index 000000000..e92b354e1 --- /dev/null +++ b/Mailman/interfaces/rules.py @@ -0,0 +1,45 @@ +# Copyright (C) 2007 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Interface describing the basics of rules.""" + +from zope.interface import Interface, Attribute + + + +class IRule(Interface): + """A basic rule.""" + + name = Attribute('Rule name; must be unique.') + + description = Attribute('A brief description of the rule.') + + record = Attribute( + """Should this rule's success or failure be recorded? + + This is a boolean; if True then this rule's hit or miss will be + recorded in a message header. If False, it won't. + """) + + def check(mlist, msg, msgdata): + """Run the rule. + + :param mlist: The mailing list object. + :param msg: The message object. + :param msgdata: The message metadata. + :returns: a boolean specifying whether the rule matched or not. + """ diff --git a/Mailman/queue/__init__.py b/Mailman/queue/__init__.py index c415834ba..6a1873d60 100644 --- a/Mailman/queue/__init__.py +++ b/Mailman/queue/__init__.py @@ -140,7 +140,13 @@ class Switchboard: msg = cPickle.load(fp) data = cPickle.load(fp) if data.get('_parsemsg'): + # Calculate the original size of the text now so that we won't + # have to generate the message later when we do size restriction + # checking. + original_size = len(msg) msg = email.message_from_string(msg, Message.Message) + msg.original_size = original_size + data['original_size'] = original_size return msg, data def finish(self, filebase, preserve=False): diff --git a/Mailman/queue/command.py b/Mailman/queue/command.py index dfbfd1255..8628411de 100644 --- a/Mailman/queue/command.py +++ b/Mailman/queue/command.py @@ -88,8 +88,8 @@ class Results: assert isinstance(body, basestring) lines = body.splitlines() # Use no more lines than specified - self.commands.extend(lines[:config.DEFAULT_MAIL_COMMANDS_MAX_LINES]) - self.ignored.extend(lines[config.DEFAULT_MAIL_COMMANDS_MAX_LINES:]) + self.commands.extend(lines[:config.EMAIL_COMMANDS_MAX_LINES]) + self.ignored.extend(lines[config.EMAIL_COMMANDS_MAX_LINES:]) def process(self): # Now, process each line until we find an error. The first diff --git a/Mailman/queue/docs/OVERVIEW.txt b/Mailman/queue/docs/OVERVIEW.txt new file mode 100644 index 000000000..643fa8a5c --- /dev/null +++ b/Mailman/queue/docs/OVERVIEW.txt @@ -0,0 +1,78 @@ +Alias overview +============== + +A typical Mailman list exposes nine aliases which point to seven different +wrapped scripts. E.g. for a list named `mylist', you'd have: + + mylist-bounces -> bounces + mylist-confirm -> confirm + mylist-join -> join (-subscribe is an alias) + mylist-leave -> leave (-unsubscribe is an alias) + mylist-owner -> owner + mylist -> post + mylist-request -> request + +-request, -join, and -leave are a robot addresses; their sole purpose is to +process emailed commands, although the latter two are hardcoded to +subscription and unsubscription requests. -bounces is the automated bounce +processor, and all messages to list members have their return address set to +-bounces. If the bounce processor fails to extract a bouncing member address, +it can optionally forward the message on to the list owners. + +-owner is for reaching a human operator with minimal list interaction (i.e. no +bounce processing). -confirm is another robot address which processes replies +to VERP-like confirmation notices. + +So delivery flow of messages look like this: + + joerandom ---> mylist ---> list members + | | + | |[bounces] + | mylist-bounces <---+ <-------------------------------+ + | | | + | +--->[internal bounce processing] | + | ^ | | + | | | [bounce found] | + | [bounces *] +--->[register and discard] | + | | | | | + | | | |[*] | + | [list owners] |[no bounce found] | | + | ^ | | | + | | | | | + +-------> mylist-owner <--------+ | | + | | | + | data/owner-bounces.mbox <--[site list] <---+ | + | | + +-------> mylist-join--+ | + | | | + +------> mylist-leave--+ | + | | | + | v | + +-------> mylist-request | + | | | + | +---> [command processor] | + | | | + +-----> mylist-confirm ----> +---> joerandom | + | | + |[bounces] | + +----------------------+ + +A person can send an email to the list address (for posting), the -owner +address (to reach the human operator), or the -confirm, -join, -leave, and +-request mailbots. Message to the list address are then forwarded on to the +list membership, with bounces directed to the -bounces address. + +[*] Messages sent to the -owner address are forwarded on to the list +owner/moderators. All -owner destined messages have their bounces directed to +the site list -bounces address, regardless of whether a human sent the message +or the message was crafted internally. The intention here is that the site +owners want to be notified when one of their list owners' addresses starts +bouncing (yes, the will be automated in a future release). + +Any messages to site owners has their bounces directed to a special +"loop-killer" address, which just dumps the message into +data/owners-bounces.mbox. + +Finally, message to any of the mailbots causes the requested action to be +performed. Results notifications are sent to the author of the message, which +all bounces pointing back to the -bounces address. diff --git a/Mailman/queue/docs/incoming.txt b/Mailman/queue/docs/incoming.txt new file mode 100644 index 000000000..04c0cfa04 --- /dev/null +++ b/Mailman/queue/docs/incoming.txt @@ -0,0 +1,198 @@ +The incoming queue runner +========================= + +This runner's sole purpose in life is to decide the disposition of the +message. It can either be accepted for delivery, rejected (i.e. bounced), +held for moderator approval, or discarded. + +The runner operates by processing chains on a message/metadata pair in the +context of a mailing list. Each mailing list may have a 'start chain' where +processing begins, with a global default. This chain is processed with the +message eventually ending up in one of the four disposition states described +above. + + >>> from Mailman.app.lifecycle import create_list + >>> mlist = create_list(u'_xtest@example.com') + >>> mlist.start_chain + u'built-in' + + +Accepted messages +----------------- + +We have a message that is going to be sent to the mailing list. This message +is so perfectly fine for posting that it will be accepted and forward to the +prep queue. + + >>> msg = message_from_string("""\ + ... From: aperson@example.com + ... To: _xtest@example.com + ... Subject: My first post + ... Message-ID: <first> + ... + ... First post! + ... """) + +Normally, the upstream mail server would drop the message in the incoming +queue, but this is an effective simulation. + + >>> from Mailman.inject import inject + >>> inject(u'_xtest@example.com', msg) + +The incoming queue runner runs until it is empty. + + >>> from Mailman.queue.incoming import IncomingRunner + >>> from Mailman.tests.helpers import make_testable_runner + >>> incoming = make_testable_runner(IncomingRunner) + >>> incoming.run() + +And now the message is in the prep queue. + + >>> from Mailman.configuration import config + >>> from Mailman.queue import Switchboard + >>> prep_queue = Switchboard(config.PREPQUEUE_DIR) + >>> len(prep_queue.files) + 1 + >>> incoming_queue = Switchboard(config.INQUEUE_DIR) + >>> len(incoming_queue.files) + 0 + >>> from Mailman.tests.helpers import get_queue_messages + >>> item = get_queue_messages(prep_queue)[0] + >>> print item.msg.as_string() + From: aperson@example.com + To: _xtest@example.com + Subject: My first post + Message-ID: <first> + X-Mailman-Rule-Misses: approved; emergency; loop; administrivia; + implicit-dest; + max-recipients; max-size; news-moderation; no-subject; + suspicious-header + <BLANKLINE> + First post! + <BLANKLINE> + >>> sorted(item.msgdata.items()) + [...('envsender', u'noreply@example.com')...('tolist', True)...] + + +Held messages +------------- + +The list moderator sets the emergency flag on the mailing list. The built-in +chain will now hold all posted messages, so nothing will show up in the prep +queue. + + # XXX This checks the vette log file because there is no other evidence + # that this chain has done anything. + >>> import os + >>> fp = open(os.path.join(config.LOG_DIR, 'vette')) + >>> fp.seek(0, 2) + + >>> mlist.emergency = True + >>> mlist.web_page_url = u'http://archives.example.com/' + >>> inject(u'_xtest@example.com', msg) + >>> file_pos = fp.tell() + >>> incoming.run() + >>> len(prep_queue.files) + 0 + >>> len(incoming_queue.files) + 0 + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: ... HOLD: _xtest@example.com post from aperson@example.com held, + message-id=<first>: n/a + <BLANKLINE> + + >>> mlist.emergency = False + + +Discarded messages +------------------ + +Another possibility is that the message would get immediately discarded. The +built-in chain does not have such a disposition by default, so let's craft a +new chain and set it as the mailing list's start chain. + + >>> from Mailman.chains.base import Chain, Link + >>> from Mailman.interfaces import LinkAction + >>> truth_rule = config.rules['truth'] + >>> discard_chain = config.chains['discard'] + >>> test_chain = Chain('always-discard', u'Testing discards') + >>> link = Link(truth_rule, LinkAction.jump, discard_chain) + >>> test_chain.append_link(link) + >>> mlist.start_chain = u'always-discard' + + >>> inject(u'_xtest@example.com', msg) + >>> file_pos = fp.tell() + >>> incoming.run() + >>> len(prep_queue.files) + 0 + >>> len(incoming_queue.files) + 0 + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: ... DISCARD: <first> + <BLANKLINE> + + >>> del config.chains['always-discard'] + + +Rejected messages +----------------- + +Similar to discarded messages, a message can be rejected, or bounced back to +the original sender. Again, the built-in chain doesn't support this so we'll +just create a new chain that does. + + >>> reject_chain = config.chains['reject'] + >>> test_chain = Chain('always-reject', u'Testing rejections') + >>> link = Link(truth_rule, LinkAction.jump, reject_chain) + >>> test_chain.append_link(link) + >>> mlist.start_chain = u'always-reject' + +The virgin queue needs to be cleared out due to artifacts from the previous +tests above. + + >>> virgin_queue = Switchboard(config.VIRGINQUEUE_DIR) + >>> ignore = get_queue_messages(virgin_queue) + + >>> inject(u'_xtest@example.com', msg) + >>> file_pos = fp.tell() + >>> incoming.run() + >>> len(prep_queue.files) + 0 + >>> len(incoming_queue.files) + 0 + + >>> len(virgin_queue.files) + 1 + >>> item = get_queue_messages(virgin_queue)[0] + >>> print item.msg.as_string() + Subject: My first post + From: _xtest-owner@example.com + To: aperson@example.com + ... + Content-Type: text/plain; charset="us-ascii" + MIME-Version: 1.0 + Content-Transfer-Encoding: 7bit + <BLANKLINE> + [No bounce details are available] + ... + Content-Type: message/rfc822 + MIME-Version: 1.0 + <BLANKLINE> + From: aperson@example.com + To: _xtest@example.com + Subject: My first post + Message-ID: <first> + <BLANKLINE> + First post! + <BLANKLINE> + ... + >>> sorted(item.msgdata.items()) + [...('recips', [u'aperson@example.com'])...] + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: ... REJECT: <first> + <BLANKLINE> + + >>> del config.chains['always-reject'] diff --git a/Mailman/docs/news-runner.txt b/Mailman/queue/docs/news.txt index bc6619f50..bc6619f50 100644 --- a/Mailman/docs/news-runner.txt +++ b/Mailman/queue/docs/news.txt diff --git a/Mailman/docs/outgoing.txt b/Mailman/queue/docs/outgoing.txt index ba2c6430b..ba2c6430b 100644 --- a/Mailman/docs/outgoing.txt +++ b/Mailman/queue/docs/outgoing.txt diff --git a/Mailman/docs/runner.txt b/Mailman/queue/docs/runner.txt index 5e5a88d8c..5e5a88d8c 100644 --- a/Mailman/docs/runner.txt +++ b/Mailman/queue/docs/runner.txt diff --git a/Mailman/docs/switchboard.txt b/Mailman/queue/docs/switchboard.txt index 299aba499..299aba499 100644 --- a/Mailman/docs/switchboard.txt +++ b/Mailman/queue/docs/switchboard.txt diff --git a/Mailman/queue/incoming.py b/Mailman/queue/incoming.py index 6118a7ca0..649ce2213 100644 --- a/Mailman/queue/incoming.py +++ b/Mailman/queue/incoming.py @@ -1,4 +1,4 @@ -# Copyright (C) 1998-2007 by the Free Software Foundation, Inc. +# Copyright (C) 1998-2008 by the Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License @@ -12,102 +12,26 @@ # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. -"""Incoming queue runner.""" - -# A typical Mailman list exposes nine aliases which point to seven different -# wrapped scripts. E.g. for a list named `mylist', you'd have: -# -# mylist-bounces -> bounces (-admin is a deprecated alias) -# mylist-confirm -> confirm -# mylist-join -> join (-subscribe is an alias) -# mylist-leave -> leave (-unsubscribe is an alias) -# mylist-owner -> owner -# mylist -> post -# mylist-request -> request -# -# -request, -join, and -leave are a robot addresses; their sole purpose is to -# process emailed commands in a Majordomo-like fashion (although the latter -# two are hardcoded to subscription and unsubscription requests). -bounces is -# the automated bounce processor, and all messages to list members have their -# return address set to -bounces. If the bounce processor fails to extract a -# bouncing member address, it can optionally forward the message on to the -# list owners. -# -# -owner is for reaching a human operator with minimal list interaction -# (i.e. no bounce processing). -confirm is another robot address which -# processes replies to VERP-like confirmation notices. -# -# So delivery flow of messages look like this: -# -# joerandom ---> mylist ---> list members -# | | -# | |[bounces] -# | mylist-bounces <---+ <-------------------------------+ -# | | | -# | +--->[internal bounce processing] | -# | ^ | | -# | | | [bounce found] | -# | [bounces *] +--->[register and discard] | -# | | | | | -# | | | |[*] | -# | [list owners] |[no bounce found] | | -# | ^ | | | -# | | | | | -# +-------> mylist-owner <--------+ | | -# | | | -# | data/owner-bounces.mbox <--[site list] <---+ | -# | | -# +-------> mylist-join--+ | -# | | | -# +------> mylist-leave--+ | -# | | | -# | v | -# +-------> mylist-request | -# | | | -# | +---> [command processor] | -# | | | -# +-----> mylist-confirm ----> +---> joerandom | -# | | -# |[bounces] | -# +----------------------+ -# -# A person can send an email to the list address (for posting), the -owner -# address (to reach the human operator), or the -confirm, -join, -leave, and -# -request mailbots. Message to the list address are then forwarded on to the -# list membership, with bounces directed to the -bounces address. -# -# [*] Messages sent to the -owner address are forwarded on to the list -# owner/moderators. All -owner destined messages have their bounces directed -# to the site list -bounces address, regardless of whether a human sent the -# message or the message was crafted internally. The intention here is that -# the site owners want to be notified when one of their list owners' addresses -# starts bouncing (yes, the will be automated in a future release). -# -# Any messages to site owners has their bounces directed to a special -# "loop-killer" address, which just dumps the message into -# data/owners-bounces.mbox. -# -# Finally, message to any of the mailbots causes the requested action to be -# performed. Results notifications are sent to the author of the message, -# which all bounces pointing back to the -bounces address. +"""Incoming queue runner. +This runner's sole purpose in life is to decide the disposition of the +message. It can either be accepted for delivery, rejected (i.e. bounced), +held for moderator approval, or discarded. - -import os -import sys -import logging +When accepted, the message is forwarded on to the `prep queue` where it is +prepared for delivery. Rejections, discards, and holds are processed +immediately. +""" -from cStringIO import StringIO -from Mailman import Errors + +from Mailman.app.chains import process from Mailman.configuration import config from Mailman.queue import Runner -log = logging.getLogger('mailman.error') -vlog = logging.getLogger('mailman.vette') - class IncomingRunner(Runner): @@ -115,59 +39,8 @@ class IncomingRunner(Runner): def _dispose(self, mlist, msg, msgdata): if msgdata.get('envsender') is None: - msg['envsender'] = mlist.no_reply_address - # Process the message through a handler pipeline. The handler - # pipeline can actually come from one of three places: the message - # metadata, the mlist, or the global pipeline. - # - # If a message was requeued due to an uncaught exception, its metadata - # will contain the retry pipeline. Use this above all else. - # Otherwise, if the mlist has a `pipeline' attribute, it should be - # used. Final fallback is the global pipeline. - pipeline = self._get_pipeline(mlist, msg, msgdata) - msgdata['pipeline'] = pipeline - more = self._dopipeline(mlist, msg, msgdata, pipeline) - if not more: - del msgdata['pipeline'] - config.db.commit() - return more - - # Overridable - def _get_pipeline(self, mlist, msg, msgdata): - # We must return a copy of the list, otherwise, the first message that - # flows through the pipeline will empty it out! - return msgdata.get('pipeline', - getattr(mlist, 'pipeline', - config.GLOBAL_PIPELINE))[:] - - def _dopipeline(self, mlist, msg, msgdata, pipeline): - while pipeline: - handler = pipeline.pop(0) - modname = 'Mailman.Handlers.' + handler - __import__(modname) - try: - pid = os.getpid() - sys.modules[modname].process(mlist, msg, msgdata) - # Failsafe -- a child may have leaked through. - if pid <> os.getpid(): - log.error('child process leaked thru: %s', modname) - os._exit(1) - except Errors.DiscardMessage: - # Throw the message away; we need do nothing else with it. - vlog.info('Message discarded, msgid: %s', - msg.get('message-id', 'n/a')) - return 0 - except Errors.HoldMessage: - # Let the approval process take it from here. The message no - # longer needs to be queued. - return 0 - except Errors.RejectMessage, e: - mlist.bounce_message(msg, e) - return 0 - except: - # Push this pipeline module back on the stack, then re-raise - # the exception. - pipeline.insert(0, handler) - raise - # We've successfully completed handling of this message - return 0 + msgdata['envsender'] = mlist.no_reply_address + # Process the message through the mailing list's start chain. + process(mlist, msg, msgdata, mlist.start_chain) + # Do not keep this message queued. + return False diff --git a/Mailman/rules/__init__.py b/Mailman/rules/__init__.py new file mode 100644 index 000000000..a78e9bd05 --- /dev/null +++ b/Mailman/rules/__init__.py @@ -0,0 +1,50 @@ +# Copyright (C) 2007 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The built in rule set.""" + +__all__ = ['initialize'] +__metaclass__ = type + + +import os +import sys + +from Mailman.interfaces import IRule + + + +def initialize(): + """Initialize the built-in rules. + + Rules are auto-discovered by searching for IRule implementations in all + importable modules in this subpackage. + """ + # Find all rules found in all modules inside our package. + import Mailman.rules + here = os.path.dirname(Mailman.rules.__file__) + for filename in os.listdir(here): + basename, extension = os.path.splitext(filename) + if extension <> '.py': + continue + module_name = 'Mailman.rules.' + basename + __import__(module_name, fromlist='*') + module = sys.modules[module_name] + for name in module.__all__: + rule = getattr(module, name) + if IRule.implementedBy(rule): + yield rule diff --git a/Mailman/rules/administrivia.py b/Mailman/rules/administrivia.py new file mode 100644 index 000000000..76848fecf --- /dev/null +++ b/Mailman/rules/administrivia.py @@ -0,0 +1,98 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The administrivia rule.""" + +__all__ = ['Administrivia'] +__metaclass__ = type + + +from email.iterators import typed_subpart_iterator +from zope.interface import implements + +from Mailman.configuration import config +from Mailman.i18n import _ +from Mailman.interfaces import IRule + + +# The list of email commands we search for in the Subject header and payload. +# We probably should get this information from the actual implemented +# commands. +EMAIL_COMMANDS = { + # keyword: (minimum #args, maximum #args) + 'confirm': (1, 1), + 'help': (0, 0), + 'info': (0, 0), + 'lists': (0, 0), + 'options': (0, 0), + 'password': (2, 2), + 'remove': (0, 0), + 'set': (3, 3), + 'subscribe': (0, 3), + 'unsubscribe': (0, 1), + 'who': (0, 2), + } + + + +class Administrivia: + """The administrivia rule.""" + implements(IRule) + + name = 'administrivia' + description = _('Catch mis-addressed email commands.') + record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + # The list must have the administrivia check enabled. + if not mlist.administrivia: + return False + # First check the Subject text. + lines_to_check = [] + subject = msg.get('subject', '') + if subject <> '': + lines_to_check.append(subject) + # Search only the first text/plain subpart of the message. There's + # really no good way to find email commands in any other content type. + for part in typed_subpart_iterator(msg, 'text', 'plain'): + payload = part.get_payload(decode=True) + lines = payload.splitlines() + # Count lines without using enumerate() because blank lines in the + # payload don't count against the maximum examined. + lineno = 0 + for line in lines: + line = line.strip() + if line == '': + continue + lineno += 1 + if lineno > config.EMAIL_COMMANDS_MAX_LINES: + break + lines_to_check.append(line) + # Only look at the first text/plain part. + break + # For each line we're checking, split the line into words. Then see + # if it looks like a command with the min-to-max number of arguments. + for line in lines_to_check: + words = [word.lower() for word in line.split()] + if words[0] not in EMAIL_COMMANDS: + # This is not an administrivia command. + continue + minargs, maxargs = EMAIL_COMMANDS[words[0]] + if minargs <= len(words) - 1 <= maxargs: + return True + return False diff --git a/Mailman/rules/any.py b/Mailman/rules/any.py new file mode 100644 index 000000000..c0755b58f --- /dev/null +++ b/Mailman/rules/any.py @@ -0,0 +1,41 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Check if any previous rules have matched.""" + +__all__ = ['Any'] +__metaclass__ = type + + +from zope.interface import implements + +from Mailman.i18n import _ +from Mailman.interfaces import IRule + + + +class Any: + """Look for any previous rule match.""" + implements(IRule) + + name = 'any' + description = _('Look for any previous rule hit.') + record = False + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + return len(msgdata.get('rules', [])) > 0 diff --git a/Mailman/rules/approved.py b/Mailman/rules/approved.py new file mode 100644 index 000000000..f3c1dc412 --- /dev/null +++ b/Mailman/rules/approved.py @@ -0,0 +1,117 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Look for moderator pre-approval.""" + +__all__ = ['Approved'] +__metaclass__ = type + + +import re +from email.iterators import typed_subpart_iterator +from zope.interface import implements + +from Mailman.i18n import _ +from Mailman.interfaces import IRule + + +EMPTYSTRING = u'' + + + +class Approved: + """Look for moderator pre-approval.""" + implements(IRule) + + name = 'approved' + description = _('The message has a matching Approve or Approved header.') + record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + # See if the message has an Approved or Approve header with a valid + # moderator password. Also look at the first non-whitespace line in + # the file to see if it looks like an Approved header. + missing = object() + password = msg.get('approved', msg.get('approve', missing)) + if password is missing: + # Find the first text/plain part in the message + part = None + stripped = False + for part in typed_subpart_iterator(msg, 'text', 'plain'): + break + payload = part.get_payload(decode=True) + if payload is not None: + lines = payload.splitlines(True) + for lineno, line in enumerate(lines): + if line.strip() <> '': + break + if ':' in line: + header, value = line.split(':', 1) + if header.lower() in ('approved', 'approve'): + password = value.strip() + # Now strip the first line from the payload so the + # password doesn't leak. + del lines[lineno] + reset_payload(part, EMPTYSTRING.join(lines)) + stripped = True + if stripped: + # Now try all the text parts in case it's + # multipart/alternative with the approved line in HTML or + # other text part. We make a pattern from the Approved line + # and delete it from all text/* parts in which we find it. It + # would be better to just iterate forward, but email + # compatability for pre Python 2.2 returns a list, not a true + # iterator. + # + # This will process all the multipart/alternative parts in the + # message as well as all other text parts. We shouldn't find + # the pattern outside the multipart/alternative parts, but if + # we do, it is probably best to delete it anyway as it does + # contain the password. + # + # Make a pattern to delete. We can't just delete a line + # because line of HTML or other fancy text may include + # additional message text. This pattern works with HTML. It + # may not work with rtf or whatever else is possible. + pattern = header + ':(\s| )*' + re.escape(password) + for part in typed_subpart_iterator(msg, 'text'): + payload = part.get_payload(decode=True) + if payload is not None: + if re.search(pattern, payload): + reset_payload(part, re.sub(pattern, '', payload)) + else: + del msg['approved'] + del msg['approve'] + return password is not missing and password == mlist.moderator_password + + + +def reset_payload(part, payload): + # Set decoded payload maintaining content-type, charset, format and delsp. + charset = part.get_content_charset() or 'us-ascii' + content_type = part.get_content_type() + format = part.get_param('format') + delsp = part.get_param('delsp') + del part['content-transfer-encoding'] + del part['content-type'] + part.set_payload(payload, charset) + part.set_type(content_type) + if format: + part.set_param('Format', format) + if delsp: + part.set_param('DelSp', delsp) diff --git a/Mailman/rules/docs/administrivia.txt b/Mailman/rules/docs/administrivia.txt new file mode 100644 index 000000000..de802fa85 --- /dev/null +++ b/Mailman/rules/docs/administrivia.txt @@ -0,0 +1,100 @@ +Administrivia +============= + +The 'administrivia' rule matches when the message contains some common email +commands in the Subject header or first few lines of the payload. This is +used to catch messages posted to the list which should have been sent to the +-request robot address. + + >>> from Mailman.configuration import config + >>> mlist = config.db.list_manager.create(u'_xtest@example.com') + >>> mlist.administrivia = True + >>> rule = config.rules['administrivia'] + >>> rule.name + 'administrivia' + +For example, if the Subject header contains the word 'unsubscribe', the rule +matches. + + >>> msg_1 = message_from_string(u"""\ + ... From: aperson@example.com + ... Subject: unsubscribe + ... + ... """) + >>> rule.check(mlist, msg_1, {}) + True + +Similarly, if the body of the message contains the word 'subscribe' in the +first few lines of text, the rule matches. + + >>> msg_2 = message_from_string(u"""\ + ... From: aperson@example.com + ... Subject: I wish to join your list + ... + ... subscribe + ... """) + >>> rule.check(mlist, msg_2, {}) + True + +In both cases, administrivia checking can be disabled. + + >>> mlist.administrivia = False + >>> rule.check(mlist, msg_1, {}) + False + >>> rule.check(mlist, msg_2, {}) + False + +To make the administrivia heuristics a little more robust, the rule actually +looks for a minimum and maximum number of arguments, so that it really does +seem like a mis-addressed email command. In this case, the 'confirm' command +requires at least one argument. We don't give that here so the rule will not +match. + + >>> mlist.administrivia = True + >>> msg = message_from_string(u"""\ + ... From: aperson@example.com + ... Subject: confirm + ... + ... """) + >>> rule.check(mlist, msg, {}) + False + +But a real 'confirm' message will match. + + >>> msg = message_from_string(u"""\ + ... From: aperson@example.com + ... Subject: confirm 12345 + ... + ... """) + >>> rule.check(mlist, msg, {}) + True + +We don't show all the other possible email commands, but you get the idea. + + +Non-administrivia +----------------- + +Of course, messages that don't contain administrivia, don't match the rule. + + >>> msg = message_from_string(u"""\ + ... From: aperson@example.com + ... Subject: examine + ... + ... persuade + ... """) + >>> rule.check(mlist, msg, {}) + False + +Also, only text/plain parts are checked for administrivia, so any email +commands in other content type subparts are ignored. + + >>> msg = message_from_string(u"""\ + ... From: aperson@example.com + ... Subject: some administrivia + ... Content-Type: text/x-special + ... + ... subscribe + ... """) + >>> rule.check(mlist, msg, {}) + False diff --git a/Mailman/docs/approve.txt b/Mailman/rules/docs/approve.txt index 56afc1dd4..32367a76b 100644 --- a/Mailman/docs/approve.txt +++ b/Mailman/rules/docs/approve.txt @@ -13,98 +13,71 @@ approval queue. This has several use cases: In order to support this, a mailing list can be given a 'moderator password' which is shared among all the administrators. - >>> from Mailman.Handlers.Approve import process >>> from Mailman.configuration import config >>> mlist = config.db.list_manager.create(u'_xtest@example.com') + >>> mlist.moderator_password = u'abcxyz' + +The 'approved' rule determines whether the message contains the proper +approval or not. + >>> rule = config.rules['approved'] + >>> rule.name + 'approved' -Short circuiting ----------------- -The message may have been approved by some other means, as evident in the -message metadata. In this case, the handler returns immediately. +No approval +----------- + +If the message has no Approve or Approved header, then the rule does not +match. >>> msg = message_from_string(u"""\ ... From: aperson@example.com ... ... An important message. ... """) - >>> msgdata = {'approved': True} - >>> process(mlist, msg, msgdata) - >>> print msg.as_string() - From: aperson@example.com - <BLANKLINE> - An important message. - <BLANKLINE> - >>> msgdata - {'approved': True} + >>> rule.check(mlist, msg, {}) + False +If the message has an Approve or Approved header with a value that does not +match the moderator password, then the rule does not match. However, the +header is still removed. -The Approved header -------------------- + >>> msg['Approve'] = u'12345' + >>> rule.check(mlist, msg, {}) + False + >>> print msg['approve'] + None -If the moderator password is given in an Approved header, then the message -gets sent through with no further posting moderation. The Approved header is -not stripped in this handler module, but instead in the Cleanse module. This -ensures that no moderator approval password in the headers will leak out. + >>> del msg['approve'] + >>> msg['Approved'] = u'12345' + >>> rule.check(mlist, msg, {}) + False + >>> print msg['approved'] + None - >>> mlist.moderator_password = u'abcxyz' - >>> msg['Approved'] = u'abcxyz' - >>> msgdata = {} - >>> process(mlist, msg, msgdata) - >>> print msg.as_string() - From: aperson@example.com - Approved: abcxyz - <BLANKLINE> - An important message. - <BLANKLINE> - >>> sorted(msgdata.items()) - [('adminapproved', True), ('approved', True)] + >>> del msg['approved'] -But if the wrong password is given, then the message is not marked as being -approved. The header is still removed though. - >>> del msg['Approved'] - >>> msg['Approved'] = u'123456' - >>> msgdata = {} - >>> process(mlist, msg, msgdata) - >>> print msg.as_string() - From: aperson@example.com - Approved: 123456 - <BLANKLINE> - An important message. - <BLANKLINE> - >>> msgdata - {} +Using an approval header +------------------------ -In the spirit of being liberal in what you accept, using an Approve header is -completely synonymous. +If the moderator password is given in an Approve header, then the rule +matches, and the Approve header is stripped. - >>> del msg['Approved'] >>> msg['Approve'] = u'abcxyz' - >>> msgdata = {} - >>> process(mlist, msg, msgdata) - >>> print msg.as_string() - From: aperson@example.com - Approve: abcxyz - <BLANKLINE> - An important message. - <BLANKLINE> - >>> sorted(msgdata.items()) - [('adminapproved', True), ('approved', True)] + >>> rule.check(mlist, msg, {}) + True + >>> print msg['approve'] + None - >>> del msg['Approve'] - >>> msg['Approve'] = u'123456' - >>> msgdata = {} - >>> process(mlist, msg, msgdata) - >>> print msg.as_string() - From: aperson@example.com - Approve: 123456 - <BLANKLINE> - An important message. - <BLANKLINE> - >>> msgdata - {} +Similarly, for the Approved header. + + >>> msg['Approved'] = u'abcxyz' + >>> rule.check(mlist, msg, {}) + True + >>> print msg['approved'] + None Using a pseudo-header @@ -113,17 +86,20 @@ Using a pseudo-header Different mail user agents have varying degrees to which they support custom headers like Approve and Approved. For this reason, Mailman also supports using a 'pseudo-header', which is really just the first non-whitespace line in -the payload of the message of the message. If this pseudo-header looks like a -matching Approve or Approved header, the message is similarly allowed to pass. +the payload of the message. If this pseudo-header looks like a matching +Approve or Approved header, the message is similarly allowed to pass. >>> msg = message_from_string(u"""\ ... From: aperson@example.com ... - ... Approved: abcxyz + ... Approve: abcxyz ... An important message. ... """) - >>> msgdata = {} - >>> process(mlist, msg, msgdata) + >>> rule.check(mlist, msg, {}) + True + +The pseudo-header is removed. + >>> print msg.as_string() From: aperson@example.com Content-Transfer-Encoding: 7bit @@ -132,17 +108,18 @@ matching Approve or Approved header, the message is similarly allowed to pass. <BLANKLINE> An important message. <BLANKLINE> - >>> sorted(msgdata.items()) - [('adminapproved', True), ('approved', True)] + +Similarly for the Approved header. >>> msg = message_from_string(u"""\ ... From: aperson@example.com ... - ... Approve: abcxyz + ... Approved: abcxyz ... An important message. ... """) - >>> msgdata = {} - >>> process(mlist, msg, msgdata) + >>> rule.check(mlist, msg, {}) + True + >>> print msg.as_string() From: aperson@example.com Content-Transfer-Encoding: 7bit @@ -151,8 +128,6 @@ matching Approve or Approved header, the message is similarly allowed to pass. <BLANKLINE> An important message. <BLANKLINE> - >>> sorted(msgdata.items()) - [('adminapproved', True), ('approved', True)] As before, a mismatch in the pseudo-header does not approve the message, but the pseudo-header line is still removed. @@ -160,11 +135,12 @@ the pseudo-header line is still removed. >>> msg = message_from_string(u"""\ ... From: aperson@example.com ... - ... Approved: 123456 + ... Approve: 123456 ... An important message. ... """) - >>> msgdata = {} - >>> process(mlist, msg, msgdata) + >>> rule.check(mlist, msg, {}) + False + >>> print msg.as_string() From: aperson@example.com Content-Transfer-Encoding: 7bit @@ -173,17 +149,18 @@ the pseudo-header line is still removed. <BLANKLINE> An important message. <BLANKLINE> - >>> msgdata - {} + +Similarly for the Approved header. >>> msg = message_from_string(u"""\ ... From: aperson@example.com ... - ... Approve: 123456 + ... Approved: 123456 ... An important message. ... """) - >>> msgdata = {} - >>> process(mlist, msg, msgdata) + >>> rule.check(mlist, msg, {}) + False + >>> print msg.as_string() From: aperson@example.com Content-Transfer-Encoding: 7bit @@ -192,8 +169,6 @@ the pseudo-header line is still removed. <BLANKLINE> An important message. <BLANKLINE> - >>> msgdata - {} MIME multipart support @@ -211,18 +186,21 @@ used with MIME documents. ... --AAA ... Content-Type: application/x-ignore ... - ... Approved: 123456 + ... Approve: 123456 ... The above line will be ignored. ... ... --AAA ... Content-Type: text/plain ... - ... Approved: abcxyz + ... Approve: abcxyz ... An important message. ... --AAA-- ... """) - >>> msgdata = {} - >>> process(mlist, msg, msgdata) + >>> rule.check(mlist, msg, {}) + True + +Like before, the pseudo-header is removed, but only from the text parts. + >>> print msg.as_string() From: aperson@example.com MIME-Version: 1.0 @@ -231,7 +209,7 @@ used with MIME documents. --AAA Content-Type: application/x-ignore <BLANKLINE> - Approved: 123456 + Approve: 123456 The above line will be ignored. <BLANKLINE> --AAA @@ -242,8 +220,8 @@ used with MIME documents. An important message. --AAA-- <BLANKLINE> - >>> sorted(msgdata.items()) - [('adminapproved', True), ('approved', True)] + +The same goes for the Approved message. >>> msg = message_from_string(u"""\ ... From: aperson@example.com @@ -253,18 +231,21 @@ used with MIME documents. ... --AAA ... Content-Type: application/x-ignore ... - ... Approve: 123456 + ... Approved: 123456 ... The above line will be ignored. ... ... --AAA ... Content-Type: text/plain ... - ... Approve: abcxyz + ... Approved: abcxyz ... An important message. ... --AAA-- ... """) - >>> msgdata = {} - >>> process(mlist, msg, msgdata) + >>> rule.check(mlist, msg, {}) + True + +And the header is removed. + >>> print msg.as_string() From: aperson@example.com MIME-Version: 1.0 @@ -273,7 +254,7 @@ used with MIME documents. --AAA Content-Type: application/x-ignore <BLANKLINE> - Approve: 123456 + Approved: 123456 The above line will be ignored. <BLANKLINE> --AAA @@ -284,8 +265,6 @@ used with MIME documents. An important message. --AAA-- <BLANKLINE> - >>> sorted(msgdata.items()) - [('adminapproved', True), ('approved', True)] Here, the correct password is in the non-text/plain part, so it is ignored. @@ -307,8 +286,11 @@ Here, the correct password is in the non-text/plain part, so it is ignored. ... An important message. ... --AAA-- ... """) - >>> msgdata = {} - >>> process(mlist, msg, msgdata) + >>> rule.check(mlist, msg, {}) + False + +And yet the pseudo-header is still stripped. + >>> print msg.as_string() From: aperson@example.com MIME-Version: 1.0 @@ -327,8 +309,8 @@ Here, the correct password is in the non-text/plain part, so it is ignored. <BLANKLINE> An important message. --AAA-- - >>> msgdata - {} + +As before, the same goes for the Approved header. >>> msg = message_from_string(u"""\ ... From: aperson@example.com @@ -338,18 +320,21 @@ Here, the correct password is in the non-text/plain part, so it is ignored. ... --AAA ... Content-Type: application/x-ignore ... - ... Approve: abcxyz + ... Approved: abcxyz ... The above line will be ignored. ... ... --AAA ... Content-Type: text/plain ... - ... Approve: 123456 + ... Approved: 123456 ... An important message. ... --AAA-- ... """) - >>> msgdata = {} - >>> process(mlist, msg, msgdata) + >>> rule.check(mlist, msg, {}) + False + +And the pseudo-header is removed. + >>> print msg.as_string() From: aperson@example.com MIME-Version: 1.0 @@ -358,7 +343,7 @@ Here, the correct password is in the non-text/plain part, so it is ignored. --AAA Content-Type: application/x-ignore <BLANKLINE> - Approve: abcxyz + Approved: abcxyz The above line will be ignored. <BLANKLINE> --AAA @@ -368,8 +353,14 @@ Here, the correct password is in the non-text/plain part, so it is ignored. <BLANKLINE> An important message. --AAA-- - >>> msgdata - {} + + +Stripping text/html parts +------------------------- + +Because some mail readers will include both a text/plain part and a text/html +alternative, the 'approved' rule has to search the alternatives and strip +anything that looks like an Approve or Approved headers. >>> msg = message_from_string(u"""\ ... From: aperson@example.com @@ -377,30 +368,100 @@ Here, the correct password is in the non-text/plain part, so it is ignored. ... Content-Type: multipart/mixed; boundary="AAA" ... ... --AAA - ... Content-Type: application/x-ignore + ... Content-Type: text/html + ... + ... <html> + ... <head></head> + ... <body> + ... <b>Approved: abcxyz</b> + ... <p>The above line will be ignored. + ... </body> + ... </html> + ... + ... --AAA + ... Content-Type: text/plain ... ... Approved: abcxyz - ... The above line will be ignored. + ... An important message. + ... --AAA-- + ... """) + >>> rule.check(mlist, msg, {}) + True + +And the header-like text in the text/html part was stripped. + + >>> print msg.as_string() + From: aperson@example.com + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="AAA" + <BLANKLINE> + --AAA + Content-Transfer-Encoding: 7bit + MIME-Version: 1.0 + Content-Type: text/html; charset="us-ascii" + <BLANKLINE> + <html> + <head></head> + <body> + <b></b> + <p>The above line will be ignored. + </body> + </html> + <BLANKLINE> + --AAA + Content-Transfer-Encoding: 7bit + MIME-Version: 1.0 + Content-Type: text/plain; charset="us-ascii" + <BLANKLINE> + An important message. + --AAA-- + <BLANKLINE> + +This is true even if the rule does not match. + + >>> msg = message_from_string(u"""\ + ... From: aperson@example.com + ... MIME-Version: 1.0 + ... Content-Type: multipart/mixed; boundary="AAA" + ... + ... --AAA + ... Content-Type: text/html + ... + ... <html> + ... <head></head> + ... <body> + ... <b>Approve: 123456</b> + ... <p>The above line will be ignored. + ... </body> + ... </html> ... ... --AAA ... Content-Type: text/plain ... - ... Approved: 123456 + ... Approve: 123456 ... An important message. ... --AAA-- ... """) - >>> msgdata = {} - >>> process(mlist, msg, msgdata) + >>> rule.check(mlist, msg, {}) + False + >>> print msg.as_string() From: aperson@example.com MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="AAA" <BLANKLINE> --AAA - Content-Type: application/x-ignore + Content-Transfer-Encoding: 7bit + MIME-Version: 1.0 + Content-Type: text/html; charset="us-ascii" <BLANKLINE> - Approved: abcxyz - The above line will be ignored. + <html> + <head></head> + <body> + <b></b> + <p>The above line will be ignored. + </body> + </html> <BLANKLINE> --AAA Content-Transfer-Encoding: 7bit @@ -410,5 +471,3 @@ Here, the correct password is in the non-text/plain part, so it is ignored. An important message. --AAA-- <BLANKLINE> - >>> msgdata - {} diff --git a/Mailman/rules/docs/emergency.txt b/Mailman/rules/docs/emergency.txt new file mode 100644 index 000000000..aecbcb90d --- /dev/null +++ b/Mailman/rules/docs/emergency.txt @@ -0,0 +1,74 @@ +Emergency +========= + +When the mailing list has its emergency flag set, all messages posted to the +list are held for moderator approval. + + >>> from Mailman.app.lifecycle import create_list + >>> mlist = create_list(u'_xtest@example.com') + >>> mlist.web_page_url = u'http://www.example.com/' + >>> msg = message_from_string("""\ + ... From: aperson@example.com + ... To: _xtest@example.com + ... Subject: My first post + ... Message-ID: <first> + ... + ... An important message. + ... """) + +The emergency rule is matched as part of the built-in chain. The emergency +rule matches if the flag is set on the mailing list. + + >>> from Mailman.app.chains import process + >>> mlist.emergency = True + >>> process(mlist, msg, {}, 'built-in') + +There are two messages in the virgin queue. The one addressed to the original +sender will contain a token we can use to grab the held message out of the +pending requests. + + >>> from Mailman.queue import Switchboard + >>> virginq = Switchboard(config.VIRGINQUEUE_DIR) + + >>> def get_held_message(): + ... import re + ... qfiles = [] + ... for filebase in virginq.files: + ... qmsg, qdata = virginq.dequeue(filebase) + ... virginq.finish(filebase) + ... qfiles.append(qmsg) + ... from operator import itemgetter + ... qfiles.sort(key=itemgetter('to')) + ... cookie = None + ... for line in qfiles[1].get_payload().splitlines(): + ... mo = re.search('confirm/[^/]+/(?P<cookie>.*)$', line) + ... if mo: + ... cookie = mo.group('cookie') + ... break + ... assert cookie is not None, 'No confirmation token found' + ... data = config.db.pendings.confirm(cookie) + ... requestdb = config.db.requests.get_list_requests(mlist) + ... rkey, rdata = requestdb.get_request(data['id']) + ... return config.db.message_store.get_message_by_id( + ... rdata['_mod_message_id']) + + >>> msg = get_held_message() + >>> print msg.as_string() + From: aperson@example.com + To: _xtest@example.com + Subject: My first post + Message-ID: <first> + X-Mailman-Rule-Hits: emergency + X-Mailman-Rule-Misses: approved + X-Message-ID-Hash: RXJU4JL6N2OUN3OYMXXPPSCR7P7JE2BW + <BLANKLINE> + An important message. + <BLANKLINE> + +However, if the message metadata has a 'moderator_approved' key set, then even +if the mailing list has its emergency flag set, the message still goes through +to the membership. + + >>> process(mlist, msg, dict(moderator_approved=True), 'built-in') + >>> len(virginq.files) + 0 diff --git a/Mailman/rules/docs/header-matching.txt b/Mailman/rules/docs/header-matching.txt new file mode 100644 index 000000000..fbd0ff65f --- /dev/null +++ b/Mailman/rules/docs/header-matching.txt @@ -0,0 +1,145 @@ +Header matching +=============== + +Mailman can do pattern based header matching during its normal rule +processing. There is a set of site-wide default header matchines specified in +the configuaration file under the HEADER_MATCHES variable. + + >>> from Mailman.app.lifecycle import create_list + >>> mlist = create_list(u'_xtest@example.com') + +Because the default HEADER_MATCHES variable is empty when the configuration +file is read, we'll just extend the current header matching chain with a +pattern that matches 4 or more stars, discarding the message if it hits. + + >>> from Mailman.configuration import config + >>> chain = config.chains['header-match'] + >>> chain.extend('x-spam-score', '[*]{4,}', 'discard') + +First, if the message has no X-Spam-Score header, the message passes through +the chain untouched (i.e. no disposition). + + >>> msg = message_from_string("""\ + ... From: aperson@example.com + ... To: _xtest@example.com + ... Subject: Not spam + ... Message-ID: <one> + ... + ... This is a message. + ... """) + + >>> from Mailman.app.chains import process + +Pass through is seen as nothing being in the log file after processing. + + # XXX This checks the vette log file because there is no other evidence + # that this chain has done anything. + >>> import os + >>> fp = open(os.path.join(config.LOG_DIR, 'vette')) + >>> fp.seek(0, 2) + >>> file_pos = fp.tell() + >>> process(mlist, msg, {}, 'header-match') + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: + <BLANKLINE> + +Now, if the header exists but does not match, then it also passes through +untouched. + + >>> msg['X-Spam-Score'] = '***' + >>> del msg['subject'] + >>> msg['Subject'] = 'This is almost spam' + >>> del msg['message-id'] + >>> msg['Message-ID'] = '<two>' + >>> file_pos = fp.tell() + >>> process(mlist, msg, {}, 'header-match') + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: + <BLANKLINE> + +But now if the header matches, then the message gets discarded. + + >>> del msg['x-spam-score'] + >>> msg['X-Spam-Score'] = '****' + >>> del msg['subject'] + >>> msg['Subject'] = 'This is spam, but barely' + >>> del msg['message-id'] + >>> msg['Message-ID'] = '<three>' + >>> file_pos = fp.tell() + >>> process(mlist, msg, {}, 'header-match') + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: ... DISCARD: <three> + <BLANKLINE> + +For kicks, let's show a message that's really spammy. + + >>> del msg['x-spam-score'] + >>> msg['X-Spam-Score'] = '**********' + >>> del msg['subject'] + >>> msg['Subject'] = 'This is really spammy' + >>> del msg['message-id'] + >>> msg['Message-ID'] = '<four>' + >>> file_pos = fp.tell() + >>> process(mlist, msg, {}, 'header-match') + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: ... DISCARD: <four> + <BLANKLINE> + +Flush out the extended header matching rules. + + >>> chain.flush() + + +List-specific header matching +----------------------------- + +Each mailing list can also be configured with a set of header matching regular +expression rules. These are used to impose list-specific header filtering +with the same semantics as the global `HEADER_MATCHES` variable. + +The list administrator wants to match not on four stars, but on three plus +signs, but only for the current mailing list. + + >>> mlist.header_matches = [('x-spam-score', '[+]{3,}', 'discard')] + +A message with a spam score of two pluses does not match. + + >>> del msg['x-spam-score'] + >>> msg['X-Spam-Score'] = '++' + >>> del msg['message-id'] + >>> msg['Message-ID'] = '<five>' + >>> file_pos = fp.tell() + >>> process(mlist, msg, {}, 'header-match') + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: + +A message with a spam score of three pluses does match. + + >>> del msg['x-spam-score'] + >>> msg['X-Spam-Score'] = '+++' + >>> del msg['message-id'] + >>> msg['Message-ID'] = '<six>' + >>> file_pos = fp.tell() + >>> process(mlist, msg, {}, 'header-match') + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: ... DISCARD: <six> + <BLANKLINE> + +As does a message with a spam score of four pluses. + + >>> del msg['x-spam-score'] + >>> msg['X-Spam-Score'] = '+++' + >>> del msg['message-id'] + >>> msg['Message-ID'] = '<seven>' + >>> file_pos = fp.tell() + >>> process(mlist, msg, {}, 'header-match') + >>> fp.seek(file_pos) + >>> print 'LOG:', fp.read() + LOG: ... DISCARD: <seven> + <BLANKLINE> diff --git a/Mailman/rules/docs/implicit-dest.txt b/Mailman/rules/docs/implicit-dest.txt new file mode 100644 index 000000000..5a7f06c0c --- /dev/null +++ b/Mailman/rules/docs/implicit-dest.txt @@ -0,0 +1,76 @@ +Implicit destination +==================== + +The 'implicit-dest' rule matches when the mailing list's posting address is +not explicitly mentioned in the set of message recipients. + + >>> from Mailman.configuration import config + >>> mlist = config.db.list_manager.create(u'_xtest@example.com') + >>> rule = config.rules['implicit-dest'] + >>> rule.name + 'implicit-dest' + +This rule matches messages that have implicit destination, meaning that the +mailing list's posting address isn't included in the explicit recipients. + + >>> mlist.require_explicit_destination = True + >>> mlist.acceptable_aliases = u'' + >>> msg = message_from_string(u"""\ + ... From: aperson@example.org + ... Subject: An implicit message + ... + ... """) + >>> rule.check(mlist, msg, {}) + True + +You can disable implicit destination checks for the mailing list. + + >>> mlist.require_explicit_destination = False + >>> rule.check(mlist, msg, {}) + False + +Even with some recipients, if the posting address is not included, the rule +will match. + + >>> mlist.require_explicit_destination = True + >>> msg['To'] = 'myfriend@example.com' + >>> rule.check(mlist, msg, {}) + True + +Add the posting address as a recipient and the rule will no longer match. + + >>> msg['Cc'] = '_xtest@example.com' + >>> rule.check(mlist, msg, {}) + False + +Alternatively, if one of the acceptable aliases is in the recipients list, +then the rule will not match. + + >>> del msg['cc'] + >>> rule.check(mlist, msg, {}) + True + >>> mlist.acceptable_aliases = u'myfriend@example.com' + >>> rule.check(mlist, msg, {}) + False + +A message gated from NNTP will obviously have an implicit destination. Such +gated messages will not be held for implicit destination because it's assumed +that Mailman pulled it from the appropriate news group. + + >>> rule.check(mlist, msg, dict(fromusenet=True)) + False + + +Alias patterns +-------------- + +It's also possible to specify an alias pattern, i.e. a regular expression to +match against the recipients. For example, we can say that if there is a +recipient in the example.net domain, then the rule does not match. + + >>> mlist.acceptable_aliases = u'^.*@example.net' + >>> rule.check(mlist, msg, {}) + True + >>> msg['To'] = 'you@example.net' + >>> rule.check(mlist, msg, {}) + False diff --git a/Mailman/rules/docs/loop.txt b/Mailman/rules/docs/loop.txt new file mode 100644 index 000000000..8fe86cf45 --- /dev/null +++ b/Mailman/rules/docs/loop.txt @@ -0,0 +1,49 @@ +Posting loops +============= + +To avoid a posting loop, Mailman has a rule to check for the existence of an +X-BeenThere header with the value of the list's posting address. + + >>> from Mailman.configuration import config + >>> mlist = config.db.list_manager.create(u'_xtest@example.com') + >>> rule = config.rules['loop'] + >>> rule.name + 'loop' + +The header could be missing, in which case the rule does not match. + + >>> msg = message_from_string(u"""\ + ... From: aperson@example.com + ... + ... An important message. + ... """) + >>> rule.check(mlist, msg, {}) + False + +The header could be present, but not match the list's posting address. + + >>> msg['X-BeenThere'] = u'not-this-list@example.com' + >>> rule.check(mlist, msg, {}) + False + +If the header is present and does match the posting address, the rule +matches. + + >>> del msg['x-beenthere'] + >>> msg['X-BeenThere'] = mlist.posting_address + >>> rule.check(mlist, msg, {}) + True + +Even if there are multiple X-BeenThere headers, as long as one with the +posting address exists, the rule matches. + + >>> msg = message_from_string(u"""\ + ... From: aperson@example.com + ... X-BeenThere: not-this-list@example.com + ... X-BeenThere: _xtest@example.com + ... X-BeenThere: foo@example.com + ... + ... An important message. + ... """) + >>> rule.check(mlist, msg, {}) + True diff --git a/Mailman/rules/docs/max-size.txt b/Mailman/rules/docs/max-size.txt new file mode 100644 index 000000000..0d64b0cf7 --- /dev/null +++ b/Mailman/rules/docs/max-size.txt @@ -0,0 +1,40 @@ +Message size +============ + +The 'message-size' rule matches when the posted message is bigger than a +specified maximum. Generally this is used to prevent huge attachments from +getting posted to the list. This value is calculated in terms of KB (1024 +bytes). + + >>> from Mailman.configuration import config + >>> mlist = config.db.list_manager.create(u'_xtest@example.com') + >>> rule = config.rules['max-size'] + >>> rule.name + 'max-size' + +For example, setting the maximum message size to 1 means that any message +bigger than that will match the rule. + + >>> mlist.max_message_size = 1 # 1024 bytes + >>> one_line = u'x' * 79 + >>> big_body = u'\n'.join([one_line] * 15) + >>> msg = message_from_string(u"""\ + ... From: aperson@example.com + ... To: _xtest@example.com + ... + ... """ + big_body) + >>> rule.check(mlist, msg, {}) + True + +Setting the maximum message size to zero means no size check is performed. + + >>> mlist.max_message_size = 0 + >>> rule.check(mlist, msg, {}) + False + +Of course, if the maximum size is larger than the message's size, then it's +still okay. + + >>> mlist.max_message_size = msg.original_size/1024.0 + 1 + >>> rule.check(mlist, msg, {}) + False diff --git a/Mailman/rules/docs/moderation.txt b/Mailman/rules/docs/moderation.txt new file mode 100644 index 000000000..cab8f20d3 --- /dev/null +++ b/Mailman/rules/docs/moderation.txt @@ -0,0 +1,70 @@ +Member moderation +================= + +Each user has a moderation flag. When set, and the list is set to moderate +postings, then only members with a cleared moderation flag will be able to +email the list without having those messages be held for approval. The +'moderation' rule determines whether the message should be moderated or not. + + >>> from Mailman.configuration import config + >>> mlist = config.db.list_manager.create(u'_xtest@example.com') + >>> rule = config.rules['moderation'] + >>> rule.name + 'moderation' + +In the simplest case, the sender is not a member of the mailing list, so the +moderation rule can't match. + + >>> msg = message_from_string(u"""\ + ... From: aperson@example.org + ... To: _xtest@example.com + ... Subject: A posted message + ... + ... """) + >>> rule.check(mlist, msg, {}) + False + +Let's add the message author as a non-moderated member. + + >>> user = config.db.user_manager.create_user( + ... u'aperson@example.org', u'Anne Person') + >>> address = list(user.addresses)[0] + >>> from Mailman.interfaces import MemberRole + >>> member = address.subscribe(mlist, MemberRole.member) + >>> member.is_moderated + False + >>> rule.check(mlist, msg, {}) + False + +Once the member's moderation flag is set though, the rule matches. + + >>> member.is_moderated = True + >>> rule.check(mlist, msg, {}) + True + + +Non-members +----------- + +There is another, related rule for matching non-members, which simply matches +if the sender is /not/ a member of the mailing list. + + >>> rule = config.rules['non-member'] + >>> rule.name + 'non-member' + +If the sender is a member of this mailing list, the rule does not match. + + >>> rule.check(mlist, msg, {}) + False + +But if the sender is not a member of this mailing list, the rule matches. + + >>> msg = message_from_string(u"""\ + ... From: bperson@example.org + ... To: _xtest@example.com + ... Subject: A posted message + ... + ... """) + >>> rule.check(mlist, msg, {}) + True diff --git a/Mailman/rules/docs/news-moderation.txt b/Mailman/rules/docs/news-moderation.txt new file mode 100644 index 000000000..f32919ce5 --- /dev/null +++ b/Mailman/rules/docs/news-moderation.txt @@ -0,0 +1,37 @@ +Newsgroup moderation +==================== + +The 'news-moderation' rule matches all messages posted to mailing lists that +gateway to a moderated newsgroup. The reason for this is that such messages +must get forwarded on to the newsgroup moderator. From there it will get +posted to the newsgroup, and from there, gated to the mailing list. It's a +circuitous route, but it works nonetheless by holding all messages posted +directly to the mailing list. + + >>> from Mailman.configuration import config + >>> mlist = config.db.list_manager.create(u'_xtest@example.com') + >>> rule = config.rules['news-moderation'] + >>> rule.name + 'news-moderation' + +Set the list configuraiton variable to enable newsgroup moderation. + + >>> from Mailman.interfaces import NewsModeration + >>> mlist.news_moderation = NewsModeration.moderated + +And now all messages will match the rule. + + >>> msg = message_from_string(u"""\ + ... From: aperson@example.org + ... Subject: An announcment + ... + ... Great things are happening. + ... """) + >>> rule.check(mlist, msg, {}) + True + +When moderation is turned off, the rule does not match. + + >>> mlist.news_moderation = NewsModeration.none + >>> rule.check(mlist, msg, {}) + False diff --git a/Mailman/rules/docs/no-subject.txt b/Mailman/rules/docs/no-subject.txt new file mode 100644 index 000000000..3627ac03f --- /dev/null +++ b/Mailman/rules/docs/no-subject.txt @@ -0,0 +1,34 @@ +No Subject header +================= + +This rule matches if the message has no Subject header, or if the header is +the empty string when stripped. + + >>> from Mailman.configuration import config + >>> mlist = config.db.list_manager.create(u'_xtest@example.com') + >>> rule = config.rules['no-subject'] + >>> rule.name + 'no-subject' + +A message with a non-empty subject does not match the rule. + + >>> msg = message_from_string(u"""\ + ... From: aperson@example.org + ... To: _xtest@example.com + ... Subject: A posted message + ... + ... """) + >>> rule.check(mlist, msg, {}) + False + +Delete the Subject header and the rule matches. + + >>> del msg['subject'] + >>> rule.check(mlist, msg, {}) + True + +Even a Subject header with only whitespace still matches the rule. + + >>> msg['Subject'] = u' ' + >>> rule.check(mlist, msg, {}) + True diff --git a/Mailman/rules/docs/recipients.txt b/Mailman/rules/docs/recipients.txt new file mode 100644 index 000000000..21d04b8ae --- /dev/null +++ b/Mailman/rules/docs/recipients.txt @@ -0,0 +1,41 @@ +Maximum number of recipients +============================ + +The 'max-recipients' rule matches when there are more than the maximum allowed +number of explicit recipients addressed by the message. + + >>> from Mailman.configuration import config + >>> mlist = config.db.list_manager.create(u'_xtest@example.com') + >>> rule = config.rules['max-recipients'] + >>> rule.name + 'max-recipients' + +In this case, we'll create a message with 5 recipients. These include all +addresses in the To and CC headers. + + >>> msg = message_from_string(u"""\ + ... From: aperson@example.com + ... To: _xtest@example.com, bperson@example.com + ... Cc: cperson@example.com + ... Cc: dperson@example.com (Dan Person) + ... To: Elly Q. Person <eperson@example.com> + ... + ... Hey folks! + ... """) + +For backward compatibility, the message must have fewer than the maximum +number of explicit recipients. + + >>> mlist.max_num_recipients = 5 + >>> rule.check(mlist, msg, {}) + True + + >>> mlist.max_num_recipients = 6 + >>> rule.check(mlist, msg, {}) + False + +Zero means any number of recipients are allowed. + + >>> mlist.max_num_recipients = 0 + >>> rule.check(mlist, msg, {}) + False diff --git a/Mailman/rules/docs/rules.txt b/Mailman/rules/docs/rules.txt new file mode 100644 index 000000000..d2d291331 --- /dev/null +++ b/Mailman/rules/docs/rules.txt @@ -0,0 +1,70 @@ +Rules +===== + +Rules are applied to each message as part of a rule chain. Individual rules +simply return a boolean specifying whether the rule matches or not. Chain +links determine what happens when a rule matches. + + +All rules +--------- + +Rules are maintained in the configuration object as a dictionary mapping rule +names to rule objects. + + >>> from zope.interface.verify import verifyObject + >>> from Mailman.configuration import config + >>> from Mailman.interfaces import IRule + >>> for rule_name in sorted(config.rules): + ... rule = config.rules[rule_name] + ... print rule_name, verifyObject(IRule, rule) + administrivia True + any True + approved True + emergency True + implicit-dest True + loop True + max-recipients True + max-size True + moderation True + news-moderation True + no-subject True + non-member True + suspicious-header True + truth True + +You can get a rule by name. + + >>> rule = config.rules['emergency'] + >>> verifyObject(IRule, rule) + True + + +Rule checks +----------- + +Individual rules can be checked to see if they match, by running the rule's +`check()` method. This returns a boolean indicating whether the rule was +matched or not. + + >>> mlist = config.db.list_manager.create(u'_xtest@example.com') + >>> msg = message_from_string("""\ + ... From: aperson@example.com + ... + ... An important message. + ... """) + +For example, the emergency rule just checks to see if the emergency flag is +set on the mailing list, and the message has not been pre-approved by the list +administrator. + + >>> rule.name + 'emergency' + >>> mlist.emergency = False + >>> rule.check(mlist, msg, {}) + False + >>> mlist.emergency = True + >>> rule.check(mlist, msg, {}) + True + >>> rule.check(mlist, msg, dict(moderator_approved=True)) + False diff --git a/Mailman/rules/docs/suspicious.txt b/Mailman/rules/docs/suspicious.txt new file mode 100644 index 000000000..6b0eeda35 --- /dev/null +++ b/Mailman/rules/docs/suspicious.txt @@ -0,0 +1,36 @@ +Suspicious headers +================== + +Suspicious headers are a way for Mailman to hold messages that match a +particular regular expression. This mostly historical feature is fairly +confusing to users, and the list attribute that controls this is misnamed. + + >>> from Mailman.configuration import config + >>> mlist = config.db.list_manager.create(u'_xtest@example.com') + >>> rule = config.rules['suspicious-header'] + >>> rule.name + 'suspicious-header' + +Set the so-called suspicious header configuration variable. + + >>> mlist.bounce_matching_headers = u'From: .*person@(blah.)?example.com' + >>> msg = message_from_string(u"""\ + ... From: aperson@example.com + ... To: _xtest@example.com + ... Subject: An implicit message + ... + ... """) + >>> rule.check(mlist, msg, {}) + True + +But if the header doesn't match the regular expression, the rule won't match. +This one comes from a .org address. + + >>> msg = message_from_string(u"""\ + ... From: aperson@example.org + ... To: _xtest@example.com + ... Subject: An implicit message + ... + ... """) + >>> rule.check(mlist, msg, {}) + False diff --git a/Mailman/rules/docs/truth.txt b/Mailman/rules/docs/truth.txt new file mode 100644 index 000000000..baa40772a --- /dev/null +++ b/Mailman/rules/docs/truth.txt @@ -0,0 +1,10 @@ +Truth +===== + +The 'truth' rule always matches. This makes it useful as a terminus rule for +unconditionally jumping to another chain. + + >>> from Mailman.configuration import config + >>> rule = config.rules['truth'] + >>> rule.check(False, False, False) + True diff --git a/Mailman/rules/emergency.py b/Mailman/rules/emergency.py new file mode 100644 index 000000000..6d924b399 --- /dev/null +++ b/Mailman/rules/emergency.py @@ -0,0 +1,44 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The emergency hold rule.""" + +__all__ = ['Emergency'] +__metaclass__ = type + + +from zope.interface import implements + +from Mailman.i18n import _ +from Mailman.interfaces import IRule + + + +class Emergency: + """The emergency hold rule.""" + implements(IRule) + + name = 'emergency' + description = _( + """The mailing list is in emergency hold and this message was not + pre-approved by the list administrator. + """) + record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + return mlist.emergency and not msgdata.get('moderator_approved') diff --git a/Mailman/rules/implicit_dest.py b/Mailman/rules/implicit_dest.py new file mode 100644 index 000000000..1b459caed --- /dev/null +++ b/Mailman/rules/implicit_dest.py @@ -0,0 +1,95 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The implicit destination rule.""" + +__all__ = ['ImplicitDestination'] +__metaclass__ = type + + +import re +from email.utils import getaddresses +from zope.interface import implements + +from Mailman.i18n import _ +from Mailman.interfaces import IRule + + + +class ImplicitDestination: + """The implicit destination rule.""" + implements(IRule) + + name = 'implicit-dest' + description = _('Catch messages with implicit destination.') + record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + # Implicit destination checking must be enabled in the mailing list. + if not mlist.require_explicit_destination: + return False + # Messages gated from NNTP will always have an implicit destination so + # are never checked. + if msgdata.get('fromusenet'): + return False + # Calculate the list of acceptable aliases. If the alias starts with + # a caret (i.e. ^), then it's a regular expression to match against. + aliases = set() + alias_patterns = set() + for alias in mlist.acceptable_aliases.splitlines(): + alias = alias.strip().lower() + if alias.startswith('^'): + alias_patterns.add(alias) + elif '@' in alias: + aliases.add(alias) + else: + # This is not a regular expression, nor a fully-qualified + # email address, so skip it. + pass + # Add the list's posting address, i.e. the explicit address, to the + # set of acceptable aliases. + aliases.add(mlist.posting_address) + # Look at all the recipients. If the recipient is any acceptable + # alias (or the explicit posting address), then this rule does not + # match. If not, then add it to the set of recipients we'll check + # against the alias patterns later. + recipients = set() + for header in ('to', 'cc', 'resent-to', 'resent-cc'): + for fullname, address in getaddresses(msg.get_all(header, [])): + address = address.lower() + if address in aliases: + return False + recipients.add(address) + # Now for all alias patterns, see if any of the recipients matches a + # pattern. If so, then this rule does not match. + for pattern in alias_patterns: + escaped = re.escape(pattern) + for recipient in recipients: + try: + if re.match(pattern, recipient, re.IGNORECASE): + return False + except re.error: + # The pattern is a malformed regular expression. Try + # matching again with the pattern escaped. + try: + if re.match(escaped, recipient, re.IGNORECASE): + return False + except re.error: + pass + # Nothing matched. + return True diff --git a/Mailman/rules/loop.py b/Mailman/rules/loop.py new file mode 100644 index 000000000..93bd1241d --- /dev/null +++ b/Mailman/rules/loop.py @@ -0,0 +1,44 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Look for a posting loop.""" + +__all__ = ['Loop'] +__metaclass__ = type + + +from zope.interface import implements + +from Mailman.i18n import _ +from Mailman.interfaces import IRule + + + +class Loop: + """Look for a posting loop.""" + implements(IRule) + + name = 'loop' + description = _("""Look for a posting loop, via the X-BeenThere header.""") + record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + # Has this message already been posted to this list? + been_theres = [value.strip().lower() + for value in msg.get_all('x-beenthere', [])] + return mlist.posting_address in been_theres diff --git a/Mailman/rules/max_recipients.py b/Mailman/rules/max_recipients.py new file mode 100644 index 000000000..e4fb3faf3 --- /dev/null +++ b/Mailman/rules/max_recipients.py @@ -0,0 +1,48 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The maximum number of recipients rule.""" + +__all__ = ['MaximumRecipients'] +__metaclass__ = type + + +from email.utils import getaddresses +from zope.interface import implements + +from Mailman.i18n import _ +from Mailman.interfaces import IRule + + + +class MaximumRecipients: + """The maximum number of recipients rule.""" + implements(IRule) + + name = 'max-recipients' + description = _('Catch messages with too many explicit recipients.') + record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + # Zero means any number of recipients are allowed. + if mlist.max_num_recipients == 0: + return False + # Figure out how many recipients there are + recipients = getaddresses(msg.get_all('to', []) + + msg.get_all('cc', [])) + return len(recipients) >= mlist.max_num_recipients diff --git a/Mailman/rules/max_size.py b/Mailman/rules/max_size.py new file mode 100644 index 000000000..e54b68c9c --- /dev/null +++ b/Mailman/rules/max_size.py @@ -0,0 +1,46 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The maximum message size rule.""" + +__all__ = ['MaximumSize'] +__metaclass__ = type + + +from zope.interface import implements + +from Mailman.i18n import _ +from Mailman.interfaces import IRule + + + +class MaximumSize: + """The implicit destination rule.""" + implements(IRule) + + name = 'max-size' + description = _('Catch messages that are bigger than a specified maximum.') + record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + if mlist.max_message_size == 0: + return False + assert hasattr(msg, 'original_size'), ( + 'Message was not sized on initial parsing.') + # The maximum size is specified in 1024 bytes. + return msg.original_size / 1024.0 > mlist.max_message_size diff --git a/Mailman/rules/moderation.py b/Mailman/rules/moderation.py new file mode 100644 index 000000000..29a478662 --- /dev/null +++ b/Mailman/rules/moderation.py @@ -0,0 +1,66 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Membership related rules.""" + +__all__ = [ + 'Moderation', + 'NonMember', + ] +__metaclass__ = type + + +from zope.interface import implements + +from Mailman.i18n import _ +from Mailman.interfaces import IRule + + + +class Moderation: + """The member moderation rule.""" + implements(IRule) + + name = 'moderation' + description = _('Match messages sent by moderated members.') + record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + for sender in msg.get_senders(): + member = mlist.members.get_member(sender) + if member is not None and member.is_moderated: + return True + return False + + + +class NonMember: + """The non-membership rule.""" + implements(IRule) + + name = 'non-member' + description = _('Match messages sent by non-members.') + record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + for sender in msg.get_senders(): + if mlist.members.get_member(sender) is not None: + # The sender is a member of the mailing list. + return False + return True diff --git a/Mailman/rules/news_moderation.py b/Mailman/rules/news_moderation.py new file mode 100644 index 000000000..56c9fef37 --- /dev/null +++ b/Mailman/rules/news_moderation.py @@ -0,0 +1,44 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The news moderation rule.""" + +__all__ = ['ModeratedNewsgroup'] +__metaclass__ = type + + +from zope.interface import implements + +from Mailman.i18n import _ +from Mailman.interfaces import IRule, NewsModeration + + + +class ModeratedNewsgroup: + """The news moderation rule.""" + implements(IRule) + + name = 'news-moderation' + description = _( + u"""Match all messages posted to a mailing list that gateways to a + moderated newsgroup. + """) + record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + return mlist.news_moderation == NewsModeration.moderated diff --git a/Mailman/rules/no_subject.py b/Mailman/rules/no_subject.py new file mode 100644 index 000000000..4a8c6ac99 --- /dev/null +++ b/Mailman/rules/no_subject.py @@ -0,0 +1,42 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The no-Subject header rule.""" + +__all__ = ['NoSubject'] +__metaclass__ = type + + +from zope.interface import implements + +from Mailman.i18n import _ +from Mailman.interfaces import IRule + + + +class NoSubject: + """The no-Subject rule.""" + implements(IRule) + + name = 'no-subject' + description = _('Catch messages with no, or empty, Subject headers.') + record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + subject = msg.get('subject', '').strip() + return subject == '' diff --git a/Mailman/rules/suspicious.py b/Mailman/rules/suspicious.py new file mode 100644 index 000000000..4936734e5 --- /dev/null +++ b/Mailman/rules/suspicious.py @@ -0,0 +1,93 @@ +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""The historical 'suspicious header' rule.""" + +__all__ = ['SuspiciousHeader'] +__metaclass__ = type + + +import re +from email.utils import getaddresses +from zope.interface import implements + +from Mailman.i18n import _ +from Mailman.interfaces import IRule + + + +class SuspiciousHeader: + """The historical 'suspicious header' rule.""" + implements(IRule) + + name = 'suspicious-header' + description = _('Catch messages with suspicious headers.') + record = True + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + return (mlist.bounce_matching_headers and + has_matching_bounce_header(mlist, msg)) + + + +def _parse_matching_header_opt(mlist): + """Return a list of triples [(field name, regex, line), ...].""" + # - Blank lines and lines with '#' as first char are skipped. + # - Leading whitespace in the matchexp is trimmed - you can defeat + # that by, eg, containing it in gratuitous square brackets. + all = [] + for line in mlist.bounce_matching_headers.splitlines(): + line = line.strip() + # Skip blank lines and lines *starting* with a '#'. + if not line or line.startswith('#'): + continue + i = line.find(':') + if i < 0: + # This didn't look like a header line. BAW: should do a + # better job of informing the list admin. + log.error('bad bounce_matching_header line: %s\n%s', + mlist.real_name, line) + else: + header = line[:i] + value = line[i+1:].lstrip() + try: + cre = re.compile(value, re.IGNORECASE) + except re.error, e: + # The regexp was malformed. BAW: should do a better + # job of informing the list admin. + log.error("""\ +bad regexp in bounce_matching_header line: %s +\n%s (cause: %s)""", mlist.real_name, value, e) + else: + all.append((header, cre, line)) + return all + + +def has_matching_bounce_header(mlist, msg): + """Does the message have a matching bounce header? + + :param mlist: The mailing list the message is destined for. + :param msg: The email message object. + :return: True if a header field matches a regexp in the + bounce_matching_header mailing list variable. + """ + for header, cre, line in _parse_matching_header_opt(mlist): + for value in msg.get_all(header, []): + if cre.search(value): + return True + return False diff --git a/Mailman/rules/truth.py b/Mailman/rules/truth.py new file mode 100644 index 000000000..d3cfc30f3 --- /dev/null +++ b/Mailman/rules/truth.py @@ -0,0 +1,41 @@ +# Copyright (C) 2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""A rule which always matches.""" + +__all__ = ['Truth'] +__metaclass__ = type + + +from zope.interface import implements + +from Mailman.i18n import _ +from Mailman.interfaces import IRule + + + +class Truth: + """Look for any previous rule match.""" + implements(IRule) + + name = 'truth' + description = _('A rule which always matches.') + record = False + + def check(self, mlist, msg, msgdata): + """See `IRule`.""" + return True diff --git a/Mailman/templates/en/__init__.py b/Mailman/templates/en/__init__.py index e69de29bb..e69de29bb 100755..100644 --- a/Mailman/templates/en/__init__.py +++ b/Mailman/templates/en/__init__.py diff --git a/Mailman/templates/en/postauth.txt b/Mailman/templates/en/postauth.txt index a10727716..19f1e384a 100644 --- a/Mailman/templates/en/postauth.txt +++ b/Mailman/templates/en/postauth.txt @@ -1,7 +1,7 @@ As list administrator, your authorization is requested for the following mailing list posting: - List: %(listname)s@%(hostname)s + List: %(listname)s From: %(sender)s Subject: %(subject)s Reason: %(reason)s diff --git a/Mailman/tests/bounces/__init__.py b/Mailman/tests/bounces/__init__.py index e69de29bb..e69de29bb 100755..100644 --- a/Mailman/tests/bounces/__init__.py +++ b/Mailman/tests/bounces/__init__.py diff --git a/Mailman/tests/helpers.py b/Mailman/tests/helpers.py new file mode 100644 index 000000000..1b24f11e6 --- /dev/null +++ b/Mailman/tests/helpers.py @@ -0,0 +1,64 @@ +# Copyright (C) 2008 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Various test helpers.""" + +__metaclass__ = type +__all__ = [ + 'get_queue_messages', + 'make_testable_runner', + ] + + + +def make_testable_runner(runner_class): + """Create a queue runner that runs until its queue is empty. + + :param runner_class: An IRunner + :return: A runner instance. + """ + + class EmptyingRunner(runner_class): + """Stop processing when the queue is empty.""" + + def _doperiodic(self): + """Stop when the queue is empty.""" + self._stop = (len(self._switchboard.files) == 0) + + return EmptyingRunner() + + + +class _Bag: + def __init__(self, **kws): + for key, value in kws.items(): + setattr(self, key, value) + + +def get_queue_messages(queue): + """Return and clear all the messages in the given queue. + + :param queue: An ISwitchboard + :return: A list of 2-tuples where each item contains the message and + message metadata. + """ + messages = [] + for filebase in queue.files: + msg, msgdata = queue.dequeue(filebase) + messages.append(_Bag(msg=msg, msgdata=msgdata)) + queue.finish(filebase) + return messages diff --git a/Mailman/tests/smtplistener.py b/Mailman/tests/smtplistener.py index 565772b1d..977726247 100644 --- a/Mailman/tests/smtplistener.py +++ b/Mailman/tests/smtplistener.py @@ -1,4 +1,4 @@ -# Copyright (C) 2007 by the Free Software Foundation, Inc. +# Copyright (C) 2007-2008 by the Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License @@ -15,8 +15,11 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, # USA. +"""A test SMTP listener.""" + import sys import smtpd +import signal import mailbox import asyncore import optparse @@ -29,34 +32,68 @@ DEFAULT_PORT = 9025 class Channel(smtpd.SMTPChannel): - def smtp_EXIT(self, arg): - raise asyncore.ExitNow + """A channel that can reset the mailbox.""" + + def __init__(self, server, conn, addr): + smtpd.SMTPChannel.__init__(self, server, conn, addr) + # Stash this here since the subclass uses private attributes. :( + self._server = server + + def smtp_RSET(self, arg): + """Respond to RSET and clear the mailbox.""" + self._server.clear_mailbox() + smtpd.SMTPChannel.smtp_RSET(self, arg) + def send(self, data): + """Silence the bloody asynchat/asyncore broken pipe errors!""" + try: + return smtpd.SMTPChannel.send(self, data) + except socket.error: + # Nothing here can affect the outcome, and these messages are just + # plain annoying! So ignore them. + pass + + class Server(smtpd.SMTPServer): - def __init__(self, localaddr, mboxfile): + """An SMTP server that stores messages to a mailbox.""" + + def __init__(self, localaddr, mailbox_path): smtpd.SMTPServer.__init__(self, localaddr, None) - self._mbox = mailbox.mbox(mboxfile) + self._mailbox = mailbox.Maildir(mailbox_path) def handle_accept(self): + """Handle connections by creating our own Channel object.""" conn, addr = self.accept() Channel(self, conn, addr) def process_message(self, peer, mailfrom, rcpttos, data): + """Process a message by adding it to the mailbox.""" msg = message_from_string(data) msg['X-Peer'] = peer msg['X-MailFrom'] = mailfrom msg['X-RcptTo'] = COMMASPACE.join(rcpttos) - self._mbox.add(msg) + self._mailbox.add(msg) + self._mailbox.clean() - def close(self): - self._mbox.flush() - self._mbox.close() + + +def handle_signal(*ignore): + """Handle signal sent by parent to kill the process.""" + asyncore.socket_map.clear() def main(): - parser = optparse.OptionParser(usage='%prog mboxfile') + parser = optparse.OptionParser(usage="""\ +%prog [options] mboxfile + +This starts a process listening on a specified host and port (by default +localhost:9025) for SMTP conversations. All messages this process receives +are stored in a specified mbox file for the parent process to investigate. + +This SMTP server responds to RSET commands by clearing the mbox file. +""") parser.add_option('-a', '--address', type='string', default=None, help='host:port to listen on') @@ -77,12 +114,14 @@ def main(): host, port = opts.address.split(':', 1) port = int(port) + # Catch the parent's exit signal, and also C-c. + signal.signal(signal.SIGTERM, handle_signal) + signal.signal(signal.SIGINT, handle_signal) + server = Server((host, port), mboxfile) - try: - asyncore.loop() - except asyncore.ExitNow: - asyncore.close_all() - server.close() + asyncore.loop() + asyncore.close_all() + server.close() return 0 diff --git a/Mailman/tests/test_documentation.py b/Mailman/tests/test_documentation.py index 390ba6a66..ad00ba19c 100644 --- a/Mailman/tests/test_documentation.py +++ b/Mailman/tests/test_documentation.py @@ -18,7 +18,6 @@ """Harness for testing Mailman's documentation.""" import os -import pdb import doctest import unittest @@ -31,12 +30,25 @@ from Mailman.app.styles import style_manager from Mailman.configuration import config +DOT = '.' COMMASPACE = ', ' def specialized_message_from_string(text): - return message_from_string(text, Message) + """Parse text into a message object. + + This is specialized in the sense that an instance of Mailman's own Message + object is returned, and this message object has an attribute + `original_size` which is the pre-calculated size in bytes of the message's + text representation. + """ + # This mimic what Switchboard.dequeue() does when parsing a message from + # text into a Message instance. + original_size = len(text) + message = message_from_string(text, Message) + message.original_size = original_size + return message def setup(testobj): @@ -66,7 +78,12 @@ def cleaning_teardown(testobj): def test_suite(): suite = unittest.TestSuite() - docsdir = os.path.join(os.path.dirname(Mailman.__file__), 'docs') + topdir = os.path.dirname(Mailman.__file__) + packages = [] + for dirpath, dirnames, filenames in os.walk(topdir): + if 'docs' in dirnames: + docsdir = os.path.join(dirpath, 'docs')[len(topdir)+1:] + packages.append(docsdir) # Under higher verbosity settings, report all doctest errors, not just the # first one. flags = (doctest.ELLIPSIS | @@ -74,13 +91,15 @@ def test_suite(): doctest.REPORT_NDIFF) if config.opts.verbosity <= 2: flags |= doctest.REPORT_ONLY_FIRST_FAILURE - for filename in os.listdir(docsdir): - if os.path.splitext(filename)[1] == '.txt': - test = doctest.DocFileSuite( - 'docs/' + filename, - package=Mailman, - optionflags=flags, - setUp=setup, - tearDown=cleaning_teardown) - suite.addTest(test) + # Add all the doctests in all subpackages. + for docsdir in packages: + for filename in os.listdir(os.path.join('Mailman', docsdir)): + if os.path.splitext(filename)[1] == '.txt': + test = doctest.DocFileSuite( + os.path.join(docsdir, filename), + package='Mailman', + optionflags=flags, + setUp=setup, + tearDown=cleaning_teardown) + suite.addTest(test) return suite diff --git a/docs/NEWS.txt b/docs/NEWS.txt index 2cc28963e..89dc9c06d 100644 --- a/docs/NEWS.txt +++ b/docs/NEWS.txt @@ -1,10 +1,11 @@ Mailman - The GNU Mailing List Management System -Copyright (C) 1998-2007 by the Free Software Foundation, Inc. +Copyright (C) 1998-2008 by the Free Software Foundation, Inc. 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA Here is a history of user visible changes to Mailman. -3.0 alpha 1 (XX-XXX-200X) +3.0 alpha 1 -- "Leave That Thing Alone" +(XX-XXX-200X) User visible changes @@ -40,10 +41,22 @@ Here is a history of user visible changes to Mailman. instead. The substitution variable $fqdn_listname has been added. DEFAULT_MSG_FOOTER in Defaults.py.in hsa been updated accordingly. + - The KNOWN_SPAMMERS global variable is replaced with HEADER_MATCHES. The + mailing list's header_filter_rules variable is replaced with + header_matches which has the same semantics as HEADER_MATCHES, but is + list-specific. + + - DEFAULT_MAIL_COMMANDS_MAX_LINES -> EMAIL_COMMANDS_MAX_LINES + Architecture - - SQLAlchemy/Elixir based storage for all list and user data, with default - storage in a SQLite database. + - Implementation of a chain-of-rules based approach for deciding whether a + message should initially be accepted, held for approval, + rejected/bounced, or discarded. This replaces most of the disposition + handlers in the pipeline. + + - The Storm ORM is used for data storage, with the SQLite backend as the + default relational database. - Zope interfaces are used to describe the major components. @@ -126,6 +139,9 @@ Here is a history of user visible changes to Mailman. been removed. The domain that the list is being added to must already exist. + - Backport the ability to specify additional footer interpolation + variables by the message metadata 'decoration-data' key. + Bug fixes and other patches - Removal of DomainKey/DKIM signatures is now controlled by Defaults.py @@ -148,6 +164,9 @@ Here is a history of user visible changes to Mailman. incorrectly suppressed in messages that Mailman sends directly to users. + - The 'adminapproved' metadata key is renamed 'moderator_approved'. + + 2.1.9 (12-Sep-2006) Security @@ -88,6 +88,8 @@ Any other spelling is incorrect.""", 'mailman.database' : 'stock = Mailman.database:StockDatabase', 'mailman.mta' : 'stock = Mailman.MTA:Manual', 'mailman.styles' : 'default = Mailman.app.styles:DefaultStyle', + 'mailman.mta' : 'stock = Mailman.MTA:Manual', + 'mailman.rules' : 'default = Mailman.rules:initialize', }, # Third-party requirements. install_requires = [ |