Move the pending database into the SQLAlchemy/Elixir layer. The old

Pending.py module is removed.  Added an interface to this functionality such
that any IPendable (essentially a key/value mapping) can be associated with a
token, and that token can be confirmed and has a lifetime.  Any keys and
values can be stored, as long as both are unicodes.

Added a doctest.

Modified initialization of the database layer to support pluggability via
setuptools.  No longer is this layer initialized from a module, but now it's
instantiated from a class that implements IDatabase.  The StockDatabase class
implements the SQLAchemy/Elixir layer, but this can be overridden in a
setup.py.  Bye bye MANAGERS_INIT_FUNCTION, we hardly knew ye.

Added a package Mailman.app which will contain certain application specific
functionality.  Right now, the only there there is an IRegistar
implementation, which didn't seem to fit anywhere else.

Speaking of which, the IRegistrar interface implements all the logic related
to registration and verification of email addresses.  Think the equivalent of
MailList.AddMember() except generalized out of a mailing list context.  This
latter will eventually go away.  The IRegistrar sends the confirmation email.

Added an IDomain interface, though the only implementation of this so far
lives in the registration.txt doctest.  This defines the context necessary for
domain-level things, like address confirmation.

A bunch of other cleanups in modules that are necessary due to the refactoring
of Pending, but don't affect anything that's actually tested yet, so I won't
vouch for them (except that they don't throw errors on import!).

Clean up Defaults.py; also turn the functions seconds(), minutes(), hours()
and days() into their datetime.timedelta equivalents.

Consolidated the bogus email address exceptions.

In some places where appropriate, use email 4.0 module names instead of the
older brand.

Switch from Mailman.Utils.unique_message_id() to email.utils.make_msgid()
everywhere.  This is because we need to allow sending not in the context of a
mailing list (i.e. domain-wide address confirmation message).  So we can't use
a Message-ID generator that requires a mailing list.  OTOH, this breaks
Message-ID collision detection in the mail->news gateway.  I'll fix that
eventually.

Remove the 'verified' row on the Address table.  Now verification is checked
by Address.verified_on not being None.

2 files changed, 488 insertions, 0 deletions

diff --git a/Mailman/docs/pending.txt b/Mailman/docs/pending.txt
new file mode 100644
index 000000000..518dadf87
--- /dev/null
+++ b/Mailman/docs/pending.txt
@@ -0,0 +1,104 @@
+The pending database
+====================
+
+The pending database is where various types of events which need confirmation
+are stored.  These can include email address registration events, held
+messages (but only for user confirmation), auto-approvals, and probe bounces.
+This is not where messages held for administrator approval are kept.
+
+    >>> from Mailman.configuration import config
+    >>> from Mailman.database import flush
+    >>> from Mailman.interfaces import IPendable, IPending
+    >>> from zope.interface import implements
+    >>> from zope.interface.verify import verifyObject
+
+In order to pend an event, you first need a pending database, which is
+available by adapting the list manager.
+
+    >>> pendingdb = IPending(config.list_manager)
+    >>> verifyObject(IPending, pendingdb)
+    True
+
+The pending database can add any IPendable to the database, returning a token
+that can be used in urls and such.
+
+    >>> class SimplePendable(dict):
+    ...     implements(IPendable)
+    >>> subscription = SimplePendable(
+    ...     type='subscription',
+    ...     address='aperson@example.com',
+    ...     realname='Anne Person',
+    ...     language='en',
+    ...     password='xyz')
+    >>> token = pendingdb.add(subscription)
+    >>> flush()
+    >>> len(token)
+    40
+
+There's not much you can do with tokens except to 'confirm' them, which
+basically means returning the IPendable structure (as a dict) from the
+database that matches the token.  If the token isn't in the database, None is
+returned.
+
+    >>> pendable = pendingdb.confirm('missing')
+    >>> print pendable
+    None
+    >>> pendable = pendingdb.confirm(token)
+    >>> flush()
+    >>> sorted(pendable.items())
+    [('address', 'aperson@example.com'),
+     ('language', 'en'),
+     ('password', 'xyz'),
+     ('realname', 'Anne Person'),
+     ('type', 'subscription')]
+
+After confirmation, the token is no longer in the database.
+
+    >>> pendable = pendingdb.confirm(token)
+    >>> print pendable
+    None
+
+There are a few other things you can do with the pending database.  When you
+confirm a token, you can leave it in the database, or in otherwords, not
+expunge it.
+
+    >>> event_1 = SimplePendable(type='one')
+    >>> token_1 = pendingdb.add(event_1)
+    >>> event_2 = SimplePendable(type='two')
+    >>> token_2 = pendingdb.add(event_2)
+    >>> event_3 = SimplePendable(type='three')
+    >>> token_3 = pendingdb.add(event_3)
+    >>> flush()
+    >>> pendable = pendingdb.confirm(token_1, expunge=False)
+    >>> flush()
+    >>> pendable.items()
+    [('type', 'one')]
+    >>> pendable = pendingdb.confirm(token_1, expunge=True)
+    >>> flush()
+    >>> pendable.items()
+    [('type', 'one')]
+    >>> pendable = pendingdb.confirm(token_1)
+    >>> flush()
+    >>> print pendable
+    None
+
+An event can be given a lifetime when it is pended, otherwise it just uses a
+default lifetime.
+
+    >>> from datetime import timedelta
+    >>> yesterday = timedelta(days=-1)
+    >>> event_4 = SimplePendable(type='four')
+    >>> token_4 = pendingdb.add(event_4, lifetime=yesterday)
+    >>> flush()
+
+Every once in a while the pending database is cleared of old records.
+
+    >>> pendingdb.evict()
+    >>> flush()
+    >>> pendable = pendingdb.confirm(token_4)
+    >>> print pendable
+    None
+    >>> pendable = pendingdb.confirm(token_2)
+    >>> pendable.items()
+    [('type', 'two')]
+    >>> flush()
diff --git a/Mailman/docs/registration.txt b/Mailman/docs/registration.txt
new file mode 100644
index 000000000..1d59f184b
--- /dev/null
+++ b/Mailman/docs/registration.txt
@@ -0,0 +1,384 @@
+Address registration
+====================
+
+When a user wants to join a mailing list -- any mailing list -- in the running
+instance, he or she must first register with Mailman.  The only thing they
+must supply is an email address, although there is additional information they
+may supply.  All registered email addresses must be verified before Mailman
+will send them any list traffic.
+
+    >>> from Mailman.app.registrar import Registrar
+    >>> from Mailman.configuration import config
+    >>> from Mailman.database import flush
+    >>> from Mailman.interfaces import IRegistrar
+
+The IUserManager manages users, but it does so at a fairly low level.
+Specifically, it does not handle verifications, email address syntax validity
+checks, etc.  The IRegistrar is the interface to the object handling all this
+stuff.
+
+Create a dummy domain, which will provide the context for the verification
+email message.
+
+    >>> from zope.interface import implements
+    >>> from Mailman.interfaces import IDomain
+    >>> class TestDomain(object):
+    ...     implements(IDomain)
+    ...     def __init__(self):
+    ...         self.domain_name = 'example.com'
+    ...         self.description = 'mail.example.com'
+    ...         self.contact_address = 'postmaster@mail.example.com'
+    ...         self.base_url = 'http://mail.example.com'
+    ...     def confirm_address(self, token=''):
+    ...         return 'confirm+%s@example.com' % token
+    ...     def confirm_url(self, token=''):
+    ...         return self.base_url + '/confirm/' + token
+    ...     def __conform__(self, protocol):
+    ...         if protocol is IRegistrar:
+    ...             return Registrar(self)
+    ...         return None
+    >>> domain = TestDomain()
+
+Get a registrar by adapting a context to the interface.
+
+    >>> from zope.interface.verify import verifyObject
+    >>> registrar = IRegistrar(domain)
+    >>> verifyObject(IRegistrar, registrar)
+    True
+
+Here is a helper function to check the token strings.
+
+    >>> def check_token(token):
+    ...     assert isinstance(token, basestring), 'Not a string'
+    ...     assert len(token) == 40, 'Unexpected length: %d' % len(token)
+    ...     assert token.isalnum(), 'Not alphanumeric'
+    ...     print 'ok'
+
+Here is a helper function to extract tokens from confirmation messages.
+
+    >>> import re
+    >>> cre = re.compile('http://mail.example.com/confirm/(.*)')
+    >>> def extract_token(msg):
+    ...     mo = cre.search(qmsg.get_payload())
+    ...     return mo.group(1)
+
+
+Invalid email addresses
+-----------------------
+
+The only piece of information you need to register is the email address.
+Some amount of sanity checks are performed on the email address, although
+honestly, not as much as probably should be done.  Still, some patently bad
+addresses are rejected outright.
+
+    >>> registrar.register('')
+    Traceback (most recent call last):
+    ...
+    InvalidEmailAddress: ''
+    >>> registrar.register('some name@example.com')
+    Traceback (most recent call last):
+    ...
+    InvalidEmailAddress: 'some name@example.com'
+    >>> registrar.register('<script>@example.com')
+    Traceback (most recent call last):
+    ...
+    InvalidEmailAddress: '<script>@example.com'
+    >>> registrar.register('\xa0@example.com')
+    Traceback (most recent call last):
+    ...
+    InvalidEmailAddress: '\xa0@example.com'
+    >>> registrar.register('noatsign')
+    Traceback (most recent call last):
+    ...
+    InvalidEmailAddress: 'noatsign'
+    >>> registrar.register('nodom@ain')
+    Traceback (most recent call last):
+    ...
+    InvalidEmailAddress: 'nodom@ain'
+
+
+Register an email address
+-------------------------
+
+Registration of an unknown address creates nothing until the confirmation step
+is complete.  No IUser or IAddress is created at registration time, but a
+record is added to the pending database, and the token for that record is
+returned.
+
+    >>> token = registrar.register('aperson@example.com', 'Anne Person')
+    >>> flush()
+    >>> check_token(token)
+    ok
+
+There should be no records in the user manager for this address yet.
+
+    >>> usermgr = config.user_manager
+    >>> print usermgr.get_user('aperson@example.com')
+    None
+    >>> print usermgr.get_address('aperson@example.com')
+    None
+
+But this address is waiting for confirmation.
+
+    >>> from Mailman.interfaces import IPending
+    >>> pendingdb = IPending(config.db)
+    >>> sorted(pendingdb.confirm(token, expunge=False).items())
+    [('address', 'aperson@example.com'),
+     ('real_name', 'Anne Person'),
+     ('type', 'registration')]
+
+
+Verification by email
+---------------------
+
+There is also a verification email sitting in the virgin queue now.  This
+message is sent to the user in order to verify the registered address.
+
+    >>> from Mailman.Queue.Switchboard import Switchboard
+    >>> switchboard = Switchboard(config.VIRGINQUEUE_DIR)
+    >>> len(switchboard.files)
+    1
+    >>> filebase = switchboard.files[0]
+    >>> qmsg, qdata = switchboard.dequeue(filebase)
+    >>> switchboard.finish(filebase)
+    >>> print qmsg.as_string()
+    MIME-Version: 1.0
+    Content-Type: text/plain; charset="us-ascii"
+    Content-Transfer-Encoding: 7bit
+    Subject: confirm ...
+    From: confirm+...@example.com
+    To: aperson@example.com
+    Message-ID: <...>
+    Date: ...
+    Precedence: bulk
+    <BLANKLINE>
+    Email Address Registration Confirmation
+    <BLANKLINE>
+    Hello, this is the GNU Mailman server at example.com.
+    <BLANKLINE>
+    We have received a registration request for the email address
+    <BLANKLINE>
+        aperson@example.com
+    <BLANKLINE>
+    Before you can start using GNU Mailman at this site, you must first
+    confirm that this is your email address.  You can do this by replying to
+    this message, keeping the Subject header intact.  Or you can visit this
+    web page
+    <BLANKLINE>
+        http://mail.example.com/confirm/...
+    <BLANKLINE>
+    If you do not wish to register this email address simply disregard this
+    message.  If you think you are being maliciously subscribed to the list, or
+    have any other questions, you may contact
+    <BLANKLINE>
+        postmaster@mail.example.com
+    <BLANKLINE>
+    >>> sorted(qdata.items())
+    [('_parsemsg', False),
+     ('nodecorate', True),
+     ('received_time', ...),
+     ('recips', ['aperson@example.com']),
+     ('reduced_list_headers', True),
+     ('version', 3)]
+
+The confirmation token shows up in several places, each of which provides an
+easy way for the user to complete the confirmation.  The token will always
+appear in a URL in the body of the message.
+
+    >>> sent_token = extract_token(qmsg)
+    >>> sent_token == token
+    True
+
+The same token will appear in the From header.
+
+    >>> qmsg['from'] == 'confirm+' + token + '@example.com'
+    True
+
+It will also appear in the Subject header.
+
+    >>> qmsg['subject'] == 'confirm ' + token
+    True
+
+The user would then validate their just registered address by clicking on a
+url or responding to the message.  Either way, the confirmation process
+extracts the token and uses that to confirm the pending registration.
+
+    >>> registrar.confirm(token)
+    True
+    >>> flush()
+
+Now, there is an IAddress in the database matching the address, as well as an
+IUser linked to this address.  The IAddress is verified.
+
+    >>> found_address = usermgr.get_address('aperson@example.com')
+    >>> found_address
+    <Address: Anne Person <aperson@example.com> [verified] at ...>
+    >>> found_user = usermgr.get_user('aperson@example.com')
+    >>> found_user
+    <User "Anne Person" at ...>
+    >>> found_user.controls(found_address.address)
+    True
+    >>> from datetime import datetime
+    >>> isinstance(found_address.verified_on, datetime)
+    True
+
+
+Non-standard registrations
+--------------------------
+
+If you try to confirm a registration token twice, of course only the first one
+will work.  The second one is ignored.
+
+    >>> token = registrar.register('bperson@example.com')
+    >>> flush()
+    >>> check_token(token)
+    ok
+    >>> filebase = switchboard.files[0]
+    >>> qmsg, qdata = switchboard.dequeue(filebase)
+    >>> switchboard.finish(filebase)
+    >>> sent_token = extract_token(qmsg)
+    >>> token == sent_token
+    True
+    >>> registrar.confirm(token)
+    True
+    >>> flush()
+    >>> registrar.confirm(token)
+    False
+
+If an address is in the system, but that address is not linked to a user yet
+and the address is not yet validated, then no user is created until the
+confirmation step is completed.
+
+    >>> usermgr.create_address('cperson@example.com')
+    <Address: cperson@example.com [not verified] at ...>
+    >>> flush()
+    >>> token = registrar.register('cperson@example.com', 'Claire Person')
+    >>> flush()
+    >>> print usermgr.get_user('cperson@example.com')
+    None
+    >>> filebase = switchboard.files[0]
+    >>> qmsg, qdata = switchboard.dequeue(filebase)
+    >>> switchboard.finish(filebase)
+    >>> registrar.confirm(token)
+    True
+    >>> flush()
+    >>> usermgr.get_user('cperson@example.com')
+    <User "Claire Person" at ...>
+    >>> usermgr.get_address('cperson@example.com')
+    <Address: cperson@example.com [verified] at ...>
+
+If an address being registered has already been verified, linked or not to a
+user, then registration sends no confirmation.
+
+    >>> print registrar.register('cperson@example.com')
+    None
+    >>> len(switchboard.files)
+    0
+
+But if the already verified address is not linked to a user, then a user is
+created now and they are linked, with no confirmation necessary.
+
+    >>> address = usermgr.create_address('dperson@example.com', 'Dave Person')
+    >>> address.verified_on = datetime.now()
+    >>> flush()
+    >>> print usermgr.get_user('dperson@example.com')
+    None
+    >>> print registrar.register('dperson@example.com')
+    None
+    >>> flush()
+    >>> len(switchboard.files)
+    0
+    >>> usermgr.get_user('dperson@example.com')
+    <User "Dave Person" at ...>
+
+
+Discarding
+----------
+
+A confirmation token can also be discarded, say if the user changes his or her
+mind about registering.  When discarded, no IAddress or IUser is created.
+
+    >>> token = registrar.register('eperson@example.com', 'Elly Person')
+    >>> check_token(token)
+    ok
+    >>> flush()
+    >>> registrar.discard(token)
+    >>> flush()
+    >>> print pendingdb.confirm(token)
+    None
+    >>> print usermgr.get_address('eperson@example.com')
+    None
+    >>> print usermgr.get_user('eperson@example.com')
+    None
+
+
+Registering a new address for an existing user
+----------------------------------------------
+
+When a new address for an existing user is registered, there isn't too much
+different except that the new address will still need to be verified before it
+can be used.
+
+    >>> dperson = usermgr.get_user('dperson@example.com')
+    >>> dperson
+    <User "Dave Person" at ...>
+    >>> from operator import attrgetter
+    >>> sorted((addr for addr in dperson.addresses), key=attrgetter('address'))
+    [<Address: Dave Person <dperson@example.com> [verified] at ...>]
+    >>> dperson.register('david.person@example.com', 'David Person')
+    <Address: David Person <david.person@example.com> [not verified] at ...>
+    >>> flush()
+    >>> token = registrar.register('david.person@example.com')
+    >>> flush()
+    >>> filebase = switchboard.files[0]
+    >>> qmsg, qdata = switchboard.dequeue(filebase)
+    >>> switchboard.finish(filebase)
+    >>> registrar.confirm(token)
+    True
+    >>> user = usermgr.get_user('david.person@example.com')
+    >>> user is dperson
+    True
+    >>> user
+    <User "Dave Person" at ...>
+    >>> sorted((addr for addr in user.addresses), key=attrgetter('address'))
+    [<Address: David Person <david.person@example.com> [verified] at ...>,
+     <Address: Dave Person <dperson@example.com> [verified] at ...>]
+
+
+Corner cases
+------------
+
+If you try to confirm a token that doesn't exist in the pending database, the
+confirm method will just return None.
+
+    >>> registrar.confirm('no token')
+    False
+
+Likewise, if you try to confirm, through the IUserRegistrar interface, a token
+that doesn't match a registration even, you will get None.  However, the
+pending even matched with that token will still be removed.
+
+    >>> from Mailman.interfaces import IPendable, IPending
+    >>> pendingdb = IPending(config.db)
+    >>> class SimplePendable(dict):
+    ...     implements(IPendable)
+    >>> pendable = SimplePendable(type='foo', bar='baz')
+    >>> token = pendingdb.add(pendable)
+    >>> flush()
+    >>> registrar.confirm(token)
+    False
+    >>> flush()
+    >>> print pendingdb.confirm(token)
+    None
+
+If somehow the pending registration event doesn't have an address in its
+record, you will also get None back, and the record will be removed.
+
+    >>> pendable = SimplePendable(type='registration', foo='bar')
+    >>> token = pendingdb.add(pendable)
+    >>> flush()
+    >>> registrar.confirm(token)
+    False
+    >>> flush()
+    >>> print pendingdb.confirm(token)
+    None