Add a working (though not yet complete) test for the LMTP runner.

Refactor the TestableMaster class so that doctests don't need to do all the
threading and event fiddling themselves.  Hide all that in the test class.

In bin/master, add the -r/--runner option to allow overriding the set of queue
runners to start from the command line.  This is much more convenient than
fiddling the config file (which is still supported), and it allows us to start
a subset of the runners from the TestableMaster class.

Refactor the calculation of queue runner shortcut names into
Configuration.add_qrunner().  This way, it's easy to share between bin/qrunner
and bin/master.

Use pkg_resource in bin/testall to create the test configuration file.  We're
still using mailman.__file__ as the target to os.walk() though, so this
conversion isn't complete.

In bin/testall, update to the new convention of putting .options and
.arguments on the parser instance, so we only need to pass back one thing.

In test_documentation.py, use config.verbosity instead of
config.options.verbosity.  Wind that through to bin/testall while we're at
it.

Update loginit.py so that defaults for all the logging options can be found in
a [*] section.  This is better than [DEFAULT] because the latter requires
%-interpolation and still requires each sublogger to be specified explicitly.
Now we just set values in the [*] section and have them apply to all loggers.

Allow for passing bin/testall's -v and -e options to any subprocesses that get
created, e.g. via the TestableMaster.  Now testall will write a logging.cfg
file with a [*] entry as appropriate.

It's DEFAULT_DATABASE_URL now, not SQLALCHEMY_ENGINE_URL.

StockDatabase._reset() requires that the store be rolled back before it's
reset.  Otherwise if there are outstanding transactions, the reset will fail
with an OperationalError.

Update the LMTPRunner.  Have it return a 501 error if the message has defects.
Most defective messages are spam.  Make the LMTPRunner startable and
stoppable.  Just because it's asyncore based doesn't mean that's not possible
<wink>.

Enable the LMTP runner in tests.

3 files changed, 53 insertions, 5 deletions

diff --git a/mailman/tests/helpers.py b/mailman/tests/helpers.py
index 4dc289458..71b0914cb 100644
--- a/mailman/tests/helpers.py
+++ b/mailman/tests/helpers.py
@@ -23,6 +23,7 @@ __metaclass__ = type
 __all__ = [
     'TestableMaster',
     'digest_mbox',
+    'get_lmtp_client',
     'get_queue_messages',
     'make_testable_runner',
     ]
@@ -31,6 +32,8 @@ __all__ = [
 import os
 import time
 import errno
+import signal
+import socket
 import mailbox
 import smtplib
 import tempfile
@@ -102,12 +105,27 @@ def digest_mbox(mlist):
 class TestableMaster(Master):
     """A testable master loop watcher."""
 
-    def __init__(self, event):
+    def __init__(self):
         super(TestableMaster, self).__init__(
             restartable=False, config_file=config.filename)
-        self._event = event
+        self.event = threading.Event()
+        self.thread = threading.Thread(target=self.loop)
         self._started_kids = None
 
+    def start(self, *qrunners):
+        """Start the master."""
+        self.start_qrunners(qrunners)
+        self.thread.start()
+        # Wait until all the children are definitely started, or timeout.
+        self.event.wait(2.0)
+
+    def stop(self):
+        """Stop the master by killing all the children."""
+        for pid in self.qrunner_pids:
+            os.kill(pid, signal.SIGTERM)
+        self.cleanup()
+        self.thread.join()
+
     def loop(self):
         """Wait until all the qrunners are actually running before looping."""
         starting_kids = set(self._kids)
@@ -125,7 +143,7 @@ class TestableMaster(Master):
         # testing environment, even after all have exited.
         self._started_kids = set(self._kids)
         # Let the blocking thread know everything's running.
-        self._event.set()
+        self.event.set()
         super(TestableMaster, self).loop()
 
     @property
@@ -173,9 +191,38 @@ class SMTPServer:
             else:
                 self._messages.append(message)
             yield message
-            
+
     def clear(self):
         """Clear all messages from the queue."""
         # Just throw these away.
         list(self._messages)
         self._messages = []
+
+
+
+class LMTP(smtplib.SMTP):
+    """Like a normal SMTP client, but for LMTP."""
+    def lhlo(self, name=''):
+        self.putcmd('lhlo', name or self.local_hostname)
+        code, msg = self.getreply()
+        self.helo_resp = msg
+        return code, msg
+
+
+def get_lmtp_client():
+    """Return a connected LMTP client."""
+    # It's possible the process has started but is not yet accepting
+    # connections.  Wait a little while.
+    lmtp = LMTP()
+    for attempts in range(3):
+        try:
+            response = lmtp.connect(config.LMTP_HOST, config.LMTP_PORT)
+            print response
+            return lmtp
+        except socket.error, error:
+            if error[0] == errno.ECONNREFUSED:
+                time.sleep(1)
+            else:
+                raise
+    else:
+        raise RuntimeError('Connection refused')
diff --git a/mailman/tests/test_documentation.py b/mailman/tests/test_documentation.py
index d55e2b4bb..d11d4bd70 100644
--- a/mailman/tests/test_documentation.py
+++ b/mailman/tests/test_documentation.py
@@ -88,7 +88,7 @@ def test_suite():
     flags = (doctest.ELLIPSIS |
              doctest.NORMALIZE_WHITESPACE |
              doctest.REPORT_NDIFF)
-    if config.opts.verbosity <= 2:
+    if config.verbosity <= 2:
         flags |= doctest.REPORT_ONLY_FIRST_FAILURE
     # Add all the doctests in all subpackages.
     for docsdir in packages:
diff --git a/mailman/tests/testing.cfg.in b/mailman/tests/testing.cfg.in
index 074806cb1..e7b23ac6d 100644
--- a/mailman/tests/testing.cfg.in
+++ b/mailman/tests/testing.cfg.in
@@ -7,6 +7,7 @@
 SMTPPORT                = 10825
 MAX_RESTARTS            = 1
 MTA                     = None
+USE_LMTP                = Yes
 
 add_domain('example.com', 'www.example.com')