7 files changed, 252 insertions, 59 deletions

diff --git a/README.rst b/README.rst
index c87b7eccb..a9995c652 100644
--- a/README.rst
+++ b/README.rst
@@ -23,8 +23,8 @@ and there is a community driven wiki at
 
     http://wiki.list.org
 
-For more information on Mailman, see the above web sites, or the documentation
-provided with this software.  Start at src/mailman/docs/README.txt.
+For more information on Mailman, see the above web sites, or the
+:ref:`documentation provided with this software <start-here>`.
 
 
 Table of Contents
@@ -40,6 +40,7 @@ Table of Contents
     src/mailman/docs/8-miles-high
     src/mailman/docs/WebUIin5
     src/mailman/docs/ArchiveUIin5
+    src/mailman/docs/DEVELOP
 
 
 Mailman modules
@@ -50,20 +51,20 @@ These documents are generated from the doctest suite.
 .. toctree::
     :glob:
 
-    src/mailman/bin/docs/*
-    src/mailman/commands/docs/*
-    src/mailman/database/docs/*
     src/mailman/model/docs/*
-    src/mailman/core/docs/*
-    src/mailman/app/docs/*
-    src/mailman/styles/docs/*
     src/mailman/runners/docs/*
-    src/mailman/handlers/docs/*
-    src/mailman/rest/docs/*
     src/mailman/chains/docs/*
     src/mailman/rules/docs/*
+    src/mailman/handlers/docs/*
+    src/mailman/rest/docs/*
+    src/mailman/core/docs/*
+    src/mailman/app/docs/*
+    src/mailman/styles/docs/*
     src/mailman/archiving/docs/*
     src/mailman/mta/docs/*
+    src/mailman/database/docs/*
+    src/mailman/bin/docs/*
+    src/mailman/commands/docs/*
     src/mailman/docs/NEWS
     src/mailman/docs/STYLEGUIDE
     src/mailman/docs/ACKNOWLEDGMENTS
diff --git a/buildout.cfg b/buildout.cfg
index ef5d619d2..594e6ea5a 100644
--- a/buildout.cfg
+++ b/buildout.cfg
@@ -4,7 +4,7 @@ parts =
     tags
     test
 unzip = true
-develop = .
+develop = . /home/barry/projects/mailman/zope.interface
 
 [interpreter]
 recipe = z3c.recipe.scripts
diff --git a/src/mailman/docs/8-miles-high.rst b/src/mailman/docs/8-miles-high.rst
index 812d78d72..6e92ca817 100644
--- a/src/mailman/docs/8-miles-high.rst
+++ b/src/mailman/docs/8-miles-high.rst
@@ -11,7 +11,7 @@ Notes from the PyCon 2012 Mailman Sprint
 *These are notes from the Mailman sprint at PyCon 2012.  They are not
 terribly well organized, nor fully fleshed out.  Please edit and push
 branches to Launchpad at lp:mailman or post patches to
-<https://bugs.launchpad.net/mailman>.*
+the Mailman bug tracker at <https://bugs.launchpad.net/mailman>.*
 
 The intent of this document is to provide a view of Mailman 3's workflow and
 structures from "eight miles high".
@@ -139,7 +139,7 @@ The default set of rules looks something like this:
 Configuration
 =============
 
-Uses lazr.config.
+Uses `lazr.config`_, essentially an "ini"-style configuration format.
 
 Each Runner's configuration object knows whether it should be started
 when the Mailman daemon starts, and what queue the Runner manages.
@@ -187,3 +187,6 @@ Each list *style* is a named object.  Its attributes are functions used to
 apply the relevant style settings to the mailing list *at creation time*.
 Since these are functions, they can be composed in various ways, to create
 substyles, *etc*.
+
+
+.. _`lazr.config`: http://pypi.python.org/pypi/lazr.config
diff --git a/src/mailman/docs/DATABASE.rst b/src/mailman/docs/DATABASE.rst
index 762d60a11..229bfe8ed 100644
--- a/src/mailman/docs/DATABASE.rst
+++ b/src/mailman/docs/DATABASE.rst
@@ -11,12 +11,12 @@ Currently, Mailman is known to work with either the default SQLite3 database,
 or PostgreSQL.  (Volunteers to port it to other databases are welcome!).  If
 you want to use SQLite3, you generally don't need to change anything, but if
 you want Mailman to use PostgreSQL, you'll need to set that up first, and then
-change a configuration variable in your `/etc/mailman.cfg` file.
+change a configuration variable in your ``/etc/mailman.cfg`` file.
 
 Two configuration variables control which database Mailman uses.  The first
 names the class implementing the database interface.  The second names the
 Storm URL for connecting to the database.  Both variables live in the
-`[database]` section of the configuration file.
+``[database]`` section of the configuration file.
 
 
 SQLite3
@@ -24,10 +24,10 @@ SQLite3
 
 As mentioned, if you want to use SQLite3 in the default configuration, you
 generally don't need to change anything.  However, if you want to change where
-the SQLite3 database is stored, you can change the `url` variable in the
-`[database]` section.  By default, the database is stored in the *data
-directory* in the `mailman.db` file.  Here's how you'd force Mailman to store
-its database in `/var/lib/mailman/sqlite.db` file::
+the SQLite3 database is stored, you can change the ``url`` variable in the
+``[database]`` section.  By default, the database is stored in the *data
+directory* in the ``mailman.db`` file.  Here's how you'd force Mailman to
+store its database in ``/var/lib/mailman/sqlite.db`` file::
 
     [database]
     url: sqlite:////var/lib/mailman/sqlite.db
@@ -51,9 +51,9 @@ You would then need to set both the `class` and `url` variables in
 That should be it.
 
 Note that if you want to run the full test suite against PostgreSQL, you
-should make these changes to the `mailman/testing/test.cfg` file (yes,
-eventually we'll make this easier), start up PostgreSQL and run `bin/test` as
-normal.
+should make these changes to the ``mailman/testing/testing.cfg`` file (yes,
+eventually we'll make this easier), start up PostgreSQL and run ``bin/test``
+as normal.
 
 If you have any problems, you may need to delete the database and re-create
 it::
diff --git a/src/mailman/docs/DEVELOP.rst b/src/mailman/docs/DEVELOP.rst
new file mode 100644
index 000000000..c9e1bd596
--- /dev/null
+++ b/src/mailman/docs/DEVELOP.rst
@@ -0,0 +1,156 @@
+==================
+Developing Mailman
+==================
+
+The following documentation is generated from the internal developer
+documentation.  This documentation is also used by the test suite.
+
+For now, this will have to suffice as an overview of the Mailman system.
+
+
+Object model
+============
+
+Every major component of the system is defined in an interface.  Look through
+``src/mailman/interfaces`` for an understanding of the system components.
+Mailman objects which are stored in the database, are defined by *model*
+classes.  Objects such as *mailing lists*, *users*, *members*, and *addresses*
+are primary objects within the system.
+
+Obviously, the *mailing list* is the central object which holds all the
+configuration settings for a particular mailing list.  A mailing list is
+associated with a *domain*, and all mailing lists are managed by the *mailing
+list manager*.
+
+*Users* represent people, and have a *user id* and a *display name*.  Users
+are linked to *addresses* which represent a single email address.  One user
+can be linked to many addresses, but an address is only linked to one user.
+Addresses can be *verified* or *not verified*.  Mailman will deliver email
+only to *verified* addresses.
+
+Users and addresses are managed by the *user manager*.
+
+A *member* is created when a user subscribes to a mailing list, either via
+their *preferred address* or explicitly via one of their linked and verified
+addresses.  Members link an address to a mailing list, and represent regular
+members, digest members, list owners, and list moderators.  Members can even
+represent *non-members* (i.e. people not yet subscribed to the mailing list)
+for various moderation purposes.
+
+
+Process model
+=============
+
+Messages move around inside the Mailman system by way of *queue* directories
+managed by the *switchboard*.  For example, when a message is first received
+by Mailman, it is moved to the *in* (for "incoming") queue.  During the
+processing of this message, it -- or copies of it -- may be moved to other
+queues such as the *out* queue (for outgoing email), the *archive* queue (for
+sending to the archivers), the *digest* queue (for composing digests), etc.
+
+A message in a queue is represented by a single file, a ``.pck`` file.  This
+file contains two objects, serialized as `Python pickles`_.  The first object
+is the message being processed, already parsed into a more efficient internal
+representation.  The second object is a metadata dictionary that records
+additional information about the message as it is being processed.
+
+``.pck`` files only exist for messages moving between different system queues.
+There is no ``.pck`` file for messages while they are actively being
+processed.
+
+Each queue directory is associated with a *runner* process which wakes up
+every so often.  When the runner wakes up, it examines all the ``.pck`` files
+in FIFO order, deserializing the message and metadata objects, processing
+them.  If the message needs further processing in a different queue, it will
+be re-serialized back into a ``.pck`` file.  If not (e.g. because processing
+of the message is complete), then no ``.pck`` file is written.
+
+The Mailman system uses a few other runners which don't process messages in a
+queue.  You can think of these as fairly typical server process, and examples
+include the LMTP server, and the HTTP server for processing REST commands.
+
+All of the runners are managed by a *master watcher* process.  When you type
+``bin/mailman start`` you are actually starting the master.  Based on
+configuration options, the master will start the appropriate runners as
+subprocesses, and it will watch for the clean exiting of these subprocesses
+when ``bin/mailman stop`` is called.
+
+
+Rules and chains
+================
+
+When a message is first received for posting to a mailing list, Mailman
+processes the message to determine whether the message is appropriate for the
+mailing list.  If so, it *approves* the message and it gets posted.  Mailman
+can also *discard* the message so that no further processing occurs.  Mailman
+can also *reject* the message, bouncing it back to the original sender, usual
+with some indication of why the message was rejected.  Mailman can also *hold*
+the message for moderator approval.
+
+*Moderation* is the phase of processing that determines which of the above
+four dispositions will occur for the newly posted message.  Moderation does
+not generally change the message, but it may record information in the
+metadata dictionary.  Moderation is performed by the *in* queue runner.
+
+Each step in the moderation phase is performed by applying a *rule* to the
+message and asking whether the rule *hits* or *misses*.  Each rule is *linked*
+to an action which is taken if the rule hits (i.e. matches).  If the rule
+misses (i.e. doesn't match), then the next rule is tried.  All of the
+rule/action links are strung together sequentially into a *chain*, and every
+mailing list has a *start chain* where rule processing begins.
+
+Actually, every mailing list has *two* start chains, one for regular postings
+to the mailing list, and another for posting to the owners of the mailing
+list.
+
+To recap: when a message comes into Mailman for posting to a mailing list, the
+incoming runner finds the destination mailing list, determines whether the
+message is for the entire list membership, or the list owners, and retrieves
+the appropriate start chain.  The message is then passed to the chain, where
+each link in the chain first checks to see if its rule matches, and if so, it
+executes the linked action.  This action is usually one of the typical
+*accept*, *reject*, *discard*, and *hold*, but other actions are possible.
+
+As you might imagine, you can write new rules, compose them into new chains,
+and configure a mailing list to use your custom chain when processing the
+message during the moderation phase.
+
+
+Pipeline of handlers
+====================
+
+Once a message is accepted for posting to the mailing list, the message is
+usually modified in a number of different ways.  For example, some message
+headers may be added or removed, some MIME parts might be scrubbed, added, or
+rearranged, and various informative headers and footers may be added to the
+message.
+
+The process of preparing the message for the list membership (as well as the
+digests, archivers, and NNTP) falls to the *pipeline of handlers* managed by
+the *pipeline* queue.
+
+The pipeline of handlers is similar to the processing chain, except here, a
+handler can make any modifications to the message it wants, and there is no
+rule decision or action.  The message and metadata simply flow through a
+sequence of handlers arranged in a named pipeline.  Some of the handlers
+modify the message in ways described above, and others copy the message to the
+outgoing, NNTP, archiver, or digester queues.
+
+As with chains, each mailing list has two pipelines, one for posting to the
+list membership, and the other for posting to the list's owners.
+
+Of course, you can define new handlers, compose them into new pipelines, and
+change a mailing list's pipelines.
+
+
+Other bits and pieces
+=====================
+
+There are lots of other pieces to the Mailman puzzle, such as the REST API,
+the set of core functionality (logging, initialization, event handling, etc.),
+mailing list *styles*, the API for integrating external archivers and mail
+servers.  The database layer is an critical piece, and Mailman has an
+extensive set of command line commands, and email commands.
+
+
+.. _`Python pickles`: http://docs.python.org/2/library/pickle.html
diff --git a/src/mailman/docs/MTA.rst b/src/mailman/docs/MTA.rst
index ccfd18c0e..aa53981f8 100644
--- a/src/mailman/docs/MTA.rst
+++ b/src/mailman/docs/MTA.rst
@@ -21,7 +21,7 @@ for the popular open source mail servers.
 
 Note that Mailman provides lots of configuration variables that you can use to
 tweak performance for your operating environment.  See the
-`src/mailman/config/schema.cfg` file for details.
+``src/mailman/config/schema.cfg`` file for details.
 
 
 Exim
@@ -40,7 +40,7 @@ Mailman settings
 ----------------
 
 You need to tell Mailman that you are using the Postfix mail server.  In your
-`mailman.cfg` file, add the following section::
+``mailman.cfg`` file, add the following section::
 
     [mta]
     incoming: mailman.mta.postfix.LMTP
@@ -51,8 +51,8 @@ You need to tell Mailman that you are using the Postfix mail server.  In your
     smtp_port: 25
 
 Some of these settings are already the default, so take a look at Mailman's
-`src/mailman/config/schema.cfg` file for details.  You'll need to change the
-`lmtp_host` and `smtp_host` to the appropriate host names of course.
+``src/mailman/config/schema.cfg`` file for details.  You'll need to change the
+``lmtp_host`` and ``smtp_host`` to the appropriate host names of course.
 Generally, Postfix will listen for incoming SMTP connections on port 25.
 Postfix will deliver via LMTP over port 24 by default, however if you are not
 running Mailman as root, you'll need to change this to a higher port number,
@@ -63,7 +63,7 @@ Basic Postfix connections
 -------------------------
 
 There are several ways to hook Postfix up to Mailman, so here are the simplest
-instructions.  The following settings should be added to Postfix's `main.cf`
+instructions.  The following settings should be added to Postfix's ``main.cf``
 file.
 
 Mailman supports a technique called `Variable Envelope Return Path`_ (VERP) to
@@ -76,14 +76,14 @@ handle Mailman's VERP'd messages::
 
 In older versions of Postfix, unknown local recipients generated a temporary
 failure.  It's much better (and the default in newer Postfix releases) to
-treat them as permanent failures.  You can add this to your `main.cf` file if
-needed (use the `postconf`_ to check the defaults)::
+treat them as permanent failures.  You can add this to your ``main.cf`` file
+if needed (use the `postconf`_ command to check the defaults)::
 
     unknown_local_recipient_reject_code = 550
 
-While generally not necessary if you set `recipient_delimiter` as described
-above, it's better for Postfix to not treat `owner-` and `-request` addresses
-specially::
+While generally not necessary if you set ``recipient_delimiter`` as described
+above, it's better for Postfix to not treat ``owner-`` and ``-request``
+addresses specially::
 
     owner_request_special = no
 
@@ -93,9 +93,9 @@ Transport maps
 
 By default, Mailman works well with Postfix transport maps as a way to deliver
 incoming messages to Mailman's LMTP server.  Mailman will automatically write
-the correct transport map when its `bin/mailman aliases` command is run, or
+the correct transport map when its ``bin/mailman aliases`` command is run, or
 whenever a mailing list is created or removed via other commands.  To connect
-Postfix to Mailman's LMTP server, add the following to Postfix's `main.cf`
+Postfix to Mailman's LMTP server, add the following to Postfix's ``main.cf``
 file::
 
     transport_maps =
@@ -105,8 +105,8 @@ file::
     relay_domains =
         hash:/path-to-mailman/var/data/postfix_domains
 
-where `path-to-mailman` is replaced with the actual path that you're running
-Mailman from.  Setting `local_recipient_maps` as well as `transport_maps`
+where ``path-to-mailman`` is replaced with the actual path that you're running
+Mailman from.  Setting ``local_recipient_maps`` as well as ``transport_maps``
 allows Postfix to properly reject all messages destined for non-existent local
 users.  Setting `relay_domains`_ means Postfix will start to accept mail for
 newly added domains even if they are not part of `mydestination`_.
@@ -114,7 +114,7 @@ newly added domains even if they are not part of `mydestination`_.
 Note that if you are not using virtual domains, then `relay_domains`_ isn't
 strictly needed (but it is harmless).  All you need to do in this scenario is
 to make sure that Postfix accepts mail for your one domain, normally by
-including it in `mydestination`.
+including it in ``mydestination``.
 
 
 Postfix documentation
diff --git a/src/mailman/docs/START.rst b/src/mailman/docs/START.rst
index dbf4966c4..2c88bb808 100644
--- a/src/mailman/docs/START.rst
+++ b/src/mailman/docs/START.rst
@@ -1,3 +1,5 @@
+.. _start-here:
+
 ================================
 Getting started with GNU Mailman
 ================================
@@ -20,8 +22,8 @@ unlike Mailman 2 whose web interface and archives were tightly integrated with
 the core, Mailman 3 exposes a REST administrative interface to the web,
 communicates with archivers via decoupled interfaces, and leaves summary,
 search, and retrieval of archived messages to a separate application (a simple
-implementation is provided).  As of beta 1 the web interface and archiver are
-still at an early stage of development.  As such, some things may not work.
+implementation is provided).  The web interface (known as `Postorius`_) and
+archiver (known as `Hyperkitty`_) are in separate development.
 
 Contributions are welcome.  Please submit bug reports on the Mailman bug
 tracker at https://bugs.launchpad.net/mailman though you will currently need
@@ -33,7 +35,7 @@ Requirements
 ============
 
 Python 2.7 is required.  It can either be the default 'python' on your
-$PATH or it can be accessible via the ``python2.7`` binary.
+``$PATH`` or it can be accessible via the ``python2.7`` binary.
 If your operating system does not include Python, see http://www.python.org
 downloading and installing it from source.  Python 3 is not yet supported.
 
@@ -41,18 +43,28 @@ In this documentation, a bare ``python`` refers to the Python executable used
 to invoke ``bootstrap.py``.
 
 Mailman 3 is now based on the `zc.buildout`_ infrastructure, which greatly
-simplifies building and testing Mailman.
+simplifies testing Mailman.  Buildout is not required for installation.
 
 During the beta program, you may need some additional dependencies, such as a
 C compiler and the Python development headers and libraries.  You will need an
-internet connection.  Also `Postorius`_ (the web UI) and the `archive UI`_ are
-distributed and installed separately.
+internet connection.
 
 
 Building Mailman 3
 ==================
 
-Here are the commands to build everything in core Mailman::
+We provide several recipes for building Mailman.  All should generally work,
+but some may provide a better experience for developing Mailman versus
+deploying Mailman.
+
+
+Building for development
+------------------------
+
+The best way to build Mailman for development is to use the `zc.buildout`_
+tools.  This will download all Mailman dependencies from the `Python
+Cheeseshop`_.  The dependencies will get installed locally, but isolated from
+your system Python.  Here are the commands to build Mailman for development::
 
     % python bootstrap.py
     % bin/buildout
@@ -70,9 +82,9 @@ Build the online docs by running::
 
     % python setup.py build_sphinx
 
-(You might get warnings which you can safely ignore.)  Then visit::
+Then visit::
 
-    build/sphinx/html/README.html
+    build/sphinx/html/index.html
 
 in your browser to start reading the documentation.  Or you can just read the
 doctests by looking in all the 'doc' directories under the 'mailman' package.
@@ -80,13 +92,33 @@ Doctests are documentation first, so they should give you a pretty good idea
 how various components of Mailman 3 work.
 
 
+Building for deployment using virtualenv
+----------------------------------------
+
+`virtualenv`_ is a way to create isolated Python environments.  You can use
+virtualenv as a way to deploy Mailman without installing it into your system
+Python.  There are lots of ways to use virtualenv, but as described here, it
+will be default use any dependencies which are already installed in your
+system, downloading from the Cheeseshop only those which are missing.  Here
+are the steps to install Mailman using virtualenv::
+
+    $ virtualenv --system-site-packages /path/to/your/installation
+    $ source /path/to/your/installation/bin/activate
+    $ python setup.py install
+
+Once everything is downloaded and installed, you can initialize Mailman and
+get a display of the basic configuration settings by running::
+
+    $ mailman info -v
+
+
 Running Mailman 3
 =================
 
-What, you actually want to *run* Mailman 3?  Oh well, if you insist.  You
-will need to set up a configuration file to override the defaults and set
-things up for your environment.  Mailman is configured via the `lazr.config`_
-package which is really just a fancy ini-style configuration system.
+What, you actually want to *run* Mailman 3?  Oh well, if you insist.  You will
+need to set up a configuration file to override the defaults and set things up
+for your environment.  Mailman is configured using an "ini"-style
+configuration system.
 
 ``src/mailman/config/schema.cfg`` defines the ini-file schema and contains
 documentation for every section and configuration variable.  Sections that end
@@ -139,8 +171,8 @@ however, given Florian Fuchs' `Five Minute Guide` from his `blog post`_ on the
 Mailman wiki.  (Check the `blog post`_ for the most recent version!)
 
 
-The List Archiver
------------------
+The Archiver
+------------
 
 In Mailman 3, the archivers are decoupled from the core engine.  It is useful
 to provide a simple, standard interface for third-party archiving tools and
@@ -149,23 +181,24 @@ messages into any of a number of configured archivers, using whatever protocol
 is appropriate for that archiver.  Summary, search, and retrieval of archived
 posts are handled by a separate application.
 
-A new `archive UI`_ called Hyperkitty, based on the `notmuch mail indexer`_
+A new archive UI called `Hyperkitty`_, based on the `notmuch mail indexer`_
 and `Django`_, was prototyped at the PyCon 2012 sprint by Toshio Kuratomi, and
 like the web UI it is also in early alpha as of Mailman 3 beta 1.  The
-"hyperkitty" archiver is very loosely coupled to Mailman 3 core.  In fact, any
-email application that speaks LMTP or SMTP will be able to use hyperkitty.
+Hyperkitty archiver is very loosely coupled to Mailman 3 core.  In fact, any
+email application that speaks LMTP or SMTP will be able to use Hyperkitty.
 
-A `five minute guide to hyperkitty`_ is based on Toshio Kuratomi's README.
+A `five minute guide to Hyperkitty`_ is based on Toshio Kuratomi's README.
 
 
 .. _`zc.buildout`: http://pypi.python.org/pypi/zc.buildout
-.. _`lazr.config`: http://pypi.python.org/pypi/lazr.config
 .. _`Postorius`: https://launchpad.net/postorius
-.. _`archive UI`: https://launchpad.net/hyperkitty
+.. _`Hyperkitty`: https://launchpad.net/hyperkitty
 .. _`Django`: http://djangoproject.org/
 .. _`REST client module`: https://launchpad.net/mailman.client
 .. _`Five Minute Guide the Web UI`: WebUIin5.html
 .. _`blog post`: http://wiki.list.org/display/DEV/A+5+minute+guide+to+get+the+Mailman+web+UI+running
 .. _`notmuch mail indexer`: http://notmuchmail.org
-.. _`five minute guide to hyperkitty`: ArchiveUIin5.html
+.. _`five minute guide to Hyperkitty`: ArchiveUIin5.html
 .. _`Pycon 2012 sprint`: https://us.pycon.org/2012/community/sprints/projects/
+.. _`Python Cheeseshop`: http://pypi.python.org/pypi
+.. _`virtualenv`: http://www.virtualenv.org/en/latest/