From e6b14e5a509fa28d8bf871d88260be0b7749c72b Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Sat, 4 Feb 2017 17:08:14 -0500 Subject: Extensive reorganization --- src/mailman/config/docs/config.rst | 16 +- src/mailman/docs/ARCHITECTURE.rst | 164 --------------- src/mailman/docs/ArchiveUIin5.rst | 108 ---------- src/mailman/docs/CONTRIBUTE.rst | 245 ---------------------- src/mailman/docs/DATABASE.rst | 134 ------------ src/mailman/docs/INSTALL.rst | 120 ----------- src/mailman/docs/INTRODUCTION.rst | 126 ----------- src/mailman/docs/MTA.rst | 412 ------------------------------------ src/mailman/docs/RELEASENOTES.rst | 26 --- src/mailman/docs/STYLEGUIDE.rst | 33 +-- src/mailman/docs/WebUIin5.rst | 101 --------- src/mailman/docs/architecture.rst | 188 +++++++++++++++++ src/mailman/docs/contribute.rst | 247 ++++++++++++++++++++++ src/mailman/docs/database.rst | 132 ++++++++++++ src/mailman/docs/hyperkitty.rst | 108 ++++++++++ src/mailman/docs/install.rst | 105 ++++++++++ src/mailman/docs/introduction.rst | 111 ++++++++++ src/mailman/docs/mta.rst | 413 +++++++++++++++++++++++++++++++++++++ src/mailman/docs/postorius.rst | 83 ++++++++ src/mailman/docs/release-notes.rst | 26 +++ src/mailman/rest/docs/basic.rst | 52 +++-- src/mailman/rest/docs/rest.rst | 49 ++++- 22 files changed, 1519 insertions(+), 1480 deletions(-) delete mode 100644 src/mailman/docs/ARCHITECTURE.rst delete mode 100644 src/mailman/docs/ArchiveUIin5.rst delete mode 100644 src/mailman/docs/CONTRIBUTE.rst delete mode 100644 src/mailman/docs/DATABASE.rst delete mode 100644 src/mailman/docs/INSTALL.rst delete mode 100644 src/mailman/docs/INTRODUCTION.rst delete mode 100644 src/mailman/docs/MTA.rst delete mode 100644 src/mailman/docs/RELEASENOTES.rst delete mode 100644 src/mailman/docs/WebUIin5.rst create mode 100644 src/mailman/docs/architecture.rst create mode 100644 src/mailman/docs/contribute.rst create mode 100644 src/mailman/docs/database.rst create mode 100644 src/mailman/docs/hyperkitty.rst create mode 100644 src/mailman/docs/install.rst create mode 100644 src/mailman/docs/introduction.rst create mode 100644 src/mailman/docs/mta.rst create mode 100644 src/mailman/docs/postorius.rst create mode 100644 src/mailman/docs/release-notes.rst (limited to 'src/mailman') diff --git a/src/mailman/config/docs/config.rst b/src/mailman/config/docs/config.rst index af9b35061..16c3d40bc 100644 --- a/src/mailman/config/docs/config.rst +++ b/src/mailman/config/docs/config.rst @@ -1,6 +1,8 @@ -======================= - Mailman Configuration -======================= +.. _configuration: + +===================== + Configuring Mailman +===================== This is Mailman's default configuration, directly included from the source code. The format is standard "ini"-style. @@ -30,10 +32,10 @@ You will need to restart Mailman for any changes to take effect. schema.cfg ========== -``schema.cfg`` includes templates for several configuration options that are -instantiated inside of ``mailman.cfg``. Sections that are named with a suffix -of ``.master`` or ``.template`` (e.g. ``paths.master``) are "template" -sections which require an instantiation in ``mailman.cfg`` [#]_. +``schema.cfg`` defines the ini-file schema and contains documentation for +every section and configuration variable. Sections that are named with a +suffix of ``.master`` or ``.template`` (e.g. ``paths.master``) are "template" +sections which must be defined in the ``mailman.cfg`` file [#]_. .. literalinclude:: ../schema.cfg diff --git a/src/mailman/docs/ARCHITECTURE.rst b/src/mailman/docs/ARCHITECTURE.rst deleted file mode 100644 index 3208ec842..000000000 --- a/src/mailman/docs/ARCHITECTURE.rst +++ /dev/null @@ -1,164 +0,0 @@ -====================== -Mailman 3 architecture -====================== - -This is a brief overview of the internal architecture of the Mailman 3 core -delivery engine. You should start here if you want to understand how Mailman -works at the 1000 foot level. Another good source of architectural -information is available in the chapter written by Barry Warsaw for the -`Architecture of Open Source Applications`_. - - -User model -========== - -Every major component of the system is defined by an interface. Look through -``src/mailman/interfaces`` for an understanding of the system components. -Mailman objects which are stored in the database, are defined by *model* -classes. Objects such as *mailing lists*, *users*, *members*, and *addresses* -are primary objects within the system. - -The *mailing list* is the central object which holds all the configuration -settings for a particular mailing list. A mailing list is associated with a -*domain*, and all mailing lists are managed (i.e. created, destroyed, looked -up) via the *mailing list manager*. - -*Users* represent people, and have a *user id* and a *display name*. Users -are linked to *addresses* which represent a single email address. One user -can be linked to many addresses, but an address is only linked to one user. -Addresses can be *verified* or *not verified*. Mailman will deliver email -only to *verified* addresses. - -Users and addresses are managed by the *user manager*. - -A *member* is created by linking a *subscriber* to a mailing list. -Subscribers can be: - -* A user, which become members through their *preferred address*. -* An address, which can be linked or unlinked to a user, but must be verified. - -Members also have a *role*, representing regular members, digest members, list -owners, and list moderators. Members can even have the *non-member* role -(i.e. people not yet subscribed to the mailing list) for various moderation -purposes. - - -Process model -============= - -Messages move around inside the Mailman system by way of *queue* directories -managed by the *switchboard*. For example, when a message is first received -by Mailman, it is moved to the *in* (for "incoming") queue. During the -processing of this message, it -- or copies of it -- may be moved to other -queues such as the *out* queue (for outgoing email), the *archive* queue (for -sending to the archivers), the *digest* queue (for composing digests), etc. - -A message in a queue is represented by a single file, a ``.pck`` file. This -file contains two objects, serialized as `Python pickles`_. The first object -is the message being processed, already parsed into a `more efficient internal -representation`_. The second object is a metadata dictionary that records -additional information about the message as it is being processed. - -``.pck`` files only exist for messages moving between different system queues. -There is no ``.pck`` file for messages while they are actively being -processed. - -Each queue directory is associated with a *runner* process which wakes up -every so often. When the runner wakes up, it examines all the ``.pck`` files -in FIFO order, deserializing the message and metadata objects, processing -them. If the message needs further processing in a different queue, it will -be re-serialized back into a ``.pck`` file. If not (e.g. because processing -of the message is complete), then no ``.pck`` file is written. - -The Mailman system uses a few other runners which don't process messages in a -queue. You can think of these as fairly typical server process, and examples -include the LMTP server, and the HTTP server for processing REST commands. - -All of the runners are managed by a *master watcher* process. When you type -``mailman start`` you are actually starting the master. Based on -configuration options, the master will start the appropriate runners as -subprocesses, and it will watch for the clean exiting of these subprocesses -when ``mailman stop`` is called. - - -Rules and chains -================ - -When a message is first received for posting to a mailing list, Mailman -processes the message to determine whether the message is appropriate for the -mailing list. If so, it *accepts* the message and it gets posted. Mailman -can also *discard* the message so that no further processing occurs. Mailman -can also *reject* the message, bouncing it back to the original sender, -usually with some indication of why the message was rejected. Mailman can -also *hold* the message for moderator approval. - -*Moderation* is the phase of processing that determines which of the above -four dispositions will occur for the newly posted message. Moderation does -not generally change the message, but it may record information in the -metadata dictionary. Moderation is performed by the *in* queue runner. - -Each step in the moderation phase applies a *rule* to the message and asks -whether the rule *hits* or *misses*. Each rule is linked to an *action* which -is taken if the rule hits (i.e. matches). If the rule misses (i.e. doesn't -match), then the next rule is tried. All of the rule/action links are strung -together sequentially into a *chain*, and every mailing list has a *start -chain* where rule processing begins. - -Actually, every mailing list has *two* start chains, one for regular postings -to the mailing list, and another for posting to the owners of the mailing -list. - -To recap: when a message comes into Mailman for posting to a mailing list, the -incoming runner finds the destination mailing list, determines whether the -message is for the entire list membership, or the list owners, and retrieves -the appropriate start chain. The message is then passed to the chain, where -each link in the chain first checks to see if its rule matches, and if so, it -executes the linked action. This action is usually one of *accept*, *reject*, -*discard*, and *hold*, but other actions are possible, such as executing a -function or jumping to another chain. - -As you might imagine, you can write new rules, compose them into new chains, -and configure a mailing list to use your custom chain when processing the -message during the moderation phase. - - -Pipeline of handlers -==================== - -Once a message is accepted for posting to the mailing list, the message is -usually modified in a number of different ways. For example, some message -headers may be added or removed, some MIME parts might be scrubbed, added, or -rearranged, and various informative headers and footers may be added to the -message. - -The process of preparing the message for the list membership (as well as the -digests, archivers, and NNTP) falls to the *pipeline of handlers* managed by -the *pipeline* queue. - -The pipeline of handlers is similar to the processing chain, except here, a -handler can make any modifications to the message it wants, and there is no -rule decision or action. The message and metadata simply flow through a -sequence of handlers arranged in a named pipeline. Some of the handlers -modify the message in ways described above, and others copy the message to the -outgoing, NNTP, archiver, or digester queues. - -As with chains, each mailing list has two pipelines, one for posting to the -list membership, and the other for posting to the list's owners. - -Of course, you can define new handlers, compose them into new pipelines, and -change a mailing list's pipelines. - - -Other bits and pieces -===================== - -There are lots of other pieces to the Mailman puzzle, such as the REST API, -the set of core functionality (logging, initialization, event handling, etc.), -mailing list *styles*, the API for integrating external archivers and mail -servers. The database layer is an critical piece, and Mailman has an -extensive set of command line commands, and email commands. - - -.. _`Architecture of Open Source Applications`: http://www.aosabook.org/en/mailman.html -.. _`Python pickles`: http://docs.python.org/2/library/pickle.html -.. _`more efficient internal representation`: https://docs.python.org/3/library/email.html diff --git a/src/mailman/docs/ArchiveUIin5.rst b/src/mailman/docs/ArchiveUIin5.rst deleted file mode 100644 index d4b24c92c..000000000 --- a/src/mailman/docs/ArchiveUIin5.rst +++ /dev/null @@ -1,108 +0,0 @@ -===================================== -Set up the archive ui in five minutes -===================================== - -.. note:: - This document is way out of date. If you have access to the Web, - the most recent explanation of setting up a full Mailman 3 system, - including Postorius and HyperKitty, is the `Development Setup Guide`_ at - `FedoraHosted`_. If you must work offline, this document may be of some - use, but be careful. - -The `hyperkitty`_ application aims at providing an interface to visualize and -explore Mailman archives. - -This is a `Django`_ project. - -Requirements -============ - -- A mail archive in `maildir format`_ (no, you don't need a running Mailman - 3!) Eventually hyperkitty will support `mbox format`_ for backward - compatibility with *Pipermail*, and *zipped maildirs* seem like a good idea - to save space. **Beware:** Although you'd think that we would be able to - manipulate the venerable *mbox* format safely and efficiently, that doesn't - seem to be the case. *Maildir* archives are **strongly** preferred, because - they are more robust to program bugs (whether in Mailman, hyperkitty, or in - the originating MUA!) -- Django is the web framework that supports the UI. -- `bunch`_ DOES WHAT? -- The `notmuch mail indexer`_ is used to generate indexes (and requires - `Xapian`_). -- hyperkitty itself, which is a UI, and not responsible for maintaining the - message archive itself. (Since the archive is in `maildir format`_, any - modern MTA or MDA can build one for you.) - - -Get it running (under virtualenv): ----------------------------------- - -It is generally a good idea to use `virtualenv`_ to create a stable -environment for your Python applications. - -- Create the virtualenv:: - - % virtualenv mailman3 - -- Activate the virtualenv:: - - % cd mailman3 - % source bin/activate - -You don't *have* to use virtualenv, though, and if you don't want to, just -omit the preceding steps. Continue with these steps. - -- Install Django and dependencies:: - - % easy_install django - % easy_install bunch - -- Install notmuch -- these are bindings that come with the notmuch C library. - The easiest way is probably to install them for your OS vendor and then - symlink them into the virtualenv similar to this:: - - % yum install -y python-notmuch - -- Note: on a multiarch system like Fedora, the directories may be lib64 rather - than lib on 64 bit systems. Next:: - - % cd lib/python2.7/site-packages - % ln -s /usr/lib/python2.7/site-packages/notmuch . - -- Note: this is the version of notmuch I tested with; others may work:: - - % ln -s /usr/lib/python2.7/site-packages/notmuch-0.11-py2.7.egg-info . - -- Install the hyperkitty sources:: - - % git clone https://github.com/hyperkitty/kittystore.git - % git clone https://github.com/hyperkitty/hyperkitty.git - % git clone https://github.com/hyperkitty/hyperkitty_standalone.git - - -Running hyperkitty ------------------- - -- Start it:: - - % cd hyperkitty - -- Put the static content where it should be:: - - % python manage.py collectstatic - -- Run the Django server:: - - % python manage.py runserver - - -.. _`Development Setup Guide`: https://fedorahosted.org/hyperkitty/wiki/DevelopmentSetupGuide -.. _`FedoraHosted`: https://fedorahosted.org/ -.. _`hyperkitty`: https://fedorahosted.org/hyperkitty/ -.. _`Django`: http://djangoproject.org/ -.. _`notmuch mail indexer`: http://notmuchmail.org -.. _`bunch`: http://pypi.python.org/pypi/bunch -.. _`Xapian`: http://xapian.org/ -.. _`maildir format`: http://en.wikipedia.org/wiki/Maildir -.. _`mbox format`: http://en.wikipedia.org/wiki/Mbox -.. _`virtualenv`: http://pypi.python.org/pypi/virtualenv diff --git a/src/mailman/docs/CONTRIBUTE.rst b/src/mailman/docs/CONTRIBUTE.rst deleted file mode 100644 index 15c542e3e..000000000 --- a/src/mailman/docs/CONTRIBUTE.rst +++ /dev/null @@ -1,245 +0,0 @@ -========================= -Contributing to Mailman 3 -========================= - -Copyright (C) 2008-2017 by the Free Software Foundation, Inc. - - -How to contribute -================= - -We accept `merge requests`_ and `bug reports`_ on GitLab. We prefer if every -merge request is linked to a bug report, because we can more easily manage the -priority of bug reports. For more substantial contributions, we may ask you -to sign a `copyright assignment`_ to the Free Software Foundation, the owner -of the GNU Mailman copyright. If you'd like to jump start your copyright -assignment, please contact the GNU Mailman `steering committee`_. - - -Contact Us -========== - -Contributions of code, problem reports, and feature requests are welcome. -Please submit bug reports on the Mailman bug tracker at -https://gitlab.com/mailman/mailman/issues (you need to have a login on GitLab -to do so). You can also send email to the mailman-developers@python.org -mailing list, or ask on IRC channel ``#mailman`` on Freenode. - - -Get the sources -=============== - -The Mailman 3 source code is version controlled using Git. You can get a -local copy by running this command:: - - $ git clone https://gitlab.com/mailman/mailman.git - -or if you have a GitLab account and prefer ssh:: - - $ git clone git@gitlab.com:mailman/mailman.git - - -Testing Mailman 3 -================= - -To run the Mailman test suite, just use the `tox`_ command:: - - $ tox - -`tox` creates a virtual environment (virtualenv_) for you, installs all the -dependencies into that virtualenv, and runs the test suite from that -virtualenv. By default it does not use the `--system-site-packages` so it -downloads everything from the `Python Cheeseshop`_. - -A bare ``tox`` command will try to run several test suites, which might take a -long time, and/or require versions of Python or other components you might not -have installed. You can run ``tox -l`` to list the test suite *environments* -available. Very often, when you want to run the full test suite in the -quickest manner with components that should be available everywhere, run one -of these command, depending on which version of Python 3 you have:: - - $ tox -e py36-nocov - $ tox -e py35-nocov - $ tox -e py34-nocov - -You can run individual tests in any given environment by providing additional -positional arguments. For example, to run only the tests that match a -specific pattern:: - - $ tox -e py35-nocov -- -P user - -You can see all the other arguments supported by the test suite by running:: - - $ tox -e py35-nocov -- --help - -You also have access to the virtual environments created by tox, and you can -use this run the virtual environment's Python executable, or run the -``mailman`` command locally, e.g.:: - - $ .tox/py35-nocov/bin/python - $ .tox/py35-nocov/bin/mailman --help - -If you want to set up the virtual environment without running the full test -suite, you can do this:: - - $ tox -e py35-nocov --notest -r - - -Testing with PostgreSQL and MySQL -================================= - -By default, the test suite runs with the built-in SQLite database engine. If -you want to run the full test suite against the PostgreSQL or MySQL databases, -set the database up as described in :doc:`DATABASE`. - -For PostgreSQL, then create a `postgres.cfg` file any where you want. This -`postgres.cfg` file will contain the ``[database]`` section for PostgreSQL, -e.g.:: - - [database] - class: mailman.database.postgresql.PostgreSQLDatabase - url: postgres://myuser:mypassword@mypghost/mailman - -Then run the test suite like so:: - - $ MAILMAN_EXTRA_TESTING_CFG=/path/to/postgres.cfg tox -e py35-pg - -You can combine these ways to invoke Mailman, so if you want to run an -individual test against PostgreSQL, you could do:: - - $ MAILMAN_EXTRA_TESTING_CFG=/path/to/postgres.cfg tox -e py35-pg -- -P user - -Note that the path specified in `MAILMAN_EXTRA_TESTING_CFG` must be an -absolute path or some tests will fail. - - -Building for development -======================== - -To build Mailman for development purposes, you can create a virtual -environment outside of tox. You need to have the `virtualenv`_ program -installed, or you can use Python 3's built-in `pyvenv`_ command. - -First, create a virtual environment (venv). The directory you install the -venv into is up to you, but for purposes of this document, we'll install it -into ``/tmp/mm3``:: - - $ python3 -m venv /tmp/mm3 - -Now, activate the virtual environment and set it up for development:: - - % source /tmp/mm3/bin/activate - % python setup.py develop - -Sit back and have some Kombucha while you wait for everything to download and -install. - - -Building the documentation -========================== - -To build the documentation, you need some additional dependencies. The only -one you probably need from your OS vendor is `graphiz`. E.g. On Debian or -Ubuntu, you can do:: - - $ sudo apt install graphiz - -All other dependencies should be automatically installed as needed. Build the -documentation by running:: - - $ tox -e docs - -Then visit:: - - build/sphinx/html/index.html - - -Mailman Shell -============= - -This documentation has examples which use the Mailman shell to interact with -Mailman. To start the shell type ``mailman shell`` in your terminal. - -There are some testings functions which need to be imported first before you -use them. They can be imported from the modules available in -``mailman.testing``. For example, to use ``dump_list`` you first need to -import it from the ``mailman.testing.documentation`` module. - -.. Of course, *this* doctest doesn't have these preloaded... - >>> from zope.component import getUtility - >>> from mailman.interfaces.listmanager import IListManager - -The shell automatically initializes the Mailman system, loads all the -available interfaces, and configures the `Zope Component Architecture`_ (ZCA) -which is used to access all the software components in Mailman. So for -example, if you wanted to get access to the list manager component, you could -do:: - - $ mailman shell - Welcome to the GNU Mailman shell - - >>> list_manager = getUtility(IListManager) - - -Related projects -================ - -What you are looking at right now is the Mailman Core. It's "just" the -message delivery engine, but it's designed to work with a web user interface -for list members and administrators, and an archiver. The GNU Mailman project -also develops a web ui and archiver, but these are available in separate git -repositories. - - -Mailman Web UI --------------- - -The Mailman 3 web UI, called *Postorius*, interfaces to core Mailman engine -via the REST client API. This architecture makes it possible for users with -other needs to adapt the web UI, or even replace it entirely, with a -reasonable amount of effort. However, as a core feature of Mailman, the web -UI emphasizes usability over modularity at first, so most users should use the -web UI described here. Postorius_ is a Django_ application. - - -The Archiver -~~~~~~~~~~~~ - -In Mailman 3, the archivers are decoupled from the Core. Instead, Mailman 3 -provides a simple, standard interface for third-party archiving tools and -services. For this reason, Mailman 3 defines a formal interface to insert -messages into any of a number of configured archivers, using whatever protocol -is appropriate for that archiver. Summary, search, and retrieval of archived -posts are handled by a separate application. - -A new archive UI called `HyperKitty`_, based on the `notmuch mail indexer`_ -was prototyped at the `Pycon 2012 sprint`_ by Toshio Kuratomi. The HyperKitty -archiver is very loosely coupled to Mailman 3 core. In fact, any email -application that speaks LMTP or SMTP will be able to use HyperKitty. -HyperKitty is also a Django application. - - -REST API Python bindings -~~~~~~~~~~~~~~~~~~~~~~~~ - -Mailman 3 provides a REST API for administrative purposes, and this is used by -both HyperKitty and Postorius. You can of course use any HTTP client to speak -to it, but we provide official Python bindings (for both Python 2 and 3) in a -package we call `mailman.client`_. - - -.. _`merge requests`: https://gitlab.com/mailman/mailman/merge_requests -.. _`bug reports`: https://gitlab.com/mailman/mailman/issues -.. _`copyright assignment`: https://www.fsf.org/licensing/assigning.html/?searchterm=copyright%20assignment -.. _`steering committee`: mailto:mailman-cabal@python.org -.. _tox: https://testrun.org/tox/latest/ -.. _`Zope Component Architecture`: https://pypi.python.org/pypi/zope.component -.. _`Postorius`: https://gitlab.com/mailman/postorius -.. _`Django`: http://djangoproject.org/ -.. _`HyperKitty`: https://gitlab.com/mailman/hyperkitty -.. _`notmuch mail indexer`: http://notmuchmail.org -.. _`mailman.client`: https://gitlab.com/mailman/mailmanclient -.. _`Pycon 2012 sprint`: https://us.pycon.org/2012/community/sprints/projects/ -.. _`Python Cheeseshop`: http://pypi.python.org/pypi -.. _`virtualenv`: http://www.virtualenv.org/en/latest/ -.. _`pyvenv`: https://docs.python.org/3/library/venv.html diff --git a/src/mailman/docs/DATABASE.rst b/src/mailman/docs/DATABASE.rst deleted file mode 100644 index c85731fb3..000000000 --- a/src/mailman/docs/DATABASE.rst +++ /dev/null @@ -1,134 +0,0 @@ -======================== -Setting up your database -======================== - -Mailman uses the SQLAlchemy_ ORM to provide persistence of data in a -relational database. By default, Mailman uses Python's built-in SQLite3_ -database, however, SQLAlchemy is compatible with PostgreSQL_ and MySQL, among -possibly others. - -Currently, Mailman is known to work with the SQLite3, PostgreSQL, and MySQL -databases. (Volunteers to port it to other databases are welcome!). If you -want to use SQLite3, you generally don't need to change anything, but if you -want Mailman to use PostgreSQL or MySQL, you'll need to set those up first, -and then change a configuration variable in your ``/etc/mailman.cfg`` file. - -Two configuration variables control which database Mailman uses. The first -names the class implementing the database interface. The second names the URL -for connecting to the database. Both variables live in the ``[database]`` -section of the configuration file. - - -SQLite3 -======= - -As mentioned, if you want to use SQLite3 in the default configuration, you -generally don't need to change anything. However, if you want to change where -the SQLite3 database is stored, you can change the ``url`` variable in the -``[database]`` section. By default, the database is stored in the *data -directory* in the ``mailman.db`` file. Here's how you'd force Mailman to -store its database in ``/var/lib/mailman/sqlite.db`` file:: - - [database] - url: sqlite:////var/lib/mailman/sqlite.db - - -PostgreSQL -========== - -First, you need to configure PostgreSQL itself. This `Ubuntu article`_ may -help. Let's say you create the `mailman` database in PostgreSQL via:: - - $ sudo -u postgres createdb -O $USER mailman - -You would also need the python driver `psycopg2` for Postgresql.:: - - $ pip install psycopg2 - -You would then need to set both the `class` and `url` variables in -`mailman.cfg` like so:: - - [database] - class: mailman.database.postgresql.PostgreSQLDatabase - url: postgres://myuser:mypassword@mypghost/mailman - -That should be it. - -If you have any problems, you may need to delete the database and re-create -it:: - - $ sudo -u postgres dropdb mailman - $ sudo -u postgres createdb -O myuser mailman - -My thanks to Stephen A. Goss for his contribution of PostgreSQL support. - - -MySQL -===== - -First, you need to configure MySQL itself. Lets say you create the `mailman` -database in MySQL via:: - - mysql> CREATE DATABASE mailman; - -You would also need the Python driver `pymysql` for MySQL.:: - - $ pip install pymysql - -You would then need to set both the `class` and `url` variables in -`mailman.cfg` like so:: - - [database] - class: mailman.database.mysql.MySQLDatabase - url: mysql+pymysql://myuser:mypassword@mymysqlhost/mailman?charset=utf8&use_unicode=1 - -The last part of the url specifies the charset that client expects from the -server and to use Unicode via the flag `use_unicode`. You can find more about -these options on the `SQLAlchemy's MySQL page`_. - -If you have any problems, you may need to delete the database and re-create -it:: - - mysql> DROP DATABASE mailman; - mysql> CREATE DATABASE mailman; - - -Database Migrations -=================== - -Mailman uses `Alembic`_ to manage database migrations. Let's say you change -something in the models, what steps are needed to reflect that change in the -database schema? You need to create and enter a virtual environment, install -Mailman into that, and then run the ``alembic`` command. For example:: - - $ python3 -m venv /tmp/mm3 - $ source /tmp/mm3/bin/activate - $ python setup.py develop - $ mailman info - $ alembic -c src/mailman/config/alembic.cfg revision --autogenerate -m - "" - $ deactivate - -This would create a new migration which would automatically be migrated to the -database on the next run of Mailman. - -People upgrading Mailman from previous versions need not do anything manually, -as soon as a new migration is added in the sources, it will be automatically -reflected in the schema on first-run post-update. - -**Note:** When auto-generating migrations using Alembic, be sure to check -the created migration before adding it to the version control. You will have -to manually change some of the special data types defined in -``mailman.database.types``. For example, ``mailman.database.types.Enum()`` -needs to be changed to ``sa.Integer()``, as the ``Enum`` type stores just the -integer in the database. A more complex migration would be needed for -``UUID`` depending upon the database layer to be used. - - -.. _SQLAlchemy: http://www.sqlalchemy.org/ -.. _SQLite3: http://docs.python.org/library/sqlite3.html -.. _PostgreSQL: http://www.postgresql.org/ -.. _MySQL: http://dev.mysql.com/ -.. _`Ubuntu article`: https://help.ubuntu.com/community/PostgreSQL -.. _`Alembic`: https://alembic.readthedocs.org/en/latest/ -.. _`SQLAlchemy's MySQL page`: http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#unicode diff --git a/src/mailman/docs/INSTALL.rst b/src/mailman/docs/INSTALL.rst deleted file mode 100644 index 8a4722190..000000000 --- a/src/mailman/docs/INSTALL.rst +++ /dev/null @@ -1,120 +0,0 @@ -==================== -Installing Mailman 3 -==================== - -Copyright (C) 2008-2017 by the Free Software Foundation, Inc. - - -Requirements -============ - -For the Core, Python 3.4 or newer is required. It can either be the default -'python3' on your ``$PATH`` or it can be accessible via the ``python3.4``, -``python3.5``, or ``python3.6`` binary. If your operating system does not -include Python 3, see http://www.python.org for information about downloading -installers (where available) and installing it from source (when necessary or -preferred). Python 2 is not supported by the Core. - -You may need some additional dependencies, which are either available from -your OS vendor, or can be downloaded automatically from the `Python -Cheeseshop`_. - - -Documentation -============= - -The documentation for Mailman 3 is distributed throughout the sources. The -core documentation (such as this file) is found in the ``src/mailman/docs`` -directory, but much of the documentation is in module-specific places. Online -versions of the `Mailman 3 Core documentation`_ is available online. - - -Get the sources -=============== - -The Mailman 3 source code is version controlled using Git. You can get a -local copy by running this command:: - - $ git clone https://gitlab.com/mailman/mailman.git - -or if you have a GitLab account and prefer ssh:: - - $ git clone git@gitlab.com:mailman/mailman.git - - -Running Mailman 3 -================= - -You will need to set up a configuration file to override the defaults and set -things up for your environment. Mailman is configured using an "ini"-style -configuration system. - -``src/mailman/config/schema.cfg`` defines the ini-file schema and contains -documentation for every section and configuration variable. Sections that end -in ``.template`` or ``.master`` are templates that must be overridden in -actual configuration files. There is a default configuration file that -defines these basic overrides in ``src/mailman/config/mailman.cfg``. Your own -configuration file will override those. - -By default, all runtime files are put under a ``var`` directory in the current -working directory. - -Mailman searches for its configuration file using the following search path. -The first existing file found wins. - -* ``-C config`` command line option -* ``$MAILMAN_CONFIG_FILE`` environment variable -* ``./mailman.cfg`` -* ``~/.mailman.cfg`` -* ``/etc/mailman.cfg`` -* ``argv[0]/../../etc/mailman.cfg`` - -Run the ``mailman info`` command to see which configuration file Mailman will -use, and where it will put its database file. The first time you run this, -Mailman will also create any necessary run-time directories and log files. - -Try ``mailman --help`` for more details. You can use the commands -``mailman start`` to start the runner subprocess daemons, and of course -``mailman stop`` to stop them. - -Postorius_, a web UI for administration and subscriber settings, is being -developed as a separate, Django-based project. For now, the most flexible -means of configuration is via the command line and REST API. - -Note that you can also "run" Mailman from one of the virtual environments -created by tox, e.g.:: - - $ .tox/py35-nocov/bin/mailman info - - -Mailman Shell -============= - -This documentation has examples which use the Mailman shell to interact with -Mailman. To start the shell type ``mailman shell`` in your terminal. - -There are some testings functions which need to be imported first before you -use them. They can be imported from the modules available in -``mailman.testing``. For example, to use ``dump_list`` you first need to -import it from the ``mailman.testing.documentation`` module. - -.. Of course, *this* doctest doesn't have these preloaded... - >>> from zope.component import getUtility - >>> from mailman.interfaces.listmanager import IListManager - -The shell automatically initializes the Mailman system, loads all the -available interfaces, and configures the `Zope Component Architecture`_ (ZCA) -which is used to access all the software components in Mailman. So for -example, if you wanted to get access to the list manager component, you could -do:: - - $ mailman shell - Welcome to the GNU Mailman shell - - >>> list_manager = getUtility(IListManager) - - -.. _`Postorius`: https://gitlab.com/mailman/postorius -.. _`Python Cheeseshop`: http://pypi.python.org/pypi -.. _`Mailman 3 Core documentation`: https://mailman.readthedocs.io -.. _`Zope Component Architecture`: https://pypi.python.org/pypi/zope.component diff --git a/src/mailman/docs/INTRODUCTION.rst b/src/mailman/docs/INTRODUCTION.rst deleted file mode 100644 index e62dae183..000000000 --- a/src/mailman/docs/INTRODUCTION.rst +++ /dev/null @@ -1,126 +0,0 @@ -.. _start-here: - -================================================ -Mailman - The GNU Mailing List Management System -================================================ - -This is `GNU Mailman`_, a mailing list management system distributed under the -terms of the `GNU General Public License`_ (GPL) version 3 or later. - -Mailman is written in Python_, a free object-oriented programming language. -Python is available for all platforms that Mailman is supported on, which -includes GNU/Linux and most other Unix-like operating systems (e.g. Solaris, -\*BSD, MacOSX, etc.). Mailman is not supported on Windows, although web and -mail clients on any platform should be able to interact with Mailman just -fine. - -GNU Mailman 3 consists of a `suite of programs`_ that work together: - -* Mailman Core; the core delivery engine. This is where you are right now. -* Postorius; the web user interface for list members and administrators. -* HyperKitty; the web-based archiver -* Mailman client; the official Python bindings for talking to the Core's REST - administrative API. -* Mailman bundler; a convenient way to deploy Mailman 3. - -Only the Core is required. You could easily write your own web user interface -or archiver that speaks to the Core over its REST API. The REST API is a pure -HTTP-based API so you don't *have* to use Python, or even our official -bindings. And you can deploy whatever components you want using whatever -mechanisms you want. - -But we really like Postorius and HyperKitty and hope you will too! - - -Copyright -========= - -Copyright 1998-2017 by the Free Software Foundation, Inc. - -This file is part of GNU Mailman. - -GNU Mailman is free software: you can redistribute it and/or modify it under -the terms of the GNU General Public License as published by the Free Software -Foundation, either version 3 of the License, or (at your option) any later -version. - -GNU Mailman is distributed in the hope that it will be useful, but WITHOUT ANY -WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR -A PARTICULAR PURPOSE. See the GNU General Public License for more details. - -You should have received a copy of the GNU General Public License along with -GNU Mailman. If not, see . - - -Spelling -======== - -The name of this software is spelled `Mailman` with a leading capital `M` -but with a lower case second `m`. Any other spelling is incorrect. Its full -name is `GNU Mailman` but is often referred colloquially as `Mailman`. - - -History -======= - -Mailman was originally developed by John Viega in the mid-1990s. Subsequent -development (through version 1.0b3) was by Ken Manheimer. Further work -towards the 1.0 final release was a group effort, with the core contributors -being: Barry Warsaw, Ken Manheimer, Scott Cotton, Harald Meland, and John -Viega. Version 1.0 and beyond have been primarily maintained by Barry Warsaw -with contributions from many; see the `ACKNOWLEDGMENTS`_ file for details. -Jeremy Hylton helped considerably with the Pipermail code in Mailman 2.0. -Mailman 2.1 is primarily maintained by Mark Sapiro, with previous help by -Tokio Kikuchi. Barry Warsaw is the lead developer on Mailman 3. Aurélien -Bompard and Florian Fuchs lead development of Postorius and HyperKitty. -Abhilash Raj is a valued contributor to all bits and maintains the CI -infrastructure. - - -Project details -=============== - -The Mailman home page is: - -* http://www.list.org - -The community driven wiki (including the FAQ_) is at: - -* http://wiki.list.org - -Other help resources, such as on-line documentation, links to the mailing -lists and archives, etc., are available at: - -* http://www.list.org/help.html - -Mailman 3 is maintained on GitLab: - -* https://gitlab.com/groups/mailman - -You can report issues in the Core at: - -* https://gitlab.com/mailman/mailman/issues - -(GNU Mailman 2.1 is still maintained on Launchpad_.) - -There are two mailing lists you can use to contact the Mailman 3 developers. -The general development mailing list: - -* https://mail.python.org/mailman/listinfo/mailman-developers - -and the Mailman 3 users list: - -* https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ - -For now, please leave the older mailman-users mailing list for Mailman 2. - - -.. _`GNU Mailman`: http://www.list.org -.. _`GNU General Public License`: http://www.gnu.org/licenses/gpl.txt -.. _Python: http://www.python.org -.. _FAQ: http://wiki.list.org/display/DOC/Frequently+Asked+Questions -.. _`Python 3.4`: https://www.python.org/downloads/release/python-342/ -.. _`ACKNOWLEDGMENTS`: ACKNOWLEDGMENTS.html -.. _`Django`: https://www.djangoproject.com/ -.. _`suite of programs`: http://docs.mailman3.org/en/latest/ -.. _Launchpad: https://launchpad.net/mailman diff --git a/src/mailman/docs/MTA.rst b/src/mailman/docs/MTA.rst deleted file mode 100644 index ac38c3026..000000000 --- a/src/mailman/docs/MTA.rst +++ /dev/null @@ -1,412 +0,0 @@ -=========================== -Hooking up your mail server -=========================== - -Mailman needs to communicate with your *MTA* (*mail transport agent* -or *mail server*, the software which handles sending mail across the -Internet), both to accept incoming mail and to deliver outgoing mail. -Mailman itself never delivers messages to the end user. It sends them -to its immediate upstream MTA, which delivers them. In the same way, -Mailman never receives mail directly. Mail from outside always comes -via the MTA. - -Mailman accepts incoming messages from the MTA using the `Local Mail -Transfer Protocol`_ (LMTP_) interface. Mailman can use other incoming -transports, but LMTP is much more efficient than spawning a process -just to do the delivery. Most open source MTAs support LMTP for local -delivery. If yours doesn't, and you need to use a different -interface, please ask on the `mailing list or on IRC`_. - -Mailman passes all outgoing messages to the MTA using the `Simple Mail -Transfer Protocol`_ (SMTP_). - -Cooperation between Mailman and the MTA requires some configuration of -both. MTA configuration differs for each of the available MTAs, and -there is a section for each one. Instructions for Postfix and Exim (v4) -are given below. We would really appreciate a contribution of a -configuration for Sendmail, and welcome information about other popular -open source mail servers. - -Configuring Mailman to communicate with the MTA is straightforward, and -basically the same for all MTAs. Here are the default settings; if you need -to change them, edit your ``mailman.cfg`` file:: - - [mta] - incoming: mailman.mta.postfix.LMTP - outgoing: mailman.mta.deliver.deliver - lmtp_host: 127.0.0.1 - lmtp_port: 8024 - smtp_host: localhost - smtp_port: 25 - configuration: python:mailman.config.postfix - -This configuration is for a system where Mailman and the MTA are on -the same host. - -Note that the modules that configure the communication protocol (especially -``incoming``) are full-fledged Python modules, and may use these configuration -parameters to automatically configure the MTA to recognize the list addresses -and other attributes of the communication channel. This is why some -constraints on the format of attributes arise (e.g., ``lmtp_host``), even -though Mailman itself has no problem with them. - -The ``incoming`` and ``outgoing`` parameters identify the Python objects used -to communicate with the MTA. The ``python:`` scheme indicates that the paths -should be a dotted Python module specification. The ``deliver`` module used -in ``outgoing`` should be satisfactory for most MTAs. The ``postfix`` module -in ``incoming`` is specific to the Postfix MTA. See the section for your MTA -below for details on these parameters. - -``lmtp_host`` and ``lmtp_port`` are parameters which are used by -Mailman, but also will be passed to the MTA to identify the Mailman -host. The "same host" case is special; some MTAs (including Postfix) -do not recognize "localhost", and need the numerical IP address. If -they are on different hosts, ``lmtp_host`` should be set to the domain -name or IP address of the Mailman host. ``lmtp_port`` is fairly -arbitrary (there is no standard port for LMTP). Use any port -convenient for your site. "8024" is as good as any, unless another -service is using it. - -``smtp_host`` and ``smtp_port`` are parameters used to identify the -MTA to Mailman. If the MTA and Mailman are on separate hosts, -``smtp_host`` should be set to the domain name or IP address of the -MTA host. ``smtp_port`` will almost always be 25, which is the -standard port for SMTP. (Some special site configurations set it to a -different port. If you need this, you probably already know that, -know why, and what to do, too!) - -Mailman also provides many other configuration variables that you can -use to tweak performance for your operating environment. See the -``src/mailman/config/schema.cfg`` file for details. - - -Postfix -======= - -Postfix_ is an open source mail server by Wietse Venema. - - -Mailman settings ----------------- - -You need to tell Mailman that you are using the Postfix mail server. In your -``mailman.cfg`` file, add the following section:: - - [mta] - incoming: mailman.mta.postfix.LMTP - outgoing: mailman.mta.deliver.deliver - lmtp_host: mail.example.com - lmtp_port: 8024 - smtp_host: mail.example.com - smtp_port: 25 - -Some of these settings are already the default, so take a look at Mailman's -``src/mailman/config/schema.cfg`` file for details. You'll need to change the -``lmtp_host`` and ``smtp_host`` to the appropriate host names of course. -Generally, Postfix will listen for incoming SMTP connections on port 25. -Postfix will deliver via LMTP over port 24 by default, however if you are not -running Mailman as root, you'll need to change this to a higher port number, -as shown above. - - -Basic Postfix connections -------------------------- - -There are several ways to hook Postfix up to Mailman, so here are the simplest -instructions. The following settings should be added to Postfix's ``main.cf`` -file. - -Mailman supports a technique called `Variable Envelope Return Path`_ (VERP) to -disambiguate and accurately record bounces. By default Mailman's VERP -delimiter is the `+` sign, so adding this setting allows Postfix to properly -handle Mailman's VERP'd messages:: - - # Support the default VERP delimiter. - recipient_delimiter = + - -In older versions of Postfix, unknown local recipients generated a temporary -failure. It's much better (and the default in newer Postfix releases) to -treat them as permanent failures. You can add this to your ``main.cf`` file -if needed (use the `postconf`_ command to check the defaults):: - - unknown_local_recipient_reject_code = 550 - -While generally not necessary if you set ``recipient_delimiter`` as described -above, it's better for Postfix to not treat ``owner-`` and ``-request`` -addresses specially:: - - owner_request_special = no - - -Transport maps --------------- - -By default, Mailman works well with Postfix transport maps as a way to deliver -incoming messages to Mailman's LMTP server. Mailman will automatically write -the correct transport map when its ``mailman aliases`` command is run, or -whenever a mailing list is created or removed via other commands. To connect -Postfix to Mailman's LMTP server, add the following to Postfix's ``main.cf`` -file:: - - transport_maps = - hash:/path-to-mailman/var/data/postfix_lmtp - local_recipient_maps = - hash:/path-to-mailman/var/data/postfix_lmtp - relay_domains = - hash:/path-to-mailman/var/data/postfix_domains - -where ``path-to-mailman`` is replaced with the actual path that you're running -Mailman from. Setting ``local_recipient_maps`` as well as ``transport_maps`` -allows Postfix to properly reject all messages destined for non-existent local -users. Setting `relay_domains`_ means Postfix will start to accept mail for -newly added domains even if they are not part of `mydestination`_. - -Note that if you are not using virtual domains, then `relay_domains`_ isn't -strictly needed (but it is harmless). All you need to do in this scenario is -to make sure that Postfix accepts mail for your one domain, normally by -including it in ``mydestination``. - - -Postfix documentation ---------------------- - -For more information regarding how to configure Postfix, please see -the Postfix documentation at: - -.. _`The official Postfix documentation`: - http://www.postfix.org/documentation.html -.. _`The reference page for all Postfix configuration parameters`: - http://www.postfix.org/postconf.5.html -.. _`relay_domains`: http://www.postfix.org/postconf.5.html#relay_domains -.. _`mydestination`: http://www.postfix.org/postconf.5.html#mydestination - - -Exim -==== - -`Exim 4`_ is an MTA maintained by the `University of Cambridge`_ and -distributed by most open source OS distributions. - -Mailman settings ----------------- - -Add or edit a stanza like this in mailman.cfg:: - - [mta] - # For all Exim4 installations. - incoming: mailman.mta.exim4.LMTP - outgoing: mailman.mta.deliver.deliver - # Typical single host with MTA and Mailman configuration. - # Adjust to your system's configuration. - # Exim happily works with the "localhost" alias rather than IP address. - lmtp_host: localhost - smtp_host: localhost - # Mailman should not be run as root. - # Use any convenient port > 1024. 8024 is a convention, but can be - # changed if there is a conflict with other software using that port. - lmtp_port: 8024 - # smtp_port rarely needs to be set. - smtp_port: 25 - # Exim4-specific configuration parameter defaults. Currently empty. - configuration: python:mailman.config.exim4 - -For further information about these settings, see -``mailman/config/schema.cfg``. - -Exim4 configuration -------------------- - -The configuration presented below is mostly boilerplate that allows Exim to -automatically discover your list addresses, and route both posts and -administrative messages to the right Mailman services. For this reason, the -`mailman.mta.exim4` module ends up with all methods being no-ops. - -This configuration is field-tested in a Debian "conf.d"-style Exim -installation, with multiple configuration files that are assembled by a -Debian-specific script. If your Exim v4 installation is structured -differently, ignore the comments indicating location in the Debian -installation. -:: - - # /etc/exim4/conf.d/main/25_mm3_macros - # The colon-separated list of domains served by Mailman. - domainlist mm_domains=list.example.net - - MM3_LMTP_PORT=8024 - - # Assuming a typical source installation in /usr/local, with - # links to the Mailman bin directory and so on from MM3_HOME. - MM3_HOME=/usr/local/var/mailman - MM3_UID=list - MM3_GID=list - - ################################################################ - # The configuration below is boilerplate: - # you should not need to change it. - - # The path to the list receipt (used as the required file when - # matching list addresses) - MM3_LISTCHK=MM3_HOME/lists/${local_part}@${domain} - - # /etc/exim4/conf.d/router/455_mm3_router - mailman3_router: - driver = accept - domains = +mm_domains - require_files = MM3_LISTCHK - local_part_suffix_optional - local_part_suffix = -admin : \ - -bounces : -bounces+* : \ - -confirm : -confirm+* : \ - -join : -leave : \ - -owner : -request : \ - -subscribe : -unsubscribe - transport = mailman3_transport - - # /etc/exim4/conf.d/transport/55_mm3_transport - mailman3_transport: - driver = smtp - protocol = lmtp - allow_localhost - hosts = localhost - port = MM3_LMTP_PORT - -Troubleshooting ---------------- - -The most likely causes of failure to deliver to Mailman are typos in the -configuration, and errors in the ``MM3_HOME`` macro or the ``mm_domains`` -list. Mismatches in the LMTP port could be a cause. Finally, Exim's router -configuration is order-sensitive. Especially if you are being tricky and -supporting Mailman 2 and Mailman 3 at the same time, you could have one shadow -the other. - -Exim 4 documentation --------------------- - -There is `copious documentation for Exim`_. The parts most relevant to -configuring communication with Mailman 3 are the chapters on the `accept -router`_ and the `LMTP transport`_. Unless you are already familiar -with Exim configuration, you probably want to start with the chapter on -`how Exim receives and delivers mail`. - -.. _`Exim 4`: http://www.exim.org/ -.. _`University of Cambridge`: http://www.cam.ac.uk/ -.. _`copious documentation for Exim`: http://www.exim.org/docs.html -.. _`accept router`: http://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_accept_router.html -.. _`LMTP transport`: http://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_lmtp_transport.html -.. _`how Exim receives and delivers mail`: http://www.exim.org/exim-html-current/doc/html/spec_html/ch-how_exim_receives_and_delivers_mail.html - - -qmail -===== - -qmail_ is a MTA written by djb_ and, though old and not updated, still -bulletproof and occassionally in use. - -Mailman settings ----------------- - -Mostly defaults in mailman.cfg:: - - [mta] - # NullMTA is just implementing the interface and thus satisfying Mailman - # without doing anything fancy - incoming: mailman.mta.null.NullMTA - # Mailman should not be run as root. - # Use any convenient port > 1024. 8024 is a convention, but can be - # changed if there is a conflict with other software using that port. - lmtp_port: 8024 - -This will listen on localhost:8024 with LMTP and deliver outgoing -messages to localhost:25. See ``mailman/config/schema.cfg`` for more -information on these settings. - -qmail configuration -------------------- - -It is assumed that qmail is configured to use the ``.qmail*`` files in a user’s -home directory, however the instructions should easily be adaptable to other -qmail configurations. However, it is required that Mailman has a (sub)domain -respectively a namespace on its own. A helper script called ``qmail-lmtp`` is -needed and can be found in the ``contrib/`` directory of the Mailman source -tree and assumed to be on ``$PATH`` here. - -As qmail puts every namespace in the address, we have to filter it out again. -If your main domain is ``example.com`` and you assign ``lists.example.com`` to -the user ``mailman``, qmail will give you the destination address -``mailman-spam@lists.example.com`` while it should actually be -``spam@lists.example.com``. The second argument to ``qmail-lmtp`` defines -how many parts (separated by dashes) to filter out. The first argument -specifies the LMTP port of mailman. Long story short, as user mailman: -:: - - % chmod +t "$HOME" - % echo '|qmail-lmtp 1 8042' > .qmail # put appropriate values here - % ln -sf .qmail .qmail-default - % chmod -t "$HOME" - -.. _qmail: https://cr.yp.to/qmail.html -.. _djb: https://cr.yp.to - - -Sendmail -======== - -The core Mailman developers generally do not use Sendmail, so experience is -limited. Any and all contributions are welcome! The follow information from -a post by Gary Algier may be useful as a starting point, -although it describes Mailman 2: - - I have it working fine. I recently replaced a very old implementation - of sendmail and Mailman 2 on Solaris with a new one on CentOS 6. When I - did so, I used the POSTFIX_ALIAS_CMD mechanism to automatically process - the aliases. See:: - - https://mail.python.org/pipermail/mailman-users/2004-June/037518.html - - In mm_cfg.py:: - - MTA='Postfix' - POSTFIX_ALIAS_CMD = '/usr/bin/sudo /etc/mail/import-mailman-aliases' - - /etc/mail/import-mailman-aliases contains:: - - #! /bin/sh - /bin/cp /etc/mailman/aliases /etc/mail/mailman.aliases - /usr/bin/newaliases - - In /etc/sudoers.d/mailman:: - - Cmnd_Alias IMPORT_MAILMAN_ALIASES = /etc/mail/import-mailman-aliases - apache ALL= NOPASSWD: IMPORT_MAILMAN_ALIASES - mailman ALL= NOPASSWD: IMPORT_MAILMAN_ALIASES - Defaults!IMPORT_MAILMAN_ALIASES !requiretty - - In the sendmail.mc file I changed:: - - define(`ALIAS_FILE', `/etc/aliases')dnl - - to:: - - define(`ALIAS_FILE', `/etc/aliases,/etc/mail/mailman.aliases')dnl - - so that the Mailman aliases would be in a separate file. - -The main issue here is that Mailman 2 expects to receive messages from -the MTA via pipes, whereas Mailman 3 uses LMTP exclusively. Recent -Sendmail does support LMTP, so it's a matter of configuring a stock -Sendmail. But rather than using aliases, it needs to be configured to -relay to the LMTP port of Mailman. - - -.. _`mailing list or on IRC`: START.html#contact-us -.. _`Local Mail Transfer Protocol`: - http://en.wikipedia.org/wiki/Local_Mail_Transfer_Protocol -.. _LMTP: http://www.faqs.org/rfcs/rfc2033.html -.. _`Simple Mail Transfer Protocol`: - http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol -.. _SMTP: http://www.faqs.org/rfcs/rfc5321.html -.. _Postfix: http://www.postfix.org -.. _`Variable Envelope Return Path`: - http://en.wikipedia.org/wiki/Variable_envelope_return_path -.. _postconf: http://www.postfix.org/postconf.1.html diff --git a/src/mailman/docs/RELEASENOTES.rst b/src/mailman/docs/RELEASENOTES.rst deleted file mode 100644 index bccb03bf8..000000000 --- a/src/mailman/docs/RELEASENOTES.rst +++ /dev/null @@ -1,26 +0,0 @@ -=============== - Release notes -=============== - -Mailman 3 is a fully rewritten code base. The developers believe it has -sufficient functionality to provide full mailing list services. It should be -ready for production use by experienced system developers, but it may not be -easy to install or run by novices. - -It should also be possible to migrate Mailman 2.1 mailing lists to Mailman 3. -Caution, backups, and testing is recommended. - -We expect it to be possible to run Mailman 3 and Mailman 2.1 together on the -same systems, but you may need to be quite experienced with configuring your -mail server and web infrastructure. - -Mailman 3 may have bugs. - -Mailman 3 is not yet feature complete with Mailman 2.1. - -The documentation here describes the Mailman Core in great detail. Postorius, -Hyperkitty, and mailman.client are described and developed elsewhere. - -More release notes are maintained on the `Mailman wiki`_. - -.. _`Mailman wiki`: http://wiki.list.org/Mailman3 diff --git a/src/mailman/docs/STYLEGUIDE.rst b/src/mailman/docs/STYLEGUIDE.rst index 503b98db5..d5bbdc1ae 100644 --- a/src/mailman/docs/STYLEGUIDE.rst +++ b/src/mailman/docs/STYLEGUIDE.rst @@ -1,17 +1,17 @@ -============================== -GNU Mailman Coding Style Guide -============================== +================================ + GNU Mailman Coding Style Guide +================================ Copyright (C) 2002-2017 Barry A. Warsaw -Python coding style guide for GNU Mailman -========================================= +Python coding style guide for GNU Mailman Core +============================================== This document contains a style guide for Python programming, as used in GNU -Mailman. `PEP 8`_ is the basis for this style guide so its recommendations -should be followed except for the differences outlined here. This document -assumes the use of Python 3. +Mailman Core. `PEP 8`_ is the basis for this style guide so its +recommendations should be followed except for the differences outlined here. +Core is a Python 3 application, so this document assumes the use of Python 3. Much of the style guide is enforced by the command ``tox -e qa``. @@ -19,8 +19,8 @@ Much of the style guide is enforced by the command ``tox -e qa``. an example. * Public module-global names should be exported in the ``__all__`` but use the - ``@public`` decorator from the top-level ``mailman`` package to do this for - all classes and functions. + ``@public`` decorator from the public_ package to do this for all classes + and functions. * Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants. @@ -70,18 +70,19 @@ Much of the style guide is enforced by the command ``tox -e qa``. """ * Write docstrings for modules, functions, classes, and methods. Docstrings - can be omitted for special methods (e.g. __init__() or __str__()) where the - meaning is obvious. + can be omitted for special methods (e.g. ``__init__()`` or ``__str__()``) + where the meaning is obvious. -* PEP 257 describes good docstrings conventions. Note that most importantly, - the """ that ends a multiline docstring should be on a line by itself, e.g.:: +* `PEP 257`_ describes good docstrings conventions. Note that most + importantly, the ``"""`` that ends a multiline docstring should be on a line + by itself, e.g.:: """Return a foobang Optional plotz says to frobnicate the bizbaz first. """ -* For one liner docstrings, keep the closing """ on the same line. +* For one liner docstrings, keep the closing ``"""`` on the same line. * ``fill-column`` for docstrings should be 78. @@ -106,3 +107,5 @@ Much of the style guide is enforced by the command ``tox -e qa``. .. _`PEP 8`: http://www.python.org/peps/pep-0008.html .. _`GNU Mailman Python template`: https://gitlab.com/mailman/mailman/blob/master/template.py +.. _public: https://public.readthedocs.io/en/latest/ +.. _`PEP 257`: http://www.python.org/peps/pep-0257.html diff --git a/src/mailman/docs/WebUIin5.rst b/src/mailman/docs/WebUIin5.rst deleted file mode 100644 index 70716e6a7..000000000 --- a/src/mailman/docs/WebUIin5.rst +++ /dev/null @@ -1,101 +0,0 @@ -================================ -Set up Postorius in five minutes -================================ - -This is a quick guide for setting up a development environment to work on -Mailman 3's web UI, called Postorius. If all goes as planned, you should be -done within 5 minutes. This has been tested on Ubuntu 11.04. - -In order to download the components necessary you need to have the `Git`_ -version control system installed on your system. Mailman requires Python 3.4, -while mailman.client needs at least Python version 2.6. - -It's probably a good idea to set up a virtual Python environment using -`virtualenv`_. `Here is a brief HOWTO`_. You would need two separate virtual -environment one using Python version 2.6 or 2.7 (for Postorius and -mailman.client) and other using Python version 3.4 (for Mailman core). - -.. _`virtualenv`: http://pypi.python.org/pypi/virtualenv -.. _`Here is a brief HOWTO`: ./ArchiveUIin5.html#get-it-running-under-virtualenv -.. _`Git`: http://git-scm.com - - -GNU Mailman 3 -============= - -First download the latest revision of Mailman 3 from Gitlab. -:: - - $(py3) git clone git@gitlab.com:mailman/mailman.git - -Install the Core:: - - $(py3) cd mailman - $(py3) python setup.py develop - -If you get no errors you can now start Mailman:: - - $(py3) mailman start - $(py3) cd .. - -At this point Mailman will not send nor receive any real emails. But that's -fine as long as you only want to work on the components related to the REST -client or the web ui. - - -mailman.client (the Python bindings for Mailman's REST API) -=========================================================== - -Now you should switch to the virtual environment running Python version 2.6 or -2.7. Download the client from Gitlab:: - - $(py2) git clone git@gitlab.com:mailman/mailmanclient.git - -Install in development mode to be able to change the code without working -directly on the PYTHONPATH. -:: - - $(py2) cd mailmanclient - $(py2) python setup.py develop - $(py2) cd .. - - -Postorius -========= - -:: - - $(py2) git clone git@gitlab.com:mailman/postorius.git - $(py2) cd postorius - $(py2) python setup.py develop - - -Start the development server -============================ - -Postorius is a Django app which can be used with any Django project. We have -a project already developed which you can set up like this:: - - $(py2) cd example_project - $(py2) python manage.py migrate - $(py2) python manage.py runserver - -The last command will start the dev server on http://localhost:8000. - - -A note for MacOS X users (and possibly others running python 2.7) -================================================================= - -*Note: These paragraphs are struck-through on the Mailman wiki.* - -On an OS X 10.7 (Lion) system, some of these steps needed to be modified to -use python2.6 instead of python. (In particular, bzr is known to behave badly -when used python2.7 on OS X 10.7 at the moment -- hopefully this will be fixed -and no longer an issue soon.) - -You will need to install the latest version of XCode on MacOS 10.7, which is -available for free from the App Store. If you had a previous version of XCode -installed when you upgraded to 10.7, it will no longer work and will not have -automatically been upgraded, so be prepared to install again. Once you have -it installed from the App Store, you will still need to go run the installer -from ``/Applications`` to complete the installation. diff --git a/src/mailman/docs/architecture.rst b/src/mailman/docs/architecture.rst new file mode 100644 index 000000000..e1d5c4ec5 --- /dev/null +++ b/src/mailman/docs/architecture.rst @@ -0,0 +1,188 @@ +============================= + Mailman 3 Core architecture +============================= + +This is a brief overview of the internal architecture of the Mailman 3 core +delivery engine. You should start here if you want to understand how Mailman +works at the 1000 foot level. Another good source of architectural +information is available in the chapter written by Barry Warsaw for the +`Architecture of Open Source Applications`_. + + +User model +========== + +Every major component of the system is defined by an interface. Look through +``src/mailman/interfaces`` for an understanding of the system components. +Mailman objects which are stored in the database, are defined by *model* +classes. Objects such as *mailing lists*, *users*, *members*, and *addresses* +are primary objects within the system. + +The *mailing list* is the central object which holds all the configuration +settings for a particular mailing list. A mailing list is associated with a +*domain*, and all mailing lists are managed (i.e. created, destroyed, looked +up) via the *mailing list manager*. + +*Users* represent people, and have a *user id* and a *display name*. Users +are linked to *addresses* which represent a single email address. One user +can be linked to many addresses, but an address is only linked to one user. +Addresses can be *verified* or *not verified*. Mailman will deliver email +only to *verified* addresses. + +Users and addresses are managed by the *user manager*. + +A *member* is created by linking a *subscriber* to a mailing list. +Subscribers can be: + +* A user, which become members through their *preferred address*. +* An address, which can be linked or unlinked to a user, but must be verified. + +Members also have a *role*, representing regular members, digest members, list +owners, and list moderators. Members can even have the *non-member* role +(i.e. people not yet subscribed to the mailing list) for various moderation +purposes. + + +Process model +============= + +Messages move around inside the Mailman system by way of *queue* directories +managed by the *switchboard*. For example, when a message is first received +by Mailman, it is moved to the *in* (for "incoming") queue. During the +processing of this message, it -or copies of it- may be moved to other queues +such as the *out* queue (for outgoing email), the *archive* queue (for sending +to the archivers), the *digest* queue (for composing digests), etc. + +A message in a queue is represented by a single file, a ``.pck`` file. This +file contains two objects, serialized as `Python pickles`_. The first object +is the message being processed, already parsed into a `more efficient internal +representation`_. The second object is a metadata dictionary that records +additional information about the message as it is being processed. + +``.pck`` files only exist for messages moving between different system queues. +There is no ``.pck`` file for messages while they are actively being +processed. + +Each queue directory is associated with a *runner* process which wakes up +every so often. When the runner wakes up, it examines all the ``.pck`` files +in FIFO order, deserializing the message and metadata objects, and processing +them. If the message needs further processing in a different queue, it will +be re-serialized back into a ``.pck`` file. If not (e.g. because processing +of the message is complete), then no ``.pck`` file is written. + +The Mailman system uses a few other runners which don't process messages in a +queue. You can think of these as fairly typical server process, and examples +include the LMTP server, and the HTTP server for processing REST commands. + +All of the runners are managed by a *master watcher* process. When you type +``mailman start`` you are actually starting the master. Based on +configuration options, the master will start the appropriate runners as +subprocesses, and it will watch for the clean exiting of these subprocesses +when ``mailman stop`` is called. + + +Rules and chains +================ + +When a message is first received for posting to a mailing list, Mailman +processes the message to determine whether the message is appropriate for the +mailing list. If so, it *accepts* the message and it gets posted. Mailman +can *discard* the message so that no further processing occurs. Mailman can +also *reject* the message, bouncing it back to the original sender, usually +with some indication of why the message was rejected. Or, Mailman can *hold* +the message for moderator approval. + +*Moderation* is the phase of processing that determines which of the above +four dispositions will occur for the newly posted message. Moderation does +not generally change the message, but it may record information in the +metadata dictionary. Moderation is performed by the *in* queue runner. + +Each step in the moderation phase applies a *rule* to the message and asks +whether the rule *hits* or *misses*. Each rule is linked to an *action* which +is taken if the rule hits (i.e. matches). If the rule misses (i.e. doesn't +match), then the next rule is tried. All of the rule/action links are strung +together sequentially into a *chain*, and every mailing list has a *start +chain* where rule processing begins. + +Actually, every mailing list has *two* start chains, one for regular postings +to the mailing list, and another for posting to the owners of the mailing +list. + +To recap: when a message comes into Mailman for posting to a mailing list, the +incoming runner finds the destination mailing list, determines whether the +message is for the entire list membership, or the list owners, and retrieves +the appropriate start chain. The message is then passed to the chain, where +each link in the chain first checks to see if its rule matches, and if so, it +executes the linked action. This action is usually one of *accept*, *reject*, +*discard*, and *hold*, but other actions are possible, such as executing a +function, deferring action, or jumping to another chain. + +As you might imagine, you can write new rules, compose them into new chains, +and configure a mailing list to use your custom chain when processing the +message during the moderation phase. + + +Pipeline of handlers +==================== + +Once a message is accepted for posting to the mailing list, the message is +usually modified in a number of different ways. For example, some message +headers may be added or removed, some MIME parts might be scrubbed, added, or +rearranged, and various informative headers and footers may be added to the +message. + +The process of preparing the message for the list membership (as well as the +digests, archivers, and NNTP) falls to the *pipeline of handlers* managed by +the *pipeline* queue. + +The pipeline of handlers is similar to the processing chain, except here, a +handler can make any modifications to the message it wants, and there is no +rule decision or action. The message and metadata simply flow through a +sequence of handlers arranged in a named pipeline. Some of the handlers +modify the message in ways described above, and others copy the message to the +outgoing, NNTP, archiver, or digester queues. + +As with chains, each mailing list has two pipelines, one for posting to the +list membership, and the other for posting to the list's owners. + +Of course, you can define new handlers, compose them into new pipelines, and +change a mailing list's pipelines. + + +Integration and control +======================= + +Humans and external programs can interact with a running Core system in may +different ways. There's an extensive command line interface that provides +useful options to a system administrator. For external applications such as +the Postorius web user interface, and the HyperKitty archiver, the +`administrative REST API ` is the most common way to get information +into and out of the Core. + +**Note**: The REST API is an administrative API and as such it must not be +exposed to the public internet. By default, the REST server only listens on +``localhost``. + +Internally, the Python API is extensive and well-documented. Most objects in +the system are accessed through the `Zope Component Architecture`_ (ZCA). If +your Mailman installation is importable, you can write scripts directly +against the internal public Python API. + + +Other bits and pieces +===================== + +There are lots of other pieces to the Mailman puzzle, such as the set of core +functionality (logging, initialization, event handling, etc.), mailing list +*styles*, the API for integrating external archivers and mail servers. The +database layer is an critical piece, and Mailman has an extensive set of +command line commands, and email commands. + +Almost the entire system is documented in these pages, but it maybe be a bit +of a spelunking effort to find it. Improvements are welcome! + + +.. _`Architecture of Open Source Applications`: http://www.aosabook.org/en/mailman.html +.. _`Python pickles`: http://docs.python.org/2/library/pickle.html +.. _`more efficient internal representation`: https://docs.python.org/3/library/email.html +.. _`Zope Component Architecture`: https://pypi.python.org/pypi/zope.component diff --git a/src/mailman/docs/contribute.rst b/src/mailman/docs/contribute.rst new file mode 100644 index 000000000..136a4460d --- /dev/null +++ b/src/mailman/docs/contribute.rst @@ -0,0 +1,247 @@ +=========================== + Contributing to Mailman 3 +=========================== + +Copyright (C) 2008-2017 by the Free Software Foundation, Inc. + + +How to contribute +================= + +We accept `merge requests`_ and `bug reports`_ on GitLab. We prefer if every +merge request is linked to a bug report, because we can more easily manage the +priority of bug reports. For more substantial contributions, we may ask you +to sign a `copyright assignment`_ to the Free Software Foundation, the owner +of the GNU Mailman copyright. If you'd like to jump start your copyright +assignment, please contact the GNU Mailman `steering committee`_. + +Please read the :doc:`STYLEGUIDE` for required coding style guidelines. + + +Contact Us +========== + +Contributions of code, problem reports, and feature requests are welcome. +Please submit bug reports on the Mailman bug tracker at +https://gitlab.com/mailman/mailman/issues (you need to have a login on GitLab +to do so). You can also send email to the mailman-developers@python.org +mailing list, or ask on IRC channel ``#mailman`` on Freenode. + + +Get the sources +=============== + +The Mailman 3 source code is version controlled using Git. You can get a +local copy by running this command:: + + $ git clone https://gitlab.com/mailman/mailman.git + +or if you have a GitLab account and prefer ssh:: + + $ git clone git@gitlab.com:mailman/mailman.git + + +Testing Mailman 3 +================= + +To run the Mailman test suite, just use the `tox`_ command:: + + $ tox + +`tox` creates a virtual environment (virtualenv_) for you, installs all the +dependencies into that virtualenv, and runs the test suite from that +virtualenv. By default it does not use the `--system-site-packages` so it +downloads everything from the `Python Cheeseshop`_. + +A bare ``tox`` command will try to run several test suites, which might take a +long time, and/or require versions of Python or other components you might not +have installed. You can run ``tox -l`` to list the test suite *environments* +available. Very often, when you want to run the full test suite in the +quickest manner with components that should be available everywhere, run one +of these command, depending on which version of Python 3 you have:: + + $ tox -e py36-nocov + $ tox -e py35-nocov + $ tox -e py34-nocov + +You can run individual tests in any given environment by providing additional +positional arguments. For example, to run only the tests that match a +specific pattern:: + + $ tox -e py35-nocov -- -P user + +You can see all the other arguments supported by the test suite by running:: + + $ tox -e py35-nocov -- --help + +You also have access to the virtual environments created by tox, and you can +use this run the virtual environment's Python executable, or run the +``mailman`` command locally, e.g.:: + + $ .tox/py35-nocov/bin/python + $ .tox/py35-nocov/bin/mailman --help + +If you want to set up the virtual environment without running the full test +suite, you can do this:: + + $ tox -e py35-nocov --notest -r + + +Testing with PostgreSQL and MySQL +================================= + +By default, the test suite runs with the built-in SQLite database engine. If +you want to run the full test suite against the PostgreSQL or MySQL databases, +set the database up as described in :doc:`database`. + +For PostgreSQL, then create a `postgres.cfg` file any where you want. This +`postgres.cfg` file will contain the ``[database]`` section for PostgreSQL, +e.g.:: + + [database] + class: mailman.database.postgresql.PostgreSQLDatabase + url: postgres://myuser:mypassword@mypghost/mailman + +Then run the test suite like so:: + + $ MAILMAN_EXTRA_TESTING_CFG=/path/to/postgres.cfg tox -e py35-pg + +You can combine these ways to invoke Mailman, so if you want to run an +individual test against PostgreSQL, you could do:: + + $ MAILMAN_EXTRA_TESTING_CFG=/path/to/postgres.cfg tox -e py35-pg -- -P user + +Note that the path specified in `MAILMAN_EXTRA_TESTING_CFG` must be an +absolute path or some tests will fail. + + +Building for development +======================== + +To build Mailman for development purposes, you can create a virtual +environment outside of tox. You need to have the `virtualenv`_ program +installed, or you can use Python 3's built-in `pyvenv`_ command. + +First, create a virtual environment (venv). The directory you install the +venv into is up to you, but for purposes of this document, we'll install it +into ``/tmp/mm3``:: + + $ python3 -m venv /tmp/mm3 + +Now, activate the virtual environment and set it up for development:: + + % source /tmp/mm3/bin/activate + % python setup.py develop + +Sit back and have some Kombucha while you wait for everything to download and +install. + + +Building the documentation +========================== + +To build the documentation, you need some additional dependencies. The only +one you probably need from your OS vendor is `graphiz`. E.g. On Debian or +Ubuntu, you can do:: + + $ sudo apt install graphiz + +All other dependencies should be automatically installed as needed. Build the +documentation by running:: + + $ tox -e docs + +Then visit:: + + build/sphinx/html/index.html + + +Mailman Shell +============= + +This documentation has examples which use the Mailman shell to interact with +Mailman. To start the shell type ``mailman shell`` in your terminal. + +There are some testings functions which need to be imported first before you +use them. They can be imported from the modules available in +``mailman.testing``. For example, to use ``dump_list`` you first need to +import it from the ``mailman.testing.documentation`` module. + +.. Of course, *this* doctest doesn't have these preloaded... + >>> from zope.component import getUtility + >>> from mailman.interfaces.listmanager import IListManager + +The shell automatically initializes the Mailman system, loads all the +available interfaces, and configures the `Zope Component Architecture`_ (ZCA) +which is used to access all the software components in Mailman. So for +example, if you wanted to get access to the list manager component, you could +do:: + + $ mailman shell + Welcome to the GNU Mailman shell + + >>> list_manager = getUtility(IListManager) + + +Related projects +================ + +What you are looking at right now is the Mailman Core. It's "just" the +message delivery engine, but it's designed to work with a web user interface +for list members and administrators, and an archiver. The GNU Mailman project +also develops a web ui and archiver, but these are available in separate git +repositories. + + +Mailman Web UI +-------------- + +The Mailman 3 web UI, called *Postorius*, interfaces to core Mailman engine +via the REST client API. This architecture makes it possible for users with +other needs to adapt the web UI, or even replace it entirely, with a +reasonable amount of effort. However, as a core feature of Mailman, the web +UI emphasizes usability over modularity at first, so most users should use the +web UI described here. Postorius_ is a Django_ application. + + +The Archiver +~~~~~~~~~~~~ + +In Mailman 3, the archivers are decoupled from the Core. Instead, Mailman 3 +provides a simple, standard interface for third-party archiving tools and +services. For this reason, Mailman 3 defines a formal interface to insert +messages into any of a number of configured archivers, using whatever protocol +is appropriate for that archiver. Summary, search, and retrieval of archived +posts are handled by a separate application. + +A new archive UI called `HyperKitty`_, based on the `notmuch mail indexer`_ +was prototyped at the `Pycon 2012 sprint`_ by Toshio Kuratomi. The HyperKitty +archiver is very loosely coupled to Mailman 3 core. In fact, any email +application that speaks LMTP or SMTP will be able to use HyperKitty. +HyperKitty is also a Django application. + + +REST API Python bindings +~~~~~~~~~~~~~~~~~~~~~~~~ + +Mailman 3 provides a REST API for administrative purposes, and this is used by +both HyperKitty and Postorius. You can of course use any HTTP client to speak +to it, but we provide official Python bindings (for both Python 2 and 3) in a +package we call `mailman.client`_. + + +.. _`merge requests`: https://gitlab.com/mailman/mailman/merge_requests +.. _`bug reports`: https://gitlab.com/mailman/mailman/issues +.. _`copyright assignment`: https://www.fsf.org/licensing/assigning.html/?searchterm=copyright%20assignment +.. _`steering committee`: mailto:mailman-cabal@python.org +.. _tox: https://testrun.org/tox/latest/ +.. _`Zope Component Architecture`: https://pypi.python.org/pypi/zope.component +.. _`Postorius`: https://gitlab.com/mailman/postorius +.. _`Django`: http://djangoproject.org/ +.. _`HyperKitty`: https://gitlab.com/mailman/hyperkitty +.. _`notmuch mail indexer`: http://notmuchmail.org +.. _`mailman.client`: https://gitlab.com/mailman/mailmanclient +.. _`Pycon 2012 sprint`: https://us.pycon.org/2012/community/sprints/projects/ +.. _`Python Cheeseshop`: http://pypi.python.org/pypi +.. _`virtualenv`: http://www.virtualenv.org/en/latest/ +.. _`pyvenv`: https://docs.python.org/3/library/venv.html diff --git a/src/mailman/docs/database.rst b/src/mailman/docs/database.rst new file mode 100644 index 000000000..fc606a64d --- /dev/null +++ b/src/mailman/docs/database.rst @@ -0,0 +1,132 @@ +========================== + Setting up your database +========================== + +Mailman uses the SQLAlchemy_ ORM to provide persistence of data in a +relational database. By default, Mailman uses Python's built-in SQLite3_ +database, however, SQLAlchemy is compatible with PostgreSQL_ and MySQL_, among +possibly others. + +Currently, Mailman is known to work with the SQLite3, PostgreSQL, and MySQL +databases. (Volunteers to port it to other databases are welcome!). If you +want to use SQLite3, you generally don't need to change anything, but if you +want Mailman to use PostgreSQL or MySQL, you'll need to set those up first, +and then change a configuration variable in your ``/etc/mailman.cfg`` file. + +Two configuration variables control which database Mailman uses. The first +names the class implementing the database interface. The second names the URL +for connecting to the database. Both variables live in the ``[database]`` +section of the configuration file. + + +SQLite3 +======= + +As mentioned, if you want to use SQLite3 in the default configuration, you +generally don't need to change anything. However, if you want to change where +the SQLite3 database is stored, you can change the ``url`` variable in the +``[database]`` section. By default, the database is stored in the *data +directory* in the ``mailman.db`` file. Here's how to tell Mailman to store +its database in ``/var/lib/mailman/sqlite.db`` file:: + + [database] + url: sqlite:////var/lib/mailman/sqlite.db + + +PostgreSQL +========== + +First, you need to configure PostgreSQL itself. This `Ubuntu article`_ may +help. Let's say you create the `mailman` database in PostgreSQL via:: + + $ sudo -u postgres createdb -O $USER mailman + +You would also need the Python driver `psycopg2` for PostgreSQL:: + + $ pip install psycopg2 + +You would then need to set both the `class` and `url` variables in +`mailman.cfg` like so:: + + [database] + class: mailman.database.postgresql.PostgreSQLDatabase + url: postgres://myuser:mypassword@mypghost/mailman + +If you have any problems, you may need to delete the database and re-create +it:: + + $ sudo -u postgres dropdb mailman + $ sudo -u postgres createdb -O myuser mailman + +Many thanks to Stephen A. Goss for his contribution of PostgreSQL support. + + +MySQL +===== + +First you need to configure MySQL itself. Lets say you create the `mailman` +database in MySQL via:: + + mysql> CREATE DATABASE mailman; + +You would also need the Python driver `pymysql` for MySQL.:: + + $ pip install pymysql + +You would then need to set both the `class` and `url` variables in +`mailman.cfg` like so:: + + [database] + class: mailman.database.mysql.MySQLDatabase + url: mysql+pymysql://myuser:mypassword@mymysqlhost/mailman?charset=utf8&use_unicode=1 + +The last part of the url specifies the charset that client expects from the +server and to use Unicode via the flag `use_unicode`. You can find more about +these options on the `SQLAlchemy's MySQL page`_. + +If you have any problems, you may need to delete the database and re-create +it:: + + mysql> DROP DATABASE mailman; + mysql> CREATE DATABASE mailman; + + +Database Migrations +=================== + +Mailman uses `Alembic`_ to manage database migrations. Let's say you change +something in the models, what steps are needed to reflect that change in the +database schema? You need to create and enter a virtual environment, install +Mailman into that, and then run the ``alembic`` command. For example:: + + $ python3 -m venv /tmp/mm3 + $ source /tmp/mm3/bin/activate + $ python setup.py develop + $ mailman info + $ alembic -c src/mailman/config/alembic.cfg revision --autogenerate -m + "" + $ deactivate + +This would create a new migration which would be applied to the database +automatically on the next run of Mailman. + +People upgrading Mailman from previous versions need not do anything manually, +as soon as a new migration is added in the sources, it will be automatically +reflected in the schema on first-run post-update. + +**Note:** When auto-generating migrations using Alembic, be sure to check +the created migration before adding it to the version control. You will have +to manually change some of the special data types defined in +``mailman.database.types``. For example, ``mailman.database.types.Enum()`` +needs to be changed to ``sa.Integer()``, as the ``Enum`` type stores just the +integer in the database. A more complex migration would be needed for +``UUID`` depending upon the database layer to be used. + + +.. _SQLAlchemy: http://www.sqlalchemy.org/ +.. _SQLite3: http://docs.python.org/library/sqlite3.html +.. _PostgreSQL: http://www.postgresql.org/ +.. _MySQL: http://dev.mysql.com/ +.. _`Ubuntu article`: https://help.ubuntu.com/community/PostgreSQL +.. _`Alembic`: https://alembic.readthedocs.org/en/latest/ +.. _`SQLAlchemy's MySQL page`: http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#unicode diff --git a/src/mailman/docs/hyperkitty.rst b/src/mailman/docs/hyperkitty.rst new file mode 100644 index 000000000..d56031fa9 --- /dev/null +++ b/src/mailman/docs/hyperkitty.rst @@ -0,0 +1,108 @@ +=================================== + Set up HyperKitty in five minutes +=================================== + +.. note:: + This document is way out of date. If you have access to the Web, + the most recent explanation of setting up a full Mailman 3 system, + including Postorius and HyperKitty, is the `Development Setup Guide`_ at + `FedoraHosted`_. If you must work offline, this document may be of some + use, but be careful. + +The `HyperKitty`_ application aims at providing an interface to visualize and +explore Mailman archives. + +This is a `Django`_ project. + +Requirements +============ + +- A mail archive in `maildir format`_ (no, you don't need a running Mailman + 3!) Eventually HyperKitty will support `mbox format`_ for backward + compatibility with *Pipermail*, and *zipped maildirs* seem like a good idea + to save space. **Beware:** Although you'd think that we would be able to + manipulate the venerable *mbox* format safely and efficiently, that doesn't + seem to be the case. *Maildir* archives are **strongly** preferred, because + they are more robust to program bugs (whether in Mailman, HyperKitty, or in + the originating MUA!) +- Django is the web framework that supports the UI. +- `bunch`_ DOES WHAT? +- The `notmuch mail indexer`_ is used to generate indexes (and requires + `Xapian`_). +- HyperKitty itself, which is a UI, and not responsible for maintaining the + message archive itself. (Since the archive is in `maildir format`_, any + modern MTA or MDA can build one for you.) + + +Get it running (under virtualenv): +---------------------------------- + +It is generally a good idea to use `virtualenv`_ to create a stable +environment for your Python applications. + +- Create the virtualenv:: + + % virtualenv mailman3 + +- Activate the virtualenv:: + + % cd mailman3 + % source bin/activate + +You don't *have* to use virtualenv, though, and if you don't want to, just +omit the preceding steps. Continue with these steps. + +- Install Django and dependencies:: + + % easy_install django + % easy_install bunch + +- Install notmuch -- these are bindings that come with the notmuch C library. + The easiest way is probably to install them for your OS vendor and then + symlink them into the virtualenv similar to this:: + + % yum install -y python-notmuch + +- Note: on a multiarch system like Fedora, the directories may be lib64 rather + than lib on 64 bit systems. Next:: + + % cd lib/python2.7/site-packages + % ln -s /usr/lib/python2.7/site-packages/notmuch . + +- Note: this is the version of notmuch I tested with; others may work:: + + % ln -s /usr/lib/python2.7/site-packages/notmuch-0.11-py2.7.egg-info . + +- Install the HyperKitty sources:: + + % git clone https://github.com/hyperkitty/kittystore.git + % git clone https://github.com/hyperkitty/hyperkitty.git + % git clone https://github.com/hyperkitty/hyperkitty_standalone.git + + +Running HyperKitty +------------------ + +- Start it:: + + % cd hyperkitty + +- Put the static content where it should be:: + + % python manage.py collectstatic + +- Run the Django server:: + + % python manage.py runserver + + +.. _`Development Setup Guide`: https://fedorahosted.org/hyperkitty/wiki/DevelopmentSetupGuide +.. _`FedoraHosted`: https://fedorahosted.org/ +.. _`HyperKitty`: https://fedorahosted.org/hyperkitty/ +.. _`Django`: http://djangoproject.org/ +.. _`notmuch mail indexer`: http://notmuchmail.org +.. _`bunch`: http://pypi.python.org/pypi/bunch +.. _`Xapian`: http://xapian.org/ +.. _`maildir format`: http://en.wikipedia.org/wiki/Maildir +.. _`mbox format`: http://en.wikipedia.org/wiki/Mbox +.. _`virtualenv`: http://pypi.python.org/pypi/virtualenv diff --git a/src/mailman/docs/install.rst b/src/mailman/docs/install.rst new file mode 100644 index 000000000..227fc087c --- /dev/null +++ b/src/mailman/docs/install.rst @@ -0,0 +1,105 @@ +====================== + Installing Mailman 3 +====================== + +Copyright (C) 2008-2017 by the Free Software Foundation, Inc. + + +Requirements +============ + +For the Core, Python 3.4 or newer is required. It can either be the default +'python3' on your ``$PATH`` or it can be accessible via the ``python3.4``, +``python3.5``, or ``python3.6`` binary. If your operating system does not +include Python 3, see http://www.python.org for information about downloading +installers (where available) and installing it from source (when necessary or +preferred). Python 2 is not supported by the Core. + +You may need some additional dependencies, which are either available from +your OS vendor, or can be downloaded automatically from the `Python +Cheeseshop`_. + + +Documentation +============= + +The documentation for Mailman 3 is distributed throughout the sources. The +core documentation (such as this file) is found in the ``src/mailman/docs`` +directory, but much of the documentation is in module-specific places. Online +versions of the `Mailman 3 Core documentation`_ is available online. + +Also helpful might be Mark Sapiro's documentation on `building out the +mailman3.org`_ server. + + +Get the sources +=============== + +The Mailman 3 source code is version controlled using Git. You can get a +local copy by running this command:: + + $ git clone https://gitlab.com/mailman/mailman.git + +or if you have a GitLab account and prefer ssh:: + + $ git clone git@gitlab.com:mailman/mailman.git + + +Running Mailman 3 +================= + +You will need to set up a configuration file to override the defaults and set +things up for your environment. Mailman is configured using an "ini"-style +configuration system. Usually this means creating a ``mailman.cfg`` file and +putting it in a standard search location. See the :ref:`configuration +` documentation for details. + +By default, all runtime files are put under a ``var`` directory in the current +working directory. + +Run the ``mailman info`` command to see which configuration file Mailman is +using, and where it will put its database file. The first time you run this, +Mailman will also create any necessary run-time directories and log files. + +Try ``mailman --help`` for more details. You can use the commands +``mailman start`` to start the runner subprocess daemons, and of course +``mailman stop`` to stop them. + +Note that you can also run Mailman from one of the virtual environments +created by tox, e.g.:: + + $ tox -e py35-nocov --notest -r + $ .tox/py35-nocov/bin/mailman info + + +Mailman Shell +============= + +This documentation has examples which use the Mailman shell to interact with +Mailman. To start the shell type ``mailman shell`` in your terminal. + +There are some testings functions which need to be imported first before you +use them. They can be imported from the modules available in +``mailman.testing``. For example, to use ``dump_list`` you first need to +import it from the ``mailman.testing.documentation`` module. + +.. Of course, *this* doctest doesn't have these preloaded... + >>> from zope.component import getUtility + >>> from mailman.interfaces.listmanager import IListManager + +The shell automatically initializes the Mailman system, loads all the +available interfaces, and configures the `Zope Component Architecture`_ (ZCA) +which is used to access all the software components in Mailman. So for +example, if you wanted to get access to the list manager component, you could +do:: + + $ mailman shell + Welcome to the GNU Mailman shell + + >>> list_manager = getUtility(IListManager) + + +.. _`Python Cheeseshop`: http://pypi.python.org/pypi +.. _`Mailman 3 Core documentation`: https://mailman.readthedocs.io +.. _`Zope Component Architecture`: https://pypi.python.org/pypi/zope.component +.. _`building out the mailman3.org`: https://wiki.list.org/DOC/Mailman%203%20installation%20experience diff --git a/src/mailman/docs/introduction.rst b/src/mailman/docs/introduction.rst new file mode 100644 index 000000000..9814796cd --- /dev/null +++ b/src/mailman/docs/introduction.rst @@ -0,0 +1,111 @@ +.. _start-here: + +================================================ +Mailman - The GNU Mailing List Management System +================================================ + +This is `GNU Mailman`_, a mailing list management system distributed under the +terms of the `GNU General Public License`_ (GPL) version 3 or later. + +Mailman is written in Python_, a free object-oriented programming language. +Python is available for all platforms that Mailman is supported on, which +includes GNU/Linux and most other Unix-like operating systems (e.g. Solaris, +\*BSD, MacOSX, etc.). Mailman is not supported on Windows, although web and +mail clients on any platform should be able to interact with Mailman just +fine. + +GNU Mailman 3 consists of a `suite of programs`_ that work together: + +* Mailman Core; the core delivery engine. This is where you are right now. +* Postorius; the web user interface for list members and administrators. +* HyperKitty; the web-based archiver +* Mailman client; the official Python bindings for talking to the Core's REST + administrative API. + +Only the Core is required. You can write or integrate your own web user +interface or archiver that speaks to the Core over its REST API. The REST API +is a pure HTTP-based API so you don't *have* to use Python, or even our +official bindings. And you can deploy whatever components you want using +whatever mechanisms you want. + +But we really like Postorius and HyperKitty and hope you will too! + + +Copyright +========= + +Copyright 1998-2017 by the Free Software Foundation, Inc. + +This file is part of GNU Mailman. + +GNU Mailman is free software: you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free Software +Foundation, either version 3 of the License, or (at your option) any later +version. + +GNU Mailman is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR +A PARTICULAR PURPOSE. See the GNU General Public License for more details. + +You should have received a copy of the GNU General Public License along with +GNU Mailman. If not, see . + + +Spelling +======== + +The name of this software is spelled `Mailman` with a leading capital `M` +but with a lower case second `m`. Any other spelling is incorrect. Its full +name is `GNU Mailman` but is often referred colloquially as `Mailman`. + + +History +======= + +Mailman was originally developed by John Viega in the mid-1990s. Subsequent +development (through version 1.0b3) was by Ken Manheimer. Further work +towards the 1.0 final release was a group effort, with the core contributors +being: Barry Warsaw, Ken Manheimer, Scott Cotton, Harald Meland, and John +Viega. Since version 1.0 the project has been led by Barry Warsaw with +contributions from many; see the `ACKNOWLEDGMENTS`_ file for details. Jeremy +Hylton helped considerably with the Pipermail code in Mailman 2.0. Mailman +2.1 is primarily maintained by Mark Sapiro, with previous help by Tokio +Kikuchi (RIP). Barry Warsaw is the lead developer on Mailman 3. Aurélien +Bompard and Florian Fuchs lead development of Postorius and HyperKitty. +Abhilash Raj is a core developer contributing to all the bits and maintains +the CI infrastructure. + + +Project details +=============== + +* Project home page: http://www.list.org +* Documentation: https: https://mailman.readthedocs.io +* The community driven wiki (including the FAQ_): https://wiki.list.org +* Additional help resources: http://www.list.org/help.html +* Report Core bugs at: https://gitlab.com/mailman/mailman/issues +* Mailman 3 suite on GitLab: https://gitlab.com/groups/mailman + +(GNU Mailman 2.1 is still maintained on Launchpad_.) + +There are two mailing lists you can use to contact the Mailman 3 developers. +The general development mailing list: + +* https://mail.python.org/mailman/listinfo/mailman-developers + +and the Mailman 3 users list: + +* https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ + +For now, please leave the older mailman-users mailing list for Mailman 2. + + +.. _`GNU Mailman`: http://www.list.org +.. _`GNU General Public License`: http://www.gnu.org/licenses/gpl.txt +.. _Python: http://www.python.org +.. _FAQ: http://wiki.list.org/display/DOC/Frequently+Asked+Questions +.. _`Python 3.4`: https://www.python.org/downloads/release/python-342/ +.. _`ACKNOWLEDGMENTS`: ACKNOWLEDGMENTS.html +.. _`Django`: https://www.djangoproject.com/ +.. _`suite of programs`: http://docs.mailman3.org/en/latest/ +.. _Launchpad: https://launchpad.net/mailman diff --git a/src/mailman/docs/mta.rst b/src/mailman/docs/mta.rst new file mode 100644 index 000000000..2a7f3d6d9 --- /dev/null +++ b/src/mailman/docs/mta.rst @@ -0,0 +1,413 @@ +============================= + Hooking up your mail server +============================= + +Mailman needs to communicate with your *MTA* (*mail transport agent* +or *mail server*, the software which handles sending mail across the +Internet), both to accept incoming mail and to deliver outgoing mail. +Mailman itself never delivers messages to the end user. It sends them +to its immediate upstream MTA, which delivers them. In the same way, +Mailman never receives mail directly. Mail from outside always comes +via the MTA. + +Mailman accepts incoming messages from the MTA using the `Local Mail Transfer +Protocol`_ (LMTP_) interface. LMTP is much more efficient than spawning a +process just to do the delivery. Most open source MTAs support LMTP for local +delivery. If yours doesn't, and you need to use a different interface, please +ask on the `mailing list or on IRC`_. + +Mailman passes all outgoing messages to the MTA using the `Simple Mail +Transfer Protocol`_ (SMTP_). + +Cooperation between Mailman and the MTA requires some configuration of +both. MTA configuration differs for each of the available MTAs, and +there is a section for each one. Instructions for Postfix and Exim (v4) +are given below. We would really appreciate a contribution of a +configuration for Sendmail, and welcome information about other popular +open source mail servers. + +Configuring Mailman to communicate with the MTA is straightforward, and +basically the same for all MTAs. Here are the default settings; if you need +to change them, edit your ``mailman.cfg`` file:: + + [mta] + incoming: mailman.mta.postfix.LMTP + outgoing: mailman.mta.deliver.deliver + lmtp_host: 127.0.0.1 + lmtp_port: 8024 + smtp_host: localhost + smtp_port: 25 + configuration: python:mailman.config.postfix + +This configuration is for a system where Mailman and the MTA are on +the same host. + +Note that the modules that configure the communication protocol (especially +``incoming``) are full-fledged Python modules, and may use these configuration +parameters to automatically configure the MTA to recognize the list addresses +and other attributes of the communication channel. This is why some +constraints on the format of attributes arise (e.g., ``lmtp_host``), even +though Mailman itself has no problem with them. + +It is possible (although not documented here) to completely replace or +override the default mechanisms to handle both incoming and outgoing mail. +Mailman is highly customizable here! + +The ``incoming`` and ``outgoing`` parameters identify the Python objects used +to communicate with the MTA. The ``python:`` scheme indicates that the paths +should be a dotted Python module specification. The ``deliver`` module used +in ``outgoing`` should be satisfactory for most MTAs. The ``postfix`` module +in ``incoming`` is specific to the Postfix MTA. See the section for your MTA +below for details on these parameters. + +``lmtp_host`` and ``lmtp_port`` are parameters which are used by Mailman, but +also will be passed to the MTA to identify the Mailman host. The "same host" +case is special; some MTAs (including Postfix) do not recognize "localhost", +and need the numerical IP address. If they are on different hosts, +``lmtp_host`` should be set to the domain name or IP address of the Mailman +host. ``lmtp_port`` is fairly arbitrary (there is no standard port for LMTP). +Use any port convenient for your site. "8024" is as good as any, unless +another service is using it. + +``smtp_host`` and ``smtp_port`` are parameters used to identify the MTA to +Mailman. If the MTA and Mailman are on separate hosts, ``smtp_host`` should +be set to the domain name or IP address of the MTA host. ``smtp_port`` will +almost always be 25, which is the standard port for SMTP. (Some special site +configurations set it to a different port. If you need this, you probably +already know that, know why, and what to do, too!) + +Mailman also provides many other configuration variables that you can +use to tweak performance for your operating environment. See the +``src/mailman/config/schema.cfg`` file for details. + + +Postfix +======= + +Postfix_ is an open source mail server by Wietse Venema. + + +Mailman settings +---------------- + +You need to tell Mailman that you are using the Postfix mail server. In your +``mailman.cfg`` file, add the following section:: + + [mta] + incoming: mailman.mta.postfix.LMTP + outgoing: mailman.mta.deliver.deliver + lmtp_host: mail.example.com + lmtp_port: 8024 + smtp_host: mail.example.com + smtp_port: 25 + +Some of these settings are already the default, so take a look at Mailman's +``src/mailman/config/schema.cfg`` file for details. You'll need to change the +``lmtp_host`` and ``smtp_host`` to the appropriate host names of course. +Generally, Postfix will listen for incoming SMTP connections on port 25. +Postfix will deliver via LMTP over port 24 by default, however if you are not +running Mailman as root, you'll need to change this to a higher port number, +as shown above. + + +Basic Postfix connections +------------------------- + +There are several ways to hook Postfix up to Mailman, so here are the simplest +instructions. The following settings should be added to Postfix's ``main.cf`` +file. + +Mailman supports a technique called `Variable Envelope Return Path`_ (VERP) to +disambiguate and accurately record bounces. By default Mailman's VERP +delimiter is the `+` sign, so adding this setting allows Postfix to properly +handle Mailman's VERP'd messages:: + + # Support the default VERP delimiter. + recipient_delimiter = + + +In older versions of Postfix, unknown local recipients generated a temporary +failure. It's much better (and the default in newer Postfix releases) to +treat them as permanent failures. You can add this to your ``main.cf`` file +if needed (use the `postconf`_ command to check the defaults):: + + unknown_local_recipient_reject_code = 550 + +While generally not necessary if you set ``recipient_delimiter`` as described +above, it's better for Postfix to not treat ``owner-`` and ``-request`` +addresses specially:: + + owner_request_special = no + + +Transport maps +-------------- + +By default, Mailman works well with Postfix transport maps as a way to deliver +incoming messages to Mailman's LMTP server. Mailman will automatically write +the correct transport map when its ``mailman aliases`` command is run, or +whenever a mailing list is created or removed via other commands. To connect +Postfix to Mailman's LMTP server, add the following to Postfix's ``main.cf`` +file:: + + transport_maps = + hash:/path-to-mailman/var/data/postfix_lmtp + local_recipient_maps = + hash:/path-to-mailman/var/data/postfix_lmtp + relay_domains = + hash:/path-to-mailman/var/data/postfix_domains + +where ``path-to-mailman`` is replaced with the actual path that you're running +Mailman from. Setting ``local_recipient_maps`` as well as ``transport_maps`` +allows Postfix to properly reject all messages destined for non-existent local +users. Setting `relay_domains`_ means Postfix will start to accept mail for +newly added domains even if they are not part of `mydestination`_. + +Note that if you are not using virtual domains, then `relay_domains`_ isn't +strictly needed (but it is harmless). All you need to do in this scenario is +to make sure that Postfix accepts mail for your one domain, normally by +including it in ``mydestination``. + + +Postfix documentation +--------------------- + +For more information regarding how to configure Postfix, please see +the Postfix documentation at: + +.. _`The official Postfix documentation`: + http://www.postfix.org/documentation.html +.. _`The reference page for all Postfix configuration parameters`: + http://www.postfix.org/postconf.5.html +.. _`relay_domains`: http://www.postfix.org/postconf.5.html#relay_domains +.. _`mydestination`: http://www.postfix.org/postconf.5.html#mydestination + + +Exim +==== + +`Exim 4`_ is an MTA maintained by the `University of Cambridge`_ and +distributed by most open source OS distributions. + +Mailman settings +---------------- + +Add or edit a stanza like this in mailman.cfg:: + + [mta] + # For all Exim4 installations. + incoming: mailman.mta.exim4.LMTP + outgoing: mailman.mta.deliver.deliver + # Typical single host with MTA and Mailman configuration. + # Adjust to your system's configuration. + # Exim happily works with the "localhost" alias rather than IP address. + lmtp_host: localhost + smtp_host: localhost + # Mailman should not be run as root. + # Use any convenient port > 1024. 8024 is a convention, but can be + # changed if there is a conflict with other software using that port. + lmtp_port: 8024 + # smtp_port rarely needs to be set. + smtp_port: 25 + # Exim4-specific configuration parameter defaults. Currently empty. + configuration: python:mailman.config.exim4 + +For further information about these settings, see +``mailman/config/schema.cfg``. + +Exim4 configuration +------------------- + +The configuration presented below is mostly boilerplate that allows Exim to +automatically discover your list addresses, and route both posts and +administrative messages to the right Mailman services. For this reason, the +`mailman.mta.exim4` module ends up with all methods being no-ops. + +This configuration is field-tested in a Debian "conf.d"-style Exim +installation, with multiple configuration files that are assembled by a +Debian-specific script. If your Exim v4 installation is structured +differently, ignore the comments indicating location in the Debian +installation. +:: + + # /etc/exim4/conf.d/main/25_mm3_macros + # The colon-separated list of domains served by Mailman. + domainlist mm_domains=list.example.net + + MM3_LMTP_PORT=8024 + + # Assuming a typical source installation in /usr/local, with + # links to the Mailman bin directory and so on from MM3_HOME. + MM3_HOME=/usr/local/var/mailman + MM3_UID=list + MM3_GID=list + + ################################################################ + # The configuration below is boilerplate: + # you should not need to change it. + + # The path to the list receipt (used as the required file when + # matching list addresses) + MM3_LISTCHK=MM3_HOME/lists/${local_part}@${domain} + + # /etc/exim4/conf.d/router/455_mm3_router + mailman3_router: + driver = accept + domains = +mm_domains + require_files = MM3_LISTCHK + local_part_suffix_optional + local_part_suffix = -admin : \ + -bounces : -bounces+* : \ + -confirm : -confirm+* : \ + -join : -leave : \ + -owner : -request : \ + -subscribe : -unsubscribe + transport = mailman3_transport + + # /etc/exim4/conf.d/transport/55_mm3_transport + mailman3_transport: + driver = smtp + protocol = lmtp + allow_localhost + hosts = localhost + port = MM3_LMTP_PORT + +Troubleshooting +--------------- + +The most likely causes of failure to deliver to Mailman are typos in the +configuration, and errors in the ``MM3_HOME`` macro or the ``mm_domains`` +list. Mismatches in the LMTP port could be a cause. Finally, Exim's router +configuration is order-sensitive. Especially if you are being tricky and +supporting Mailman 2 and Mailman 3 at the same time, you could have one shadow +the other. + +Exim 4 documentation +-------------------- + +There is `copious documentation for Exim`_. The parts most relevant to +configuring communication with Mailman 3 are the chapters on the `accept +router`_ and the `LMTP transport`_. Unless you are already familiar +with Exim configuration, you probably want to start with the chapter on +`how Exim receives and delivers mail`. + +.. _`Exim 4`: http://www.exim.org/ +.. _`University of Cambridge`: http://www.cam.ac.uk/ +.. _`copious documentation for Exim`: http://www.exim.org/docs.html +.. _`accept router`: http://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_accept_router.html +.. _`LMTP transport`: http://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_lmtp_transport.html +.. _`how Exim receives and delivers mail`: http://www.exim.org/exim-html-current/doc/html/spec_html/ch-how_exim_receives_and_delivers_mail.html + + +qmail +===== + +qmail_ is a MTA written by djb_ and, though old and not updated, still +bulletproof and occassionally in use. + +Mailman settings +---------------- + +Mostly defaults in mailman.cfg:: + + [mta] + # NullMTA is just implementing the interface and thus satisfying Mailman + # without doing anything fancy + incoming: mailman.mta.null.NullMTA + # Mailman should not be run as root. + # Use any convenient port > 1024. 8024 is a convention, but can be + # changed if there is a conflict with other software using that port. + lmtp_port: 8024 + +This will listen on ``localhost:8024`` with LMTP and deliver outgoing messages +to ``localhost:25``. See ``mailman/config/schema.cfg`` for more information +on these settings. + +qmail configuration +------------------- + +It is assumed that qmail is configured to use the ``.qmail*`` files in a user’s +home directory, however the instructions should easily be adaptable to other +qmail configurations. However, it is required that Mailman has a (sub)domain +respectively a namespace on its own. A helper script called ``qmail-lmtp`` is +needed and can be found in the ``contrib/`` directory of the Mailman source +tree and assumed to be on ``$PATH`` here. + +As qmail puts every namespace in the address, we have to filter it out again. +If your main domain is ``example.com`` and you assign ``lists.example.com`` to +the user ``mailman``, qmail will give you the destination address +``mailman-spam@lists.example.com`` while it should actually be +``spam@lists.example.com``. The second argument to ``qmail-lmtp`` defines +how many parts (separated by dashes) to filter out. The first argument +specifies the LMTP port of mailman. Long story short, as user mailman: +:: + + % chmod +t "$HOME" + % echo '|qmail-lmtp 1 8042' > .qmail # put appropriate values here + % ln -sf .qmail .qmail-default + % chmod -t "$HOME" + +.. _qmail: https://cr.yp.to/qmail.html +.. _djb: https://cr.yp.to + + +Sendmail +======== + +The core Mailman developers generally do not use Sendmail, so experience is +limited. Any and all contributions are welcome! The follow information from +a post by Gary Algier may be useful as a starting point, +although it describes Mailman 2: + + I have it working fine. I recently replaced a very old implementation + of sendmail and Mailman 2 on Solaris with a new one on CentOS 6. When I + did so, I used the POSTFIX_ALIAS_CMD mechanism to automatically process + the aliases. See:: + + https://mail.python.org/pipermail/mailman-users/2004-June/037518.html + + In mm_cfg.py:: + + MTA='Postfix' + POSTFIX_ALIAS_CMD = '/usr/bin/sudo /etc/mail/import-mailman-aliases' + + /etc/mail/import-mailman-aliases contains:: + + #! /bin/sh + /bin/cp /etc/mailman/aliases /etc/mail/mailman.aliases + /usr/bin/newaliases + + In /etc/sudoers.d/mailman:: + + Cmnd_Alias IMPORT_MAILMAN_ALIASES = /etc/mail/import-mailman-aliases + apache ALL= NOPASSWD: IMPORT_MAILMAN_ALIASES + mailman ALL= NOPASSWD: IMPORT_MAILMAN_ALIASES + Defaults!IMPORT_MAILMAN_ALIASES !requiretty + + In the sendmail.mc file I changed:: + + define(`ALIAS_FILE', `/etc/aliases')dnl + + to:: + + define(`ALIAS_FILE', `/etc/aliases,/etc/mail/mailman.aliases')dnl + + so that the Mailman aliases would be in a separate file. + +The main issue here is that Mailman 2 expects to receive messages from +the MTA via pipes, whereas Mailman 3 uses LMTP exclusively. Recent +Sendmail does support LMTP, so it's a matter of configuring a stock +Sendmail. But rather than using aliases, it needs to be configured to +relay to the LMTP port of Mailman. + + +.. _`mailing list or on IRC`: START.html#contact-us +.. _`Local Mail Transfer Protocol`: + http://en.wikipedia.org/wiki/Local_Mail_Transfer_Protocol +.. _LMTP: http://www.faqs.org/rfcs/rfc2033.html +.. _`Simple Mail Transfer Protocol`: + http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol +.. _SMTP: http://www.faqs.org/rfcs/rfc5321.html +.. _Postfix: http://www.postfix.org +.. _`Variable Envelope Return Path`: + http://en.wikipedia.org/wiki/Variable_envelope_return_path +.. _postconf: http://www.postfix.org/postconf.1.html diff --git a/src/mailman/docs/postorius.rst b/src/mailman/docs/postorius.rst new file mode 100644 index 000000000..f1cc2b96e --- /dev/null +++ b/src/mailman/docs/postorius.rst @@ -0,0 +1,83 @@ +================================== + Set up Postorius in five minutes +================================== + +This is a quick guide for setting up a development environment to work on +Mailman 3's web UI, called Postorius. If all goes as planned, you should be +done within 5 minutes. This has been tested on Ubuntu 11.04. + +In order to download the components necessary you need to have the `Git`_ +version control system installed on your system. Mailman requires Python 3.4 +or newer, while mailman.client needs at least Python version 2.6. + +It's probably a good idea to set up a virtual Python environment using +`virtualenv`_. `Here is a brief HOWTO`_. You would need two separate virtual +environment one using Python version 2.6 or 2.7 (for Postorius and +mailman.client) and other using Python 3 (for Mailman core). + +.. _`virtualenv`: http://pypi.python.org/pypi/virtualenv +.. _`Here is a brief HOWTO`: ./ArchiveUIin5.html#get-it-running-under-virtualenv +.. _`Git`: http://git-scm.com + + +Mailman Core +============ + +First download the latest revision of Mailman 3 from Gitlab. +:: + + $(py3) git clone git@gitlab.com:mailman/mailman.git + +Install the Core:: + + $(py3) cd mailman + $(py3) python setup.py develop + +If you get no errors you can now start Mailman:: + + $(py3) mailman start + $(py3) cd .. + +At this point Mailman will not send nor receive any real emails. But that's +fine as long as you only want to work on the components related to the REST +client or the web ui. + + +mailman.client (the Python bindings for Mailman's REST API) +=========================================================== + +Now you should switch to the virtual environment running Python version 2.6 or +2.7. Download the client from Gitlab:: + + $(py2) git clone git@gitlab.com:mailman/mailmanclient.git + +Install in development mode to be able to change the code without working +directly on the PYTHONPATH. +:: + + $(py2) cd mailmanclient + $(py2) python setup.py develop + $(py2) cd .. + + +Postorius +========= + +:: + + $(py2) git clone git@gitlab.com:mailman/postorius.git + $(py2) cd postorius + $(py2) python setup.py develop + + +Start the development server +============================ + +Postorius is a Django app which can be used with any Django project. We have +a project already developed which you can set up like this:: + + $(py2) cd example_project + $(py2) python manage.py migrate + $(py2) python manage.py runserver + +The last command will start the dev server on http://localhost:8000. diff --git a/src/mailman/docs/release-notes.rst b/src/mailman/docs/release-notes.rst new file mode 100644 index 000000000..bccb03bf8 --- /dev/null +++ b/src/mailman/docs/release-notes.rst @@ -0,0 +1,26 @@ +=============== + Release notes +=============== + +Mailman 3 is a fully rewritten code base. The developers believe it has +sufficient functionality to provide full mailing list services. It should be +ready for production use by experienced system developers, but it may not be +easy to install or run by novices. + +It should also be possible to migrate Mailman 2.1 mailing lists to Mailman 3. +Caution, backups, and testing is recommended. + +We expect it to be possible to run Mailman 3 and Mailman 2.1 together on the +same systems, but you may need to be quite experienced with configuring your +mail server and web infrastructure. + +Mailman 3 may have bugs. + +Mailman 3 is not yet feature complete with Mailman 2.1. + +The documentation here describes the Mailman Core in great detail. Postorius, +Hyperkitty, and mailman.client are described and developed elsewhere. + +More release notes are maintained on the `Mailman wiki`_. + +.. _`Mailman wiki`: http://wiki.list.org/Mailman3 diff --git a/src/mailman/rest/docs/basic.rst b/src/mailman/rest/docs/basic.rst index 45ace87ec..e1a712980 100644 --- a/src/mailman/rest/docs/basic.rst +++ b/src/mailman/rest/docs/basic.rst @@ -1,21 +1,9 @@ -=========== -REST server -=========== - -Mailman exposes a REST HTTP server for administrative control. - -The server listens for connections on a configurable host name and port. +================= + Basic operation +================= -It is always protected by HTTP basic authentication using a single global -user name and password. The credentials are set in the `[webservice]` section -of the configuration using the `admin_user` and `admin_pass` properties. - -Because the REST server has full administrative access, it should always be -run only on localhost, unless you really know what you're doing. In addition -you should set the user name and password to secure values and distribute them -to any REST clients with reasonable precautions. - -The Mailman major and minor version numbers are in the URL. +In order to do anything with the REST API, you need to know its `Basic AUTH`_ +credentials, and the version of the API you wish to speak to. Credentials @@ -35,8 +23,8 @@ succeeds. 200 -Version information -=================== +System version information +========================== System version information can be retrieved from the server, in the form of a JSON encoded response. @@ -49,4 +37,30 @@ JSON encoded response. self_link: http://localhost:9001/3.0/system/versions +API Versions +============ + +The REST API exposes two versions which are almost completely identical. As +you've seen above, the ``3.0`` API is the base API. There is also a ``3.1`` +API, which can be used interchangably:: + + >>> dump_json('http://localhost:9001/3.1/system/versions') + api_version: 3.1 + http_etag: "..." + mailman_version: GNU Mailman 3... + python_version: ... + self_link: http://localhost:9001/3.1/system/versions + +The only difference is the way UUIDs are represented. UUIDs are 128-bit +unique ids for objects such as users and members. In version ``3.0`` of the +API, UUIDs are represented as 128-bit integers, but these were found to be +incompatible for some versions of JavaScript, so in API version ``3.1`` UUIDs +are represented as hex strings. + +Choose whichever API version makes sense for your application. In general, we +recommend using API ``3.1``, but most of the current documentation describes +API ``3.0``. Just make the mental substitution as you read along. + + .. _REST: http://en.wikipedia.org/wiki/REST +.. _`Basic AUTH`: https://en.wikipedia.org/wiki/Basic_auth diff --git a/src/mailman/rest/docs/rest.rst b/src/mailman/rest/docs/rest.rst index 8461e641d..ca962ad61 100644 --- a/src/mailman/rest/docs/rest.rst +++ b/src/mailman/rest/docs/rest.rst @@ -1,8 +1,51 @@ -======== -REST API -======== +.. _rest-api: + +======================================== + Mailman 3 Core administrative REST API +======================================== + +Here is extensive documentation on the Mailman Core administrative REST API. + + +The REST server +=============== + +Mailman exposes a REST HTTP server for administrative control. + +The server listens for connections on a configurable host name and port. + +It is always protected by HTTP basic authentication using a single global +user name and password. The credentials are set in the `[webservice]` section +of the configuration using the `admin_user` and `admin_pass` properties. + +Because the REST server has full administrative access, it should never be +exposed to the public internet. By default it only listens to connections on +``localhost``. Don't change this unless you really know what you're doing. +In addition you should set the user name and password to secure values and +distribute them to any REST clients with reasonable precautions. + +The Mailman major and minor version numbers are in the URL. + +You can write your own HTTP clients to speak this API, or you can use the +`official Python bindings`_. + .. toctree:: :glob: + :maxdepth: 1 + ./basic + ./collections + ./helpers + ./systemconf + ./domains + ./lists + ./listconf + ./addresses + ./users + ./membership + ./queues ./* + + +.. _`official Python bindings`: https://mailmanclient.readthedocs.io/en/latest/ -- cgit v1.2.3-70-g09d2