Cooking headers
===============

Messages that flow through the global pipeline get their headers 'cooked',
which basically means that their headers go through several mostly unrelated
transformations.  Some headers get added, others get changed.  Some of these
changes depend on mailing list settings and others depend on how the message
is getting sent through the system.  We'll take things one-by-one.

    >>> from mailman.pipeline.cook_headers import process
    >>> mlist = config.db.list_manager.create(u'_xtest@example.com')
    >>> mlist.subject_prefix = u''
    >>> mlist.include_list_post_header = False
    >>> mlist.archive = True
    >>> # XXX This will almost certainly change once we've worked out the web
    >>> # space layout for mailing lists now.
    >>> mlist.web_page_url = u'http://lists.example.com/'


Saving the original sender
--------------------------

Because the original sender headers may get deleted or changed, CookHeaders
will place the sender in the message metadata for safe keeping.

    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ...
    ... A message of great import.
    ... """)
    >>> msgdata = {}
    >>> process(mlist, msg, msgdata)
    >>> msgdata['original_sender']
    u'aperson@example.com'

But if there was no original sender, then the empty string will be saved.

    >>> msg = message_from_string("""\
    ... Subject: No original sender
    ...
    ... A message of great import.
    ... """)
    >>> msgdata = {}
    >>> process(mlist, msg, msgdata)
    >>> msgdata['original_sender']
    ''


X-BeenThere header
------------------

The X-BeenThere header is what Mailman uses to recognize messages that have
already been processed by this mailing list.  It's one small measure against
mail loops.

    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ...
    ... A message of great import.
    ... """)
    >>> process(mlist, msg, {})
    >>> msg['x-beenthere']
    u'_xtest@example.com'

Mailman appends X-BeenThere headers, so if there already is one in the
original message, the posted message will contain two such headers.

    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ... X-BeenThere: another@example.com
    ...
    ... A message of great import.
    ... """)
    >>> process(mlist, msg, {})
    >>> sorted(msg.get_all('x-beenthere'))
    [u'_xtest@example.com', u'another@example.com']


Mailman version header
----------------------

Mailman will also insert an X-Mailman-Version header...

    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ...
    ... A message of great import.
    ... """)
    >>> process(mlist, msg, {})
    >>> from mailman.version import VERSION
    >>> msg['x-mailman-version'] == VERSION
    True

...but only if one doesn't already exist.

    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ... X-Mailman-Version: 3000
    ...
    ... A message of great import.
    ... """)
    >>> process(mlist, msg, {})
    >>> msg['x-mailman-version']
    u'3000'


Precedence header
-----------------

Mailman will insert a Precedence header, which is a de-facto standard for
telling automatic reply software (e.g. vacation(1)) not to respond to this
message.

    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ...
    ... A message of great import.
    ... """)
    >>> process(mlist, msg, {})
    >>> msg['precedence']
    u'list'

But Mailman will only add that header if the original message doesn't already
have one of them.

    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ... Precedence: junk
    ...
    ... A message of great import.
    ... """)
    >>> process(mlist, msg, {})
    >>> msg['precedence']
    u'junk'


RFC 2919 and 2369 headers
-------------------------

This is a helper function for the following section.

    >>> def list_headers(msg):
    ...     print '---start---'
    ...     # Sort the List-* headers found in the message.  We need to do
    ...     # this because CookHeaders puts them in a dictionary which does
    ...     # not have a guaranteed sort order.
    ...     for header in sorted(msg.keys()):
    ...         parts = header.lower().split('-')
    ...         if 'list' not in parts:
    ...             continue
    ...         for value in msg.get_all(header):
    ...             print '%s: %s' % (header, value)
    ...     print '---end---'

These RFCs define headers for mailing list actions.  A mailing list should
generally add these headers, but not for messages that aren't crafted for a
specific list (e.g. password reminders in Mailman 2.x).

    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ...
    ... """)
    >>> process(mlist, msg, dict(_nolist=True))
    >>> list_headers(msg)
    ---start---
    ---end---

Some people don't like these headers because their mail readers aren't good
about hiding them.  A list owner can turn these headers off.

    >>> mlist.include_rfc2369_headers = False
    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ...
    ... """)
    >>> process(mlist, msg, {})
    >>> list_headers(msg)
    ---start---
    ---end---

But normally, a list will include these headers.

    >>> mlist.include_rfc2369_headers = True
    >>> mlist.include_list_post_header = True
    >>> mlist.preferred_language = u'en'
    >>> config.archivers['pipermail'].is_enabled = True
    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ... Message-ID: <12345>
    ...
    ... """)
    >>> process(mlist, msg, {})
    >>> list_headers(msg)
    ---start---
    List-Archive: <http://www.example.com/pipermail/_xtest@example.com>
    List-Help: <mailto:_xtest-request@example.com?subject=help>
    List-Id: <_xtest.example.com>
    List-Post: <mailto:_xtest@example.com>
    List-Subscribe: <http://lists.example.com/listinfo/_xtest@example.com>,
    	<mailto:_xtest-join@example.com>
    List-Unsubscribe: <http://lists.example.com/listinfo/_xtest@example.com>,
    	<mailto:_xtest-leave@example.com>
    ---end---

If the mailing list has a description, then it is included in the List-Id
header.

    >>> mlist.description = u'My test mailing list'
    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ...
    ... """)
    >>> process(mlist, msg, {})
    >>> list_headers(msg)
    ---start---
    List-Archive: <http://www.example.com/pipermail/_xtest@example.com>
    List-Help: <mailto:_xtest-request@example.com?subject=help>
    List-Id: My test mailing list <_xtest.example.com>
    List-Post: <mailto:_xtest@example.com>
    List-Subscribe: <http://lists.example.com/listinfo/_xtest@example.com>,
    	<mailto:_xtest-join@example.com>
    List-Unsubscribe: <http://lists.example.com/listinfo/_xtest@example.com>,
    	<mailto:_xtest-leave@example.com>
    ---end---

Administrative messages crafted by Mailman will have a reduced set of headers.

    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ...
    ... """)
    >>> process(mlist, msg, dict(reduced_list_headers=True))
    >>> list_headers(msg)
    ---start---
    List-Help: <mailto:_xtest-request@example.com?subject=help>
    List-Id: My test mailing list <_xtest.example.com>
    List-Subscribe: <http://lists.example.com/listinfo/_xtest@example.com>,
    	<mailto:_xtest-join@example.com>
    List-Unsubscribe: <http://lists.example.com/listinfo/_xtest@example.com>,
    	<mailto:_xtest-leave@example.com>
    X-List-Administrivia: yes
    ---end---

With the normal set of List-* headers, it's still possible to suppress the
List-Post header, which is reasonable for an announce only mailing list.

    >>> mlist.include_list_post_header = False
    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ...
    ... """)
    >>> process(mlist, msg, {})
    >>> list_headers(msg)
    ---start---
    List-Archive: <http://www.example.com/pipermail/_xtest@example.com>
    List-Help: <mailto:_xtest-request@example.com?subject=help>
    List-Id: My test mailing list <_xtest.example.com>
    List-Subscribe: <http://lists.example.com/listinfo/_xtest@example.com>,
    	<mailto:_xtest-join@example.com>
    List-Unsubscribe: <http://lists.example.com/listinfo/_xtest@example.com>,
    	<mailto:_xtest-leave@example.com>
    ---end---

And if the list isn't being archived, it makes no sense to add the
List-Archive header either.

    >>> mlist.include_list_post_header = True
    >>> mlist.archive = False
    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ...
    ... """)
    >>> process(mlist, msg, {})
    >>> list_headers(msg)
    ---start---
    List-Help: <mailto:_xtest-request@example.com?subject=help>
    List-Id: My test mailing list <_xtest.example.com>
    List-Post: <mailto:_xtest@example.com>
    List-Subscribe: <http://lists.example.com/listinfo/_xtest@example.com>,
    	<mailto:_xtest-join@example.com>
    List-Unsubscribe: <http://lists.example.com/listinfo/_xtest@example.com>,
    	<mailto:_xtest-leave@example.com>
    ---end---


Archived-At
-----------

RFC 5064 (draft) defines a new Archived-At header which contains the url to
the individual message in the archives.  The stock Pipermail archiver doesn't
support this because the url can't be calculated until after the message is
archived.  Because this is done by the archive runner, this information isn't
available to us now.

    >>> print msg['archived-at']
    None


Personalization
---------------

The To field normally contains the list posting address.  However when
messages are fully personalized, that header will get overwritten with the
address of the recipient.  The list's posting address will be added to one of
the recipient headers so that users will be able to reply back to the list.

    >>> from mailman.interfaces import Personalization, ReplyToMunging
    >>> mlist.personalize = Personalization.full
    >>> mlist.reply_goes_to_list = ReplyToMunging.no_munging
    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ...
    ... """)
    >>> process(mlist, msg, {})
    >>> print msg.as_string()
    From: aperson@example.com
    X-BeenThere: _xtest@example.com
    X-Mailman-Version: ...
    Precedence: list
    Cc: My test mailing list <_xtest@example.com>
    List-Id: My test mailing list <_xtest.example.com>
    List-Unsubscribe: <http://lists.example.com/listinfo/_xtest@example.com>,
    	<mailto:_xtest-leave@example.com>
    List-Post: <mailto:_xtest@example.com>
    List-Help: <mailto:_xtest-request@example.com?subject=help>
    List-Subscribe: <http://lists.example.com/listinfo/_xtest@example.com>,
    	<mailto:_xtest-join@example.com>
    <BLANKLINE>
    <BLANKLINE>