summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
Diffstat
-rw-r--r--src/mailman/config/postfix.cfg5
-rw-r--r--src/mailman/docs/NEWS.rst2
-rw-r--r--src/mailman/docs/mta.rst41
-rw-r--r--src/mailman/handlers/docs/filtering.rst22
-rw-r--r--src/mailman/handlers/mime_delete.py42
-rw-r--r--src/mailman/handlers/tests/data/__init__.py0
-rw-r--r--src/mailman/handlers/tests/data/collapse_alternatives.eml52
-rw-r--r--src/mailman/handlers/tests/data/msg_rfc822.eml183
-rw-r--r--src/mailman/handlers/tests/data/msg_rfc822_out.eml140
-rw-r--r--src/mailman/handlers/tests/test_mimedel.py73
-rw-r--r--src/mailman/mta/postfix.py49
-rw-r--r--src/mailman/mta/tests/test_aliases.py168
-rw-r--r--src/mailman/testing/layers.py1
13 files changed, 693 insertions, 85 deletions
diff --git a/src/mailman/config/postfix.cfg b/src/mailman/config/postfix.cfg
index 9bdba221e..cddda220a 100644
--- a/src/mailman/config/postfix.cfg
+++ b/src/mailman/config/postfix.cfg
@@ -6,3 +6,8 @@
# be appended to this string (with a separating space), so it must be
# appropriate for os.system().
postmap_command: /usr/sbin/postmap
+
+# This variable describes the type of transport maps that will be generated by
+# mailman to be used with postfix for LMTP transport. By default, it is set to
+# hash, but mailman also supports `regex` tables.
+transport_file_type: hash
diff --git a/src/mailman/docs/NEWS.rst b/src/mailman/docs/NEWS.rst
index 0dcba4450..3a2398b74 100644
--- a/src/mailman/docs/NEWS.rst
+++ b/src/mailman/docs/NEWS.rst
@@ -111,6 +111,8 @@ Bugs
(Closes: #255)
* Update documentation links for ``config.cfg`` settings. (Closes: #306)
* Disallow problematic characters in listnames. (Closes: #311)
+ * Forward port several content filtering fixes from the 2.1 branch.
+ (Closes: #330, #331, #332 and #334)
Configuration
-------------
diff --git a/src/mailman/docs/mta.rst b/src/mailman/docs/mta.rst
index 1d2bbc268..54095bc68 100644
--- a/src/mailman/docs/mta.rst
+++ b/src/mailman/docs/mta.rst
@@ -145,9 +145,12 @@ Transport maps
By default, Mailman works well with Postfix transport maps as a way to deliver
incoming messages to Mailman's LMTP server. Mailman will automatically write
the correct transport map when its ``mailman aliases`` command is run, or
-whenever a mailing list is created or removed via other commands. To connect
-Postfix to Mailman's LMTP server, add the following to Postfix's ``main.cf``
-file::
+whenever a mailing list is created or removed via other commands. Mailman
+supports two type of transport map tables for Postfix, namely ``hash`` and
+``regexp``. Tables using hash are processed by ``postmap`` command. To use this
+format, you should have ``postmap`` command available on the host running
+Mailman. It is also the default one of the two. To connect Postfix to
+Mailman's LMTP server, add the following to Postfix's ``main.cf`` file::
transport_maps =
hash:/path-to-mailman/var/data/postfix_lmtp
@@ -167,6 +170,36 @@ strictly needed (but it is harmless). All you need to do in this scenario is
to make sure that Postfix accepts mail for your one domain, normally by
including it in ``mydestination``.
+Regular Expression tables remove the additional dependency of having ``postmap``
+command available to Mailman. If you want to use ``regexp`` or Regular
+Expression tables, then add the following to Postfix's ``main.cf`` file::
+
+ transport_maps =
+ regexp:/path-to-mailman/var/data/postfix_lmtp
+ local_recipient_maps =
+ regexp:/path-to-mailman/var/data/postfix_lmtp
+ relay_domains =
+ regexp:/path-to-mailman/var/data/postfix_domains
+
+You will also have to instruct Mailman to generate regexp tables instead of hash
+tables by adding the following configuration to ``mailman.cfg``::
+
+ [mta]
+ incoming: mailman.mta.postfix.LMTP
+ outgoing: mailman.mta.deliver.deliver
+ lmtp_host: mail.example.com
+ lmtp_port: 8024
+ smtp_host: mail.example.com
+ smtp_port: 25
+ configuration: /path/to/postfix-mailman.cfg
+
+Also you will have to create another configuration file called as
+``postfix-mailman.cfg`` and add its path to the ``configuration`` parameter
+above. The ``postfix-mailman.cfg`` would look like this::
+
+ [postfix]
+ transport_file_type: regex
+
Postfix documentation
---------------------
@@ -339,7 +372,7 @@ the user ``mailman``, qmail will give you the destination address
``mailman-spam@lists.example.com`` while it should actually be
``spam@lists.example.com``. The second argument to ``qmail-lmtp`` defines
how many parts (separated by dashes) to filter out. The first argument
-specifies the LMTP port of mailman. Long story short, as user mailman:
+specifies the LMTP port of Mailman. Long story short, as user mailman:
::
% chmod +t "$HOME"
diff --git a/src/mailman/handlers/docs/filtering.rst b/src/mailman/handlers/docs/filtering.rst
index 427db4273..2d2480555 100644
--- a/src/mailman/handlers/docs/filtering.rst
+++ b/src/mailman/handlers/docs/filtering.rst
@@ -76,7 +76,8 @@ Simple multipart filtering
==========================
If one of the subparts in a ``multipart`` message matches the filter type,
-then just that subpart will be stripped.
+then just that subpart will be stripped. If that leaves just a single subpart,
+the ``multipart`` will be replaced by the subpart.
::
>>> msg = message_from_string("""\
@@ -101,17 +102,11 @@ then just that subpart will be stripped.
>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.com
- Content-Type: multipart/mixed; boundary=BOUNDARY
MIME-Version: 1.0
- X-Content-Filtered-By: Mailman/MimeDel ...
- <BLANKLINE>
- --BOUNDARY
Content-Type: image/gif
- MIME-Version: 1.0
+ X-Content-Filtered-By: Mailman/MimeDel ...
<BLANKLINE>
yyy
- --BOUNDARY--
- <BLANKLINE>
Collapsing multipart/alternative messages
@@ -128,7 +123,8 @@ Content filtering will remove the jpeg part, leaving the
``multipart/alternative`` with only a single gif subpart. Because there's
only one subpart left, the MIME structure of the message will be reorganized,
removing the inner ``multipart/alternative`` so that the outer
-``multipart/mixed`` has just a single gif subpart.
+``multipart/mixed`` has just a single gif subpart, and then the multipart is
+recast as just the subpart.
>>> mlist.collapse_alternatives = True
>>> msg = message_from_string("""\
@@ -158,17 +154,11 @@ removing the inner ``multipart/alternative`` so that the outer
>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.com
- Content-Type: multipart/mixed; boundary=BOUNDARY
MIME-Version: 1.0
- X-Content-Filtered-By: Mailman/MimeDel ...
- <BLANKLINE>
- --BOUNDARY
Content-Type: image/gif
- MIME-Version: 1.0
+ X-Content-Filtered-By: Mailman/MimeDel ...
<BLANKLINE>
yyy
- --BOUNDARY--
- <BLANKLINE>
When the outer part is a ``multipart/alternative`` and filtering leaves this
outer part with just one subpart, the entire message is converted to the left
diff --git a/src/mailman/handlers/mime_delete.py b/src/mailman/handlers/mime_delete.py
index de1e92d1c..14aee1a0e 100644
--- a/src/mailman/handlers/mime_delete.py
+++ b/src/mailman/handlers/mime_delete.py
@@ -139,6 +139,11 @@ def process(mlist, msg, msgdata):
if ctype == 'multipart/alternative':
firstalt = msg.get_payload(0)
reset_payload(msg, firstalt)
+ # Now that we've collapsed the MPA parts, go through the message
+ # and recast any multipart parts with only one sub-part as just
+ # the sub-part.
+ if msg.is_multipart():
+ recast_multipart(msg)
# If we removed some parts, make note of this
changedp = 0
if numparts != len([subpart for subpart in msg.walk()]):
@@ -222,12 +227,43 @@ def collapse_multipart_alternatives(msg):
if subpart.get_content_type() == 'multipart/alternative':
with suppress(IndexError):
firstalt = subpart.get_payload(0)
- newpayload.append(firstalt)
+ if msg.get_content_type() == 'message/rfc822':
+ # This is a multipart/alternative message in a
+ # message/rfc822 part. We treat it specially so as not to
+ # lose the headers.
+ reset_payload(subpart, firstalt)
+ newpayload.append(subpart)
+ else:
+ newpayload.append(firstalt)
+ elif subpart.is_multipart():
+ collapse_multipart_alternatives(subpart)
+ newpayload.append(subpart)
else:
newpayload.append(subpart)
msg.set_payload(newpayload)
+def recast_multipart(msg):
+ # If we're left with a multipart message with only one sub-part, recast
+ # the message to just the sub-part, but not if the part is message/rfc822
+ # because we don't want to lose the headers.
+ # Also, if this is a multipart/signed part, stop now as the original part
+ # may have had a multipart sub-part with only one sub-sub-part, the sig
+ # may still be valid and going further may break it. (LP: #1551075)
+ if msg.get_content_type() == 'multipart/signed':
+ return
+ if msg.is_multipart():
+ if (len(msg.get_payload()) == 1 and
+ msg.get_content_type() != 'message/rfc822'):
+ reset_payload(msg, msg.get_payload(0))
+ # now that we've recast this part, check the subordinate parts
+ recast_multipart(msg)
+ else:
+ # This part's OK but check deeper.
+ for part in msg.get_payload():
+ recast_multipart(part)
+
+
def to_plaintext(msg):
changedp = 0
counter = count()
@@ -263,12 +299,12 @@ def get_file_ext(m):
fext = ''
filename = m.get_filename('') or m.get_param('name', '')
if filename:
- fext = os.path.splitext(oneline(filename, 'utf-8'))[1]
+ fext = os.path.splitext(oneline(filename, 'utf-8', in_unicode=True))[1]
if len(fext) > 1:
fext = fext[1:]
else:
fext = ''
- return fext
+ return fext.lower()
@public
diff --git a/src/mailman/handlers/tests/data/__init__.py b/src/mailman/handlers/tests/data/__init__.py
new file mode 100644
index 000000000..e69de29bb
--- /dev/null
+++ b/src/mailman/handlers/tests/data/__init__.py
diff --git a/src/mailman/handlers/tests/data/collapse_alternatives.eml b/src/mailman/handlers/tests/data/collapse_alternatives.eml
new file mode 100644
index 000000000..38ee3524e
--- /dev/null
+++ b/src/mailman/handlers/tests/data/collapse_alternatives.eml
@@ -0,0 +1,52 @@
+To: mailman-users@mailman3.org
+Message-ID: <5847c753-d3c5-23ac-edbd-c102999636a2@example.com>
+Date: Fri, 19 May 2017 12:58:25 +0200
+MIME-Version: 1.0
+Content-Type: multipart/signed; micalg=pgp-sha256;
+ protocol="application/pgp-signature";
+ boundary="vFaMjPs797GQgsvxAoMXW6m0wITFbm1Hh"
+Subject: Test multipart alternative
+From: user@example.com
+
+This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
+--vFaMjPs797GQgsvxAoMXW6m0wITFbm1Hh
+Content-Type: multipart/mixed; boundary="hFtv7Gl3eCgubGHcAccJQEFmFcP6jKVWP";
+ protected-headers="v1"
+
+--hFtv7Gl3eCgubGHcAccJQEFmFcP6jKVWP
+Content-Type: multipart/alternative;
+ boundary="------------826A33926CBE213C2D92E2EE"
+
+This is a multi-part message in MIME format.
+--------------826A33926CBE213C2D92E2EE
+Content-Type: text/plain; charset=utf-8
+
+This is the plain text
+
+--------------826A33926CBE213C2D92E2EE
+Content-Type: text/html; charset="utf-8"
+
+<html><head></head><body>
+This is the html.
+</body></html>
+
+--------------826A33926CBE213C2D92E2EE--
+
+--hFtv7Gl3eCgubGHcAccJQEFmFcP6jKVWP
+Content-Type: text/plain; charset="utf-8"
+
+This is just another part
+
+--hFtv7Gl3eCgubGHcAccJQEFmFcP6jKVWP--
+
+--vFaMjPs797GQgsvxAoMXW6m0wITFbm1Hh
+Content-Type: application/pgp-signature; name="signature.asc"
+Content-Description: OpenPGP digital signature
+Content-Disposition: attachment; filename="signature.asc"
+
+-----BEGIN PGP SIGNATURE-----
+
+some data
+-----END PGP SIGNATURE-----
+
+--vFaMjPs797GQgsvxAoMXW6m0wITFbm1Hh--
diff --git a/src/mailman/handlers/tests/data/msg_rfc822.eml b/src/mailman/handlers/tests/data/msg_rfc822.eml
new file mode 100644
index 000000000..dda65095c
--- /dev/null
+++ b/src/mailman/handlers/tests/data/msg_rfc822.eml
@@ -0,0 +1,183 @@
+Message-ID: <4D9E6AEA.1060802@example.net>
+Date: Thu, 07 Apr 2011 18:54:50 -0700
+From: User <user@example.com>
+MIME-Version: 1.0
+To: Someone <someone@example.net>
+Subject: Message Subject
+Content-Type: multipart/mixed;
+ boundary="------------050603050603060608020908"
+
+This is a multi-part message in MIME format.
+--------------050603050603060608020908
+Content-Type: text/plain; charset=ISO-8859-1
+Content-Transfer-Encoding: 7bit
+
+Plain body.
+
+--------------050603050603060608020908
+Content-Type: message/rfc822
+Content-Transfer-Encoding: 7bit
+Content-Disposition: attachment
+
+Message-ID: <4D9E647F.4050308@example.net>
+Date: Thu, 07 Apr 2011 18:27:27 -0700
+From: User1 <user1@example.com>
+MIME-Version: 1.0
+To: Someone1 <someone1@example.net>
+Content-Type: multipart/mixed; boundary="------------060107040402070208020705"
+Subject: Attached Message 1 Subject
+
+This is a multi-part message in MIME format.
+--------------060107040402070208020705
+Content-Type: text/plain; charset=ISO-8859-1
+Content-Transfer-Encoding: 7bit
+
+Attached Message 1 body.
+
+--------------060107040402070208020705
+Content-Type: message/rfc822
+Content-Transfer-Encoding: 7bit
+Content-Disposition: attachment
+
+From: User2 <user2@example.com>
+To: Someone2 <someone2@example.net>
+Subject: Attached Message 2 Subject
+Date: Thu, 7 Apr 2011 19:09:35 -0500
+Message-ID: <DAE689E1FD1D493BACD15180145B4151@example.net>
+MIME-Version: 1.0
+Content-Type: multipart/mixed;
+ boundary="----=_NextPart_000_0066_01CBF557.56C6F370"
+
+This is a multi-part message in MIME format.
+
+------=_NextPart_000_0066_01CBF557.56C6F370
+Content-Type: text/plain;
+ charset="us-ascii"
+Content-Transfer-Encoding: 7bit
+
+Attached Message 2 body.
+
+------=_NextPart_000_0066_01CBF557.56C6F370
+Content-Type: message/rfc822
+Content-Transfer-Encoding: 7bit
+Content-Disposition: attachment
+
+From: User3 <user3@example.com>
+To: Someone3 <someone3@example.net>
+Subject: Attached Message 3 Subject
+Date: Thu, 7 Apr 2011 17:22:04 -0500
+Message-ID: <BANLkTi=SzfNJo-V7cvrg3nE3uOi9uxXv3g@example.net>
+MIME-Version: 1.0
+Content-Type: multipart/alternative;
+ boundary="----=_NextPart_000_0058_01CBF557.56C48270"
+
+This is a multi-part message in MIME format.
+
+------=_NextPart_000_0058_01CBF557.56C48270
+Content-Type: text/plain;
+ charset="iso-8859-1"
+Content-Transfer-Encoding: 7bit
+
+Attached Message 3 plain body.
+
+------=_NextPart_000_0058_01CBF557.56C48270
+Content-Type: text/html;
+ charset="iso-8859-1"
+Content-Transfer-Encoding: quoted-printable
+
+
+Attached Message 3 html body.
+
+------=_NextPart_000_0058_01CBF557.56C48270--
+
+------=_NextPart_000_0066_01CBF557.56C6F370
+Content-Type: message/rfc822
+Content-Transfer-Encoding: 7bit
+Content-Disposition: attachment
+
+From: User4 <user4@example.com>
+To: Someone4 <someone4@example.net>
+Subject: Attached Message 4 Subject
+Date: Thu, 7 Apr 2011 17:24:26 -0500
+Message-ID: <19CC3BDF28CF49AD988FF43B2DBC5F1D@example>
+MIME-Version: 1.0
+Content-Type: multipart/mixed;
+ boundary="----=_NextPart_000_0060_01CBF557.56C6F370"
+
+This is a multi-part message in MIME format.
+
+------=_NextPart_000_0060_01CBF557.56C6F370
+Content-Type: multipart/alternative;
+ boundary="----=_NextPart_001_0061_01CBF557.56C6F370"
+
+------=_NextPart_001_0061_01CBF557.56C6F370
+Content-Type: text/plain;
+ charset="us-ascii"
+Content-Transfer-Encoding: 7bit
+
+Attached Message 4 plain body.
+
+------=_NextPart_001_0061_01CBF557.56C6F370
+Content-Type: text/html;
+ charset="us-ascii"
+Content-Transfer-Encoding: quoted-printable
+
+Attached Message 4 html body.
+
+------=_NextPart_001_0061_01CBF557.56C6F370--
+
+------=_NextPart_000_0060_01CBF557.56C6F370
+Content-Type: message/rfc822
+Content-Transfer-Encoding: 7bit
+Content-Disposition: attachment
+
+From: User5 <user5@example.com>
+To: Someone5 <someone5@example.net>
+Subject: Attached Message 5 Subject
+Date: Thu, 7 Apr 2011 16:24:26 -0500
+Message-ID: <some_id@example>
+Content-Type: multipart/alternative;
+ boundary="----=_NextPart_000_005C_01CBF557.56C6F370"
+
+This is a multi-part message in MIME format.
+
+------=_NextPart_000_005C_01CBF557.56C6F370
+Content-Type: text/plain;
+ charset="iso-8859-1"
+Content-Transfer-Encoding: 7bit
+
+Attached Message 5 plain body.
+
+------=_NextPart_000_005C_01CBF557.56C6F370
+Content-Type: text/html;
+ charset="iso-8859-1"
+Content-Transfer-Encoding: quoted-printable
+
+Attached Message 5 html body.
+
+------=_NextPart_000_005C_01CBF557.56C6F370--
+
+------=_NextPart_000_0060_01CBF557.56C6F370
+Content-Type: text/plain;
+ name="ATT00055.txt"
+Content-Transfer-Encoding: quoted-printable
+Content-Disposition: attachment;
+ filename="ATT00055.txt"
+
+Another plain part.
+
+------=_NextPart_000_0060_01CBF557.56C6F370--
+
+------=_NextPart_000_0066_01CBF557.56C6F370--
+
+--------------060107040402070208020705
+Content-Type: text/plain; charset="us-ascii"
+MIME-Version: 1.0
+Content-Transfer-Encoding: 7bit
+Content-Disposition: inline
+
+Final plain part.
+
+--------------060107040402070208020705--
+
+--------------050603050603060608020908--
diff --git a/src/mailman/handlers/tests/data/msg_rfc822_out.eml b/src/mailman/handlers/tests/data/msg_rfc822_out.eml
new file mode 100644
index 000000000..a0f7c632d
--- /dev/null
+++ b/src/mailman/handlers/tests/data/msg_rfc822_out.eml
@@ -0,0 +1,140 @@
+Message-ID: <4D9E6AEA.1060802@example.net>
+Date: Thu, 07 Apr 2011 18:54:50 -0700
+From: User <user@example.com>
+MIME-Version: 1.0
+To: Someone <someone@example.net>
+Subject: Message Subject
+Content-Type: multipart/mixed;
+ boundary="------------050603050603060608020908"
+X-Content-Filtered-By: Mailman/MimeDel 3.1.0b5
+
+This is a multi-part message in MIME format.
+--------------050603050603060608020908
+Content-Type: text/plain; charset=ISO-8859-1
+Content-Transfer-Encoding: 7bit
+
+Plain body.
+
+--------------050603050603060608020908
+Content-Type: message/rfc822
+Content-Transfer-Encoding: 7bit
+Content-Disposition: attachment
+
+Message-ID: <4D9E647F.4050308@example.net>
+Date: Thu, 07 Apr 2011 18:27:27 -0700
+From: User1 <user1@example.com>
+MIME-Version: 1.0
+To: Someone1 <someone1@example.net>
+Content-Type: multipart/mixed; boundary="------------060107040402070208020705"
+Subject: Attached Message 1 Subject
+
+This is a multi-part message in MIME format.
+--------------060107040402070208020705
+Content-Type: text/plain; charset=ISO-8859-1
+Content-Transfer-Encoding: 7bit
+
+Attached Message 1 body.
+
+--------------060107040402070208020705
+Content-Type: message/rfc822
+Content-Transfer-Encoding: 7bit
+Content-Disposition: attachment
+
+From: User2 <user2@example.com>
+To: Someone2 <someone2@example.net>
+Subject: Attached Message 2 Subject
+Date: Thu, 7 Apr 2011 19:09:35 -0500
+Message-ID: <DAE689E1FD1D493BACD15180145B4151@example.net>
+MIME-Version: 1.0
+Content-Type: multipart/mixed;
+ boundary="----=_NextPart_000_0066_01CBF557.56C6F370"
+
+This is a multi-part message in MIME format.
+
+------=_NextPart_000_0066_01CBF557.56C6F370
+Content-Type: text/plain;
+ charset="us-ascii"
+Content-Transfer-Encoding: 7bit
+
+Attached Message 2 body.
+
+------=_NextPart_000_0066_01CBF557.56C6F370
+Content-Type: message/rfc822
+Content-Transfer-Encoding: 7bit
+Content-Disposition: attachment
+
+From: User3 <user3@example.com>
+To: Someone3 <someone3@example.net>
+Subject: Attached Message 3 Subject
+Date: Thu, 7 Apr 2011 17:22:04 -0500
+Message-ID: <BANLkTi=SzfNJo-V7cvrg3nE3uOi9uxXv3g@example.net>
+MIME-Version: 1.0
+Content-Type: text/plain;
+ charset="iso-8859-1"
+Content-Transfer-Encoding: 7bit
+
+Attached Message 3 plain body.
+
+------=_NextPart_000_0066_01CBF557.56C6F370
+Content-Type: message/rfc822
+Content-Transfer-Encoding: 7bit
+Content-Disposition: attachment
+
+From: User4 <user4@example.com>
+To: Someone4 <someone4@example.net>
+Subject: Attached Message 4 Subject
+Date: Thu, 7 Apr 2011 17:24:26 -0500
+Message-ID: <19CC3BDF28CF49AD988FF43B2DBC5F1D@example>
+MIME-Version: 1.0
+Content-Type: multipart/mixed;
+ boundary="----=_NextPart_000_0060_01CBF557.56C6F370"
+
+This is a multi-part message in MIME format.
+
+------=_NextPart_000_0060_01CBF557.56C6F370
+Content-Type: text/plain;
+ charset="us-ascii"
+Content-Transfer-Encoding: 7bit
+
+Attached Message 4 plain body.
+
+------=_NextPart_000_0060_01CBF557.56C6F370
+Content-Type: message/rfc822
+Content-Transfer-Encoding: 7bit
+Content-Disposition: attachment
+
+From: User5 <user5@example.com>
+To: Someone5 <someone5@example.net>
+Subject: Attached Message 5 Subject
+Date: Thu, 7 Apr 2011 16:24:26 -0500
+Message-ID: <some_id@example>
+Content-Type: text/plain;
+ charset="iso-8859-1"
+Content-Transfer-Encoding: 7bit
+
+Attached Message 5 plain body.
+
+------=_NextPart_000_0060_01CBF557.56C6F370
+Content-Type: text/plain;
+ name="ATT00055.txt"
+Content-Transfer-Encoding: quoted-printable
+Content-Disposition: attachment;
+ filename="ATT00055.txt"
+
+Another plain part.
+
+------=_NextPart_000_0060_01CBF557.56C6F370--
+
+------=_NextPart_000_0066_01CBF557.56C6F370--
+
+--------------060107040402070208020705
+Content-Type: text/plain; charset="us-ascii"
+MIME-Version: 1.0
+Content-Transfer-Encoding: 7bit
+Content-Disposition: inline
+
+Final plain part.
+
+--------------060107040402070208020705--
+
+--------------050603050603060608020908--
diff --git a/src/mailman/handlers/tests/test_mimedel.py b/src/mailman/handlers/tests/test_mimedel.py
index 5e536134f..01cac0f20 100644
--- a/src/mailman/handlers/tests/test_mimedel.py
+++ b/src/mailman/handlers/tests/test_mimedel.py
@@ -19,11 +19,13 @@
import os
import sys
+import email
import shutil
import tempfile
import unittest
from contextlib import ExitStack, contextmanager
+from io import StringIO
from mailman.app.lifecycle import create_list
from mailman.config import config
from mailman.handlers import mime_delete
@@ -35,6 +37,7 @@ from mailman.testing.helpers import (
LogFileMark, configuration, get_queue_messages,
specialized_message_from_string as mfs)
from mailman.testing.layers import ConfigLayer
+from pkg_resources import resource_filename
from zope.component import getUtility
@@ -215,3 +218,73 @@ MIME-Version: 1.0
msg['x-content-filtered-by'].startswith('Mailman/MimeDel'))
payload_lines = msg.get_payload().splitlines()
self.assertEqual(payload_lines[0], 'Converted text/html to text/plain')
+
+
+class TestMiscellaneous(unittest.TestCase):
+ """Test various miscellaneous filtering actions."""
+
+ layer = ConfigLayer
+
+ def setUp(self):
+ self._mlist = create_list('test@example.com')
+ self._mlist.collapse_alternatives = True
+ self._mlist.filter_content = True
+ self._mlist.filter_extensions = ['xlsx']
+
+ def test_collapse_alternatives(self):
+ email_file = resource_filename(
+ 'mailman.handlers.tests.data', 'collapse_alternatives.eml')
+ with open(email_file) as fp:
+ msg = email.message_from_file(fp)
+ process = config.handlers['mime-delete'].process
+ process(self._mlist, msg, {})
+ structure = StringIO()
+ email.iterators._structure(msg, fp=structure)
+ self.assertEqual(structure.getvalue(), """\
+multipart/signed
+ multipart/mixed
+ text/plain
+ text/plain
+ application/pgp-signature
+""")
+
+ def test_msg_rfc822(self):
+ email_file = resource_filename(
+ 'mailman.handlers.tests.data', 'msg_rfc822.eml')
+ email_file2 = resource_filename(
+ 'mailman.handlers.tests.data', 'msg_rfc822_out.eml')
+ with open(email_file) as fp:
+ msg = email.message_from_file(fp)
+ process = config.handlers['mime-delete'].process
+ process(self._mlist, msg, {})
+ with open(email_file2) as fp:
+ self.assertEqual(msg.as_string(), fp.read())
+
+ def test_mixed_case_ext_and_recast(self):
+ msg = mfs("""\
+From: anne@example.com
+To: test@example.com
+Subject: Testing mixed extension
+Message-ID: <ant>
+MIME-Version: 1.0
+Content-Type: multipart/mixed; boundary="AAAA"
+
+--AAAA
+Content-Type: text/plain; charset="utf-8"
+
+Plain text
+
+--AAAA
+Content-Type: application/octet-stream; name="test.xlsX"
+Content-Disposition: attachment; filename="test.xlsX"
+
+spreadsheet
+
+--AAAA--
+""")
+ process = config.handlers['mime-delete'].process
+ process(self._mlist, msg, {})
+ self.assertEqual(msg['content-type'], 'text/plain; charset="utf-8"')
+ self.assertEqual(msg.get_payload(), """\
+Plain text
+""")
diff --git a/src/mailman/mta/postfix.py b/src/mailman/mta/postfix.py
index d819533a3..6e71d0651 100644
--- a/src/mailman/mta/postfix.py
+++ b/src/mailman/mta/postfix.py
@@ -56,6 +56,8 @@ class LMTP:
# Locate and read the Postfix specific configuration file.
mta_config = external_configuration(config.mta.configuration)
self.postmap_command = mta_config.get('postfix', 'postmap_command')
+ self.transport_file_type = mta_config.get(
+ 'postfix', 'transport_file_type')
def create(self, mlist):
"""See `IMailTransportAgentLifecycle`."""
@@ -84,19 +86,21 @@ class LMTP:
self._generate_domains_file(fp)
# Atomically rename to the intended path.
os.rename(domains_path_new, domains_path)
- # Now, run the postmap command on both newly generated files. If
- # one files, still try the other one.
- errors = []
- for path in (lmtp_path, domains_path):
- command = self.postmap_command + ' ' + path
- status = (os.system(command) >> 8) & 0xff
- if status:
- msg = 'command failure: %s, %s, %s'
- errstr = os.strerror(status)
- log.error(msg, command, status, errstr)
- errors.append(msg % (command, status, errstr))
- if errors:
- raise RuntimeError(NL.join(errors))
+ # If the transport_file_type is 'hash' then run the postmap command
+ # on newly generated file to convert them in to hash table like
+ # Postfix wants.
+ if self.transport_file_type == 'hash':
+ errors = []
+ for path in (lmtp_path, domains_path):
+ command = self.postmap_command + ' ' + path
+ status = (os.system(command) >> 8) & 0xff
+ if status:
+ msg = 'command failure: %s, %s, %s'
+ errstr = os.strerror(status)
+ log.error(msg, command, status, errstr)
+ errors.append(msg % (command, status, errstr))
+ if errors:
+ raise RuntimeError(NL.join(errors))
def _generate_lmtp_file(self, fp):
# The format for Postfix's LMTP transport map is defined here:
@@ -125,12 +129,23 @@ class LMTP:
file=fp)
for mlist in sorted(by_domain[domain], key=sort_key):
aliases = list(utility.aliases(mlist))
- width = max(len(alias) for alias in aliases) + 3
- print(ALIASTMPL.format(aliases.pop(0), config, width), file=fp)
+ width = max(len(alias) for alias in aliases) + \
+ aliases[0].count('.') + 7
+ print(ALIASTMPL.format(self._decorate(aliases.pop(0)),
+ config, width), file=fp)
for alias in aliases:
- print(ALIASTMPL.format(alias, config, width), file=fp)
+ print(ALIASTMPL.format(self._decorate(alias),
+ config, width), file=fp)
print(file=fp)
+ def _decorate(self, name):
+ # Postfix regex tables need regex matching listname or domains. This
+ # method just decorates the name to be printed in the transport map
+ # file or relay domains file.
+ if self.transport_file_type == 'regex':
+ return '/^{}$/'.format(name).replace('.', '\.')
+ return name
+
def _generate_domains_file(self, fp):
# Uniquify the domains, then sort them alphabetically.
domains = set()
@@ -145,5 +160,5 @@ class LMTP:
# you're on your own.
""".format(now().replace(microsecond=0)), file=fp)
for domain in sorted(domains):
- print('{0} {0}'.format(domain), file=fp)
+ print('{} {}'.format(self._decorate(domain), domain), file=fp)
print(file=fp)
diff --git a/src/mailman/mta/tests/test_aliases.py b/src/mailman/mta/tests/test_aliases.py
index d5cea3ecb..b8090c86d 100644
--- a/src/mailman/mta/tests/test_aliases.py
+++ b/src/mailman/mta/tests/test_aliases.py
@@ -154,15 +154,15 @@ example.com example.com
contents = _strip_header(fp.read())
self.assertMultiLineEqual(contents, """\
# Aliases which are visible only in the @example.com domain.
-test@example.com lmtp:[127.0.0.1]:9024
-test-bounces@example.com lmtp:[127.0.0.1]:9024
-test-confirm@example.com lmtp:[127.0.0.1]:9024
-test-join@example.com lmtp:[127.0.0.1]:9024
-test-leave@example.com lmtp:[127.0.0.1]:9024
-test-owner@example.com lmtp:[127.0.0.1]:9024
-test-request@example.com lmtp:[127.0.0.1]:9024
-test-subscribe@example.com lmtp:[127.0.0.1]:9024
-test-unsubscribe@example.com lmtp:[127.0.0.1]:9024
+test@example.com lmtp:[127.0.0.1]:9024
+test-bounces@example.com lmtp:[127.0.0.1]:9024
+test-confirm@example.com lmtp:[127.0.0.1]:9024
+test-join@example.com lmtp:[127.0.0.1]:9024
+test-leave@example.com lmtp:[127.0.0.1]:9024
+test-owner@example.com lmtp:[127.0.0.1]:9024
+test-request@example.com lmtp:[127.0.0.1]:9024
+test-subscribe@example.com lmtp:[127.0.0.1]:9024
+test-unsubscribe@example.com lmtp:[127.0.0.1]:9024
""")
def test_two_lists(self):
@@ -185,25 +185,25 @@ example.com example.com
contents = _strip_header(fp.read())
self.assertMultiLineEqual(contents, """\
# Aliases which are visible only in the @example.com domain.
-other@example.com lmtp:[127.0.0.1]:9024
-other-bounces@example.com lmtp:[127.0.0.1]:9024
-other-confirm@example.com lmtp:[127.0.0.1]:9024
-other-join@example.com lmtp:[127.0.0.1]:9024
-other-leave@example.com lmtp:[127.0.0.1]:9024
-other-owner@example.com lmtp:[127.0.0.1]:9024
-other-request@example.com lmtp:[127.0.0.1]:9024
-other-subscribe@example.com lmtp:[127.0.0.1]:9024
-other-unsubscribe@example.com lmtp:[127.0.0.1]:9024
+other@example.com lmtp:[127.0.0.1]:9024
+other-bounces@example.com lmtp:[127.0.0.1]:9024
+other-confirm@example.com lmtp:[127.0.0.1]:9024
+other-join@example.com lmtp:[127.0.0.1]:9024
+other-leave@example.com lmtp:[127.0.0.1]:9024
+other-owner@example.com lmtp:[127.0.0.1]:9024
+other-request@example.com lmtp:[127.0.0.1]:9024
+other-subscribe@example.com lmtp:[127.0.0.1]:9024
+other-unsubscribe@example.com lmtp:[127.0.0.1]:9024
-test@example.com lmtp:[127.0.0.1]:9024
-test-bounces@example.com lmtp:[127.0.0.1]:9024
-test-confirm@example.com lmtp:[127.0.0.1]:9024
-test-join@example.com lmtp:[127.0.0.1]:9024
-test-leave@example.com lmtp:[127.0.0.1]:9024
-test-owner@example.com lmtp:[127.0.0.1]:9024
-test-request@example.com lmtp:[127.0.0.1]:9024
-test-subscribe@example.com lmtp:[127.0.0.1]:9024
-test-unsubscribe@example.com lmtp:[127.0.0.1]:9024
+test@example.com lmtp:[127.0.0.1]:9024
+test-bounces@example.com lmtp:[127.0.0.1]:9024
+test-confirm@example.com lmtp:[127.0.0.1]:9024
+test-join@example.com lmtp:[127.0.0.1]:9024
+test-leave@example.com lmtp:[127.0.0.1]:9024
+test-owner@example.com lmtp:[127.0.0.1]:9024
+test-request@example.com lmtp:[127.0.0.1]:9024
+test-subscribe@example.com lmtp:[127.0.0.1]:9024
+test-unsubscribe@example.com lmtp:[127.0.0.1]:9024
""")
def test_two_lists_two_domains(self):
@@ -229,24 +229,102 @@ example.net example.net
contents = _strip_header(fp.read())
self.assertMultiLineEqual(contents, """\
# Aliases which are visible only in the @example.com domain.
-test@example.com lmtp:[127.0.0.1]:9024
-test-bounces@example.com lmtp:[127.0.0.1]:9024
-test-confirm@example.com lmtp:[127.0.0.1]:9024
-test-join@example.com lmtp:[127.0.0.1]:9024
-test-leave@example.com lmtp:[127.0.0.1]:9024
-test-owner@example.com lmtp:[127.0.0.1]:9024
-test-request@example.com lmtp:[127.0.0.1]:9024
-test-subscribe@example.com lmtp:[127.0.0.1]:9024
-test-unsubscribe@example.com lmtp:[127.0.0.1]:9024
+test@example.com lmtp:[127.0.0.1]:9024
+test-bounces@example.com lmtp:[127.0.0.1]:9024
+test-confirm@example.com lmtp:[127.0.0.1]:9024
+test-join@example.com lmtp:[127.0.0.1]:9024
+test-leave@example.com lmtp:[127.0.0.1]:9024
+test-owner@example.com lmtp:[127.0.0.1]:9024
+test-request@example.com lmtp:[127.0.0.1]:9024
+test-subscribe@example.com lmtp:[127.0.0.1]:9024
+test-unsubscribe@example.com lmtp:[127.0.0.1]:9024
# Aliases which are visible only in the @example.net domain.
-other@example.net lmtp:[127.0.0.1]:9024
-other-bounces@example.net lmtp:[127.0.0.1]:9024
-other-confirm@example.net lmtp:[127.0.0.1]:9024
-other-join@example.net lmtp:[127.0.0.1]:9024
-other-leave@example.net lmtp:[127.0.0.1]:9024
-other-owner@example.net lmtp:[127.0.0.1]:9024
-other-request@example.net lmtp:[127.0.0.1]:9024
-other-subscribe@example.net lmtp:[127.0.0.1]:9024
-other-unsubscribe@example.net lmtp:[127.0.0.1]:9024
+other@example.net lmtp:[127.0.0.1]:9024
+other-bounces@example.net lmtp:[127.0.0.1]:9024
+other-confirm@example.net lmtp:[127.0.0.1]:9024
+other-join@example.net lmtp:[127.0.0.1]:9024
+other-leave@example.net lmtp:[127.0.0.1]:9024
+other-owner@example.net lmtp:[127.0.0.1]:9024
+other-request@example.net lmtp:[127.0.0.1]:9024
+other-subscribe@example.net lmtp:[127.0.0.1]:9024
+other-unsubscribe@example.net lmtp:[127.0.0.1]:9024
+""")
+
+ def test_missing_postmap_command_raises_runtime_errorr(self):
+ # Changing the postmap command to false will always
+ # return a non-zero exit code.
+ self.postfix.postmap_command = 'false'
+ # Generating postmap hash files will raise a runtimerror.
+ with self.assertRaises(RuntimeError):
+ self.postfix.regenerate(self.tempdir)
+ # Now change the command back to true will make the
+ # command run normally.
+ self.postfix.postmap_command = 'true'
+ self.postfix.regenerate(self.tempdir)
+ # There should be two files in the tempdir.
+ self.assertEqual(sorted(os.listdir(self.tempdir)),
+ ['postfix_domains', 'postfix_lmtp'])
+
+ def test_aliases_regex(self):
+ # Test aliases generation for regex maps for postfix.
+ # Set the transport map type to regex.
+ self.postfix.transport_file_type = 'regex'
+ self.postfix.regenerate(self.tempdir)
+ # The domains file just contains the example.com domain.
+ with open(os.path.join(self.tempdir, 'postfix_domains')) as fp:
+ contents = _strip_header(fp.read())
+ self.assertMultiLineEqual(contents, """\
+/^example\.com$/ example.com
+""")
+
+ # the lmtp file contains transport mapping to the lmtp server.
+ with open(os.path.join(self.tempdir, 'postfix_lmtp')) as fp:
+ contents = _strip_header(fp.read())
+ self.assertMultiLineEqual(contents, """\
+# Aliases which are visible only in the @example.com domain.
+/^test@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-bounces@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-confirm@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-join@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-leave@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-owner@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-request@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-subscribe@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-unsubscribe@example\.com$/ lmtp:[127.0.0.1]:9024
+""")
+
+ def test_aliases_regex_with_dots(self):
+ # Test regex is generated for listnames with multiple dots.
+ self.postfix.transport_file_type = 'regex'
+ create_list('test.list.name.dots@example.com')
+ self.postfix.regenerate(self.tempdir)
+ with open(os.path.join(self.tempdir, 'postfix_domains')) as fp:
+ contents = _strip_header(fp.read())
+ self.assertMultiLineEqual(contents, """\
+/^example\.com$/ example.com
+""")
+ with open(os.path.join(self.tempdir, 'postfix_lmtp')) as fp:
+ contents = _strip_header(fp.read())
+ self.assertMultiLineEqual(contents, """\
+# Aliases which are visible only in the @example.com domain.
+/^test@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-bounces@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-confirm@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-join@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-leave@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-owner@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-request@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-subscribe@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test-unsubscribe@example\.com$/ lmtp:[127.0.0.1]:9024
+
+/^test\.list\.name\.dots@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test\.list\.name\.dots-bounces@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test\.list\.name\.dots-confirm@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test\.list\.name\.dots-join@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test\.list\.name\.dots-leave@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test\.list\.name\.dots-owner@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test\.list\.name\.dots-request@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test\.list\.name\.dots-subscribe@example\.com$/ lmtp:[127.0.0.1]:9024
+/^test\.list\.name\.dots-unsubscribe@example\.com$/ lmtp:[127.0.0.1]:9024
""")
diff --git a/src/mailman/testing/layers.py b/src/mailman/testing/layers.py
index f08b81fca..dbed1fccf 100644
--- a/src/mailman/testing/layers.py
+++ b/src/mailman/testing/layers.py
@@ -106,6 +106,7 @@ class ConfigLayer(MockAndMonkeyLayer):
print(dedent("""
[postfix]
postmap_command: true
+ transport_file_type: hash
"""), file=fp)
test_config = dedent("""
[mailman]
generated by cgit v1.2.3-70-g09d2 (git 2.51.2) at 2025-11-08 12:03:00 +0000