Add .get() to our Message subclass, which ensures that returned

values are unicodes if they come from the base class as a string.

Get rid of the 'global id'.  Now use just Message-ID.  Rename
X-List-ID-Hash to X-Message-ID-Hash.  Do not take Date header into
account when calculating this hash.

Because of the above change, the assumption is that there will be no
Message-ID collisions.  Therefore, get rid of IMessageStore
.get_message(), .get_messages_by_message_id() and
.get_messages_by_hash().  Instead, it's now .get_message_by_id() and
.get_message_by_hash() both of which return the message object or
None.

Message.hash -> Message.message_id_hash

When storing a message in the message store, the final path component
has the entire hash, not just the leftover parts after directory
prefix splitting.

MessageStore.delete_message() deletes the file too.

Doctests clean up message store messages though the message store
instead of directly off the filesystem.

4 files changed, 58 insertions, 89 deletions

diff --git a/Mailman/docs/hold.txt b/Mailman/docs/hold.txt
index 56a10206f..a93953435 100644
--- a/Mailman/docs/hold.txt
+++ b/Mailman/docs/hold.txt
@@ -130,8 +130,7 @@ that Mailman pulled it from the appropriate news group.
     From: aperson@example.org
     Subject: An implicit message
     Message-ID: ...
-    X-List-ID-Hash: ...
-    X-List-Sequence-Number: ...
+    X-Message-ID-Hash: ...
     <BLANKLINE>
     <BLANKLINE>
     >>> print msgdata
@@ -263,8 +262,7 @@ one to the original author.
     <BLANKLINE>
     From: aperson@example.com
     Message-ID: ...
-    X-List-ID-Hash: ...
-    X-List-Sequence-Number: ...
+    X-Message-ID-Hash: ...
     <BLANKLINE>
     <BLANKLINE>
     --...
@@ -350,12 +348,12 @@ The message itself is held in the message store.
 
     >>> rkey, rdata = config.db.requests.get_list_requests(mlist).get_request(
     ...     data['id'])
-    >>> msg = config.db.message_store.get_message(rdata['_mod_global_id'])
+    >>> msg = config.db.message_store.get_message_by_id(
+    ...     rdata['_mod_message_id'])
     >>> print msg.as_string()
     From: aperson@example.com
     Message-ID: ...
-    X-List-ID-Hash: ...
-    X-List-Sequence-Number: ...
+    X-Message-ID-Hash: ...
     <BLANKLINE>
     <BLANKLINE>
  
diff --git a/Mailman/docs/messagestore.txt b/Mailman/docs/messagestore.txt
index 9b44a7e59..012376a14 100644
--- a/Mailman/docs/messagestore.txt
+++ b/Mailman/docs/messagestore.txt
@@ -1,14 +1,11 @@
 The message store
 =================
 
-The message store is a collection of messages keyed off of unique global
-identifiers.  A global id for a message is calculated relative to the message
-store's base URL and its components are stored as headers on the message.  One
-piece of information is the X-List-ID-Hash, a base-32 encoding of the SHA1
-hash of the message's Message-ID header, which the message must have.  The
-second piece of information is supplied by the message store; it is a sequence
-number that will uniquely identify the message even when the X-List-ID-Hash
-collides.
+The message store is a collection of messages keyed off of Message-ID and
+X-Message-ID-Hash headers.  Either of these values can be combined with the
+message's List-Archive header to create a globally unique URI to the message
+object in the internet facing interface of the message store.  The
+X-Message-ID-Hash is the Base32 SHA1 hash of the Message-ID.
 
     >>> from Mailman.configuration import config
     >>> store = config.db.message_store
@@ -30,12 +27,11 @@ However, if the message has a Message-ID header, it can be stored.
 
     >>> msg['Message-ID'] = '<87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>'
     >>> store.add(msg)
-    1
+    'AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35'
     >>> print msg.as_string()
     Subject: An important message
     Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
-    X-List-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
-    X-List-Sequence-Number: 1
+    X-Message-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
     <BLANKLINE>
     This message is very important.
     <BLANKLINE>
@@ -44,59 +40,33 @@ However, if the message has a Message-ID header, it can be stored.
 Finding messages
 ----------------
 
-There are several ways to find a message given some or all of the information
-created above.  Because Message-IDs are not guaranteed unique, looking up
-messages with that key resturns a collection.  The collection may be empty if
-there are no matches.
-
-    >>> list(store.get_messages_by_message_id(u'nothing'))
-    []
+There are several ways to find a message given either the Message-ID or
+X-Message-ID-Hash headers.  In either case, if no matching message is found,
+None is returned.
 
-Given an existing Message-ID, all matching messages will be found.
-
-    >>> msgs = list(store.get_messages_by_message_id(msg['message-id']))
-    >>> len(msgs)
-    1
-    >>> print msgs[0].as_string()
-    Subject: An important message
-    Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
-    X-List-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
-    X-List-Sequence-Number: 1
-    <BLANKLINE>
-    This message is very important.
-    <BLANKLINE>
+    >>> print store.get_message_by_id(u'nothing')
+    None
+    >>> print store.get_message_by_hash(u'nothing')
+    None
 
-Similarly, we can find messages by the ID hash.
+Given an existing Message-ID, the message can be found.
 
-    >>> list(store.get_messages_by_hash('nothing'))
-    []
-    >>> msgs = list(store.get_messages_by_hash(msg['x-list-id-hash']))
-    >>> len(msgs)
-    1
-    >>> print msgs[0].as_string()
+    >>> message = store.get_message_by_id(msg['message-id'])
+    >>> print message.as_string()
     Subject: An important message
     Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
-    X-List-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
-    X-List-Sequence-Number: 1
+    X-Message-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
     <BLANKLINE>
     This message is very important.
     <BLANKLINE>
 
-We can also get a single message by using it's relative global ID.  This
-returns None if there is no match.
+Similarly, we can find messages by the X-Message-ID-Hash:
 
-    >>> print store.get_message('nothing')
-    None
-    >>> print store.get_message('nothing/1')
-    None
-    >>> id_hash = msg['x-list-id-hash']
-    >>> seqno = msg['x-list-sequence-number']
-    >>> global_id = id_hash + '/' + seqno
-    >>> print store.get_message(global_id).as_string()
+    >>> message = store.get_message_by_hash(msg['x-message-id-hash'])
+    >>> print message.as_string()
     Subject: An important message
     Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
-    X-List-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
-    X-List-Sequence-Number: 1
+    X-Message-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
     <BLANKLINE>
     This message is very important.
     <BLANKLINE>
@@ -108,14 +78,13 @@ Iterating over all messages
 The message store provides a means to iterate over all the messages it
 contains.
 
-    >>> msgs = list(store.messages)
-    >>> len(msgs)
+    >>> messages = list(store.messages)
+    >>> len(messages)
     1
-    >>> print msgs[0].as_string()
+    >>> print messages[0].as_string()
     Subject: An important message
     Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
-    X-List-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
-    X-List-Sequence-Number: 1
+    X-Message-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
     <BLANKLINE>
     This message is very important.
     <BLANKLINE>
@@ -124,20 +93,22 @@ contains.
 Deleting messages from the store
 --------------------------------
 
-The global relative ID is the key into the message store.  If you try to
-delete a global ID that isn't in the store, you get an exception.
+You delete a message from the storage service by providing the Message-ID for
+the message you want to delete.  If you try to delete a Message-ID that isn't
+in the store, you get an exception.
 
-    >>> store.delete_message('nothing')
+    >>> store.delete_message(u'nothing')
     Traceback (most recent call last):
     ...
-    KeyError: 'nothing'
+    LookupError: nothing
 
 But if you delete an existing message, it really gets deleted.
 
-    >>> store.delete_message(global_id)
+    >>> message_id = message['message-id']
+    >>> store.delete_message(message_id)
     >>> list(store.messages)
     []
-    >>> print store.get_message(global_id)
+    >>> print store.get_message_by_id(message_id)
+    None
+    >>> print store.get_message_by_hash(message['x-message-id-hash'])
     None
-    >>> list(store.get_messages_by_message_id(msg['message-id']))
-    []
diff --git a/Mailman/docs/requests.txt b/Mailman/docs/requests.txt
index 7a395ce94..ea4dcc75d 100644
--- a/Mailman/docs/requests.txt
+++ b/Mailman/docs/requests.txt
@@ -231,6 +231,9 @@ this case, we won't include any additional metadata.
 
 We can also hold a message with some additional metadata.
 
+    # Delete the Message-ID from the previous hold so we don't try to store
+    # collisions in the message storage.
+    >>> del msg['message-id']
     >>> msgdata = dict(sender='aperson@example.com',
     ...                approved=True,
     ...                received_time=123.45)
@@ -308,8 +311,7 @@ indicates that the message has been approved.
     To: alist@example.com
     Subject: Something important
     Message-ID: ...
-    X-List-ID-Hash: ...
-    X-List-Sequence-Number: ...
+    X-Message-ID-Hash: ...
     X-Mailman-Approved-At: ...
     <BLANKLINE>
     Here's something important about our mailing list.
@@ -338,26 +340,21 @@ is deleted.
     ... """)
     >>> id_4 = moderator.hold_message(mlist, msg, {}, 'Needs approval')
     >>> moderator.handle_message(mlist, id_4, Action.discard)
-    >>> msgs = config.db.message_store.get_messages_by_message_id(u'<12345>')
-    >>> list(msgs)
-    []
+    >>> print config.db.message_store.get_message_by_id(u'<12345>')
+    None
 
 But if we ask to preserve the message when we discard it, it will be held in
 the message store after disposition.
 
     >>> id_4 = moderator.hold_message(mlist, msg, {}, 'Needs approval')
     >>> moderator.handle_message(mlist, id_4, Action.discard, preserve=True)
-    >>> msgs = config.db.message_store.get_messages_by_message_id(u'<12345>')
-    >>> msgs = list(msgs)
-    >>> len(msgs)
-    1
-    >>> print msgs[0].as_string()
+    >>> stored_msg = config.db.message_store.get_message_by_id(u'<12345>')
+    >>> print stored_msg.as_string()
     From: aperson@example.org
     To: alist@example.com
     Subject: Something important
     Message-ID: <12345>
-    X-List-ID-Hash: 4CF7EAU3SIXBPXBB5S6PEUMO62MWGQN6
-    X-List-Sequence-Number: 1
+    X-Message-ID-Hash: 4CF7EAU3SIXBPXBB5S6PEUMO62MWGQN6
     <BLANKLINE>
     Here's something important about our mailing list.
     <BLANKLINE>
@@ -366,6 +363,10 @@ Orthogonal to preservation, the message can also be forwarded to another
 address.  This is helpful for getting the message into the inbox of one of the
 moderators.
 
+    # Set a new Message-ID from the previous hold so we don't try to store
+    # collisions in the message storage.
+    >>> del msg['message-id']
+    >>> msg['Message-ID'] = u'<abcde>'
     >>> id_4 = moderator.hold_message(mlist, msg, {}, 'Needs approval')
     >>> moderator.handle_message(mlist, id_4, Action.discard,
     ...                          forward=[u'zperson@example.com'])
@@ -383,9 +384,8 @@ moderators.
     From: aperson@example.org
     To: alist@example.com
     Subject: Something important
-    Message-ID: <12345>
-    X-List-ID-Hash: 4CF7EAU3SIXBPXBB5S6PEUMO62MWGQN6
-    X-List-Sequence-Number: ...
+    Message-ID: <abcde>
+    X-Message-ID-Hash: EN2R5UQFMOUTCL44FLNNPLSXBIZW62ER
     <BLANKLINE>
     Here's something important about our mailing list.
     <BLANKLINE>
diff --git a/Mailman/docs/subject-munging.txt b/Mailman/docs/subject-munging.txt
index 388a02564..12bc098f7 100644
--- a/Mailman/docs/subject-munging.txt
+++ b/Mailman/docs/subject-munging.txt
@@ -36,7 +36,7 @@ the new Subject header because it gets converted from a string to an
 email.header.Header instance which has an unhelpful repr.
 
     >>> msgdata['origsubj']
-    ''
+    u''
     >>> print msg['subject']
     [XTest] (no subject)
 
@@ -52,7 +52,7 @@ the beginning of the header's value.
     >>> msgdata = {}
     >>> process(mlist, msg, msgdata)
     >>> msgdata['origsubj']
-    'Something important'
+    u'Something important'
     >>> print msg['subject']
     [XTest] Something important