The start of a message store definition. Whether this will end up being used

for the archive or not is left to be seen.

Define an interface, test, and implementation of a basic message store using
globally unique identifiers of the form: archive/hash/seqno

- archive is the base url of the archive, e.g. http://archives.example.com.
  This is available in the List-Archive header.
- hash is the base32 encoded sha1 hash of the message's Message-ID and Date
  headers, which it must have.  This is available in the X-List-ID-Hash
  header.
- seqno is a sequence number specific to the archive which will uniquely
  identify the message should there be a Message-ID/Date collision.  this is
  available in the X-List-Sequence-Number header.

Added a MESSAGES_DIR variable to the config.

Added a .message_store attribute to the config.

2 files changed, 169 insertions, 1 deletions

diff --git a/Mailman/docs/archives.txt b/Mailman/docs/archives.txt
index 682ee8777..1bed66e1a 100644
--- a/Mailman/docs/archives.txt
+++ b/Mailman/docs/archives.txt
@@ -25,7 +25,6 @@ A helper function.
     ...         msg, msgdata = switchboard.dequeue(filebase)
     ...         switchboard.finish(filebase)
 
-
 The purpose of the ToArchive handler is to make a simple decision as to
 whether the message should get archived and if so, to drop the message in the
 archiving queue.  Really the most important things are to determine when a
diff --git a/Mailman/docs/messagestore.txt b/Mailman/docs/messagestore.txt
new file mode 100644
index 000000000..ace95e914
--- /dev/null
+++ b/Mailman/docs/messagestore.txt
@@ -0,0 +1,169 @@
+The message store
+=================
+
+The message store is a collection of messages keyed off of unique global
+identifiers.  A global id for a message is calculated relative to the message
+store's base URL and its components are stored as headers on the message.  One
+piece of information is the X-List-ID-Hash, a base-32 encoding of the SHA1
+hash of the message's Message-ID and Date headers, which the message must
+have.   The second piece of information is supplied by the message store; it
+is a sequence number that will uniquely identify the message even when the
+X-List-ID-Hash collides.
+
+    >>> from email import message_from_string
+    >>> from Mailman.configuration import config
+    >>> from Mailman.database import flush
+    >>> store = config.message_store
+
+If you try to add a message to the store which is missing either the
+Message-ID header or the Date header, you will get a ValueError.
+
+    >>> msg = message_from_string("""\
+    ... Subject: An important message
+    ...
+    ... This message is very important.
+    ... """)
+    >>> store.add(msg)
+    Traceback (most recent call last):
+    ...
+    ValueError: Exactly one Message-ID and one Date header required
+
+Adding a Message-ID header alone doesn't help.
+
+    >>> msg['Message-ID'] = '<87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>'
+    >>> store.add(msg)
+    Traceback (most recent call last):
+    ...
+    ValueError: Exactly one Message-ID and one Date header required
+
+Neither does adding just a Date header.
+
+    >>> del msg['message-id']
+    >>> msg['Date'] = 'Wed, 04 Jul 2007 16:49:58 +0900'
+    >>> store.add(msg)
+    Traceback (most recent call last):
+    ...
+    ValueError: Exactly one Message-ID and one Date header required
+
+However, having them both is all you need.
+
+    >>> msg['Message-ID'] = '<87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>'
+    >>> store.add(msg)
+    1
+    >>> flush()
+    >>> print msg.as_string()
+    Subject: An important message
+    Date: Wed, 04 Jul 2007 16:49:58 +0900
+    Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
+    X-List-ID-Hash: RXTJ357KFOTJP3NFJA6KMO65X7VQOHJI
+    X-List-Sequence-Number: 1
+    <BLANKLINE>
+    This message is very important.
+    <BLANKLINE>
+
+
+Finding messages
+----------------
+
+There are several ways to find a message given some or all of the information
+created above.  Because Message-IDs are not guaranteed unique, looking up
+messages with that key resturns a collection.  The collection may be empty if
+there are no matches.
+
+    >>> list(store.get_messages_by_message_id('nothing'))
+    []
+
+Given an existing Message-ID, all matching messages will be found.
+
+    >>> msgs = list(store.get_messages_by_message_id(msg['message-id']))
+    >>> len(msgs)
+    1
+    >>> print msgs[0].as_string()
+    Subject: An important message
+    Date: Wed, 04 Jul 2007 16:49:58 +0900
+    Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
+    X-List-ID-Hash: RXTJ357KFOTJP3NFJA6KMO65X7VQOHJI
+    X-List-Sequence-Number: 1
+    <BLANKLINE>
+    This message is very important.
+    <BLANKLINE>
+
+Similarly, we can find messages by the ID hash.
+
+    >>> list(store.get_messages_by_hash('nothing'))
+    []
+    >>> msgs = list(store.get_messages_by_hash(msg['x-list-id-hash']))
+    >>> len(msgs)
+    1
+    >>> print msgs[0].as_string()
+    Subject: An important message
+    Date: Wed, 04 Jul 2007 16:49:58 +0900
+    Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
+    X-List-ID-Hash: RXTJ357KFOTJP3NFJA6KMO65X7VQOHJI
+    X-List-Sequence-Number: 1
+    <BLANKLINE>
+    This message is very important.
+    <BLANKLINE>
+
+We can also get a single message by using it's relative global ID.  This
+returns None if there is no match.
+
+    >>> print store.get_message('nothing')
+    None
+    >>> print store.get_message('nothing/1')
+    None
+    >>> id_hash = msg['x-list-id-hash']
+    >>> seqno = msg['x-list-sequence-number']
+    >>> global_id = id_hash + '/' + seqno
+    >>> print store.get_message(global_id).as_string()
+    Subject: An important message
+    Date: Wed, 04 Jul 2007 16:49:58 +0900
+    Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
+    X-List-ID-Hash: RXTJ357KFOTJP3NFJA6KMO65X7VQOHJI
+    X-List-Sequence-Number: 1
+    <BLANKLINE>
+    This message is very important.
+    <BLANKLINE>
+
+
+Iterating over all messages
+---------------------------
+
+The message store provides a means to iterate over all the messages it
+contains.
+
+    >>> msgs = list(store.messages)
+    >>> len(msgs)
+    1
+    >>> print msgs[0].as_string()
+    Subject: An important message
+    Date: Wed, 04 Jul 2007 16:49:58 +0900
+    Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
+    X-List-ID-Hash: RXTJ357KFOTJP3NFJA6KMO65X7VQOHJI
+    X-List-Sequence-Number: 1
+    <BLANKLINE>
+    This message is very important.
+    <BLANKLINE>
+
+
+Deleting messages from the store
+--------------------------------
+
+The global relative ID is the key into the message store.  If you try to
+delete a global ID that isn't in the store, you get an exception.
+
+    >>> store.delete_message('nothing')
+    Traceback (most recent call last):
+    ...
+    KeyError: 'nothing'
+
+But if you delete an existing message, it really gets deleted.
+
+    >>> store.delete_message(global_id)
+    >>> flush()
+    >>> list(store.messages)
+    []
+    >>> print store.get_message(global_id)
+    None
+    >>> list(store.get_messages_by_message_id(msg['message-id']))
+    []