diff options
| author | bwarsaw | 2006-12-29 22:20:25 +0000 |
|---|---|---|
| committer | bwarsaw | 2006-12-29 22:20:25 +0000 |
| commit | f4a456a83b630feb294724ab462c87ca1ce1c3ae (patch) | |
| tree | c5c88540dae8306d11671f603d8975b01803ea16 /Mailman | |
| parent | ae185106a624bfa7888aa8722d35194d3c5150e8 (diff) | |
| download | mailman-f4a456a83b630feb294724ab462c87ca1ce1c3ae.tar.gz mailman-f4a456a83b630feb294724ab462c87ca1ce1c3ae.tar.zst mailman-f4a456a83b630feb294724ab462c87ca1ce1c3ae.zip | |
Merged revisions 8113-8121 via svnmerge from
https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch
................
r8114 | bwarsaw | 2006-12-06 00:16:54 -0500 (Wed, 06 Dec 2006) | 44 lines
Initial take on using SQLAlchemy to store list data in lieu of Python pickles.
While all the list data (including OldStyleMemberships attributes) are stored
in the database, many attributes are stored as PickleTypes binary data. This
isn't idea but it gets things working until a more sophisticated schema can be
developed.
MailList class is now a new-style class, as is required by SQLAlchemy. This
makes several things, er, interesting. Rip out all the low-level pickle
reading and writing stuff. Hook SA transaction events into Lock() and
Unlock(). Move the hooking of the _memberadaptor into InitTempVars(), which
gets called by the SQLAlchemy hooks (MailList.__init__() never is).
Add an initialize.py module which centralizes all the initialization bits that
command line scripts have to do, including configuration, logging, and atabase
initialization.
This change also converts bin/withlist to mmshell wrapper.
Update to SQLAlchemy 0.3.1.
Revamp paths.py.in considerably. There were several problems with the old
way. We no longer disable default loading of site-packages so we don't need
to add Python's site-packages back to sys.path. Also, because
site.addsitedir() causes things like .pth paths to be /appended/ to sys.path,
they actually won't override any site-installed packages. E.g. if SQLAlchemy
is installed in the system Python, our version will not override. IIUC,
setuptools-based packages can be configured to work properly in the face of
package versions, however not all packages we currently depend on are
setuptools-based. So instead, we steal a bit of stuff from site.py but change
things so the prepend .pth stuff to sys.path.
Update several modules to use True/False and whitespace normalization.
Convert from mm_cfg to config object. Modernize a few coding constructs.
Add a couple of exceptions to handle database problems.
In the export script, include the widget type in the elements. This helped in
my stupid little throw away conversion script, but I think it will be more
generally useful.
Add an interact.py module which refactors interactive interpreter access.
Mostly this is used by withlist -i, but it lets us import Mailman.interact and
drop into a prompt just about anywhere (e.g. debugging).
................
r8115 | bwarsaw | 2006-12-07 09:13:56 -0500 (Thu, 07 Dec 2006) | 22 lines
Start to flesh out more of the SQLAlchemy mechanisms.
Added a MailList.__new__() which hooks instantiation to use a query on
dbcontext to get an existing mailing list. A 'no-args' call means we're doing
a Create(), though eventually that will change too.
For now, disable the CheckVersion() call. Eventually this will be folded into
schema migration.
list_exists(): Rewrite to use the dbcontext query to determine if the named
mailing list exists or not. Requires the fqdn_listname.
Eradicate two failed member adaptors: BDBMemberAdaptor and SAMemberships.
Change the way the DBContext holds onto tables. It now keeps a dictionary
mapping the table's name to the SA Table instance. This makes it easier to
look up and use the individual tables.
Add 'web_page_url' as an attribute managed by SA, and remove a debugging
print.
................
r8116 | bwarsaw | 2006-12-11 07:27:47 -0500 (Mon, 11 Dec 2006) | 29 lines
Rework the whole dbcontext and transaction framework. SA already handles
nested transactions so we don't have to worry about them. However, we do have
the weird situation where some transactions are tied to MailList
.Lock()/.Unlock()/.Save() and some are tied to non-mlist actions. So now we
use an @txn decorator to put methods in a session transaction, but then we
also hook into the above MailList methods as possibly sub-transactions. We
use a weakref subclass to manage the MailList interface, with a dictionary
mapping MailList fqdn_listnames against transactions. The weakrefs come in by
giving us a callback when a MailList gets derefed such that we're guaranteed
to rollback any outstanding transaction.
Also, we have one global DBContext instance but rather than force the rest of
Mailman to deal with context objects, instead we expose API methods on that
object into the Mailman.database module, which the rest of the code will use.
Such methods must be prepended with 'api_' to get exposed this way.
bin/rmlist now works with the SA-backend. I refactored the code here so that
other code (namely, the test suite) can more easily and consistently remove a
mailing list. This isn't the best place for it ultimately, but it's good
enough for now.
New convenience functions Utils.split_listname(), .fqdn_listname().
Convert testall to use Mailman.initialize.initialize(). Not all tests work,
but I'm down to only 8 failures and 7 errors. Also, do a better job of
recovering from failures in setUp().
MailList.__new__() now takes keyword arguments.
................
r8117 | bwarsaw | 2006-12-11 22:58:06 -0500 (Mon, 11 Dec 2006) | 7 lines
Unit test repairs; even though the unit tests are still pretty fragile,
everything now passes with the SQLAlchemy storage of list data.
Added missing 'personalize' column. Converted mailmanctl and qrunner to
initialize() interface. Fixed _cookie_path() to not fail if SCRIPT_NAME is
not in the environment.
................
r8118 | bwarsaw | 2006-12-27 18:45:41 -0500 (Wed, 27 Dec 2006) | 21 lines
Utils.list_names(): Use a database query to get all the list names.
dbcontext.py: Added api_get_list_names() to support Utils.list_names().
listdata.py: Added two additional MailList attributes which need to be stored
in the database. The first is 'admin_member_chunksize' which isn't modifiable
from the web. The second is 'password' which holds the list's password.
HTMLFormatObject: item strings can now be unicodes.
bin/list_lists.py: Must call initialize() to get the database properly
initialized, not just config.load(). This will be a common theme.
SecurityManager.py:
- Remove md5 and crypt support
- Added mailman.debug logger, though it will be only used during
debugging.
- The 'secret' can be a unicode now.
- A few coding style updates; repr() instead of backticks, 'key in dict'
instead of 'dict.has_key(key)'
................
r8119 | bwarsaw | 2006-12-27 19:13:09 -0500 (Wed, 27 Dec 2006) | 2 lines
genaliases.py: config.load() -> initialize()
................
r8120 | bwarsaw | 2006-12-27 19:17:26 -0500 (Wed, 27 Dec 2006) | 9 lines
Blocked revisions 8113 via svnmerge
........
r8113 | bwarsaw | 2006-12-05 23:54:30 -0500 (Tue, 05 Dec 2006) | 3 lines
Initialized merge tracking via "svnmerge" with revisions "1-8112" from
https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch
........
................
r8121 | bwarsaw | 2006-12-28 23:34:52 -0500 (Thu, 28 Dec 2006) | 20 lines
Remove SIGTERM handling from all the CGI scripts. This messes with HTTPRunner
because when you issue "mailmanctl stop" after the signal handler has been
installed, the process will get a SIGTERM, the signal handler will run, and
the process will exit with a normal zero code. This will cause mailmanctl to
try to restart the HTTPRunner.
I don't think we need that stuff at all when running under wsgi with a
SQLAlchemy backend. If mailmanctl kills the HTTPRunner in the middle of the
process, I believe (but have not tested) that the transaction should get
properly rolled back at process exit. We need to make sure about this, and
also we need to test the signal handling functionality under traditional CGI
environment (if we even still want to support that).
Also, make sure that we don't try to initialize the loggers twice in qrunner.
This was the cause of all the double entries in logs/qrunner.
Fix a coding style nit in mailmanctl.py.
De-DOS-ify line endings in loginit.py.
................
Diffstat (limited to 'Mailman')
49 files changed, 1326 insertions, 1689 deletions
diff --git a/Mailman/Autoresponder.py b/Mailman/Autoresponder.py index aa29e733c..bfa3d3c09 100644 --- a/Mailman/Autoresponder.py +++ b/Mailman/Autoresponder.py @@ -1,32 +1,29 @@ -# Copyright (C) 1998,1999,2000,2001,2002 by the Free Software Foundation, Inc. +# Copyright (C) 1998-2006 by the Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. -"""MailList mixin class managing the autoresponder. -""" - -from Mailman import mm_cfg -from Mailman.i18n import _ +"""MailList mixin class managing the autoresponder.""" class Autoresponder: def InitVars(self): # configurable - self.autorespond_postings = 0 - self.autorespond_admin = 0 + self.autorespond_postings = False + self.autorespond_admin = False # this value can be # 0 - no autoresponse on the -request line # 1 - autorespond, but discard the original message @@ -40,4 +37,3 @@ class Autoresponder: self.postings_responses = {} self.admin_responses = {} self.request_responses = {} - diff --git a/Mailman/BDBMemberAdaptor.py b/Mailman/BDBMemberAdaptor.py deleted file mode 100644 index 4cd8eaf30..000000000 --- a/Mailman/BDBMemberAdaptor.py +++ /dev/null @@ -1,636 +0,0 @@ -# Copyright (C) 2003-2006 by the Free Software Foundation, Inc. -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -"""A MemberAdaptor based on the Berkeley database wrapper for Python. - -Requires Python 2.2.2 or newer, and PyBSDDB3 4.1.3 or newer. -""" - -# To use, put the following in a file called extend.py in the mailing list's -# directory: -# -# from Mailman.BDBMemberAdaptor import extend -# -# that's it! - -import os -import new -import time -import errno -import struct -import cPickle as pickle - -try: - # Python 2.3 - from bsddb import db -except ImportError: - # earlier Pythons - from bsddb3 import db - -from Mailman import mm_cfg -from Mailman import Utils -from Mailman import Errors -from Mailman import MemberAdaptor -from Mailman.MailList import MailList - -STORAGE_VERSION = 'BA01' -FMT = '>BHB' -FMTSIZE = struct.calcsize(FMT) - -REGDELIV = 1 -DIGDELIV = 2 -REGFLAG = struct.pack('>B', REGDELIV) -DIGFLAG = struct.pack('>B', DIGDELIV) - -# Positional arguments for _unpack() -CPADDR = 0 -PASSWD = 1 -LANG = 2 -NAME = 3 -DIGEST = 4 -OPTIONS = 5 -STATUS = 6 - - - -class BDBMemberAdaptor(MemberAdaptor.MemberAdaptor): - def __init__(self, mlist): - self._mlist = mlist - # metainfo -- {key -> value} - # This table contains storage metadata information. The keys and - # values are simple strings of variable length. Here are the - # valid keys: - # - # version - the version of the database - # - # members -- {address | rec} - # For all regular delivery members, this maps from the member's - # key to their data record, which is a string concatenated of the - # following: - # - # -- fixed data (as a packed struct) - # + 1-byte digest or regular delivery flag - # + 2-byte option flags - # + 1-byte delivery status - # -- variable data (as a pickle of a tuple) - # + their case preserved address or '' - # + their plaintext password - # + their chosen language - # + their realname or '' - # - # status -- {address | status+time} - # Maps the member's key to their delivery status and change time. - # These are passed as a tuple and are pickled for storage. - # - # topics -- {address | topicstrings} - # Maps the member's key to their topic strings, concatenated and - # separated by SEP - # - # bounceinfo -- {address | bounceinfo} - # Maps the member's key to their bounceinfo, as a pickle - # - # Make sure the database directory exists - path = os.path.join(mlist.fullpath(), 'member.db') - exists = False - try: - os.mkdir(path, 02775) - except OSError, e: - if e.errno <> errno.EEXIST: raise - exists = True - # Create the environment - self._env = env = db.DBEnv() - if exists: - # We must join an existing environment, otherwise we'll get - # DB_RUNRECOVERY errors when the second process to open the - # environment begins a transaction. I don't get it. - env.open(path, db.DB_JOINENV) - else: - env.open(path, - db.DB_CREATE | - db.DB_RECOVER | - db.DB_INIT_MPOOL | - db.DB_INIT_TXN - ) - self._txn = None - self._tables = [] - self._metainfo = self._setupDB('metainfo') - self._members = self._setupDB('members') - self._status = self._setupDB('status') - self._topics = self._setupDB('topics') - self._bounceinfo = self._setupDB('bounceinfo') - # Check the database version number - version = self._metainfo.get('version') - if version is None: - # Initialize - try: - self.txn_begin() - self._metainfo.put('version', STORAGE_VERSION, txn=self._txn) - except: - self.txn_abort() - raise - else: - self.txn_commit() - else: - # Currently there's nothing to upgrade - assert version == STORAGE_VERSION - - def _setupDB(self, name): - d = db.DB(self._env) - openflags = db.DB_CREATE - # db 4.1 requires that databases be opened in a transaction. We'll - # use auto commit, but only if that flag exists (i.e. we're using at - # least db 4.1). - try: - openflags |= db.DB_AUTO_COMMIT - except AttributeError: - pass - d.open(name, db.DB_BTREE, openflags) - self._tables.append(d) - return d - - def _close(self): - self.txn_abort() - for d in self._tables: - d.close() - # Checkpoint the database twice, as recommended by Sleepycat - self._checkpoint() - self._checkpoint() - self._env.close() - - def _checkpoint(self): - self._env.txn_checkpoint(0, 0, db.DB_FORCE) - - def txn_begin(self): - assert self._txn is None - self._txn = self._env.txn_begin() - - def txn_commit(self): - assert self._txn is not None - self._txn.commit() - self._checkpoint() - self._txn = None - - def txn_abort(self): - if self._txn is not None: - self._txn.abort() - self._checkpoint() - self._txn = None - - def _unpack(self, member): - # Assume member is a LCE (i.e. lowercase key) - rec = self._members.get(member.lower()) - assert rec is not None - fixed = struct.unpack(FMT, rec[:FMTSIZE]) - vari = pickle.loads(rec[FMTSIZE:]) - return vari + fixed - - def _pack(self, member, cpaddr, passwd, lang, name, digest, flags, status): - # Assume member is a LCE (i.e. lowercase key) - fixed = struct.pack(FMT, digest, flags, status) - vari = pickle.dumps((cpaddr, passwd, lang, name)) - self._members.put(member.lower(), fixed+vari, txn=self._txn) - - # MemberAdaptor writeable interface - - def addNewMember(self, member, **kws): - assert self._mlist.Locked() - # Make sure this address isn't already a member - if self.isMember(member): - raise Errors.MMAlreadyAMember, member - # Parse the keywords - digest = False - password = Utils.MakeRandomPassword() - language = self._mlist.preferred_language - realname = None - if kws.has_key('digest'): - digest = kws['digest'] - del kws['digest'] - if kws.has_key('password'): - password = kws['password'] - del kws['password'] - if kws.has_key('language'): - language = kws['language'] - del kws['language'] - if kws.has_key('realname'): - realname = kws['realname'] - del kws['realname'] - # Assert that no other keywords are present - if kws: - raise ValueError, kws.keys() - # Should we store the case-preserved address? - if Utils.LCDomain(member) == member.lower(): - cpaddress = '' - else: - cpaddress = member - # Calculate the realname - if realname is None: - realname = '' - # Calculate the digest flag - if digest: - digest = DIGDELIV - else: - digest = REGDELIV - self._pack(member.lower(), - cpaddress, password, language, realname, - digest, self._mlist.new_member_options, - MemberAdaptor.ENABLED) - - def removeMember(self, member): - txn = self._txn - assert txn is not None - assert self._mlist.Locked() - self.__assertIsMember(member) - key = member.lower() - # Remove the table entries - self._members.delete(key, txn=txn) - if self._status.has_key(key): - self._status.delete(key, txn=txn) - if self._topics.has_key(key): - self._topics.delete(key, txn=txn) - if self._bounceinfo.has_key(key): - self._bounceinfo.delete(key, txn=txn) - - def changeMemberAddress(self, member, newaddress, nodelete=0): - assert self._mlist.Locked() - self.__assertIsMember(member) - okey = member.lower() - nkey = newaddress.lower() - txn = self._txn - assert txn is not None - # First, store a new member record, changing the case preserved addr. - # Then delete the old record. - cpaddr, passwd, lang, name, digest, flags, sts = self._unpack(okey) - self._pack(nkey, newaddress, passwd, lang, name, digest, flags, sts) - if not nodelete: - self._members.delete(okey, txn) - # Copy over the status times, topics, and bounce info, if present - timestr = self._status.get(okey) - if timestr is not None: - self._status.put(nkey, timestr, txn=txn) - if not nodelete: - self._status.delete(okey, txn) - topics = self._topics.get(okey) - if topics is not None: - self._topics.put(nkey, topics, txn=txn) - if not nodelete: - self._topics.delete(okey, txn) - binfo = self._bounceinfo.get(nkey) - if binfo is not None: - self._binfo.put(nkey, binfo, txn=txn) - if not nodelete: - self._binfo.delete(okey, txn) - - def setMemberPassword(self, member, password): - assert self._mlist.Locked() - self.__assertIsMember(member) - member = member.lower() - cpaddr, oldpw, lang, name, digest, flags, status = self._unpack(member) - self._pack(member, cpaddr, password, lang, name, digest, flags, status) - - def setMemberLanguage(self, member, language): - assert self._mlist.Locked() - self.__assertIsMember(member) - member = member.lower() - cpaddr, passwd, olang, name, digest, flags, sts = self._unpack(member) - self._pack(member, cpaddr, passwd, language, name, digest, flags, sts) - - def setMemberOption(self, member, flag, value): - assert self._mlist.Locked() - self.__assertIsMember(member) - member = member.lower() - cpaddr, passwd, lang, name, digest, options, sts = self._unpack(member) - # Sanity check for the digest flag - if flag == mm_cfg.Digests: - if value: - # Be sure the list supports digest delivery - if not self._mlist.digestable: - raise Errors.CantDigestError - digest = DIGDELIV - else: - # Be sure the list supports regular delivery - if not self._mlist.nondigestable: - raise Errors.MustDigestError - # When toggling off digest delivery, we want to be sure to set - # things up so that the user receives one last digest, - # otherwise they may lose some email - self._mlist.one_last_digest[member] = cpaddr - digest = REGDELIV - else: - if value: - options |= flag - else: - options &= ~flag - self._pack(member, cpaddr, passwd, lang, name, digest, options, sts) - - def setMemberName(self, member, realname): - assert self._mlist.Locked() - self.__assertIsMember(member) - member = member.lower() - cpaddr, passwd, lang, oldname, digest, flags, sts = self._unpack( - member) - self._pack(member, cpaddr, passwd, lang, realname, digest, flags, sts) - - def setMemberTopics(self, member, topics): - assert self._mlist.Locked() - self.__assertIsMember(member) - member = member.lower() - if topics: - self._topics.put(member, SEP.join(topics), txn=self._txn) - elif self._topics.has_key(member): - # No record is the same as no topics - self._topics.delete(member, self._txn) - - def setDeliveryStatus(self, member, status): - assert status in (MemberAdaptor.ENABLED, MemberAdaptor.UNKNOWN, - MemberAdaptor.BYUSER, MemberAdaptor.BYADMIN, - MemberAdaptor.BYBOUNCE) - assert self._mlist.Locked() - self.__assertIsMember(member) - if status == MemberAdaptor.ENABLED: - # Enable by resetting their bounce info - self.setBounceInfo(member, None) - else: - # Pickle up the status an the current time and store that in the - # database. Use binary mode. - data = pickle.dumps((status, time.time()), 1) - self._status.put(member.lower(), data, txn=self._txn) - - def setBounceInfo(self, member, info): - assert self._mlist.Locked() - self.__assertIsMember(member) - member = member.lower() - if info is None: - # This means to reset the bounce and delivery status information - if self._bounceinfo.has_key(member): - self._bounceinfo.delete(member, self._txn) - if self._status.has_key(member): - self._status.delete(member, self._txn) - else: - # Use binary mode - data = pickle.dumps(info, 1) - self._status.put(member, data, txn=self._txn) - - # The readable interface - - # BAW: It would be more efficient to simply return the iterator, but - # modules like admin.py can't handle that yet. They requires lists. - def getMembers(self): - return list(_AllMembersIterator(self._members)) - - def getRegularMemberKeys(self): - return list(_DeliveryMemberIterator(self._members, REGFLAG)) - - def getDigestMemberKeys(self): - return list(_DeliveryMemberIterator(self._members, DIGFLAG)) - - def __assertIsMember(self, member): - if not self.isMember(member): - raise Errors.NotAMemberError, member - - def isMember(self, member): - return self._members.has_key(member.lower()) - - def getMemberKey(self, member): - self.__assertIsMember(member) - return member.lower() - - def getMemberCPAddress(self, member): - self.__assertIsMember(member) - cpaddr = self._unpack(member)[CPADDR] - if cpaddr: - return cpaddr - return member - - def getMemberCPAddresses(self, members): - rtn = [] - for member in members: - member = member.lower() - if self._members.has_key(member): - rtn.append(self._unpack(member)[CPADDR]) - else: - rtn.append(None) - return rtn - - def authenticateMember(self, member, response): - self.__assertIsMember(member) - passwd = self._unpack(member)[PASSWD] - if passwd == response: - return passwd - return False - - def getMemberPassword(self, member): - self.__assertIsMember(member) - return self._unpack(member)[PASSWD] - - def getMemberLanguage(self, member): - if not self.isMember(member): - return self._mlist.preferred_language - lang = self._unpack(member)[LANG] - if lang in self._mlist.GetAvailableLanguages(): - return lang - return self._mlist.preferred_language - - def getMemberOption(self, member, flag): - self.__assertIsMember(member) - if flag == mm_cfg.Digests: - return self._unpack(member)[DIGEST] == DIGDELIV - options = self._unpack(member)[OPTIONS] - return bool(options & flag) - - def getMemberName(self, member): - self.__assertIsMember(member) - name = self._unpack(member)[NAME] - return name or None - - def getMemberTopics(self, member): - self.__assertIsMember(member) - topics = self._topics.get(member.lower(), '') - if not topics: - return [] - return topics.split(SEP) - - def getDeliveryStatus(self, member): - self.__assertIsMember(member) - data = self._status.get(member.lower()) - if data is None: - return MemberAdaptor.ENABLED - status, when = pickle.loads(data) - return status - - def getDeliveryStatusChangeTime(self, member): - self.__assertIsMember(member) - data = self._status.get(member.lower()) - if data is None: - return 0 - status, when = pickle.loads(data) - return when - - # BAW: see above, re iterators - def getDeliveryStatusMembers(self, status=(MemberAdaptor.UNKNOWN, - MemberAdaptor.BYUSER, - MemberAdaptor.BYADMIN, - MemberAdaptor.BYBOUNCE)): - return list(_StatusMemberIterator(self._members, self._status, status)) - - def getBouncingMembers(self): - return list(_BouncingMembersIterator(self._bounceinfo)) - - def getBounceInfo(self, member): - self.__assertIsMember(member) - return self._bounceinfo.get(member.lower()) - - - -class _MemberIterator: - def __init__(self, table): - self._table = table - self._c = table.cursor() - - def __iter__(self): - raise NotImplementedError - - def next(self): - raise NotImplementedError - - def close(self): - if self._c: - self._c.close() - self._c = None - - def __del__(self): - self.close() - - -class _AllMembersIterator(_MemberIterator): - def __iter__(self): - return _AllMembersIterator(self._table) - - def next(self): - rec = self._c.next() - if rec: - return rec[0] - self.close() - raise StopIteration - - -class _DeliveryMemberIterator(_MemberIterator): - def __init__(self, table, flag): - _MemberIterator.__init__(self, table) - self._flag = flag - - def __iter__(self): - return _DeliveryMemberIterator(self._table, self._flag) - - def next(self): - rec = self._c.next() - while rec: - addr, data = rec - if data[0] == self._flag: - return addr - rec = self._c.next() - self.close() - raise StopIteration - - -class _StatusMemberIterator(_MemberIterator): - def __init__(self, table, statustab, status): - _MemberIterator.__init__(self, table) - self._statustab = statustab - self._status = status - - def __iter__(self): - return _StatusMemberIterator(self._table, - self._statustab, - self._status) - - def next(self): - rec = self._c.next() - while rec: - addr = rec[0] - data = self._statustab.get(addr) - if data is None: - status = MemberAdaptor.ENABLED - else: - status, when = pickle.loads(data) - if status in self._status: - return addr - rec = self._c.next() - self.close() - raise StopIteration - - -class _BouncingMembersIterator(_MemberIterator): - def __iter__(self): - return _BouncingMembersIterator(self._table) - - def next(self): - rec = self._c.next() - if rec: - return rec[0] - self.close() - raise StopIteration - - - -# For extend.py -def fixlock(mlist): - def Lock(self, timeout=0): - MailList.Lock(self, timeout) - try: - self._memberadaptor.txn_begin() - except: - MailList.Unlock(self) - raise - mlist.Lock = new.instancemethod(Lock, mlist, MailList) - - -def fixsave(mlist): - def Save(self): - self._memberadaptor.txn_commit() - MailList.Save(self) - mlist.Save = new.instancemethod(Save, mlist, MailList) - - -def fixunlock(mlist): - def Unlock(self): - # It's fine to abort the transaction even if there isn't one in - # process, say because the Save() already committed it - self._memberadaptor.txn_abort() - MailList.Unlock(self) - mlist.Unlock = new.instancemethod(Unlock, mlist, MailList) - - -def extend(mlist): - mlist._memberadaptor = BDBMemberAdaptor(mlist) - fixlock(mlist) - fixsave(mlist) - fixunlock(mlist) - # To make sure we got everything, let's actually delete the - # OldStyleMemberships dictionaries. Assume if it has one, it has all - # attributes. - try: - del mlist.members - del mlist.digest_members - del mlist.passwords - del mlist.language - del mlist.user_options - del mlist.usernames - del mlist.topics_userinterest - del mlist.delivery_status - del mlist.bounce_info - except AttributeError: - pass - # BAW: How can we ensure that the BDBMemberAdaptor is closed? diff --git a/Mailman/Bouncer.py b/Mailman/Bouncer.py index aabe9d5ad..d18d70917 100644 --- a/Mailman/Bouncer.py +++ b/Mailman/Bouncer.py @@ -24,19 +24,20 @@ import logging from email.MIMEMessage import MIMEMessage from email.MIMEText import MIMEText +from Mailman import Defaults from Mailman import MemberAdaptor from Mailman import Message from Mailman import Pending from Mailman import Utils from Mailman import i18n -from Mailman import mm_cfg +from Mailman.configuration import config EMPTYSTRING = '' # This constant is supposed to represent the day containing the first midnight # after the epoch. We'll add (0,)*6 to this tuple to get a value appropriate # for time.mktime(). -ZEROHOUR_PLUSONEDAY = time.localtime(mm_cfg.days(1))[:3] +ZEROHOUR_PLUSONEDAY = time.localtime(Defaults.days(1))[:3] def _(s): return s @@ -81,19 +82,19 @@ class _BounceInfo: class Bouncer: def InitVars(self): # Configurable... - self.bounce_processing = mm_cfg.DEFAULT_BOUNCE_PROCESSING - self.bounce_score_threshold = mm_cfg.DEFAULT_BOUNCE_SCORE_THRESHOLD - self.bounce_info_stale_after = mm_cfg.DEFAULT_BOUNCE_INFO_STALE_AFTER + self.bounce_processing = config.DEFAULT_BOUNCE_PROCESSING + self.bounce_score_threshold = config.DEFAULT_BOUNCE_SCORE_THRESHOLD + self.bounce_info_stale_after = config.DEFAULT_BOUNCE_INFO_STALE_AFTER self.bounce_you_are_disabled_warnings = \ - mm_cfg.DEFAULT_BOUNCE_YOU_ARE_DISABLED_WARNINGS + config.DEFAULT_BOUNCE_YOU_ARE_DISABLED_WARNINGS self.bounce_you_are_disabled_warnings_interval = \ - mm_cfg.DEFAULT_BOUNCE_YOU_ARE_DISABLED_WARNINGS_INTERVAL + config.DEFAULT_BOUNCE_YOU_ARE_DISABLED_WARNINGS_INTERVAL self.bounce_unrecognized_goes_to_list_owner = \ - mm_cfg.DEFAULT_BOUNCE_UNRECOGNIZED_GOES_TO_LIST_OWNER + config.DEFAULT_BOUNCE_UNRECOGNIZED_GOES_TO_LIST_OWNER self.bounce_notify_owner_on_disable = \ - mm_cfg.DEFAULT_BOUNCE_NOTIFY_OWNER_ON_DISABLE + config.DEFAULT_BOUNCE_NOTIFY_OWNER_ON_DISABLE self.bounce_notify_owner_on_removal = \ - mm_cfg.DEFAULT_BOUNCE_NOTIFY_OWNER_ON_REMOVAL + config.DEFAULT_BOUNCE_NOTIFY_OWNER_ON_REMOVAL # Not configurable... # # This holds legacy member related information. It's keyed by the @@ -153,7 +154,7 @@ class Bouncer: # Now that we've adjusted the bounce score for this bounce, let's # check to see if the disable-by-bounce threshold has been reached. if info.score >= self.bounce_score_threshold: - if mm_cfg.VERP_PROBES: + if config.VERP_PROBES: log.info('sending %s list probe to: %s (score %s >= %s)', self.internal_name(), member, info.score, self.bounce_score_threshold) @@ -168,7 +169,7 @@ class Bouncer: cookie = self.pend_new(Pending.RE_ENABLE, self.internal_name(), member) info.cookie = cookie # Disable them - if mm_cfg.VERP_PROBES: + if config.VERP_PROBES: log.info('%s: %s disabling due to probe bounce received', self.internal_name(), member) else: diff --git a/Mailman/Cgi/admin.py b/Mailman/Cgi/admin.py index 3fbbd4ca4..d0da502bf 100644 --- a/Mailman/Cgi/admin.py +++ b/Mailman/Cgi/admin.py @@ -22,7 +22,6 @@ import re import cgi import sha import sys -import signal import urllib import logging @@ -127,40 +126,8 @@ def main(): # The html page document doc = Document() doc.set_language(mlist.preferred_language) - - # From this point on, the MailList object must be locked. However, we - # must release the lock no matter how we exit. try/finally isn't enough, - # because of this scenario: user hits the admin page which may take a long - # time to render; user gets bored and hits the browser's STOP button; - # browser shuts down socket; server tries to write to broken socket and - # gets a SIGPIPE. Under Apache 1.3/mod_cgi, Apache catches this SIGPIPE - # (I presume it is buffering output from the cgi script), then turns - # around and SIGTERMs the cgi process. Apache waits three seconds and - # then SIGKILLs the cgi process. We /must/ catch the SIGTERM and do the - # most reasonable thing we can in as short a time period as possible. If - # we get the SIGKILL we're screwed (because it's uncatchable and we'll - # have no opportunity to clean up after ourselves). - # - # This signal handler catches the SIGTERM, unlocks the list, and then - # exits the process. The effect of this is that the changes made to the - # MailList object will be aborted, which seems like the only sensible - # semantics. - # - # BAW: This may not be portable to other web servers or cgi execution - # models. - def sigterm_handler(signum, frame, mlist=mlist): - # Make sure the list gets unlocked... - mlist.Unlock() - # ...and ensure we exit, otherwise race conditions could cause us to - # enter MailList.Save() while we're in the unlocked state, and that - # could be bad! - sys.exit(0) - mlist.Lock() try: - # Install the emergency shutdown signal handler - signal.signal(signal.SIGTERM, sigterm_handler) - if cgidata.keys(): # There are options to change change_options(mlist, category, subcat, cgidata, doc) @@ -190,10 +157,6 @@ def main(): print doc.Format() mlist.Save() finally: - # Now be sure to unlock the list. It's okay if we get a signal here - # because essentially, the signal handler will do the same thing. And - # unlocking is unconditional, so it's not an error if we unlock while - # we're already unlocked. mlist.Unlock() diff --git a/Mailman/Cgi/admindb.py b/Mailman/Cgi/admindb.py index a589d1019..33b4b2627 100644 --- a/Mailman/Cgi/admindb.py +++ b/Mailman/Cgi/admindb.py @@ -23,7 +23,6 @@ import sys import time import email import errno -import signal import logging from urllib import quote_plus, unquote_plus @@ -134,27 +133,8 @@ def main(): if qs and isinstance(qs, list): details = qs[0] - # We need a signal handler to catch the SIGTERM that can come from Apache - # when the user hits the browser's STOP button. See the comment in - # admin.py for details. - # - # BAW: Strictly speaking, the list should not need to be locked just to - # read the request database. However the request database asserts that - # the list is locked in order to load it and it's not worth complicating - # that logic. - def sigterm_handler(signum, frame, mlist=mlist): - # Make sure the list gets unlocked... - mlist.Unlock() - # ...and ensure we exit, otherwise race conditions could cause us to - # enter MailList.Save() while we're in the unlocked state, and that - # could be bad! - sys.exit(0) - mlist.Lock() try: - # Install the emergency shutdown signal handler - signal.signal(signal.SIGTERM, sigterm_handler) - realname = mlist.real_name if not cgidata.keys(): # If this is not a form submission (i.e. there are no keys in the diff --git a/Mailman/Cgi/create.py b/Mailman/Cgi/create.py index 378fa8f37..8c11f22e5 100644 --- a/Mailman/Cgi/create.py +++ b/Mailman/Cgi/create.py @@ -21,7 +21,6 @@ import os import cgi import sha import sys -import signal import logging from Mailman import Errors @@ -161,18 +160,7 @@ def process_request(doc, cgidata): fqdn_listname = '%s@%s' % (listname, email_host) # We've got all the data we need, so go ahead and try to create the list mlist = MailList.MailList() - # See admin.py for why we need to set up the signal handler. - def sigterm_handler(signum, frame): - # Make sure the list gets unlocked... - mlist.Unlock() - # ...and ensure we exit, otherwise race conditions could cause us to - # enter MailList.Save() while we're in the unlocked state, and that - # could be bad! - sys.exit(0) try: - # Install the emergency shutdown signal handler - signal.signal(signal.SIGTERM, sigterm_handler) - pw = sha.new(password).hexdigest() # Guarantee that all newly created files have the proper permission. # proper group ownership should be assured by the autoconf script @@ -205,10 +193,6 @@ def process_request(doc, cgidata): mlist.default_member_moderation = moderate mlist.Save() finally: - # Now be sure to unlock the list. It's okay if we get a signal here - # because essentially, the signal handler will do the same thing. And - # unlocking is unconditional, so it's not an error if we unlock while - # we're already unlocked. mlist.Unlock() # Now do the MTA-specific list creation tasks if config.MTA: diff --git a/Mailman/Cgi/options.py b/Mailman/Cgi/options.py index fceeaaaf6..3b86aeb99 100644 --- a/Mailman/Cgi/options.py +++ b/Mailman/Cgi/options.py @@ -20,7 +20,6 @@ import os import cgi import sys -import signal import urllib import logging @@ -369,13 +368,6 @@ address. Upon confirmation, any other mailing list containing the address _('Addresses may not be blank')) print doc.Format() return - - # Standard sigterm handler. - def sigterm_handler(signum, frame, mlist=mlist): - mlist.Unlock() - sys.exit(0) - - signal.signal(signal.SIGTERM, sigterm_handler) if set_address: if cpuser is None: cpuser = user @@ -463,15 +455,6 @@ address. Upon confirmation, any other mailing list containing the address print doc.Format() return - # Standard signal handler - def sigterm_handler(signum, frame, mlist=mlist): - mlist.Unlock() - sys.exit(0) - - # Okay, zap them. Leave them sitting at the list's listinfo page. We - # must own the list lock, and we want to make sure the user (BAW: and - # list admin?) is informed of the removal. - signal.signal(signal.SIGTERM, sigterm_handler) mlist.Lock() needapproval = False try: @@ -582,7 +565,6 @@ address. Upon confirmation, any other mailing list containing the address # Now, lock the list and perform the changes mlist.Lock() try: - signal.signal(signal.SIGTERM, sigterm_handler) # `values' is a tuple of flags and the web values for flag, newval in newvals: # Handle language settings differently @@ -926,23 +908,10 @@ def lists_of_member(mlist, user): def change_password(mlist, user, newpw, confirmpw): - # This operation requires the list lock, so let's set up the signal - # handling so the list lock will get released when the user hits the - # browser stop button. - def sigterm_handler(signum, frame, mlist=mlist): - # Make sure the list gets unlocked... - mlist.Unlock() - # ...and ensure we exit, otherwise race conditions could cause us to - # enter MailList.Save() while we're in the unlocked state, and that - # could be bad! - sys.exit(0) - # Must own the list lock! mlist.Lock() try: - # Install the emergency shutdown signal handler - signal.signal(signal.SIGTERM, sigterm_handler) - # change the user's password. The password must already have been + # Change the user's password. The password must already have been # compared to the confirmpw and otherwise been vetted for # acceptability. mlist.setMemberPassword(user, newpw) @@ -973,9 +942,6 @@ def global_options(mlist, user, globalopts): # Must own the list lock! mlist.Lock() try: - # Install the emergency shutdown signal handler - signal.signal(signal.SIGTERM, sigterm_handler) - if globalopts.enable is not None: mlist.setDeliveryStatus(user, globalopts.enable) diff --git a/Mailman/Cgi/subscribe.py b/Mailman/Cgi/subscribe.py index 6aee0e6f8..5c3e5382b 100644 --- a/Mailman/Cgi/subscribe.py +++ b/Mailman/Cgi/subscribe.py @@ -20,7 +20,6 @@ import os import cgi import sys -import signal import logging from Mailman import Errors @@ -76,27 +75,8 @@ def main(): i18n.set_language(language) doc.set_language(language) - # We need a signal handler to catch the SIGTERM that can come from Apache - # when the user hits the browser's STOP button. See the comment in - # admin.py for details. - # - # BAW: Strictly speaking, the list should not need to be locked just to - # read the request database. However the request database asserts that - # the list is locked in order to load it and it's not worth complicating - # that logic. - def sigterm_handler(signum, frame, mlist=mlist): - # Make sure the list gets unlocked... - mlist.Unlock() - # ...and ensure we exit, otherwise race conditions could cause us to - # enter MailList.Save() while we're in the unlocked state, and that - # could be bad! - sys.exit(0) - mlist.Lock() try: - # Install the emergency shutdown signal handler - signal.signal(signal.SIGTERM, sigterm_handler) - process_form(mlist, doc, cgidata, language) mlist.Save() finally: diff --git a/Mailman/Defaults.py.in b/Mailman/Defaults.py.in index f539f4e39..4c4a0a822 100644 --- a/Mailman/Defaults.py.in +++ b/Mailman/Defaults.py.in @@ -100,20 +100,18 @@ HTML_TO_PLAIN_TEXT_COMMAND = '/usr/bin/lynx -dump %(filename)s' # Database options ##### -# Specify the name of the membership adaptor class that implements the -# MemberAdaptor interface you want to use. The first is traditional -# pickle-based adapter that was standard for Mailman 2.1 and earlier. -MEMBER_ADAPTOR_CLASS = 'Mailman.OldStyleMemberships.OldStyleMemberships' +# Use this to set the SQLAlchemy database engine URL. You generally have one +# primary database connection for all of Mailman. List data and most rosters +# will store their data in this database, although external rosters may access +# other databases in their own way. This string support substitutions using +# any variable in the Configuration object. +SQLALCHEMY_ENGINE_URL = 'sqlite:///$DATA_DIR/mailman.db' -# This is the SQLAlchemy-based adaptor which lets you store all membership -# data in any of several supported RDBMs. -#MEMBER_ADAPTOR_CLASS = 'Mailman.SAMemberships.SAMemberships' +# For debugging purposes +SQLALCHEMY_ECHO = False -# If you choose the SAMemberships adaptor, set this variable to specify the -# connection url to the backend database engine. Specify the placeholder -# $listdir for the directory that list data is stored in, and $listname for -# the name of the mailing list. -SQLALCHEMY_ENGINE_URL = 'sqlite:///$listdir/members.db' +# XXX REMOVE ME +MEMBER_ADAPTOR_CLASS = 'Mailman.OldStyleMemberships.OldStyleMemberships' diff --git a/Mailman/Digester.py b/Mailman/Digester.py index 2d27b8442..98cb40dd3 100644 --- a/Mailman/Digester.py +++ b/Mailman/Digester.py @@ -1,50 +1,50 @@ -# Copyright (C) 1998,1999,2000,2001,2002 by the Free Software Foundation, Inc. +# Copyright (C) 1998-2006 by the Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. """Mixin class with list-digest handling methods and settings.""" import os -from stat import ST_SIZE import errno -from Mailman import mm_cfg -from Mailman import Utils from Mailman import Errors +from Mailman import Utils from Mailman.Handlers import ToDigest +from Mailman.configuration import config from Mailman.i18n import _ class Digester: def InitVars(self): - # Configurable - self.digestable = mm_cfg.DEFAULT_DIGESTABLE - self.digest_is_default = mm_cfg.DEFAULT_DIGEST_IS_DEFAULT - self.mime_is_default_digest = mm_cfg.DEFAULT_MIME_IS_DEFAULT_DIGEST - self.digest_size_threshhold = mm_cfg.DEFAULT_DIGEST_SIZE_THRESHHOLD - self.digest_send_periodic = mm_cfg.DEFAULT_DIGEST_SEND_PERIODIC - self.next_post_number = 1 - self.digest_header = mm_cfg.DEFAULT_DIGEST_HEADER - self.digest_footer = mm_cfg.DEFAULT_DIGEST_FOOTER - self.digest_volume_frequency = mm_cfg.DEFAULT_DIGEST_VOLUME_FREQUENCY - # Non-configurable. + # Configurable + self.digestable = config.DEFAULT_DIGESTABLE + self.digest_is_default = config.DEFAULT_DIGEST_IS_DEFAULT + self.mime_is_default_digest = config.DEFAULT_MIME_IS_DEFAULT_DIGEST + self.digest_size_threshhold = config.DEFAULT_DIGEST_SIZE_THRESHHOLD + self.digest_send_periodic = config.DEFAULT_DIGEST_SEND_PERIODIC + self.next_post_number = 1 + self.digest_header = config.DEFAULT_DIGEST_HEADER + self.digest_footer = config.DEFAULT_DIGEST_FOOTER + self.digest_volume_frequency = config.DEFAULT_DIGEST_VOLUME_FREQUENCY + # Non-configurable. self.one_last_digest = {} - self.digest_members = {} - self.next_digest_number = 1 + self.digest_members = {} + self.next_digest_number = 1 self.digest_last_sent_at = 0 def send_digest_now(self): @@ -55,7 +55,7 @@ class Digester: try: mboxfp = None # See if there's a digest pending for this mailing list - if os.stat(digestmbox)[ST_SIZE] > 0: + if os.stat(digestmbox).st_size > 0: mboxfp = open(digestmbox) ToDigest.send_digests(self, mboxfp) os.unlink(digestmbox) @@ -63,10 +63,11 @@ class Digester: if mboxfp: mboxfp.close() except OSError, e: - if e.errno <> errno.ENOENT: raise + if e.errno <> errno.ENOENT: + raise # List has no outstanding digests - return 0 - return 1 + return False + return True def bump_digest_volume(self): self.volume += 1 diff --git a/Mailman/Errors.py b/Mailman/Errors.py index 26333a688..f2d4fad19 100644 --- a/Mailman/Errors.py +++ b/Mailman/Errors.py @@ -175,3 +175,19 @@ class HostileSubscriptionError(MailmanError): """A cross-subscription attempt was made.""" # This exception gets raised when an invitee attempts to use the # invitation to cross-subscribe to some other mailing list. + + + +# Database exceptions +class DatabaseError(MailmanError): + """A problem with the database occurred.""" + + +class SchemaVersionMismatchError(DatabaseError): + def __init__(self, got): + self._got = got + + def __str__(self): + from Mailman.Version import DATABASE_SCHEMA_VERSION + return 'Incompatible database schema version (got: %d, expected: %d)' \ + % (self._got, DATABASE_SCHEMA_VERSION) diff --git a/Mailman/GatewayManager.py b/Mailman/GatewayManager.py index 094f0c76d..04c630cbe 100644 --- a/Mailman/GatewayManager.py +++ b/Mailman/GatewayManager.py @@ -1,4 +1,4 @@ -# Copyright (C) 1998,1999,2000,2001,2002 by the Free Software Foundation, Inc. +# Copyright (C) 1998-2006 by the Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License @@ -12,7 +12,8 @@ # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. """Mixin class for configuring Usenet gateway. @@ -21,18 +22,18 @@ gateway and cron/gate_news for the news->mail gateway. """ -from Mailman import mm_cfg -from Mailman.i18n import _ +from Mailman.configuration import config + class GatewayManager: def InitVars(self): # Configurable - self.nntp_host = mm_cfg.DEFAULT_NNTP_HOST + self.nntp_host = config.DEFAULT_NNTP_HOST self.linked_newsgroup = '' - self.gateway_to_news = 0 - self.gateway_to_mail = 0 - self.news_prefix_subject_too = 1 + self.gateway_to_news = False + self.gateway_to_mail = False + self.news_prefix_subject_too = True # In patch #401270, this was called newsgroup_is_moderated, but the # semantics weren't quite the same. - self.news_moderation = 0 + self.news_moderation = False diff --git a/Mailman/MTA/Postfix.py b/Mailman/MTA/Postfix.py index 595581718..2c6f8c60e 100644 --- a/Mailman/MTA/Postfix.py +++ b/Mailman/MTA/Postfix.py @@ -284,7 +284,7 @@ def create(mlist, cgi=False, nolock=False, quiet=False): if not nolock: lock = makelock() lock.lock() - # Do the aliases file, which need to be done in any case + # Do the aliases file, which always needs to be done try: if config.USE_LMTP: _do_create(mlist, TRPTFILE, _addtransport) diff --git a/Mailman/MailList.py b/Mailman/MailList.py index 555717eff..43ae3046c 100644 --- a/Mailman/MailList.py +++ b/Mailman/MailList.py @@ -46,6 +46,7 @@ from Mailman import Errors from Mailman import LockFile from Mailman import Utils from Mailman import Version +from Mailman import database from Mailman.UserDesc import UserDesc from Mailman.configuration import config @@ -84,9 +85,31 @@ slog = logging.getLogger('mailman.subscribe') # Use mixins here just to avoid having any one chunk be too large. -class MailList(HTMLFormatter, Deliverer, ListAdmin, +class MailList(object, HTMLFormatter, Deliverer, ListAdmin, Archiver, Digester, SecurityManager, Bouncer, GatewayManager, Autoresponder, TopicMgr, Pending.Pending): + def __new__(cls, *args, **kws): + # Search positional and keyword arguments to find the name of the + # existing list that is being opened, with the latter taking + # precedence. If no name can be found, then make sure there are no + # arguments, otherwise it's an error. + if 'name' in kws: + listname = kws.pop('name') + elif not args: + if not kws: + # We're probably being created from the ORM layer, so just let + # the super class do its thing. + return super(MailList, cls).__new__(cls, *args, **kws) + raise ValueError("'name' argument required'") + else: + listname = args[0] + fqdn_listname = Utils.fqdn_listname(listname) + listname, hostname = Utils.split_listname(fqdn_listname) + mlist = database.find_list(listname, hostname) + if not mlist: + raise Errors.MMUnknownListError(fqdn_listname) + return mlist + # # A MailList object's basic Python object model support # @@ -100,13 +123,6 @@ class MailList(HTMLFormatter, Deliverer, ListAdmin, baseclass.__init__(self) # Initialize volatile attributes self.InitTempVars(name, check_version) - # Attach a membership adaptor instance. - parts = config.MEMBER_ADAPTOR_CLASS.split(DOT) - adaptor_class = parts.pop() - adaptor_module = DOT.join(parts) - __import__(adaptor_module) - mod = sys.modules[adaptor_module] - self._memberadaptor = getattr(mod, adaptor_class)(self) # This extension mechanism allows list-specific overrides of any # method (well, except __init__(), InitTempVars(), and InitVars() # I think). Note that fullpath() will return None when we're creating @@ -134,6 +150,8 @@ class MailList(HTMLFormatter, Deliverer, ListAdmin, self.Load(name, check_version) def __getattr__(self, name): + if name.startswith('_'): + return super(MailList, self).__getattr__(name) # Because we're using delegation, we want to be sure that attribute # access to a delegated member function gets passed to the # sub-objects. This of course imposes a specific name resolution @@ -147,7 +165,7 @@ class MailList(HTMLFormatter, Deliverer, ListAdmin, except AttributeError: pass else: - raise AttributeError, name + raise AttributeError(name) def __repr__(self): if self.Locked(): @@ -162,6 +180,7 @@ class MailList(HTMLFormatter, Deliverer, ListAdmin, # Lock management # def Lock(self, timeout=0): + database.lock(self) self.__lock.lock(timeout) self._memberadaptor.lock() # Must reload our database for consistency. Watch out for lists that @@ -173,6 +192,7 @@ class MailList(HTMLFormatter, Deliverer, ListAdmin, raise def Unlock(self): + database.unlock(self) self.__lock.unlock(unconditionally=True) self._memberadaptor.unlock() @@ -281,6 +301,13 @@ class MailList(HTMLFormatter, Deliverer, ListAdmin, # def InitTempVars(self, name, check_version=True): """Set transient variables of this and inherited classes.""" + # Attach a membership adaptor instance. + parts = config.MEMBER_ADAPTOR_CLASS.split(DOT) + adaptor_class = parts.pop() + adaptor_module = DOT.join(parts) + __import__(adaptor_module) + mod = sys.modules[adaptor_module] + self._memberadaptor = getattr(mod, adaptor_class)(self) # The timestamp is set whenever we load the state from disk. If our # timestamp is newer than the modtime of the config.pck file, we don't # need to reload, otherwise... we do. @@ -519,7 +546,7 @@ class MailList(HTMLFormatter, Deliverer, ListAdmin, raise Errors.MMListAlreadyExistsError(fqdn_listname) # Validate the admin's email address Utils.ValidateEmail(admin_email) - self._internal_name = listname + self._internal_name = self.list_name = listname self._full_path = os.path.join(config.LIST_DATA_DIR, fqdn_listname) Utils.makedirs(self._full_path) # Don't use Lock() since that tries to load the non-existant config.pck @@ -537,206 +564,35 @@ class MailList(HTMLFormatter, Deliverer, ListAdmin, self.available_languages = langs url_host = config.domains[email_host] self.web_page_url = config.DEFAULT_URL_PATTERN % url_host + database.add_list(self) - # - # Database and filesystem I/O - # - def __save(self, dict): - # Save the file as a binary pickle, and rotate the old version to a - # backup file. We must guarantee that config.pck is always valid so - # we never rotate unless the we've successfully written the temp file. - # We use pickle now because marshal is not guaranteed to be compatible - # between Python versions. - fname = os.path.join(self.fullpath(), 'config.pck') - fname_tmp = fname + '.tmp.%s.%d' % (socket.gethostname(), os.getpid()) - fname_last = fname + '.last' - fp = None - try: - fp = open(fname_tmp, 'w') - # Use a binary format... it's more efficient. - cPickle.dump(dict, fp, 1) - fp.flush() - if config.SYNC_AFTER_WRITE: - os.fsync(fp.fileno()) - fp.close() - except IOError, e: - elog.error('Failed config.pck write, retaining old state.\n%s', e) - if fp is not None: - os.unlink(fname_tmp) - raise - # Now do config.pck.tmp.xxx -> config.pck -> config.pck.last rotation - # as safely as possible. - try: - # might not exist yet - os.unlink(fname_last) - except OSError, e: - if e.errno <> errno.ENOENT: raise - try: - # might not exist yet - os.link(fname, fname_last) - except OSError, e: - if e.errno <> errno.ENOENT: raise - os.rename(fname_tmp, fname) - # Reset the timestamp - self.__timestamp = os.path.getmtime(fname) - def Save(self): # Refresh the lock, just to let other processes know we're still # interested in it. This will raise a NotLockedError if we don't have # the lock (which is a serious problem!). TBD: do we need to be more # defensive? self.__lock.refresh() + # Commit the database transaction + database.save(self) + # The member adaptor may have its own save operation self._memberadaptor.save() - # copy all public attributes to serializable dictionary - dict = {} - for key, value in self.__dict__.items(): - if key[0] == '_' or isinstance(value, MethodType): - continue - dict[key] = value - # Make config.pck unreadable by `other', as it contains all the - # list members' passwords (in clear text). - omask = os.umask(007) - try: - self.__save(dict) - finally: - os.umask(omask) - self.SaveRequestsDb() + self.SaveRequestsDb() self.CheckHTMLArchiveDir() - def __load(self, dbfile): - # Attempt to load and unserialize the specified database file. This - # could actually be a config.db (for pre-2.1alpha3) or config.pck, - # i.e. a marshal or a binary pickle. Actually, it could also be a - # .last backup file if the primary storage file was corrupt. The - # decision on whether to unpickle or unmarshal is based on the file - # extension, but we always save it using pickle (since only it, and - # not marshal is guaranteed to be compatible across Python versions). - # - # On success return a 2-tuple of (dictionary, None). On error, return - # a 2-tuple of the form (None, errorobj). - if dbfile.endswith('.db') or dbfile.endswith('.db.last'): - loadfunc = marshal.load - elif dbfile.endswith('.pck') or dbfile.endswith('.pck.last'): - loadfunc = cPickle.load - else: - assert 0, 'Bad database file name' - try: - # Check the mod time of the file first. If it matches our - # timestamp, then the state hasn't change since the last time we - # loaded it. Otherwise open the file for loading, below. If the - # file doesn't exist, we'll get an EnvironmentError with errno set - # to ENOENT (EnvironmentError is the base class of IOError and - # OSError). - mtime = os.path.getmtime(dbfile) - if mtime <= self.__timestamp: - # File is not newer - return None, None - fp = open(dbfile) - except EnvironmentError, e: - if e.errno <> errno.ENOENT: raise - # The file doesn't exist yet - return None, e - try: - try: - d = loadfunc(fp) - if not isinstance(d, dict): - return None, 'Load() expected to return a dictionary' - except (EOFError, ValueError, TypeError, MemoryError, - cPickle.PicklingError, cPickle.UnpicklingError), e: - return None, e - finally: - fp.close() - # Update timestamp - self.__timestamp = mtime - return d, None - def Load(self, fqdn_listname=None, check_version=True): if fqdn_listname is None: fqdn_listname = self.fqdn_listname if not Utils.list_exists(fqdn_listname): raise Errors.MMUnknownListError(fqdn_listname) self._memberadaptor.load() - # We first try to load config.pck, which contains the up-to-date - # version of the database. If that fails, perhaps because it's - # corrupted or missing, we'll try to load the backup file - # config.pck.last. - # - # Should both of those fail, we'll look for config.db and - # config.db.last for backwards compatibility with pre-2.1alpha3 - pfile = os.path.join(self.fullpath(), 'config.pck') - plast = pfile + '.last' - dfile = os.path.join(self.fullpath(), 'config.db') - dlast = dfile + '.last' - for file in (pfile, plast, dfile, dlast): - dict, e = self.__load(file) - if dict is None: - if e is not None: - # Had problems with this file; log it and try the next one. - elog.error("couldn't load config file %s\n%s", file, e) - else: - # We already have the most up-to-date state - return - else: - break - else: - # Nothing worked, so we have to give up - elog.error('All %s fallbacks were corrupt, giving up', - self.internal_name()) - raise Errors.MMCorruptListDatabaseError, e - # Now, if we didn't end up using the primary database file, we want to - # copy the fallback into the primary so that the logic in Save() will - # still work. For giggles, we'll copy it to a safety backup. Note we - # MUST do this with the underlying list lock acquired. - if file == plast or file == dlast: - elog.error('fixing corrupt config file, using: %s', file) - unlock = True - try: - try: - self.__lock.lock() - except LockFile.AlreadyLockedError: - unlock = False - self.__fix_corrupt_pckfile(file, pfile, plast, dfile, dlast) - finally: - if unlock: - self.__lock.unlock() - # Copy the loaded dictionary into the attributes of the current - # mailing list object, then run sanity check on the data. - self.__dict__.update(dict) if check_version: - self.CheckVersion(dict) + # XXX for now disable version checks. We'll fold this into schema + # updates eventually. + #self.CheckVersion(dict) self.CheckValues() - def __fix_corrupt_pckfile(self, file, pfile, plast, dfile, dlast): - if file == plast: - # Move aside any existing pickle file and delete any existing - # safety file. This avoids EPERM errors inside the shutil.copy() - # calls if those files exist with different ownership. - try: - os.rename(pfile, pfile + '.corrupt') - except OSError, e: - if e.errno <> errno.ENOENT: raise - try: - os.remove(pfile + '.safety') - except OSError, e: - if e.errno <> errno.ENOENT: raise - shutil.copy(file, pfile) - shutil.copy(file, pfile + '.safety') - elif file == dlast: - # Move aside any existing marshal file and delete any existing - # safety file. This avoids EPERM errors inside the shutil.copy() - # calls if those files exist with different ownership. - try: - os.rename(dfile, dfile + '.corrupt') - except OSError, e: - if e.errno <> errno.ENOENT: raise - try: - os.remove(dfile + '.safety') - except OSError, e: - if e.errno <> errno.ENOENT: raise - shutil.copy(file, dfile) - shutil.copy(file, dfile + '.safety') # diff --git a/Mailman/Makefile.in b/Mailman/Makefile.in index 5caf04719..2a7bc5e55 100644 --- a/Mailman/Makefile.in +++ b/Mailman/Makefile.in @@ -44,7 +44,7 @@ SHELL= /bin/sh MODULES= $(srcdir)/*.py SUBDIRS= Cgi Archiver Handlers Bouncers Queue MTA Gui Commands \ - bin testing + bin database testing # Modes for directories and executables created by the install # process. Default to group-writable directories but diff --git a/Mailman/Pending.py b/Mailman/Pending.py index 0c98822bd..eb3b7950e 100644 --- a/Mailman/Pending.py +++ b/Mailman/Pending.py @@ -12,7 +12,8 @@ # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. """Track pending actions which require confirmation.""" @@ -23,7 +24,8 @@ import errno import random import cPickle -from Mailman import mm_cfg +from Mailman.configuration import config + # Types of pending records CHANGE_OF_ADDRESS = 'C' @@ -43,6 +45,7 @@ _ALLKEYS = ( ) _missing = object() +_default = object() @@ -54,7 +57,7 @@ class Pending: """Create a new entry in the pending database, returning cookie for it. """ assert op in _ALLKEYS, 'op: %s' % op - lifetime = kws.get('lifetime', mm_cfg.PENDING_REQUEST_LIFE) + lifetime = kws.get('lifetime', config.PENDING_REQUEST_LIFE) # We try the main loop several times. If we get a lock error somewhere # (for instance because someone broke the lock) we simply try again. assert self.Locked() @@ -108,7 +111,7 @@ class Pending: for cookie in evictions.keys(): if not db.has_key(cookie): del evictions[cookie] - db['version'] = mm_cfg.PENDING_FILE_SCHEMA_VERSION + db['version'] = config.PENDING_FILE_SCHEMA_VERSION tmpfile = '%s.tmp.%d.%d' % (self.__pendfile, os.getpid(), now) omask = os.umask(007) try: @@ -145,8 +148,10 @@ class Pending: self.__save(db) return content - def pend_repend(self, cookie, data, lifetime=mm_cfg.PENDING_REQUEST_LIFE): + def pend_repend(self, cookie, data, lifetime=_default): assert self.Locked() + if lifetime is _default: + lifetime = config.PENDING_REQUEST_LIFE db = self.__load() db[cookie] = data db['evictions'][cookie] = time.time() + lifetime @@ -173,9 +178,9 @@ def _update(olddb): # subscription language. Best we can do here is use the server # default. db[cookie] = (SUBSCRIPTION,) + data[:-1] + \ - (mm_cfg.DEFAULT_SERVER_LANGUAGE,) + (config.DEFAULT_SERVER_LANGUAGE,) # The old database format kept the timestamp as the time the request # was made. The new format keeps it as the time the request should be # evicted. - evictions[cookie] = data[-1] + mm_cfg.PENDING_REQUEST_LIFE + evictions[cookie] = data[-1] + config.PENDING_REQUEST_LIFE return db diff --git a/Mailman/Queue/HTTPRunner.py b/Mailman/Queue/HTTPRunner.py index e542dc9a3..e2c053629 100644 --- a/Mailman/Queue/HTTPRunner.py +++ b/Mailman/Queue/HTTPRunner.py @@ -61,6 +61,8 @@ server = make_server(config.HTTP_HOST, config.HTTP_PORT, qlog.info('HTTPRunner qrunner started.') -server.serve_forever() -# We'll never get here, but just in case... -qlog.info('HTTPRunner qrunner exiting.') +try: + server.serve_forever() +except: + qlog.exception('HTTPRunner qrunner exiting.') + raise diff --git a/Mailman/SAMemberships.py b/Mailman/SAMemberships.py deleted file mode 100644 index dec76fe58..000000000 --- a/Mailman/SAMemberships.py +++ /dev/null @@ -1,330 +0,0 @@ -# Copyright (C) 2006 by the Free Software Foundation, Inc. -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, -# USA. - -"""An experimental SQLAlchemy-based membership adaptor.""" - -# XXX THIS FILE DOES NOT YET WORK! - -import os -import re -import time - -from sqlalchemy import * -from string import Template - -from Mailman import Defaults -from Mailman import Errors -from Mailman import MemberAdaptor -from Mailman import Utils -from Mailman.configuration import config - -NUL = '\0' - - - -# Python classes representing the data in the SQLAlchemy database. These will -# be associated with tables via object mappers. - -class Member(object): - def __init__(self, mlist, - address, realname=None, password=None, - digests_p=False, language=None): - self.lckey = address.lower() - self.address = address - self.realname = realname - self.password = password or Utils.MakeRandomPassword() - self.language = language or mlist.preferred_language - self.digests_p = digests_p - self.options = mlist.new_member_options - self.topics = '' - self.status = MemberAdaptor.ENABLED - # XXX This should really be a datetime - self.disable_time = 0 - - - -_table = None -_metadata = None -_mapper = None - -def create_table(): - global _table, _metadata, _mapper - - if _table: - return - _metadata = MetaData('table metadata') - _table = Table( - 'members', _metadata, - Column('member_id', Integer, primary_key=True), - Column('lckey', Unicode(), index=True, nullable=False), - Column('address', Unicode(), index=True, nullable=False), - Column('realname', Unicode()), - Column('password', Unicode()), - Column('language', String(2)), - Column('digest', Boolean), - Column('options', Integer), - Column('status', Integer), - Column('disable_time', Float), - ) - _mapper = mapper(Member, _table) - - - -class SAMemberships(MemberAdaptor.MemberAdaptor): - def __init__(self, mlist): - self._mlist = mlist - self._metadata = None - self._session = None - self._txn = None - - def _connect(self): - create_table() - # We cannot connect in the __init__() because our adaptor requires the - # fqdn_listname to exist. In MailList.Create() that won't be the case. - # - # Calculate the engine url, expanding placeholder variables. - engine_url = Template(config.SQLALCHEMY_ENGINE_URL).substitute( - {'listname' : self._mlist.fqdn_listname, - 'listdir' : os.path.join(config.LIST_DATA_DIR, - self._mlist.fqdn_listname), - }) - print 'engine_url:', engine_url - self._engine = create_engine(engine_url) - self._session = create_session(bind_to=self._engine) - self._session.bind_table(_table, self._engine) - self._session.bind_mapper(_mapper, self._engine) - # XXX There must be a better way to figure out whether the tables need - # to be created or not. - try: - _table.create() - except exceptions.SQLError: - pass - - # - # The transaction interface - # - - def load(self): - if self._session is None: - self._connect() - assert self._txn is None - self._session.clear() - self._txn = self._session.create_transaction() - - def lock(self): - pass - - def save(self): - # When a MailList is first Create()'d, the load() callback doesn't get - # called, so there will be no transaction. - if self._txn: - self._txn.commit() - self._txn = None - - def unlock(self): - if self._txn is not None: - # The MailList has not been saved, but it is being unlocked, so - # throw away all pending changes. - self._txn.rollback() - self._txn = None - - # - # The readable interface - # - - def getMembers(self): - return [m.lckey for m in self._session.query(Member).select_by()] - - def getRegularMemberKeys(self): - query = self._session.query(Member) - return [m.lckey for m in query.select(Member.c.digests_p == False)] - - def getDigestMemberKeys(self): - query = self._session.query(Member) - return [m.lckey for m in query.select(Member.c.digests_p == True)] - - def _get_member(self, member): - members = self._session.query(Member).select_by(lckey=member.lower()) - if not members: - return None - assert len(members) == 1 - return members[0] - - def _get_member_strict(self, member): - member_obj = self._get_member(member) - if not member_obj: - raise Errors.NotAMemberError(member) - return member_obj - - def isMember(self, member): - return bool(self._get_member(member)) - - def getMemberKey(self, member): - self._get_member_strict(member) - return member.lower() - - def getMemberCPAddress(self, member): - return self._get_member(member).address - - def getMemberCPAddresses(self, members): - query = self._session.query(Member) - return [user.address for user in query.select( - in_(Member.c.lckey, [m.lower() for m in members]))] - - def getMemberPassword(self, member): - return self._get_member_strict(member).password - - def authenticateMember(self, member, response): - return self._get_member_strict(member).password == response - - def getMemberLanguage(self, member): - member = self._get_member(member) - if member and member.language in self._mlist.GetAvailableLanguages(): - return member.language - return self._mlist.preferred_language - - def getMemberOption(self, member, flag): - return bool(self._get_member_strict(member).options & flag) - - def getMemberName(self, member): - return self._get_member_strict(member).realname - - def getMemberTopics(self, member): - topics = self._get_member_strict(member).topics - if not topics: - return [] - return topics.split(NUL) - - def getDeliveryStatus(self, member): - return self._get_member_strict(member).status - - def getDeliveryStatusChangeTime(self, member): - member = self._get_member_strict(member) - if member.status == MemberAdaptor.ENABLED: - return 0 - return member.disable_time - - def getDeliveryStatusMembers(self, status=(MemberAdaptor.UNKNOWN, - MemberAdaptor.BYUSER, - MemberAdaptor.BYADMIN, - MemberAdaptor.BYBOUNCE)): - query = self._session.query(Member) - return [user.lckey for user in query.select( - in_(Member.c.status, status))] - - def getBouncingMembers(self): - "XXX" - - def getBounceInfo(self): - "XXX" - - # - # The writable interface - # - - def addNewMember(self, member, **kws): - assert self._mlist.Locked() - if self.isMember(member): - raise Errors.MMAlreadyAMember(member) - try: - new_member = Member(self._mlist, member, **kws) - self._session.save(new_member) - self._session.flush() - except TypeError: - # Transform exception to API specification - raise ValueError - - def removeMember(self, memberkey): - assert self._mlist.Locked() - member = self._get_member_strict(memberkey) - self._session.delete(member) - - def changeMemberAddress(self, memberkey, newaddress, nodelete=False): - assert self._mlist.Locked() - member = self._get_member_strict(memberkey) - # First, add the new member from the previous data - self.addNewMember(newaddress, member.realname, member.password, - member.digests_p, member.language) - new_member = self._get_member(newaddress) - assert new_member - new_member.options = member.options - new_member.topics = member.topics - new_member.status = MemberAdaptor.ENABLED - new_member.disable_time = 0 - if not nodelete: - self._session.delete(member) - - def setMemberPassword(self, member, password): - assert self._mlist.Locked() - self._get_member_strict(member).password = password - - def setMemberLanguage(self, member, language): - assert self._mlist.Locked() - self._get_member_strict(member).language = language - - def setMemberOption(self, member, flag, value): - assert self._mlist.Locked() - member = self._get_member_strict(member) - # XXX the OldStyleMemberships adaptor will raise CantDigestError, - # MustDigestError, AlreadyReceivingDigests, and - # AlreadyReceivingRegularDeliveries in certain cases depending on the - # configuration of the mailing list and the member's delivery status. - # These semantics are not defined in the API so to keep things simple, - # I am not reproducing them here. Ideally, adaptors should not be - # doing semantic integrity checks, but I'm also not going to change - # the OldStyleMemberships adaptor. - # - # We still need to handle digests differently, because they aren't - # really represented as a unique flag in the options bitfield. - if flag == Defaults.Digests: - member.digests_p = bool(value) - else: - if value: - member.options |= flag - else: - member.options &= ~flag - - def setMemberName(self, member, realname): - assert self._mlist.Locked() - self._get_member_strict(member).realname = realname - - def setMemberTopics(self, member, topics): - assert self._mlist.Locked() - # For simplicity, we represent a user's topics of interest as a - # null-joined string, which will be split properly by the accessor. - if not topics: - topics = None - else: - topics = NUL.join(topics) - self._get_member_strict(member).topics = topics - - def setDeliveryStatus(self, member, status): - assert status in (MemberAdaptor.ENABLED, MemberAdaptor.UNKNOWN, - MemberAdaptor.BYUSER, MemberAdaptor.BYADMIN, - MemberAdaptor.BYBOUNCE) - assert self._mlist.Locked() - member = self._get_member_strict(member) - if status == MemberAdaptor.ENABLED: - # XXX zap bounce info - disable_time = 0 - else: - disable_time = time.time() - member.disable_time = disable_time - member.status = status - - def setBounceInfo(self, member, info): - "XXX" diff --git a/Mailman/SecurityManager.py b/Mailman/SecurityManager.py index 3865071b5..6740a958f 100644 --- a/Mailman/SecurityManager.py +++ b/Mailman/SecurityManager.py @@ -28,7 +28,7 @@ # # Each cookie has the following ingredients: the authorization context's # secret (i.e. the password, and a timestamp. We generate an SHA1 hex -# digest of these ingredients, which we call the `mac'. We then marshal +# digest of these ingredients, which we call the 'mac'. We then marshal # up a tuple of the timestamp and the mac, hexlify that and return that as # a cookie keyed off the authcontext. Note that authenticating the user # also requires the user's email address to be included in the cookie. @@ -48,7 +48,6 @@ import os import re -import md5 import sha import time import urllib @@ -64,19 +63,15 @@ from Mailman import Errors from Mailman import Utils from Mailman.configuration import config -try: - import crypt -except ImportError: - crypt = None - log = logging.getLogger('mailman.error') +dlog = logging.getLogger('mailman.debug') + +SLASH = '/' class SecurityManager: def InitVars(self): - # We used to set self.password here, from a crypted_password argument, - # but that's been removed when we generalized the mixin architecture. # self.password is really a SecurityManager attribute, but it's set in # MailList.InitVars(). self.mod_password = None @@ -144,50 +139,15 @@ class SecurityManager: if ok: return Defaults.AuthSiteAdmin elif ac == Defaults.AuthListAdmin: - def cryptmatchp(response, secret): - try: - salt = secret[:2] - if crypt and crypt.crypt(response, salt) == secret: - return True - return False - except TypeError: - # BAW: Hard to say why we can get a TypeError here. - # SF bug report #585776 says crypt.crypt() can raise - # this if salt contains null bytes, although I don't - # know how that can happen (perhaps if a MM2.0 list - # with USE_CRYPT = 0 has been updated? Doubtful. - return False # The password for the list admin and list moderator are not # kept as plain text, but instead as an sha hexdigest. The # response being passed in is plain text, so we need to - # digestify it first. Note however, that for backwards - # compatibility reasons, we'll also check the admin response - # against the crypted and md5'd passwords, and if they match, - # we'll auto-migrate the passwords to sha. + # digestify it first. key, secret = self.AuthContextInfo(ac) if secret is None: continue sharesponse = sha.new(response).hexdigest() - upgrade = ok = False if sharesponse == secret: - ok = True - elif md5.new(response).digest() == secret: - ok = upgrade = True - elif cryptmatchp(response, secret): - ok = upgrade = True - if upgrade: - save_and_unlock = False - if not self.Locked(): - self.Lock() - save_and_unlock = True - try: - self.password = sharesponse - if save_and_unlock: - self.Save() - finally: - if save_and_unlock: - self.Unlock() - if ok: return ac elif ac == Defaults.AuthListModerator: # The list moderator password must be sha'd @@ -227,21 +187,22 @@ class SecurityManager: return False def _cookie_path(self): - return '/'.join(os.environ['SCRIPT_NAME'].split('/')[:-1]) + '/' + script_name = os.environ.get('SCRIPT_NAME', '') + return SLASH.join(script_name.split(SLASH)[:-1]) + SLASH def MakeCookie(self, authcontext, user=None): key, secret = self.AuthContextInfo(authcontext, user) - if key is None or secret is None or not isinstance(secret, str): + if key is None or secret is None or not isinstance(secret, basestring): raise ValueError # Timestamp issued = int(time.time()) # Get a digest of the secret, plus other information. - mac = sha.new(secret + `issued`).hexdigest() + mac = sha.new(secret + repr(issued)).hexdigest() # Create the cookie object. c = Cookie.SimpleCookie() c[key] = binascii.hexlify(marshal.dumps((issued, mac))) c[key]['path'] = self._cookie_path() - # We use session cookies, so don't set `expires' or `max-age' keys. + # We use session cookies, so don't set 'expires' or 'max-age' keys. # Set the RFC 2109 required header. c[key]['version'] = 1 return c @@ -290,7 +251,7 @@ class SecurityManager: for k in c.keys(): if k.startswith(prefix): usernames.append(k[len(prefix):]) - # If any check out, we're golden. Note: `@'s are no longer legal + # If any check out, we're golden. Note: '@'s are no longer legal # values in cookie keys. for user in [Utils.UnobscureEmail(u) for u in usernames]: ok = self.__checkone(c, authcontext, user) @@ -307,7 +268,7 @@ class SecurityManager: key, secret = self.AuthContextInfo(authcontext, user) except Errors.NotAMemberError: return False - if not c.has_key(key) or not isinstance(secret, str): + if key not in c or not isinstance(secret, basestring): return False # Undo the encoding we performed in MakeCookie() above. BAW: I # believe this is safe from exploit because marshal can't be forced to @@ -329,7 +290,7 @@ class SecurityManager: return False # Calculate what the mac ought to be based on the cookie's timestamp # and the shared secret. - mac = sha.new(secret + `issued`).hexdigest() + mac = sha.new(secret + repr(issued)).hexdigest() if mac <> received_mac: return False # Authenticated! diff --git a/Mailman/TopicMgr.py b/Mailman/TopicMgr.py index 27f113fb6..9df1b34a8 100644 --- a/Mailman/TopicMgr.py +++ b/Mailman/TopicMgr.py @@ -1,27 +1,24 @@ -# Copyright (C) 2001,2002 by the Free Software Foundation, Inc. +# Copyright (C) 2001-2006 by the Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. -"""This class mixes in topic feature configuration for mailing lists. -""" +"""This class mixes in topic feature configuration for mailing lists.""" import re -from Mailman import mm_cfg -from Mailman.i18n import _ - class TopicMgr: @@ -46,7 +43,7 @@ class TopicMgr: # have a name or pattern are not saved when the submit button is # pressed). self.topics = [] - self.topics_enabled = 0 + self.topics_enabled = False self.topics_bodylines_limit = 5 # Non-configurable # diff --git a/Mailman/Utils.py b/Mailman/Utils.py index edbf25f31..d89d45f59 100644 --- a/Mailman/Utils.py +++ b/Mailman/Utils.py @@ -40,6 +40,7 @@ from email.Errors import HeaderParseError from string import ascii_letters, digits, whitespace from Mailman import Errors +from Mailman import database from Mailman.SafeDict import SafeDict from Mailman.configuration import config @@ -50,12 +51,13 @@ except NameError: from sets import Set as set -EMPTYSTRING = '' -UEMPTYSTRING = u'' +AT = '@' CR = '\r' -NL = '\n' DOT = '.' +EMPTYSTRING = '' IDENTCHARS = ascii_letters + digits + '_' +NL = '\n' +UEMPTYSTRING = u'' # Search for $(identifier)s strings, except that the trailing s is optional, # since that's a common mistake @@ -67,28 +69,26 @@ log = logging.getLogger('mailman.error') -def list_exists(listname): - """Return true iff list `listname' exists.""" - # The existance of any of the following file proves the list exists - # <wink>: config.pck, config.pck.last, config.db, config.db.last - # - # The former two are for 2.1alpha3 and beyond, while the latter two are - # for all earlier versions. - basepath = os.path.join(config.LIST_DATA_DIR, listname) - for ext in ('.pck', '.pck.last', '.db', '.db.last'): - dbfile = os.path.join(basepath, 'config' + ext) - if os.path.exists(dbfile): - return True - return False +def list_exists(fqdn_listname): + """Return true iff list `fqdn_listname' exists.""" + listname, hostname = split_listname(fqdn_listname) + return bool(database.find_list(listname, hostname)) def list_names(): - """Return the names of all lists in default list directory.""" - got = set() - for fn in os.listdir(config.LIST_DATA_DIR): - if list_exists(fn): - got.add(fn) - return got + """Return the fqdn names of all lists in default list directory.""" + return ['%s@%s' % (listname, hostname) + for listname, hostname in database.get_list_names()] + + +def split_listname(listname): + if AT in listname: + return listname.split(AT, 1) + return listname, config.DEFAULT_EMAIL_HOST + + +def fqdn_listname(listname): + return AT.join(split_listname(listname)) diff --git a/Mailman/Version.py b/Mailman/Version.py index 7a61ba545..474873314 100644 --- a/Mailman/Version.py +++ b/Mailman/Version.py @@ -36,6 +36,10 @@ REL_SERIAL = 1 HEX_VERSION = ((MAJOR_REV << 24) | (MINOR_REV << 16) | (MICRO_REV << 8) | (REL_LEVEL << 4) | (REL_SERIAL << 0)) + +# SQLAlchemy database schema version +DATABASE_SCHEMA_VERSION = 1 + # config.pck schema version number DATA_FILE_VERSION = 98 diff --git a/Mailman/bin/export.py b/Mailman/bin/export.py index 1419e95df..04f18ef16 100644 --- a/Mailman/bin/export.py +++ b/Mailman/bin/export.py @@ -41,6 +41,23 @@ DOLLAR_STRINGS = ('msg_header', 'msg_footer', 'autoresponse_admin_text', 'autoresponse_request_text') +TYPES = { + Defaults.Toggle : 'bool', + Defaults.Radio : 'radio', + Defaults.String : 'string', + Defaults.Text : 'text', + Defaults.Email : 'email', + Defaults.EmailList : 'email_list', + Defaults.Host : 'host', + Defaults.Number : 'number', + Defaults.FileUpload : 'upload', + Defaults.Select : 'select', + Defaults.Topics : 'topics', + Defaults.Checkbox : 'checkbox', + Defaults.EmailListEx : 'email_list_ex', + Defaults.HeaderFilter : 'header_filter', + } + class Indenter: @@ -57,7 +74,8 @@ class Indenter: assert self._indent >= 0 def write(self, s): - self._fp.write(self._indent * self._width * ' ') + if s <> '\n': + self._fp.write(self._indent * self._width * ' ') self._fp.write(s) @@ -150,13 +168,14 @@ class XMLDumper(object): continue if not is_converted and varname in DOLLAR_STRINGS: value = Utils.to_dollar(value) + widget_type = TYPES[vtype] if isinstance(value, list): - self._push_element('option', name=varname) + self._push_element('option', name=varname, type=widget_type) for v in value: self._element('value', v) self._pop_element('option') else: - self._element('option', value, name=varname) + self._element('option', value, name=varname, type=widget_type) def _dump_list(self, mlist, with_passwords): # Write list configuration values @@ -180,7 +199,7 @@ class XMLDumper(object): attrs = dict(id=member) cased = mlist.getMemberCPAddress(member) if cased <> member: - dict['original'] = cased + attrs['original'] = cased self._push_element('member', **attrs) self._element('realname', mlist.getMemberName(member)) if with_passwords: @@ -266,7 +285,7 @@ With this option, user passwords are included in cleartext. For this reason, the default is to not include passwords.""")) parser.add_option('-l', '--listname', default=[], action='append', type='string', - dest='listnames', help=_("""\ + metavar='LISTNAME', dest='listnames', help=_("""\ The list to include in the output. If not given, then all mailing lists are included in the XML output. Multiple -l flags may be given.""")) parser.add_option('-C', '--config', diff --git a/Mailman/bin/genaliases.py b/Mailman/bin/genaliases.py index 379b4bd94..2796804df 100644 --- a/Mailman/bin/genaliases.py +++ b/Mailman/bin/genaliases.py @@ -26,6 +26,7 @@ from Mailman import Utils from Mailman import Version from Mailman.configuration import config from Mailman.i18n import _ +from Mailman.initialize import initialize __i18n_templates__ = True @@ -55,7 +56,7 @@ verbosity.""")) def main(): parser, opts, args = parseargs() - config.load(opts.config) + initialize(opts.config) # Import the MTA specific module modulename = 'Mailman.MTA.' + config.MTA diff --git a/Mailman/bin/list_lists.py b/Mailman/bin/list_lists.py index c9480217d..87e855f4c 100644 --- a/Mailman/bin/list_lists.py +++ b/Mailman/bin/list_lists.py @@ -21,8 +21,8 @@ from Mailman import Defaults from Mailman import MailList from Mailman import Utils from Mailman import Version -from Mailman.configuration import config from Mailman.i18n import _ +from Mailman.initialize import initialize __i18n_templates__ = True @@ -65,7 +65,7 @@ ignored when -b is given.""")) def main(): parser, opts, args = parseargs() - config.load(opts.config) + initialize(opts.config) names = list(Utils.list_names()) names.sort() diff --git a/Mailman/bin/mailmanctl.py b/Mailman/bin/mailmanctl.py index e3a3ec866..5c85799bc 100644 --- a/Mailman/bin/mailmanctl.py +++ b/Mailman/bin/mailmanctl.py @@ -34,6 +34,7 @@ from Mailman import loginit from Mailman.MailList import MailList from Mailman.configuration import config from Mailman.i18n import _ +from Mailman.initialize import initialize __i18n_templates__ = True @@ -300,9 +301,8 @@ def main(): global elog, qlog, opts parser, opts, args = parseargs() - config.load(opts.config) + initialize(opts.config) - loginit.initialize() elog = logging.getLogger('mailman.error') qlog = logging.getLogger('mailman.qrunner') @@ -456,8 +456,8 @@ def main(): # error!) restarting = '' if opts.restart: - if ((exitstatus == None and killsig <> signal.SIGTERM) or - (killsig == None and exitstatus <> signal.SIGTERM)): + if ((exitstatus is None and killsig <> signal.SIGTERM) or + (killsig is None and exitstatus <> signal.SIGTERM)): # Then restarting = '[restarting]' qrname, slice, count, restarts = kids[pid] diff --git a/Mailman/bin/newlist.py b/Mailman/bin/newlist.py index 461cd9ee6..edd7ad7bd 100644 --- a/Mailman/bin/newlist.py +++ b/Mailman/bin/newlist.py @@ -28,6 +28,7 @@ from Mailman import Utils from Mailman import Version from Mailman import i18n from Mailman.configuration import config +from Mailman.initialize import initialize _ = i18n._ @@ -83,7 +84,7 @@ listadmin-addr and admin-password are all specified on the command line.""")) def main(): parser, opts, args = parseargs() - config.load(opts.config) + initialize(opts.config) # Set up some defaults we couldn't set up in parseargs() if opts.language is None: @@ -205,8 +206,3 @@ def main(): msg.send(mlist) finally: i18n.set_translation(otrans) - - - -if __name__ == '__main__': - main() diff --git a/Mailman/bin/qrunner.py b/Mailman/bin/qrunner.py index 1fab69d98..3b71bf448 100644 --- a/Mailman/bin/qrunner.py +++ b/Mailman/bin/qrunner.py @@ -24,6 +24,7 @@ from Mailman import Version from Mailman import loginit from Mailman.configuration import config from Mailman.i18n import _ +from Mailman.initialize import initialize __i18n_templates__ = True @@ -181,13 +182,11 @@ def main(): global log, opts parser, opts, args = parseargs() - config.load(opts.config) - # If we're not running as a subprocess of mailmanctl, then we'll log to # stderr in addition to logging to the log files. We do this by passing a # value of True to propagate, which allows the 'mailman' root logger to # see the log messages. - loginit.initialize(propagate=not opts.subproc) + initialize(opts.config, propagate_logs=not opts.subproc) log = logging.getLogger('mailman.qrunner') if opts.list: diff --git a/Mailman/bin/rmlist.py b/Mailman/bin/rmlist.py index 861e79226..4c7d4055b 100644 --- a/Mailman/bin/rmlist.py +++ b/Mailman/bin/rmlist.py @@ -20,27 +20,71 @@ import sys import shutil import optparse -from Mailman import MailList +from Mailman import Errors from Mailman import Utils from Mailman import Version +from Mailman import database +from Mailman.MailList import MailList from Mailman.configuration import config from Mailman.i18n import _ +from Mailman.initialize import initialize __i18n_templates__ = True -def remove_it(listname, filename, msg): +def remove_it(listname, filename, msg, quiet=False): if os.path.islink(filename): - print _('Removing $msg') + if not quiet: + print _('Removing $msg') os.unlink(filename) elif os.path.isdir(filename): - print _('Removing $msg') + if not quiet: + print _('Removing $msg') shutil.rmtree(filename) elif os.path.isfile(filename): os.unlink(filename) else: - print _('$listname $msg not found as $filename') + if not quiet: + print _('$listname $msg not found as $filename') + + + +def delete_list(listname, mlist=None, archives=True, quiet=False): + removeables = [] + if mlist: + # Remove the list from the database + database.remove_list(mlist) + # Do the MTA-specific list deletion tasks + if config.MTA: + modname = 'Mailman.MTA.' + config.MTA + __import__(modname) + sys.modules[modname].remove(mlist) + # Remove the list directory + removeables.append((os.path.join('lists', listname), _('list info'))) + + # Remove any stale locks associated with the list + for filename in os.listdir(config.LOCK_DIR): + fn_listname = filename.split('.')[0] + if fn_listname == listname: + removeables.append((os.path.join(config.LOCK_DIR, filename), + _('stale lock file'))) + + if archives: + removeables.extend([ + (os.path.join('archives', 'private', listname), + _('private archives')), + (os.path.join('archives', 'private', listname + '.mbox'), + _('private archives')), + (os.path.join('archives', 'public', listname), + _('public archives')), + (os.path.join('archives', 'public', listname + '.mbox'), + _('public archives')), + ]) + + for dirtmpl, msg in removeables: + path = os.path.join(config.VAR_PREFIX, dirtmpl) + remove_it(listname, path, msg, quiet) @@ -75,13 +119,12 @@ remove any residual archives.""")) def main(): parser, opts, args = parseargs() - config.load(opts.config) + initialize(opts.config) - listname = args[0].lower().strip() - if '@' not in listname: - listname = '%s@%s' % (listname, config.DEFAULT_EMAIL_HOST) - - if not Utils.list_exists(listname): + listname = Utils.fqdn_listname(args[0]) + try: + mlist = MailList(listname, lock=False) + except Errors.MMUnknownListError: if not opts.archives: print >> sys.stderr, _( 'No such list (or list already deleted): $listname') @@ -89,43 +132,12 @@ def main(): else: print _( 'No such list: ${listname}. Removing its residual archives.') + mlist = None if not opts.archives: print _('Not removing archives. Reinvoke with -a to remove them.') - removeables = [] - if Utils.list_exists(listname): - mlist = MailList.MailList(listname, lock=False) - # Do the MTA-specific list deletion tasks - if config.MTA: - modname = 'Mailman.MTA.' + config.MTA - __import__(modname) - sys.modules[modname].remove(mlist) - - removeables.append((os.path.join('lists', listname), _('list info'))) - - # Remove any stale locks associated with the list - for filename in os.listdir(config.LOCK_DIR): - fn_listname = filename.split('.')[0] - if fn_listname == listname: - removeables.append((os.path.join(config.LOCK_DIR, filename), - _('stale lock file'))) - - if opts.archives: - removeables.extend([ - (os.path.join('archives', 'private', listname), - _('private archives')), - (os.path.join('archives', 'private', listname + '.mbox'), - _('private archives')), - (os.path.join('archives', 'public', listname), - _('public archives')), - (os.path.join('archives', 'public', listname + '.mbox'), - _('public archives')), - ]) - - for dirtmpl, msg in removeables: - path = os.path.join(config.VAR_PREFIX, dirtmpl) - remove_it(listname, path, msg) + delete_list(listname, mlist, opts.archives) diff --git a/Mailman/bin/testall.py b/Mailman/bin/testall.py index 61b1cc7c7..6539e6e55 100644 --- a/Mailman/bin/testall.py +++ b/Mailman/bin/testall.py @@ -25,8 +25,8 @@ import unittest from Mailman import Version from Mailman import loginit -from Mailman.configuration import config from Mailman.i18n import _ +from Mailman.initialize import initialize __i18n_templates__ = True @@ -138,7 +138,7 @@ def main(): global basedir parser, opts, args = parseargs() - config.load(opts.config) + initialize(opts.config) if not args: args = ['.'] loginit.initialize(propagate=opts.stderr) diff --git a/Mailman/bin/update.py b/Mailman/bin/update.py index 72a40ab8c..b6953e007 100644 --- a/Mailman/bin/update.py +++ b/Mailman/bin/update.py @@ -37,6 +37,7 @@ from Mailman.OldStyleMemberships import OldStyleMemberships from Mailman.Queue.Switchboard import Switchboard from Mailman.configuration import config from Mailman.i18n import _ +from Mailman.initialize import initialize __i18n_templates__ = True @@ -658,7 +659,7 @@ def update_pending(): def main(): parser, opts, args = parseargs() - config.load(opts.config) + initialize(opts.config) # calculate the versions lastversion, thisversion = calcversions() diff --git a/Mailman/bin/withlist.py b/Mailman/bin/withlist.py new file mode 100644 index 000000000..63513d70b --- /dev/null +++ b/Mailman/bin/withlist.py @@ -0,0 +1,249 @@ +# Copyright (C) 1998-2006 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +import os +import sys +import atexit +import optparse + +from Mailman import Errors +from Mailman import MailList +from Mailman import Utils +from Mailman import Version +from Mailman import interact +from Mailman.configuration import config +from Mailman.i18n import _ +from Mailman.initialize import initialize + +__i18n_templates__ = True + +LAST_MLIST = None +VERBOSE = True +LOCK = False + + + +def exitfunc(mlist): + """Unlock a locked list, but do not implicitly Save() it.""" + if mlist.Locked(): + if VERBOSE: + listname = mlist.fqdn_listname + print >> sys.stderr, _( + 'Unlocking (but not saving) list: $listname') + mlist.Unlock() + if VERBOSE: + print >> sys.stderr, _('Finalizing') + + + +def do_list(listname, args, func): + global LAST_MLIST + + if '@' not in listname: + listname += '@' + config.DEFAULT_EMAIL_HOST + + if VERBOSE: + print >> sys.stderr, _('Loading list $listname'), + if LOCK: + print >> sys.stderr, _('(locked)') + else: + print >> sys.stderr, _('(unlocked)') + + try: + mlist = MailList.MailList(listname, lock=LOCK) + atexit.register(exitfunc, mlist) + LAST_MLIST = mlist + except Errors.MMUnknownListError: + print >> sys.stderr, _('Unknown list: $listname') + + # try to import the module and run the callable + if func: + return func(mlist, *args) + return None + + + +def parseargs(): + parser = optparse.OptionParser(version=Version.MAILMAN_VERSION, + usage=_("""\ +%prog [options] listname [args ...] + +General framework for interacting with a mailing list object. + +There are two ways to use this script: interactively or programmatically. +Using it interactively allows you to play with, examine and modify a MailList +object from Python's interactive interpreter. When running interactively, a +MailList object called 'm' will be available in the global namespace. It also +loads the class MailList into the global namespace. + +Programmatically, you can write a function to operate on a MailList object, +and this script will take care of the housekeeping (see below for examples). +In that case, the general usage syntax is: + + % bin/withlist [options] listname [args ...] + +Here's an example of how to use the -r option. Say you have a file in the +Mailman installation directory called 'listaddr.py', with the following +two functions: + + def listaddr(mlist): + print mlist.GetListEmail() + + def requestaddr(mlist): + print mlist.GetRequestEmail() + +Now, from the command line you can print the list's posting address by running +the following from the command line: + + % bin/withlist -r listaddr mylist + Loading list: mylist (unlocked) + Importing listaddr ... + Running listaddr.listaddr() ... + mylist@myhost.com + +And you can print the list's request address by running: + + % bin/withlist -r listaddr.requestaddr mylist + Loading list: mylist (unlocked) + Importing listaddr ... + Running listaddr.requestaddr() ... + mylist-request@myhost.com + +As another example, say you wanted to change the password for a particular +user on a particular list. You could put the following function in a file +called 'changepw.py': + + from Mailman.Errors import NotAMemberError + + def changepw(mlist, addr, newpasswd): + try: + mlist.setMemberPassword(addr, newpasswd) + mlist.Save() + except NotAMemberError: + print 'No address matched:', addr + +and run this from the command line: + + % bin/withlist -l -r changepw mylist somebody@somewhere.org foobar""")) + parser.add_option('-l', '--lock', + default=False, action='store_true', help=_("""\ +Lock the list when opening. Normally the list is opened unlocked (e.g. for +read-only operations). You can always lock the file after the fact by typing +'m.Lock()' + +Note that if you use this option, you should explicitly call m.Save() before +exiting, since the interpreter's clean up procedure will not automatically +save changes to the MailList object (but it will unlock the list).""")) + parser.add_option('-i', '--interactive', + default=None, action='store_true', help=_("""\ +Leaves you at an interactive prompt after all other processing is complete. +This is the default unless the -r option is given.""")) + parser.add_option('-r', '--run', + type='string', help=_("""\ +This can be used to run a script with the opened MailList object. This works +by attempting to import 'module' (which must be in the directory containing +withlist, or already be accessible on your sys.path), and then calling +'callable' from the module. callable can be a class or function; it is called +with the MailList object as the first argument. If additional args are given +on the command line, they are passed as subsequent positional args to the +callable. + +Note that 'module.' is optional; if it is omitted then a module with the name +'callable' will be imported. + +The global variable 'r' will be set to the results of this call.""")) + parser.add_option('-a', '--all', + default=False, action='store_true', help=_("""\ +This option only works with the -r option. Use this if you want to execute +the script on all mailing lists. When you use -a you should not include a +listname argument on the command line. The variable 'r' will be a list of all +the results.""")) + parser.add_option('-q', '--quiet', + default=False, action='store_true', + help=_('Suppress all status messages.')) + parser.add_option('-C', '--config', + help=_('Alternative configuration file to use')) + opts, args = parser.parse_args() + return parser, opts, args + + + +def main(): + global LAST_MLIST, LOCK, VERBOSE + + parser, opts, args = parseargs() + initialize(opts.config, not opts.quiet) + + VERBOSE = not opts.quiet + LOCK = opts.lock + + # Append our bin directory to sys.path so that any withlist scripts living + # their can be simply imported. + sys.path.append(config.BIN_DIR) + + # The default for interact is true unless -r was given + if opts.interactive is None: + if not opts.run: + opts.interactive = True + else: + opts.interactive = False + + dolist = True + if len(args) < 1 and not opts.all: + warning = _('No list name supplied.') + if opts.interactive: + # Let them keep going + print >> sys.stderr, warning + dolist = False + else: + parser.error(warning) + + if opts.all and not opts.run: + parser.error(_('--all requires --run')) + + # Try to import the module for the callable + func = None + if opts.run: + i = opts.run.find('.') + if i < 0: + module = opts.run + callable = opts.run + else: + module = opts.run[:i] + callable = opts.run[i+1:] + if VERBOSE: + print >> sys.stderr, _('Importing $module ...') + mod = __import__(module) + if VERBOSE: + print >> sys.stderr, _('Running ${module}.${callable}() ...') + func = getattr(mod, callable) + + r = None + if opts.all: + r = [do_list(listname, args, func) for listname in Utils.list_names()] + elif dolist: + listname = args.pop(0).lower().strip() + r = do_list(listname, args, func) + + # Now go to interactive mode, perhaps + if opts.interactive: + if dolist: + banner = _("The variable 'm' is the $listname MailList instance") + else: + banner = interact.DEFAULT_BANNER + overrides = dict(m=LAST_MLIST, r=r) + interact.interact(upframe=False, banner=banner, overrides=overrides) diff --git a/Mailman/configuration.py b/Mailman/configuration.py index 0cb656c89..6825d5792 100644 --- a/Mailman/configuration.py +++ b/Mailman/configuration.py @@ -148,7 +148,13 @@ class Configuration(object): E.g. 'HTTPRunner' or 'LMTPRunner'. count is the number of qrunner slices to create, by default, 1. """ - Defaults.QRUNNERS.append((name, count)) + self.QRUNNERS.append((name, count)) + + @property + def paths(self): + return dict([(k, self.__dict__[k]) + for k in self.__dict__ + if k.endswith('_DIR')]) diff --git a/Mailman/database/Makefile.in b/Mailman/database/Makefile.in new file mode 100644 index 000000000..d59f0b5f3 --- /dev/null +++ b/Mailman/database/Makefile.in @@ -0,0 +1,71 @@ +# Copyright (C) 1998-2006 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +# NOTE: Makefile.in is converted into Makefile by the configure script +# in the parent directory. Once configure has run, you can recreate +# the Makefile by running just config.status. + +# Variables set by configure + +VPATH= @srcdir@ +srcdir= @srcdir@ +bindir= @bindir@ +prefix= @prefix@ +exec_prefix= @exec_prefix@ +DESTDIR= + +CC= @CC@ +CHMOD= @CHMOD@ +INSTALL= @INSTALL@ + +DEFS= @DEFS@ + +# Customizable but not set by configure + +OPT= @OPT@ +CFLAGS= $(OPT) $(DEFS) +PACKAGEDIR= $(prefix)/Mailman/database +SHELL= /bin/sh + +MODULES= *.py + +# Modes for directories and executables created by the install +# process. Default to group-writable directories but +# user-only-writable for executables. +DIRMODE= 775 +EXEMODE= 755 +FILEMODE= 644 +INSTALL_PROGRAM=$(INSTALL) -m $(EXEMODE) + + +# Rules + +all: + +install: + for f in $(MODULES); \ + do \ + $(INSTALL) -m $(FILEMODE) $(srcdir)/$$f $(DESTDIR)$(PACKAGEDIR); \ + done + +finish: + +clean: + +distclean: + -rm *.pyc + -rm Makefile diff --git a/Mailman/database/__init__.py b/Mailman/database/__init__.py new file mode 100644 index 000000000..a2b6c97ce --- /dev/null +++ b/Mailman/database/__init__.py @@ -0,0 +1,34 @@ +# Copyright (C) 2006 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +# This module exposes the higher level interface methods that the rest of +# Mailman should use. It essentially hides the dbcontext and the SQLAlchemy +# session from all other code. The preferred way to use these methods is: +# +# from Mailman import database +# database.add_list(foo) + + +def initialize(): + from Mailman import database + from Mailman.database.dbcontext import dbcontext + + dbcontext.connect() + for attr in dir(dbcontext): + if attr.startswith('api_'): + exposed_name = attr[4:] + setattr(database, exposed_name, getattr(dbcontext, attr)) diff --git a/Mailman/database/address.py b/Mailman/database/address.py new file mode 100644 index 000000000..cea1ba072 --- /dev/null +++ b/Mailman/database/address.py @@ -0,0 +1,30 @@ +# Copyright (C) 2006 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Email addresses.""" + +from sqlalchemy import * + + + +def make_table(metadata): + table = Table( + 'Address', metadata, + Column('address_id', Integer, primary_key=True), + Column('address', Unicode(4096)), + ) + return table diff --git a/Mailman/database/dbcontext.py b/Mailman/database/dbcontext.py new file mode 100644 index 000000000..b812c9b77 --- /dev/null +++ b/Mailman/database/dbcontext.py @@ -0,0 +1,144 @@ +# Copyright (C) 2006 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +import weakref + +from sqlalchemy import * +from string import Template + +from Mailman import Version +from Mailman.configuration import config +from Mailman.database import address +from Mailman.database import listdata +from Mailman.database import version +from Mailman.database.txnsupport import txn + + + +class MlistRef(weakref.ref): + def __init__(self, mlist, callback): + super(MlistRef, self).__init__(mlist, callback) + self.fqdn_listname = mlist.fqdn_listname + + + +class DBContext(object): + def __init__(self): + self.tables = {} + self.metadata = None + self.session = None + # Special transaction used only for MailList.Lock() .Save() and + # .Unlock() interface. + self._mlist_txns = {} + + def connect(self): + # Calculate the engine url + url = Template(config.SQLALCHEMY_ENGINE_URL).safe_substitute( + config.paths) + self.metadata = BoundMetaData(url) + self.metadata.engine.echo = config.SQLALCHEMY_ECHO + # Create all the table objects, and then let SA conditionally create + # them if they don't yet exist. + version_table = None + for module in (address, listdata, version): + table = module.make_table(self.metadata) + self.tables[table.name] = table + if module is version: + version_table = table + self.metadata.create_all() + # Validate schema version, updating if necessary (XXX) + from Mailman.interact import interact + r = version_table.select(version_table.c.component=='schema').execute() + row = r.fetchone() + if row is None: + # Database has not yet been initialized + version_table.insert().execute( + component='schema', + version=Version.DATABASE_SCHEMA_VERSION) + elif row.version <> Version.DATABASE_SCHEMA_VERSION: + # XXX Update schema + raise SchemaVersionMismatchError(row.version) + self.session = create_session() + + # Cooperative method for use with @txn decorator + def _withtxn(self, meth, *args, **kws): + try: + txn = self.session.create_transaction() + rtn = meth(*args, **kws) + except: + txn.rollback() + raise + else: + txn.commit() + return rtn + + def _unlock_mref(self, mref): + txn = self._mlist_txns.pop(mref.fqdn_listname, None) + if txn is not None: + txn.rollback() + + # Higher level interface + def api_lock(self, mlist): + # Don't try to re-lock a list + if mlist.fqdn_listname in self._mlist_txns: + return + txn = self.session.create_transaction() + mref = MlistRef(mlist, self._unlock_mref) + self._mlist_txns[mlist.fqdn_listname] = txn + + def api_unlock(self, mlist): + txn = self._mlist_txns.pop(mlist.fqdn_listname, None) + if txn is not None: + txn.rollback() + + def api_save(self, mlist): + # When dealing with MailLists, .Save() will always be followed by + # .Unlock(). However lists can also be unlocked without saving. But + # if it's been locked it will always be unlocked. So the rollback in + # unlock will essentially be no-op'd if we've already saved the list. + txn = self._mlist_txns.pop(mlist.fqdn_listname, None) + if txn is not None: + txn.commit() + + @txn + def api_add_list(self, mlist): + self.session.save(mlist) + + @txn + def api_remove_list(self, mlist): + self.session.delete(mlist) + + @txn + def api_find_list(self, listname, hostname): + from Mailman.MailList import MailList + q = self.session.query(MailList) + mlists = q.select_by(list_name=listname, host_name=hostname) + assert len(mlists) <= 1, 'Duplicate mailing lists!' + if mlists: + return mlists[0] + return None + + @txn + def api_get_list_names(self): + table = self.tables['Listdata'] + results = table.select().execute() + return [(row[table.c.list_name], row[table.c.host_name]) + for row in results.fetchall()] + + + +dbcontext = DBContext() diff --git a/Mailman/database/listdata.py b/Mailman/database/listdata.py new file mode 100644 index 000000000..a90027e82 --- /dev/null +++ b/Mailman/database/listdata.py @@ -0,0 +1,167 @@ +# Copyright (C) 2006 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""SQLAlchemy based list data storage.""" + +from sqlalchemy import * + + + +def make_table(metadata): + table = Table( + 'Listdata', metadata, + # Attributes not directly modifiable via the web u/i + Column('list_id', Integer, primary_key=True), + Column('list_name', Unicode), + Column('web_page_url', Unicode), + Column('admin_member_chunksize', Integer), + # OldStyleMemberships attributes, temporarily stored as pickles. + Column('bounce_info', PickleType), + Column('delivery_status', PickleType), + Column('digest_members', PickleType), + Column('language', PickleType), + Column('members', PickleType), + Column('passwords', PickleType), + Column('topics_userinterest', PickleType), + Column('user_options', PickleType), + Column('usernames', PickleType), + # Attributes which are directly modifiable via the web u/i. The more + # complicated attributes are currently stored as pickles, though that + # will change as the schema and implementation is developed. + Column('accept_these_nonmembers', PickleType), + Column('acceptable_aliases', PickleType), + Column('admin_immed_notify', Boolean), + Column('admin_notify_mchanges', Boolean), + Column('administrivia', Boolean), + Column('advertised', Boolean), + Column('anonymous_list', Boolean), + Column('archive', Boolean), + Column('archive_private', Boolean), + Column('archive_volume_frequency', Integer), + Column('autorespond_admin', Boolean), + Column('autorespond_postings', Boolean), + Column('autorespond_requests', Integer), + Column('autoresponse_admin_text', Unicode), + Column('autoresponse_graceperiod', Integer), + Column('autoresponse_postings_text', Unicode), + Column('autoresponse_request_text', Unicode), + Column('available_languages', PickleType), + Column('ban_list', PickleType), + Column('bounce_info_stale_after', Integer), + Column('bounce_matching_headers', Unicode), + Column('bounce_notify_owner_on_disable', Boolean), + Column('bounce_notify_owner_on_removal', Boolean), + Column('bounce_processing', Boolean), + Column('bounce_score_threshold', Integer), + Column('bounce_unrecognized_goes_to_list_owner', Boolean), + Column('bounce_you_are_disabled_warnings', Integer), + Column('bounce_you_are_disabled_warnings_interval', Integer), + Column('collapse_alternatives', Boolean), + Column('convert_html_to_plaintext', Boolean), + Column('default_member_moderation', Boolean), + Column('description', Unicode), + Column('digest_footer', Unicode), + Column('digest_header', Unicode), + Column('digest_is_default', Boolean), + Column('digest_send_periodic', Boolean), + Column('digest_size_threshhold', Integer), + Column('digest_volume_frequency', Integer), + Column('digestable', Boolean), + Column('discard_these_nonmembers', PickleType), + Column('emergency', Boolean), + Column('encode_ascii_prefixes', Boolean), + Column('filter_action', Integer), + Column('filter_content', Boolean), + Column('filter_filename_extensions', PickleType), + Column('filter_mime_types', PickleType), + Column('first_strip_reply_to', Boolean), + Column('forward_auto_discards', Boolean), + Column('gateway_to_mail', Boolean), + Column('gateway_to_news', Boolean), + Column('generic_nonmember_action', Integer), + Column('goodbye_msg', Unicode), + Column('header_filter_rules', PickleType), + Column('hold_these_nonmembers', PickleType), + Column('host_name', Unicode), + Column('include_list_post_header', Boolean), + Column('include_rfc2369_headers', Boolean), + Column('info', Unicode), + Column('linked_newsgroup', Unicode), + Column('max_days_to_hold', Integer), + Column('max_message_size', Integer), + Column('max_num_recipients', Integer), + Column('member_moderation_action', Boolean), + Column('member_moderation_notice', Unicode), + Column('mime_is_default_digest', Boolean), + Column('moderator', PickleType), + Column('msg_footer', Unicode), + Column('msg_header', Unicode), + Column('new_member_options', Integer), + Column('news_moderation', Boolean), + Column('news_prefix_subject_too', Boolean), + Column('nntp_host', Unicode), + Column('nondigestable', Boolean), + Column('nonmember_rejection_notice', Unicode), + Column('obscure_addresses', Boolean), + Column('owner', PickleType), + Column('pass_filename_extensions', PickleType), + Column('pass_mime_types', PickleType), + Column('password', Unicode), + Column('personalize', Integer), + Column('preferred_language', Unicode), + Column('private_roster', Boolean), + Column('real_name', Unicode), + Column('reject_these_nonmembers', PickleType), + Column('reply_goes_to_list', Boolean), + Column('reply_to_address', Unicode), + Column('require_explicit_destination', Boolean), + Column('respond_to_post_requests', Boolean), + Column('scrub_nondigest', Boolean), + Column('send_goodbye_msg', Boolean), + Column('send_reminders', Boolean), + Column('send_welcome_msg', Boolean), + Column('subject_prefix', Unicode), + Column('subscribe_auto_approval', PickleType), + Column('subscribe_policy', Integer), + Column('topics', PickleType), + Column('topics_bodylines_limit', Integer), + Column('topics_enabled', Boolean), + Column('umbrella_list', Boolean), + Column('umbrella_member_suffix', Unicode), + Column('unsubscribe_policy', Integer), + Column('welcome_msg', Unicode), + ) + # Avoid circular imports + from Mailman.MailList import MailList + # We need to ensure MailList.InitTempVars() is called whenever a MailList + # instance is created from a row. Use a mapper extension for this. + mapper(MailList, table, extension=MailListMapperExtension()) + return table + + + +class MailListMapperExtension(MapperExtension): + def populate_instance(self, mapper, context, row, mlist, ikey, isnew): + if isnew: + # Get the list name and host name -- which are required by + # InitTempVars() from the row data. + list_name = row['listdata_list_name'] + host_name = row['listdata_host_name'] + fqdn_name = '%s@%s' % (list_name, host_name) + mlist.InitTempVars(fqdn_name) + # In all cases, let SA proceed as normal + return EXT_PASS diff --git a/Mailman/database/txnsupport.py b/Mailman/database/txnsupport.py new file mode 100644 index 000000000..30235b4af --- /dev/null +++ b/Mailman/database/txnsupport.py @@ -0,0 +1,34 @@ +# Copyright (C) 2006 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +# A transaction wrapping decorator. The basic idea is that methods in the +# DBContext that need to operate on transaction boundaries can be written to +# be transaction naive. By wrapping them in this decorator, they +# automatically become transaction safe. + +class txn(object): + def __init__(self, func): + # func is a function object, not a method (even an unbound method). + self._func = func + + def __get__(self, obj, type=None): + # Return a wrapper function that creates a bound method from the + # function, then calls it wrapped in a transaction boundary. Uses a + # non-public method called _withtxn() in the object's class. + def wrapper(*args, **kws): + return obj._withtxn(self._func.__get__(obj), *args, **kws) + return wrapper diff --git a/Mailman/database/version.py b/Mailman/database/version.py new file mode 100644 index 000000000..93b97e470 --- /dev/null +++ b/Mailman/database/version.py @@ -0,0 +1,31 @@ +# Copyright (C) 2006 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Schema versions.""" + +from sqlalchemy import * + + + +def make_table(metadata): + table = Table( + 'Version', metadata, + Column('version_id', Integer, primary_key=True), + Column('component', String(20)), + Column('version', Integer), + ) + return table diff --git a/Mailman/htmlformat.py b/Mailman/htmlformat.py index ae0007794..1bf9548ed 100644 --- a/Mailman/htmlformat.py +++ b/Mailman/htmlformat.py @@ -36,12 +36,11 @@ NL = '\n' # Format an arbitrary object. def HTMLFormatObject(item, indent): "Return a presentation of an object, invoking their Format method if any." - if type(item) == type(''): - return item - elif not hasattr(item, "Format"): - return `item` - else: + if hasattr(item, 'Format'): return item.Format(indent) + if isinstance(item, basestring): + return item + return str(item) def CaseInsensitiveKeyedDict(d): result = {} diff --git a/Mailman/initialize.py b/Mailman/initialize.py new file mode 100644 index 000000000..db26d15df --- /dev/null +++ b/Mailman/initialize.py @@ -0,0 +1,36 @@ +# Copyright (C) 2006 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Initialize all global state. + +Every entrance into the Mailman system, be it by command line, mail program, +or cgi, must call the initialize function here in order for the system's +global state to be set up properly. Typically this is called after command +line argument parsing, since some of the initialization behavior is controlled +by the command line arguments. +""" + +import Mailman.configuration +import Mailman.database +import Mailman.loginit + + + +def initialize(config=None, propagate_logs=False): + Mailman.configuration.config.load(config) + Mailman.loginit.initialize(propagate_logs) + Mailman.database.initialize() diff --git a/Mailman/interact.py b/Mailman/interact.py new file mode 100644 index 000000000..939e7224f --- /dev/null +++ b/Mailman/interact.py @@ -0,0 +1,68 @@ +# Copyright (C) 2006 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Provide an interactive prompt, mimicking the Python interpreter.""" + +import os +import sys +import code + +DEFAULT_BANNER = object() + + + +def interact(upframe=True, banner=DEFAULT_BANNER, overrides=None): + # The interactive prompt's namespace + ns = dict() + # If uplocals is true, also populate the console's locals with the locals + # of the frame that called this function (i.e. one up from here). + if upframe: + frame = sys._getframe(1) + ns.update(frame.f_globals) + ns.update(frame.f_locals) + if overrides is not None: + ns.update(overrides) + interp = code.InteractiveConsole(ns) + # Try to import the readline module, but don't worry if it's unavailable + try: + import readline + except ImportError: + pass + # Mimic the real interactive interpreter's loading of any $PYTHONSTARTUP + # file. Note that if the startup file is not prepared to be exec'd more + # than once, this could cause a problem. + startup = os.environ.get('PYTHONSTARTUP') + if startup: + try: + execfile(startup, ns) + except: + pass + # We don't want the funky console object in parentheses in the banner. + if banner is DEFAULT_BANNER: + banner = '''\ +Python %s on %s +Type "help", "copyright", "credits" or "license" for more information.''' % ( + sys.version, sys.platform) + elif not banner: + banner = None + interp.interact(banner) + # When an exception occurs in the InteractiveConsole, the various + # sys.exc_* attributes get set so that error handling works the same way + # there as it does in the built-in interpreter. Be anal about clearing + # any exception information before we're done. + sys.exc_clear() + sys.last_type = sys.last_value = sys.last_traceback = None diff --git a/Mailman/loginit.py b/Mailman/loginit.py index 85762e7ec..31a2860ec 100644 --- a/Mailman/loginit.py +++ b/Mailman/loginit.py @@ -1,132 +1,133 @@ -# Copyright (C) 2006 by the Free Software Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or
-# modify it under the terms of the GNU General Public License
-# as published by the Free Software Foundation; either version 2
-# of the License, or (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
-
-"""Logging initialization, using Python's standard logging package.
-
-This module cannot be called 'logging' because that would interfere with the
-import below. Ah, for Python 2.5 and absolute imports.
-"""
-
-import os
-import codecs
-import logging
-
-from Mailman.configuration import config
-
-FMT = '%(asctime)s (%(process)d) %(message)s'
-DATEFMT = '%b %d %H:%M:%S %Y'
-LOGGERS = (
- 'bounce',
- 'config',
- 'debug',
- 'error',
- 'fromusenet',
- 'http',
- 'locks',
- 'mischief',
- 'post',
- 'qrunner',
- 'smtp',
- 'smtp-failure',
- 'subscribe',
- 'vette',
- )
-
-_handlers = []
-
-
-
-class ReopenableFileHandler(logging.Handler):
- def __init__(self, filename):
- self._filename = filename
- self._stream = self._open()
- logging.Handler.__init__(self)
-
- def _open(self):
- return codecs.open(self._filename, 'a', 'utf-8')
-
- def flush(self):
- self._stream.flush()
-
- def emit(self, record):
- try:
- msg = self.format(record)
- fs = '%s\n'
- try:
- self._stream.write(fs % msg)
- except UnicodeError:
- self._stream.write(fs % msg.encode('string-escape'))
- self.flush()
- except:
- self.handleError(record)
-
- def close(self):
- self.flush()
- self._stream.close()
- logging.Handler.close(self)
-
- def reopen(self):
- self._stream.close()
- self._stream = self._open()
-
-
-
-def initialize(propagate=False):
- # XXX Don't call logging.basicConfig() because in Python 2.3, it adds a
- # handler to the root logger that we don't want. When Python 2.4 is the
- # minimum requirement, we can use basicConfig() with keyword arguments.
- #
- # The current set of Mailman logs are:
- #
- # error - All exceptions go to this log
- # bounce - All bounce processing logs go here
- # mischief - Various types of hostile activity
- # post - Information about messages posted to mailing lists
- # vette - Information related to admindb activity
- # smtp - Successful SMTP activity
- # smtp-failure - Unsuccessful SMTP activity
- # subscribe - Information about leaves/joins
- # config - Configuration issues
- # locks - Lock steals
- # qrunner - qrunner start/stops
- # fromusenet - Information related to the Usenet to Mailman gateway
- #
- # There was also a 'debug' logger, but that was mostly unused, so instead
- # we'll use debug level on existing loggers.
- #
- # Start by creating a common formatter and the root logger.
- formatter = logging.Formatter(fmt=FMT, datefmt=DATEFMT)
- log = logging.getLogger('mailman')
- handler = logging.StreamHandler()
- handler.setFormatter(formatter)
- log.addHandler(handler)
- log.setLevel(logging.INFO)
- # Create the subloggers
- for logger in LOGGERS:
- log = logging.getLogger('mailman.' + logger)
- # Propagation to the root logger is how we handle logging to stderr
- # when the qrunners are not run as a subprocess of mailmanctl.
- log.propagate = propagate
- handler = ReopenableFileHandler(os.path.join(config.LOG_DIR, logger))
- _handlers.append(handler)
- handler.setFormatter(formatter)
- log.addHandler(handler)
-
-
-
-def reopen():
- for handler in _handlers:
- handler.reopen()
+# Copyright (C) 2006 by the Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, +# USA. + +"""Logging initialization, using Python's standard logging package. + +This module cannot be called 'logging' because that would interfere with the +import below. Ah, for Python 2.5 and absolute imports. +""" + +import os +import codecs +import logging + +from Mailman.configuration import config + +FMT = '%(asctime)s (%(process)d) %(message)s' +DATEFMT = '%b %d %H:%M:%S %Y' +LOGGERS = ( + 'bounce', + 'config', + 'debug', + 'error', + 'fromusenet', + 'http', + 'locks', + 'mischief', + 'post', + 'qrunner', + 'smtp', + 'smtp-failure', + 'subscribe', + 'vette', + ) + +_handlers = [] + + + +class ReopenableFileHandler(logging.Handler): + def __init__(self, filename): + self._filename = filename + self._stream = self._open() + logging.Handler.__init__(self) + + def _open(self): + return codecs.open(self._filename, 'a', 'utf-8') + + def flush(self): + self._stream.flush() + + def emit(self, record): + try: + msg = self.format(record) + fs = '%s\n' + try: + self._stream.write(fs % msg) + except UnicodeError: + self._stream.write(fs % msg.encode('string-escape')) + self.flush() + except: + self.handleError(record) + + def close(self): + self.flush() + self._stream.close() + logging.Handler.close(self) + + def reopen(self): + self._stream.close() + self._stream = self._open() + + + +def initialize(propagate=False): + # XXX Don't call logging.basicConfig() because in Python 2.3, it adds a + # handler to the root logger that we don't want. When Python 2.4 is the + # minimum requirement, we can use basicConfig() with keyword arguments. + # + # The current set of Mailman logs are: + # + # error - All exceptions go to this log + # bounce - All bounce processing logs go here + # mischief - Various types of hostile activity + # post - Information about messages posted to mailing lists + # vette - Information related to admindb activity + # smtp - Successful SMTP activity + # smtp-failure - Unsuccessful SMTP activity + # subscribe - Information about leaves/joins + # config - Configuration issues + # locks - Lock steals + # qrunner - qrunner start/stops + # fromusenet - Information related to the Usenet to Mailman gateway + # + # There was also a 'debug' logger, but that was mostly unused, so instead + # we'll use debug level on existing loggers. + # + # Start by creating a common formatter and the root logger. + formatter = logging.Formatter(fmt=FMT, datefmt=DATEFMT) + log = logging.getLogger('mailman') + handler = logging.StreamHandler() + handler.setFormatter(formatter) + log.addHandler(handler) + log.setLevel(logging.INFO) + # Create the subloggers + for logger in LOGGERS: + log = logging.getLogger('mailman.' + logger) + # Propagation to the root logger is how we handle logging to stderr + # when the qrunners are not run as a subprocess of mailmanctl. + log.propagate = propagate + handler = ReopenableFileHandler(os.path.join(config.LOG_DIR, logger)) + _handlers.append(handler) + handler.setFormatter(formatter) + log.addHandler(handler) + + + +def reopen(): + for handler in _handlers: + handler.reopen() diff --git a/Mailman/testing/base.py b/Mailman/testing/base.py index a65c62bdd..cb0a78819 100644 --- a/Mailman/testing/base.py +++ b/Mailman/testing/base.py @@ -28,6 +28,7 @@ from cStringIO import StringIO from Mailman import MailList from Mailman import Utils +from Mailman.bin import rmlist from Mailman.configuration import config NL = '\n' @@ -37,9 +38,6 @@ PERMISSIONS = stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH class TestBase(unittest.TestCase): def _configure(self, fp): -## print >> fp, \ -## "MEMBER_ADAPTOR_CLASS = 'Mailman.SAMemberships.SAMemberships'" -## config.MEMBER_ADAPTOR_CLASS = 'Mailman.SAMemberships.SAMemberships' print >> fp, 'add_domain("example.com", "www.example.com")' # Only add this domain once to the current process if 'example.com' not in config.domains: @@ -79,16 +77,6 @@ class TestBase(unittest.TestCase): def tearDown(self): self._mlist.Unlock() - listname = self._mlist.fqdn_listname - for dirtmpl in ['lists/%s', - 'archives/private/%s', - 'archives/private/%s.mbox', - 'archives/public/%s', - 'archives/public/%s.mbox', - ]: - dir = os.path.join(config.VAR_PREFIX, dirtmpl % listname) - if os.path.islink(dir): - os.unlink(dir) - elif os.path.isdir(dir): - shutil.rmtree(dir) + rmlist.delete_list(self._mlist.fqdn_listname, self._mlist, + archives=True, quiet=True) os.unlink(self._config) diff --git a/Mailman/testing/emailbase.py b/Mailman/testing/emailbase.py index 53b915abb..3421efb4b 100644 --- a/Mailman/testing/emailbase.py +++ b/Mailman/testing/emailbase.py @@ -59,8 +59,12 @@ class EmailBase(TestBase): def setUp(self): TestBase.setUp(self) - # Second argument is ignored. - self._server = SinkServer(('localhost', TESTPORT), None) + try: + # Second argument is ignored. + self._server = SinkServer(('localhost', TESTPORT), None) + except: + TestBase.tearDown(self) + raise try: os.system('bin/mailmanctl -C %s -q start' % self._config) # If any errors occur in the above, be sure to manually call @@ -68,6 +72,7 @@ class EmailBase(TestBase): # setUp(). except: self.tearDown() + raise def tearDown(self): os.system('bin/mailmanctl -C %s -q stop' % self._config) diff --git a/Mailman/testing/test_handlers.py b/Mailman/testing/test_handlers.py index e36af3b0f..09ca56c29 100644 --- a/Mailman/testing/test_handlers.py +++ b/Mailman/testing/test_handlers.py @@ -143,8 +143,8 @@ Your message entitled was successfully received by the _xtest mailing list. -List info page: http://www.example.com/mailman/listinfo/_xtest -Your preferences: http://www.example.com/mailman/options/_xtest/aperson%40example.org +List info page: http://www.example.com/mailman/listinfo/_xtest@example.com +Your preferences: http://www.example.com/mailman/options/_xtest@example.com/aperson%40example.org """) # Make sure we dequeued the only message eq(len(self._sb.files()), 0) @@ -183,8 +183,8 @@ Your message entitled was successfully received by the _xtest mailing list. -List info page: http://www.example.com/mailman/listinfo/_xtest -Your preferences: http://www.example.com/mailman/options/_xtest/aperson%40example.org +List info page: http://www.example.com/mailman/listinfo/_xtest@example.com +Your preferences: http://www.example.com/mailman/options/_xtest@example.com/aperson%40example.org """) # Make sure we dequeued the only message eq(len(self._sb.files()), 0) @@ -690,13 +690,14 @@ From: aperson@example.org eq(msg['list-id'], '<_xtest.example.com>') eq(msg['list-help'], '<mailto:_xtest-request@example.com?subject=help>') eq(msg['list-unsubscribe'], - '<http://www.example.com/mailman/listinfo/_xtest>,' + '<http://www.example.com/mailman/listinfo/_xtest@example.com>,' '\n\t<mailto:_xtest-request@example.com?subject=unsubscribe>') eq(msg['list-subscribe'], - '<http://www.example.com/mailman/listinfo/_xtest>,' + '<http://www.example.com/mailman/listinfo/_xtest@example.com>,' '\n\t<mailto:_xtest-request@example.com?subject=subscribe>') eq(msg['list-post'], '<mailto:_xtest@example.com>') - eq(msg['list-archive'], '<http://www.example.com/pipermail/_xtest>') + eq(msg['list-archive'], + '<http://www.example.com/pipermail/_xtest@example.com>') def test_list_headers_with_description(self): eq = self.assertEqual @@ -710,10 +711,10 @@ From: aperson@example.org eq(unicode(msg['list-id']), u'A Test List <_xtest.example.com>') eq(msg['list-help'], '<mailto:_xtest-request@example.com?subject=help>') eq(msg['list-unsubscribe'], - '<http://www.example.com/mailman/listinfo/_xtest>,' + '<http://www.example.com/mailman/listinfo/_xtest@example.com>,' '\n\t<mailto:_xtest-request@example.com?subject=unsubscribe>') eq(msg['list-subscribe'], - '<http://www.example.com/mailman/listinfo/_xtest>,' + '<http://www.example.com/mailman/listinfo/_xtest@example.com>,' '\n\t<mailto:_xtest-request@example.com?subject=subscribe>') eq(msg['list-post'], '<mailto:_xtest@example.com>') diff --git a/Mailman/testing/test_message.py b/Mailman/testing/test_message.py index 34e5c0139..cbfabd026 100644 --- a/Mailman/testing/test_message.py +++ b/Mailman/testing/test_message.py @@ -56,11 +56,11 @@ class TestSentMessage(EmailBase): eq(qmsg['list-help'], '<mailto:_xtest-request@example.com?subject=help>') eq(qmsg['list-subscribe'], """\ -<http://www.example.com/mailman/listinfo/_xtest>, +<http://www.example.com/mailman/listinfo/_xtest@example.com>, \t<mailto:_xtest-request@example.com?subject=subscribe>""") eq(qmsg['list-id'], '<_xtest.example.com>') eq(qmsg['list-unsubscribe'], """\ -<http://www.example.com/mailman/listinfo/_xtest>, +<http://www.example.com/mailman/listinfo/_xtest@example.com>, \t<mailto:_xtest-request@example.com?subject=unsubscribe>""") eq(qmsg.get_payload(), 'About your test list') diff --git a/Mailman/testing/test_security_mgr.py b/Mailman/testing/test_security_mgr.py index 4b2515140..a8b056464 100644 --- a/Mailman/testing/test_security_mgr.py +++ b/Mailman/testing/test_security_mgr.py @@ -57,28 +57,28 @@ class TestSecurityManager(TestBase): mlist.addNewMember('aperson@dom.ain', password='xxXXxx') self.assertEqual( mlist.AuthContextInfo(config.AuthUser, 'aperson@dom.ain'), - ('_xtest+user+aperson--at--dom.ain', 'xxXXxx')) + ('_xtest%40example.com+user+aperson--at--dom.ain', 'xxXXxx')) def test_auth_context_moderator(self): mlist = self._mlist mlist.mod_password = 'yyYYyy' self.assertEqual( mlist.AuthContextInfo(config.AuthListModerator), - ('_xtest+moderator', 'yyYYyy')) + ('_xtest%40example.com+moderator', 'yyYYyy')) def test_auth_context_admin(self): mlist = self._mlist mlist.password = 'zzZZzz' self.assertEqual( mlist.AuthContextInfo(config.AuthListAdmin), - ('_xtest+admin', 'zzZZzz')) + ('_xtest%40example.com+admin', 'zzZZzz')) def test_auth_context_site(self): mlist = self._mlist mlist.password = 'aaAAaa' self.assertEqual( mlist.AuthContextInfo(config.AuthSiteAdmin), - ('_xtest+admin', 'aaAAaa')) + ('_xtest%40example.com+admin', 'aaAAaa')) def test_auth_context_huh(self): self.assertEqual( |