mailman.git/misc, branch master

Major surgery to get the setuptools based installation passing all the

2007-07-17T03:55:49Z

existing unit tests. Here's a summary of the changes. - Removed all dependent third party packages, since the setup.py file now claims all package dependencies such that they can be automatically installed from the cheeseshop. - Moved the misc directory into the Mailman package as Mailman/data. Moved templates and messages to Mailman subpackages. - Added an ILanguageManager interface, plus an implementation, so that we don't use Defaults.LC_DESCRIPTIONS directly anymore. Added a doctest for this interface and implementation. Defaults.LANGUAGES is moved into mailman.cfg. Defaults.LANGUAGE_DICT is moved to _DEFAULT_LANGUAGE_DATA, and LC_DESCRIPTIONS is removed. The calculation of the available and enabled languages is moved to the Configuration class, but this will probably still need work. Utils.GetLanguageDescr() and Utils.IsLanguage() are removed. I'd like to remove GetCharSet() eventually too, but there are too many uses of this currently, so I'm deferring it. - Utils.findtext(): Hacks added so that templates can be retrieved from the language catalog. The hack is that the template contents are used to find the translation, but in the one test case where this is actually flexed, the trailing newline in the file contents has to be trimmed. This is probably not right. - No more Defaults.py.in or mm_cfg.py! Defaults.py.in is moved to Defaults.py and is no longer created from a template file. The script called make_instance is added which creates an etc/mailman.cfg file from mailman.cfg.in (previously, mailman.cfg.sample) and /that/ file now has the small number of calculated values. In general, make_instance will not touch mailman.cfg if it exists, unless the --force option is given. CGIEXT is made the empty string by default (i.e. not generated). make_instance grows a --var-dir option. Fleshed out the --languages opton. - Defaults.py grows a DEFAULT_VAR_DIRECTORY variable, which is the default location of the 'var' directory. The Configuration class uses this as one of the directories it searches for its landmark, i.e. etc/mailman.cfg. RUNTIME_DIR is gone, as is VAR_PREFIX. - testall needs to write MAILMAN_USER, MAILMAN_UID, MAILMAN_GROUP, MAILMAN_GID, and LANGUAGES run time variables. - bin/withlist no longer needs to add config.BIN_DIR to sys.path, because in fact that variable doesn't exist any more. - Tweak the French catalog to make a test work. This is needed because of the conversion from %-strings to $-strings. - The setup.py now generates the .mo files before it does its thing. This will have to be fixed, but for now we must generate these files on setup build time instead of installation time. - Removed an unused interface.

The start of a setuptools conversion. All the Makefile.in and autoconf

2007-07-13T04:19:10Z

artifacts are removed, as is the C files which we will no longer need.

Convert failing test_message.py to doctests bounces.txt and message.txt, which

2007-07-02T03:31:21Z

of course now succeed. Rename Bouncer.py's BounceMessage() method to bounce_message() and remove the 'msgdata' parameter, which wasn't being used. Change the RejectNotice exception class to expose .notice directly, as there's no reason for this to be an accessor or property. Move the coverage.py installation to the install-packages target instead of the install-other target, so that it only gets installed once the pythonlib directory is created.

Add support for code coverage with 'testall --coverage'. However, I'm not

2007-07-02T01:49:34Z

convinced this is totally accurate as a full test run shows almost no coverage in the Mailman.database.model modules even though I /know/ they're getting executed. I'll need to figure this out, but eventually we'll convert fully to setuptools and then we'll use the nosetests to do testing and coverage.

Merge exp-elixir-branch to trunk. There is enough working to make me feel

2007-05-28T20:21:41Z

confident the Elixir branch is ready to become mainline. Also, fewer branches makes for an easier migration to a dvcs. Don't expect much of the old test suite to work, or even for much of the old functionality to work. The changes here are disruptive enough to break higher level parts of Mailman. But that's okay because I am slowly building up a new and improved test suite, which will lead to a functional system again. For now, only the doctests in Mailman/docs (and their related test harnesses) will pass, but they all do pass. Note that Mailman/docs serve as system documentation first and unit tests second. You should be able to read the doctest files to understand the underlying data model. Other changes included in this merge: - Added the Mailman.ext extension package. - zope.interfaces uses to describe major components - SQLAlchemy/Elixir used as the database model - Top level doinstall target renamed to justinstall - 3rd-party packages are now installed in pythonlib/lib/python to be more compliant with distutils standards. This allows us to use just --home instead of all the --install-* options. - No longer need to include the email package or pysqlite, as Python 2.5 is required (and comes with both packages). - munepy package is included, for Python enums - IRosterSets are added as a way to manage a collection of IRosters. Roster sets are named so that we can maintain the indirection between mailing lists and rosters, where the two are maintained in different storages. - IMailingListRosters: remove_*_roster() -> delete_*_roster() - Remove IMember interface. - Utils.list_names() -> config.list_manager.names - fqdn_listname() takes an optional hostname argument. - Added a bunch of new exceptions used throughout the new interfaces. - Make LockFile a context manager for use with the 'with' statement.

Python 2.5 is the required minimum now and that comes with the required version of the email package. So, finally! we can get rid of our copy.

2007-04-12T18:39:57Z

Update copyright years.

2007-01-19T04:38:06Z

In HTTPRunner, when we see a KeyboardInterrupt during the serve_forever(),

2007-01-05T20:57:51Z

don't re-raise the exception since that will show up in the log files. Instead just exit with a code equal to SIGTERM. Rework the way qrunners are specified in the mailman.cfg file. Always start the default number of the default set of qrunners, but allow mailman.cfg to delete some with the del_qrunner() function. Rename add_runner() to add_qrunner() and make this actually work . Both take the shortened qrunner name as the first argument (e.g. 'Arch' instead of 'ArchRunner'). Automatically start the MaildirRunner if USE_MAILDIR = Yes; same goes for LMTPRunner and USE_LMTP = Yes. In both cases, you do not need to also use add_qrunner() in your mailman.cfg file to enable them. You still do need to put "add_qrunner('HTTP')" in your mailman.cfg if you want to enable the wsgi server. This may end up being added to the default set.

A rudimentary import script. Several things are not yet imported, including

2006-12-30T22:47:12Z

header_filters and topics (both list topics and user topic selections). Everything else seems to work pretty well. dbcontext.py: Don't key the mlist transactions off of mlist.fqdn_listname because this can change. For example, if you "bin/withlist -l mylist" and then "m.host_name = 'new.example.com'" the fqdn_listname property will change and the commit machinery won't be able to find the correct transaction. Instead, store the fqdn_listname as it's seen during the api_lock() call back on the mailing list under the _txnkey attribute. Use that attribute in api_save() and api_unlock(). Upgrade to SQLAlchemy 0.3.3 Port from MM2.1 the support for multiple password schemes. Change the MailList's repr to use the fqdn_listname.

Merged revisions 8113-8121 via svnmerge from

2006-12-29T22:20:25Z

https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch ................ r8114 | bwarsaw | 2006-12-06 00:16:54 -0500 (Wed, 06 Dec 2006) | 44 lines Initial take on using SQLAlchemy to store list data in lieu of Python pickles. While all the list data (including OldStyleMemberships attributes) are stored in the database, many attributes are stored as PickleTypes binary data. This isn't idea but it gets things working until a more sophisticated schema can be developed. MailList class is now a new-style class, as is required by SQLAlchemy. This makes several things, er, interesting. Rip out all the low-level pickle reading and writing stuff. Hook SA transaction events into Lock() and Unlock(). Move the hooking of the _memberadaptor into InitTempVars(), which gets called by the SQLAlchemy hooks (MailList.__init__() never is). Add an initialize.py module which centralizes all the initialization bits that command line scripts have to do, including configuration, logging, and atabase initialization. This change also converts bin/withlist to mmshell wrapper. Update to SQLAlchemy 0.3.1. Revamp paths.py.in considerably. There were several problems with the old way. We no longer disable default loading of site-packages so we don't need to add Python's site-packages back to sys.path. Also, because site.addsitedir() causes things like .pth paths to be /appended/ to sys.path, they actually won't override any site-installed packages. E.g. if SQLAlchemy is installed in the system Python, our version will not override. IIUC, setuptools-based packages can be configured to work properly in the face of package versions, however not all packages we currently depend on are setuptools-based. So instead, we steal a bit of stuff from site.py but change things so the prepend .pth stuff to sys.path. Update several modules to use True/False and whitespace normalization. Convert from mm_cfg to config object. Modernize a few coding constructs. Add a couple of exceptions to handle database problems. In the export script, include the widget type in the elements. This helped in my stupid little throw away conversion script, but I think it will be more generally useful. Add an interact.py module which refactors interactive interpreter access. Mostly this is used by withlist -i, but it lets us import Mailman.interact and drop into a prompt just about anywhere (e.g. debugging). ................ r8115 | bwarsaw | 2006-12-07 09:13:56 -0500 (Thu, 07 Dec 2006) | 22 lines Start to flesh out more of the SQLAlchemy mechanisms. Added a MailList.__new__() which hooks instantiation to use a query on dbcontext to get an existing mailing list. A 'no-args' call means we're doing a Create(), though eventually that will change too. For now, disable the CheckVersion() call. Eventually this will be folded into schema migration. list_exists(): Rewrite to use the dbcontext query to determine if the named mailing list exists or not. Requires the fqdn_listname. Eradicate two failed member adaptors: BDBMemberAdaptor and SAMemberships. Change the way the DBContext holds onto tables. It now keeps a dictionary mapping the table's name to the SA Table instance. This makes it easier to look up and use the individual tables. Add 'web_page_url' as an attribute managed by SA, and remove a debugging print. ................ r8116 | bwarsaw | 2006-12-11 07:27:47 -0500 (Mon, 11 Dec 2006) | 29 lines Rework the whole dbcontext and transaction framework. SA already handles nested transactions so we don't have to worry about them. However, we do have the weird situation where some transactions are tied to MailList .Lock()/.Unlock()/.Save() and some are tied to non-mlist actions. So now we use an @txn decorator to put methods in a session transaction, but then we also hook into the above MailList methods as possibly sub-transactions. We use a weakref subclass to manage the MailList interface, with a dictionary mapping MailList fqdn_listnames against transactions. The weakrefs come in by giving us a callback when a MailList gets derefed such that we're guaranteed to rollback any outstanding transaction. Also, we have one global DBContext instance but rather than force the rest of Mailman to deal with context objects, instead we expose API methods on that object into the Mailman.database module, which the rest of the code will use. Such methods must be prepended with 'api_' to get exposed this way. bin/rmlist now works with the SA-backend. I refactored the code here so that other code (namely, the test suite) can more easily and consistently remove a mailing list. This isn't the best place for it ultimately, but it's good enough for now. New convenience functions Utils.split_listname(), .fqdn_listname(). Convert testall to use Mailman.initialize.initialize(). Not all tests work, but I'm down to only 8 failures and 7 errors. Also, do a better job of recovering from failures in setUp(). MailList.__new__() now takes keyword arguments. ................ r8117 | bwarsaw | 2006-12-11 22:58:06 -0500 (Mon, 11 Dec 2006) | 7 lines Unit test repairs; even though the unit tests are still pretty fragile, everything now passes with the SQLAlchemy storage of list data. Added missing 'personalize' column. Converted mailmanctl and qrunner to initialize() interface. Fixed _cookie_path() to not fail if SCRIPT_NAME is not in the environment. ................ r8118 | bwarsaw | 2006-12-27 18:45:41 -0500 (Wed, 27 Dec 2006) | 21 lines Utils.list_names(): Use a database query to get all the list names. dbcontext.py: Added api_get_list_names() to support Utils.list_names(). listdata.py: Added two additional MailList attributes which need to be stored in the database. The first is 'admin_member_chunksize' which isn't modifiable from the web. The second is 'password' which holds the list's password. HTMLFormatObject: item strings can now be unicodes. bin/list_lists.py: Must call initialize() to get the database properly initialized, not just config.load(). This will be a common theme. SecurityManager.py: - Remove md5 and crypt support - Added mailman.debug logger, though it will be only used during debugging. - The 'secret' can be a unicode now. - A few coding style updates; repr() instead of backticks, 'key in dict' instead of 'dict.has_key(key)' ................ r8119 | bwarsaw | 2006-12-27 19:13:09 -0500 (Wed, 27 Dec 2006) | 2 lines genaliases.py: config.load() -> initialize() ................ r8120 | bwarsaw | 2006-12-27 19:17:26 -0500 (Wed, 27 Dec 2006) | 9 lines Blocked revisions 8113 via svnmerge ........ r8113 | bwarsaw | 2006-12-05 23:54:30 -0500 (Tue, 05 Dec 2006) | 3 lines Initialized merge tracking via "svnmerge" with revisions "1-8112" from https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch ........ ................ r8121 | bwarsaw | 2006-12-28 23:34:52 -0500 (Thu, 28 Dec 2006) | 20 lines Remove SIGTERM handling from all the CGI scripts. This messes with HTTPRunner because when you issue "mailmanctl stop" after the signal handler has been installed, the process will get a SIGTERM, the signal handler will run, and the process will exit with a normal zero code. This will cause mailmanctl to try to restart the HTTPRunner. I don't think we need that stuff at all when running under wsgi with a SQLAlchemy backend. If mailmanctl kills the HTTPRunner in the middle of the process, I believe (but have not tested) that the transaction should get properly rolled back at process exit. We need to make sure about this, and also we need to test the signal handling functionality under traditional CGI environment (if we even still want to support that). Also, make sure that we don't try to initialize the loggers twice in qrunner. This was the cause of all the double entries in logs/qrunner. Fix a coding style nit in mailmanctl.py. De-DOS-ify line endings in loginit.py. ................