summaryrefslogtreecommitdiff
path: root/Mailman/Queue
Commit message (Collapse)AuthorAgeFilesLines
* Reorganize the qrunner infrastructure. First, the package has been renamedBarry Warsaw2007-09-2916-2120/+0
| | | | | | | | | | | | | | | | | | | | | | from Mailman.Queue to Mailman.queue (note the case change to be more PEP 8 compliant). The Switchboard and Runner classes have been moved into the package __init__.py and the previous class modules have been removed. The switchboard cache is removed; I don't think it was ultimately buying us much. Now, just import the Switchboard class and instantiate it directly. Added an IRunner interface. Renamed the ArchRunner to ArchiveRunner. bin/qrunner and bin/mailmanctl are updated accordingly. For the former, it no long accepts -r=All to run all qrunners. You can still use the short name (e.g. --runner=incoming) to run the built-in queue runners, but this design will eventually allow for plugin qrunners by allowing them to be run specifying the full package path to the class. It also now accepts a leading dot to indicate a qrunner class relative to the Mailman.queue package.
* qrunner, mailmanctl and various other repairs.Barry Warsaw2007-09-293-7/+8
| | | | | | | | | | | | The convenience methods in Defaults for getting seconds, minutes, hours, and days now returns a subtype of timedelta, which provides conversion to float and int for compatibility with interfaces that require those values (e.g. signal.alarm() and time.sleep(). In bin/make_instance, the var_dir really needs to be an absolute path, otherwise it's possible to get a var dir nested inside the var dir. More MailList object eradication.
* Repair the test suite.Barry Warsaw2007-09-271-2/+1
|
* OMGW00T: After over a decade, the MailList mixin class is gone! Well,Barry Warsaw2007-09-211-1/+2
| | | | | | | | | | | | mostly. It's no longer needed by anything in the test suite, and therefore the list manager returns database MailingList objects directly. The wrapper cruft has been removed. To accomplish this, a couple of hacks were added to the Mailman.app package, which will get cleaned up over time. The MailList module itself (and its few remaining mixins) aren't yet removed from the tree because some of the code is still not tested, and I want to leave this code around until I've finished converting it.
* Implement a context manager for Python 2.5's with statement, which isBarry Warsaw2007-09-191-6/+4
| | | | | | | | used where we used to do a try/except to temporarily change the global translation language. This makes the code shorter and cleaner. E.g. with i18n.using_language(another_language): # do something
* Refactor IDatabase interface so that the user_manager, list_manager,Barry Warsaw2007-08-021-1/+1
| | | | | message_store, and pendings 'databases' hang off the IDatabase object attached to the config object. For IPending, no adaptation is necessary.
* Repair some test suite regressions.Barry Warsaw2007-08-011-1/+1
|
* Move the pending database into the SQLAlchemy/Elixir layer. The oldBarry Warsaw2007-08-011-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pending.py module is removed. Added an interface to this functionality such that any IPendable (essentially a key/value mapping) can be associated with a token, and that token can be confirmed and has a lifetime. Any keys and values can be stored, as long as both are unicodes. Added a doctest. Modified initialization of the database layer to support pluggability via setuptools. No longer is this layer initialized from a module, but now it's instantiated from a class that implements IDatabase. The StockDatabase class implements the SQLAchemy/Elixir layer, but this can be overridden in a setup.py. Bye bye MANAGERS_INIT_FUNCTION, we hardly knew ye. Added a package Mailman.app which will contain certain application specific functionality. Right now, the only there there is an IRegistar implementation, which didn't seem to fit anywhere else. Speaking of which, the IRegistrar interface implements all the logic related to registration and verification of email addresses. Think the equivalent of MailList.AddMember() except generalized out of a mailing list context. This latter will eventually go away. The IRegistrar sends the confirmation email. Added an IDomain interface, though the only implementation of this so far lives in the registration.txt doctest. This defines the context necessary for domain-level things, like address confirmation. A bunch of other cleanups in modules that are necessary due to the refactoring of Pending, but don't affect anything that's actually tested yet, so I won't vouch for them (except that they don't throw errors on import!). Clean up Defaults.py; also turn the functions seconds(), minutes(), hours() and days() into their datetime.timedelta equivalents. Consolidated the bogus email address exceptions. In some places where appropriate, use email 4.0 module names instead of the older brand. Switch from Mailman.Utils.unique_message_id() to email.utils.make_msgid() everywhere. This is because we need to allow sending not in the context of a mailing list (i.e. domain-wide address confirmation message). So we can't use a Message-ID generator that requires a mailing list. OTOH, this breaks Message-ID collision detection in the mail->news gateway. I'll fix that eventually. Remove the 'verified' row on the Address table. Now verification is checked by Address.verified_on not being None.
* The start of a setuptools conversion. All the Makefile.in and autoconfBarry Warsaw2007-07-132-162/+0
| | | | artifacts are removed, as is the C files which we will no longer need.
* Convert failing test_message.py to doctests bounces.txt and message.txt, whichBarry Warsaw2007-07-011-1/+1
| | | | | | | | | | | | | of course now succeed. Rename Bouncer.py's BounceMessage() method to bounce_message() and remove the 'msgdata' parameter, which wasn't being used. Change the RejectNotice exception class to expose .notice directly, as there's no reason for this to be an accessor or property. Move the coverage.py installation to the install-packages target instead of the install-other target, so that it only gets installed once the pythonlib directory is created.
* Convert the rest of test_runners.py to doctests; even though incomplete, theyBarry Warsaw2007-06-283-188/+7
| | | | | | | | | | | | | | | | | | | | | | test everything the old unit tests tested. There are XXX's left in the doctests as reminders to flesh them out. Change the NNTP_REWRITE_DUPLICATE_HEADERS to use proper capitalization. Revert a change I made in the conversion of the Switchboard class: Switchboard.files is no longer a generator. The Runner implementation is cleaner if this returns a concrete list, so that's what it does now. Update the tests to reflect that. The Runner simplifies now too because it no longer needs _open_files() or the _listcache WeakValueDictionary. The standard list manager handles all this now, so just use it directly. Also change the way the Runner sets the language context in _onefile(). It still tries to set it to the preferred language of the sender, if the sender is a member of the list. Otherwise it sets it to the list's preferred language, not the system's preferred language. Removed a conditional that can't possibly happen.
* Convert the Switchboard test in test_runners.py to a doctest. Add anBarry Warsaw2007-06-273-143/+63
| | | | | | | | | | | | | | | ISwitchboard interface and modernize the Python code in the Switchboard.py implementation. The SAVE_MSGS_AS_PICKLES option is removed. Messages are always saved as pickles unless the metadata '_plaintext' key is present, though this should eventually go away too. In testall.py, put the entire VAR_PREFIX in a temporary directory. This helps the switchboard tests by not mixing their data with the installation's queue directories. The Configuration object now also ensures that all the queue and log directories exist -- one more step on the road to getting rid of the autoconf mess.
* Scrubber.py - Malformed RFC 2047 encoded filename= parameter can haveMark Sapiro2007-06-221-2/+4
| | | | | | | | a null byte or other garbage in the extension. Cleaned this. - Improved handling of None payloads. - Cleaned up a few charset coercions. OutgoingRunner.py - Made probe bounce processing and queuing of bounces conditional on having some permanent failure(s).
* Merge exp-elixir-branch to trunk. There is enough working to make me feelbwarsaw2007-05-284-21/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | confident the Elixir branch is ready to become mainline. Also, fewer branches makes for an easier migration to a dvcs. Don't expect much of the old test suite to work, or even for much of the old functionality to work. The changes here are disruptive enough to break higher level parts of Mailman. But that's okay because I am slowly building up a new and improved test suite, which will lead to a functional system again. For now, only the doctests in Mailman/docs (and their related test harnesses) will pass, but they all do pass. Note that Mailman/docs serve as system documentation first and unit tests second. You should be able to read the doctest files to understand the underlying data model. Other changes included in this merge: - Added the Mailman.ext extension package. - zope.interfaces uses to describe major components - SQLAlchemy/Elixir used as the database model - Top level doinstall target renamed to justinstall - 3rd-party packages are now installed in pythonlib/lib/python to be more compliant with distutils standards. This allows us to use just --home instead of all the --install-* options. - No longer need to include the email package or pysqlite, as Python 2.5 is required (and comes with both packages). - munepy package is included, for Python enums - IRosterSets are added as a way to manage a collection of IRosters. Roster sets are named so that we can maintain the indirection between mailing lists and rosters, where the two are maintained in different storages. - IMailingListRosters: remove_*_roster() -> delete_*_roster() - Remove IMember interface. - Utils.list_names() -> config.list_manager.names - fqdn_listname() takes an optional hostname argument. - Added a bunch of new exceptions used throughout the new interfaces. - Make LockFile a context manager for use with the 'with' statement.
* A few style fixes based on commit reviews.bwarsaw2007-03-211-1/+3
| | | | | | | | | | | | | | | | | | | | Switchboard.py - Use listname.encode('utf-8') to produce the necessary 8-bit string, instead of str(listname). Also update the preceding comment. senddigests.py - Remove an unnecessary import. Decorate.py - Remove a commented out section of code. - Remove some redundant local variables - Reorganize the section that's trying to find a usable encoding for the payload of the modified message. I don't think it really hurts much to try duplicate charsets when lcset == mcset, or when either == utf-8. Just go ahead and try them and let them fail. This simplifies the code. Also, try to get just the minimum necessary code under the UnicodeError. I think it's enough to catch the payload.encode() call.
* Fixes for i18n digest to work.tkikuchi2007-03-021-1/+1
| | | | | | | | | | | | | | | | | | Mailman/Queue/Switchboard.py: listname is returned in unicode. ( '\x80' + 'a' is OK, '\x80' + u'a' is NG) Mailman/Utils.py: Utils.oneline() is extended for returning unicode string. Mailman/Digester.py: next_post_number is not used anywhere. Mailman/database/listdata.py: Attributes added (esp. for non web u/i) Mailman/bin/senddigests.py: Initialization Mailman/Handlers/ToDigest.py: Internal string calculation is done in unicode. So, several fixes. StringIO is used because cStringIO doesn't have encoding attribute.
* Update copyright years.bwarsaw2007-01-1914-14/+14
|
* Passwords done right.bwarsaw2007-01-141-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | First off, there are several password hashing schemes added including SHA, salted-SHA, and RFC 2989 PBKDF2 (contributed by Bob Fleck). Then we encode the password using RFC 2307 style syntax. At least I think: specifically things like the PRF and iteration count for PBKDF2 are encoded the way I /think/ is intended for RFC 2307 but I could be wrong. Seems darn hard to find definitive information about that. In any event, even though CLEARTEXT passwords are supported, they are mostly deprecated, even for user passwords. It also allows us to easily update all passwords to a new hashing scheme when the existing schemes get cracked. The default scheme (specified in Defaults.py.in) is salted-SHA with a 20 byte salt (the salt length and PBKDF2 iteration counts can only be specified in the passwords.py file). These hashed passwords are used for user passwords, list owner and moderator passwords, and site and list creator passwords. Of course this means that user password reminders are impossible now. They've been ripped out of the code for a while, but now we'll need to implement password resets since user passwords cannot be recovered. bin/export has had several changes: - export no longer converts to dollar strings. Were assuming dollar strings are used by default for all new lists and any imported lists will already be converted to dollar strings. - Likewise, rip out the password scheme stuff, since cleartext passwords can never be exported, so we might as well always include the member's hashed password. - Fix exporting to stdout when that stream can only handle ascii by wrapping stdout in a utf-8 codec writer. Other changes: - add a missing import to HTTPRunner.py - Convert GUIBase.py to use Defaults.* for constants instead of mm_cfg.* - Remove pre-Python 2.4 compatibility from Utils.py. We've already said Python 2.4 will be a minimum requirement. - Change the permissions on the global password file. The default 007 umask is used and should be good enough. - bin/newlist adds the ability to specify the password scheme (or list the available schemes) for the list owner password. It is not possible to set the scheme on a per-list basis. bin/mmsitepass does the same, but for the site and list creator passwords. - Fix a nasty problem with bin/import. The comment in the code says it best: # XXX Here's what sucks. Some properties need to have # _setValue() called on the gui component, because those # methods do some pre-processing on the values before they're # applied to the MailList instance. But we don't have a good # way to find a category and sub-category that a particular # property belongs to. Plus this will probably change. So # for now, we'll just hard code the extra post-processing # here. The good news is that not all _setValue() munging # needs to be done -- for example, we've already converted # everything to dollar strings. - Set the 'debug' logger to logging.DEBUG level. It doesn't seem to make much sense for the debugging log to ignore debug messages.
* In HTTPRunner, when we see a KeyboardInterrupt during the serve_forever(),bwarsaw2007-01-051-3/+4
| | | | | | | | | | | | | | | | | don't re-raise the exception since that will show up in the log files. Instead just exit with a code equal to SIGTERM. Rework the way qrunners are specified in the mailman.cfg file. Always start the default number of the default set of qrunners, but allow mailman.cfg to delete some with the del_qrunner() function. Rename add_runner() to add_qrunner() and make this actually work <wink>. Both take the shortened qrunner name as the first argument (e.g. 'Arch' instead of 'ArchRunner'). Automatically start the MaildirRunner if USE_MAILDIR = Yes; same goes for LMTPRunner and USE_LMTP = Yes. In both cases, you do not need to also use add_qrunner() in your mailman.cfg file to enable them. You still do need to put "add_qrunner('HTTP')" in your mailman.cfg if you want to enable the wsgi server. This may end up being added to the default set.
* Clean up file permissions and umask settings. Now we set the umask to 007bwarsaw2007-01-053-26/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | during early initialization so that we're guaranteed to get the right value regardless of the shell umask used to invoke the command line script. While we're at it, we can remove almost all individual umask settings previously in the code, and make file permissions consistently -rw-rw---- (IOW, files are no longer other readable). The only subsystem that wasn't changed was the archiver, because it uses its own umask settings to ensure that private archives have the proper permissions. Eventually we'll mess with this, but if it ain't broken... Note that check_perms complains about directory permissions, but I think check_perms can be fixed (or perhaps, even removed?!). If we decide to use LMTPRunner and HTTPRunner exclusively then no outside process will be touching our files potentially with the incorrect permissions, umask, owner, or group. If we control all of our own touch points then I think we can lock out 'other'. Another open question is whether Utils.set_global_password() can have its umask setting removed. It locks permissions down so even the group can't write to the site password file, but the default umask of 007 might be good enough even for this file. Utils.makedirs() now takes an optional mode argument, which defaults to 02775 for backward compatibility. First, the default mode can probably be changed to 02770 (see above). Second, all code that was tweaking the umask in order to do a platform compatible os.mkdir() has now been refactored to use Utils.makedirs(). Another tricky thing was getting SQLite via SQLAlchemy to create its data/mailman.db file with the proper permissions. From the comment in dbcontext.py: # XXX By design of SQLite, database file creation does not honor # umask. See their ticket #1193: # http://www.sqlite.org/cvstrac/tktview?tn=1193,31 More details in that file, but the work around is to essentially 'touch' the database file if 'sqlite' is the scheme of the SQLAlchemy URL. This little pre-touch sets the right umask honoring permission and won't hurt if the file already exists. SQLite will happily keep the existing permissions, and in fact that ticket referenced above recommends doing things this way. In the Mailman.database.initialize(), create a global lock that prevents more than one process from entering this init function at the same time. It's probably not strictly necessary given that I believe all the operations in dbcontext.connect() are multi-processing safe, but it also doesn't seem to hurt and prevents race conditions regardless of the database's own safeguards (or lack thereof). Make sure nightly_gzip.py calls initialize().
* Ported from 2.1 branch:bwarsaw2007-01-021-2/+4
| | | | | | | | | | | | | Ensure that exported XML is written in utf-8, at least if we're writing to a file other than stdout. Fix a typo in getting the digest style. Update copyright years. In HTTPRunner.py, catch KeyboardInterrupt. In Python 2.5 this has been moved in the exception hierarchy so that it's no longer caught by "except Exception". import.py: Import user topic selections. Fix a typo in an error message. Catch BadDomainSpecificationErrors that can be raised in MailList.Create().
* Merged revisions 8113-8121 via svnmerge from bwarsaw2006-12-291-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch ................ r8114 | bwarsaw | 2006-12-06 00:16:54 -0500 (Wed, 06 Dec 2006) | 44 lines Initial take on using SQLAlchemy to store list data in lieu of Python pickles. While all the list data (including OldStyleMemberships attributes) are stored in the database, many attributes are stored as PickleTypes binary data. This isn't idea but it gets things working until a more sophisticated schema can be developed. MailList class is now a new-style class, as is required by SQLAlchemy. This makes several things, er, interesting. Rip out all the low-level pickle reading and writing stuff. Hook SA transaction events into Lock() and Unlock(). Move the hooking of the _memberadaptor into InitTempVars(), which gets called by the SQLAlchemy hooks (MailList.__init__() never is). Add an initialize.py module which centralizes all the initialization bits that command line scripts have to do, including configuration, logging, and atabase initialization. This change also converts bin/withlist to mmshell wrapper. Update to SQLAlchemy 0.3.1. Revamp paths.py.in considerably. There were several problems with the old way. We no longer disable default loading of site-packages so we don't need to add Python's site-packages back to sys.path. Also, because site.addsitedir() causes things like .pth paths to be /appended/ to sys.path, they actually won't override any site-installed packages. E.g. if SQLAlchemy is installed in the system Python, our version will not override. IIUC, setuptools-based packages can be configured to work properly in the face of package versions, however not all packages we currently depend on are setuptools-based. So instead, we steal a bit of stuff from site.py but change things so the prepend .pth stuff to sys.path. Update several modules to use True/False and whitespace normalization. Convert from mm_cfg to config object. Modernize a few coding constructs. Add a couple of exceptions to handle database problems. In the export script, include the widget type in the elements. This helped in my stupid little throw away conversion script, but I think it will be more generally useful. Add an interact.py module which refactors interactive interpreter access. Mostly this is used by withlist -i, but it lets us import Mailman.interact and drop into a prompt just about anywhere (e.g. debugging). ................ r8115 | bwarsaw | 2006-12-07 09:13:56 -0500 (Thu, 07 Dec 2006) | 22 lines Start to flesh out more of the SQLAlchemy mechanisms. Added a MailList.__new__() which hooks instantiation to use a query on dbcontext to get an existing mailing list. A 'no-args' call means we're doing a Create(), though eventually that will change too. For now, disable the CheckVersion() call. Eventually this will be folded into schema migration. list_exists(): Rewrite to use the dbcontext query to determine if the named mailing list exists or not. Requires the fqdn_listname. Eradicate two failed member adaptors: BDBMemberAdaptor and SAMemberships. Change the way the DBContext holds onto tables. It now keeps a dictionary mapping the table's name to the SA Table instance. This makes it easier to look up and use the individual tables. Add 'web_page_url' as an attribute managed by SA, and remove a debugging print. ................ r8116 | bwarsaw | 2006-12-11 07:27:47 -0500 (Mon, 11 Dec 2006) | 29 lines Rework the whole dbcontext and transaction framework. SA already handles nested transactions so we don't have to worry about them. However, we do have the weird situation where some transactions are tied to MailList .Lock()/.Unlock()/.Save() and some are tied to non-mlist actions. So now we use an @txn decorator to put methods in a session transaction, but then we also hook into the above MailList methods as possibly sub-transactions. We use a weakref subclass to manage the MailList interface, with a dictionary mapping MailList fqdn_listnames against transactions. The weakrefs come in by giving us a callback when a MailList gets derefed such that we're guaranteed to rollback any outstanding transaction. Also, we have one global DBContext instance but rather than force the rest of Mailman to deal with context objects, instead we expose API methods on that object into the Mailman.database module, which the rest of the code will use. Such methods must be prepended with 'api_' to get exposed this way. bin/rmlist now works with the SA-backend. I refactored the code here so that other code (namely, the test suite) can more easily and consistently remove a mailing list. This isn't the best place for it ultimately, but it's good enough for now. New convenience functions Utils.split_listname(), .fqdn_listname(). Convert testall to use Mailman.initialize.initialize(). Not all tests work, but I'm down to only 8 failures and 7 errors. Also, do a better job of recovering from failures in setUp(). MailList.__new__() now takes keyword arguments. ................ r8117 | bwarsaw | 2006-12-11 22:58:06 -0500 (Mon, 11 Dec 2006) | 7 lines Unit test repairs; even though the unit tests are still pretty fragile, everything now passes with the SQLAlchemy storage of list data. Added missing 'personalize' column. Converted mailmanctl and qrunner to initialize() interface. Fixed _cookie_path() to not fail if SCRIPT_NAME is not in the environment. ................ r8118 | bwarsaw | 2006-12-27 18:45:41 -0500 (Wed, 27 Dec 2006) | 21 lines Utils.list_names(): Use a database query to get all the list names. dbcontext.py: Added api_get_list_names() to support Utils.list_names(). listdata.py: Added two additional MailList attributes which need to be stored in the database. The first is 'admin_member_chunksize' which isn't modifiable from the web. The second is 'password' which holds the list's password. HTMLFormatObject: item strings can now be unicodes. bin/list_lists.py: Must call initialize() to get the database properly initialized, not just config.load(). This will be a common theme. SecurityManager.py: - Remove md5 and crypt support - Added mailman.debug logger, though it will be only used during debugging. - The 'secret' can be a unicode now. - A few coding style updates; repr() instead of backticks, 'key in dict' instead of 'dict.has_key(key)' ................ r8119 | bwarsaw | 2006-12-27 19:13:09 -0500 (Wed, 27 Dec 2006) | 2 lines genaliases.py: config.load() -> initialize() ................ r8120 | bwarsaw | 2006-12-27 19:17:26 -0500 (Wed, 27 Dec 2006) | 9 lines Blocked revisions 8113 via svnmerge ........ r8113 | bwarsaw | 2006-12-05 23:54:30 -0500 (Tue, 05 Dec 2006) | 3 lines Initialized merge tracking via "svnmerge" with revisions "1-8112" from https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch ........ ................ r8121 | bwarsaw | 2006-12-28 23:34:52 -0500 (Thu, 28 Dec 2006) | 20 lines Remove SIGTERM handling from all the CGI scripts. This messes with HTTPRunner because when you issue "mailmanctl stop" after the signal handler has been installed, the process will get a SIGTERM, the signal handler will run, and the process will exit with a normal zero code. This will cause mailmanctl to try to restart the HTTPRunner. I don't think we need that stuff at all when running under wsgi with a SQLAlchemy backend. If mailmanctl kills the HTTPRunner in the middle of the process, I believe (but have not tested) that the transaction should get properly rolled back at process exit. We need to make sure about this, and also we need to test the signal handling functionality under traditional CGI environment (if we even still want to support that). Also, make sure that we don't try to initialize the loggers twice in qrunner. This was the cause of all the double entries in logs/qrunner. Fix a coding style nit in mailmanctl.py. De-DOS-ify line endings in loginit.py. ................
* Support for VERP and personalization.tkikuchi2006-12-031-0/+1
|
* internal_name() to fqdn_listname change.tkikuchi2006-11-272-2/+2
|
* Postfix LMTP related brushups.tkikuchi2006-11-261-1/+1
| | | | | | | | | | | | | | - Configurable no-list error. - Ultimate loop stop address in transport -> aliases. - LMTP_ONLY_DOMAIN needs no individual transport entry. - Use of alias/lmtp is exclusive. WSGI brushups. - _cookie_path() was made simple and retain common cookie for admin/admindb/... etc. - Removed absolute=1 from admindb/confirm/create/options. configuration.py - Use of add_runner() in etc/mailman.cfg needs change. config is not loaded yet?
* Delete the .bak file from the queue for an unparseable message.msapiro2006-11-171-0/+1
|
* More work on the WSGI support. So far, I've tested most of the admin.py linksbwarsaw2006-10-151-12/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | and some of the admindb.py links. There may still be breakage in other parts of the interface and I haven't gone back to verify that traditional CGI still works. Changes: - Add wsgiref-0.1.2-py2.4.egg so that we can still do WSGI in Python 2.4, which doesn't come with wsgiref. Of course this means we /also/ have to add setuptools-0.5c3 because eggs require setuptools. - Style cleanups in HTTPRunner.py and wsgi_app.py. Also, use cStringIO instead of StringIO. - All internal links within the listinfo and admin pages are (or at least should be ;) relative now. This should make other things better, such as running Mailman over https or alternative ports. It does kind of mean that web_page_url is obsolete, but I haven't looked at whether we can completely eradicate it. - ValidateEmail(): Use ' ' in s instead of s.count(' ') > 0. - GetPathPieces(): When path is false, return the empty list instead of None, so we can still len() it. - ScriptURL(): Much simpler. To support relative urls as the default, we change the API so that it only takes a 'target' argument (i.e. the script we want to link to). It no longer takes 'absolute' or 'web_page_url', and it constructs its link from GetPathPieces(), the target, and the cgi extension. - GetRequestURI(): code style updates. - Mailman/bin/show_config.py: De-DOS-line-ending-ification. - export.py: A few modifications, although this is likely still not final (I'm still working on the import script). First, for <option> elements, don't put the value in an attribute, put it in the text body of the element. Second, put the list <option> tags in a <configuration> element. Third, put the preferred language on an <option> tag with a 'preferred_language' name attribute value. - SecurityManager: Make sure that MakeCookie() and ZapCookie() use the same 'path' cookie value by refactoring that into a separate method. That method now returns just the SCRIPT_NAME and the full listname. web_page_url doesn't enter into it. - loginit.py: Add a 'debug' logger since it's just too useful to have :) - admin.py: Remove the extra / right before the query string in ?VARHELP urls. That extra / turns out to be problematic with the relative url scheme we're using now. - Auth.py: whitespace normalization and copyright years update. Also, remove a couple of unnecessary imports. Also, make sure that the actionurl is relative. - create.py: Typo. - private.py: mm_cfg -> config object - In MailList.py: GetScriptURL() can be written in terms of Utils.ScriptURL() now.
* WSGI HTTP Server for Mailman Web interface.tkikuchi2006-10-091-0/+63
| | | | | | | | Add: HTTPRunner.py ... Start/Restart/Stop HTTP Server under Runner framework. wsgi_app.py ... WSGI to CGI wrapper. Mostly taken from scripts/driver. loginit.py ... Add http log. Time stamp is duplicated :-( Defaults.py ... HTTP_HOST and HTTP_PORT. Note that WSGI server should be used under reverse proxy environment.
* A further elaboration of the LMTP-based incoming qrunner. I believe this isbwarsaw2006-10-051-63/+104
| | | | | | | | | | | | | | | | | | | | | | now more conformant to RFC 2033. Changes include: - Make sure LMTPRunner returns a status code for each recipient accepted during the RCPT TO part of the protocol. This is required by RFC 2033. - Use email.message_from_string() directly instead of creating a StringIO. The email package will already do this for us. - Use email 4.0 package names - Create both an error and a queue logger, and write the appropriate messages to each. Note that we'll never cleanly exit out of asyncore.loop() so the trailing log message is kind of pointless. - Remove the -admin address. - Simplification, refactoring, and whitespace normalization - Updated docstring
* LMTPRunner ... This is actually a server rather than a runner but it shouldtkikuchi2006-10-021-0/+153
| | | | be convenient to put here because mailmanctl can restart it.
* Here are the patches needed in order to create new lists on my testtkikuchi2006-09-281-17/+22
| | | | | | | installation. I've tested four cases of combination of POSTFIX_STYLE_VIRTUAL_DOMAINS (Any/None) and USE_MAIL_DIR (Yes/No). Also, I've added POSTFIX_VIRTUAL_SEPARATOR = '_at_' for unique mapping of local part in the virtual-mailman/aliases files.
* - Switchboard.py Added missing call to create error logger.msapiro2006-07-221-1/+3
|
* Added robustness to Switchboards and Runners so that if a runner crashesbwarsaw2006-07-166-10/+366
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | uncleanly (e.g. segfaults the Python interpreter), messages being processed will not be lost. The vulnerability, ideas, and patches are credited to Richard Barrett and Mark Sapiro. Their original work was modified by Barry for this commit and any bugs are his fault. The basic idea is that instead of unlinking a .pck file in dequeue(), the file is renamed to a .bak file. The Switchboard grows a finish() method which then unlinks the .bak file. That class's constructor also grows a 'restore' argument (defaulting to false), which when true moves all .bak files it finds in its hash space to .pck, thereby restoring a file lost while "in flight". This relies on the fact that even with multiple qrunners, exactly one process will be responsible for one hash space slice, so it's never possible (under normal operation) for a .bak file to be renamed to .pck by some other process. Test cases for both the new Switchboard behavior and the use of that by Runner subclasses has been added. There are two things to watch out for, either of which may require some additional changes. There is some small potential to duplicate messages in various queues, if say 'mailmanctl' were improperly started more than once by a site admin. This usually won't happen unless an admin is overly eager with the mailmanctl -s switch, so we can chalk this one up to operator error. I'm not sure what more we can do about that. There's also a possibility that if we're processing a message that continually causes the Python interpreter to crash, we could end up duplicating messages endlessly. This is especially troublesome for the Outgoing runner which could conceivably cause a mail flood. I consider this the more critical issue to defend against, probably by adding a numbering scheme to the .bak file names and refusing to restore a .bak file more than say 3 times without human intervention.
* - Switchboard.py - Closed very tiny holes at the upper ends of queuemsapiro2006-07-091-3/+10
| | | | | slices that could result in unprocessable queue entries. Improved FIFO processing when two queue entries have the same timestamp.
* Fix some buglets with virtual domain support and repair unit tests broken bybwarsaw2006-07-081-1/+1
| | | | | | | | | | | | | | | | | | | | | | | this change. More unit tests should be added. misc/sitelist.cfg is removed -- this is an ex-site list. MailList.GetNoReplyEmail() -> MailList.no_reply_address (property) UserNotification._enqueue(), OwnerNotification._enqueue(): when queing the message to the virgin queue, be sure to use the fully qualified (i.e. posting) address for the list. In the MTA modules, be sure to set up the target of the mail commands as the fqdn listname because otherwise we can't find the correct list. This needs some tweaking/testing for Postfix's virtual domain support. MailList.Load() has to grow an optional argument specifying the fqdn listname. The problem is that in some situations, we can't calculate that because we don't know _internal_name, so it has to be passed in. This is mostly the case in the MailList ctor where a Load hasn't happened yet. For backward compatibility though, if it's not passed in, just use mlist.fqdn_listname.
* First crack at real virtual domain support, i.e. mailing lists with the samebwarsaw2006-07-084-10/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | name in more than one domain. - Totally eradicate MAILMAN_SITE_LIST, and in fact the entire need for a site list. The functions that the site list previously performed are either removed or supported in other ways. For example, instead of forwarding owner bounces to the site list, we now have a SITE_OWNER_ADDRESS which should point to a human, and such bounces are sent there instead. There's also a "no reply" email address that should be set up to go to devnull. For any message that never expects a reply, the sender is set to this address. - Remove the Site.py module. It was an experimental approach to trying to support virtual domains, and we're going to do it so much better now that this module is no longer necessary. Site._makedirs() -> Utils.makedir(). - VIRTUAL_HOST_OVERVIEW is completely removed, since now virtual hosts are always enabled. Virtual domains should be added to mailman.cfg by using the new add_domain() function. add_virtualhost() is gone. If no virtual domains are added explicitly, we add the default one that configure guessed (but we never add that if domains are added explicitly). - Utils.get_domain() -> Utils.get_request_domain() - withlist code cleanup and make sure that we load etc/mailman.cfg - A new base exception called MailmanException is added, from which all exceptions defined in Errors.py ultimately derive. MailmanError is retained and derives from MailmanException. - BadDomainSpecificationError is added. - Remove the -V/--virtual-host-overview option from list_lists and add instead -d/--domain and -f/--full. - bin/update probably works but needs more testing. - bin/newlist and bin/rmlist take fqdn list names, but default to the default domain if @whatever isn't given. newlist's -u/--urlhost and -e/--emailhost options are removed. The domain that the list is being added to must already exist. - Minor code cleanup in Message.py - Bump version to 2.2.0a1 - The Configuration object grows a .domain dictionary which maps email hosts to url hosts. The reverse mapping is supported, but not directly; use Configuration.get_email_host() instead. - Mailman/Cgi/create is converted from mm_cfg to config, and some minor code cleanup is performed. Also, convert to __i18n_templates__ = True. - New MailList APIs: + property .fqdn_listname + GetNoReplyEmail() + Create() API changes and refactoring.
* Massive conversion process so that Mailman can be run from a user specifiedbwarsaw2006-07-0813-93/+86
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | configuration file. While the full conversion is not yet complete, everything that seems to be required to run mailmanctl, qrunner, rmlist, and newlist have been updated. Basically, modules should no longer import mm_cfg, but instead they should import Mailman.configuration.config. The latter is an object that's guaranteed to exist, but not guaranteed to be initialized until some top-level script calls config.load(). The latter should be called with the argument to -C/--config which is a new convention the above scripts have been given. In most cases, where mm_cfg.<variable> is used config.<variable> can be used, but the exceptions are where the default value must be available before config.load() is called. Sometimes you can import Mailman.Default and get the variable from there, but other times the code has to be changed to work around this limitation. Take each on a case-by-case basis. Note that the various directories calculated from VAR_PREFIX, EXEC_PREFIX, and PREFIX are now calculated in config.py, not in Defaults.py. This way a configuration file can override the base directories and everything should work correctly. Other changes here include: - mailmanctl, qrunner, and update are switched to optparse and $-strings, and changed to the mmshell architecture - An etc directory has been added to /usr/local/mailman and a mailman.cfg.sample file is installed there. Sites should now edit an etc/mailman.cfg file to do their configurations, although the mm_cfg file is still honored. The formats of the two files are identical. - list_lists is given the -C/--config option - Some coding style fixes in bin/update, but not extensive - Get rid of nested scope hacks in qrunner.py - A start on getting EmailBase tests working (specifically test_message), although not yet complete.
* Remove most uses of the types module, in favor of isinstance checks againstbwarsaw2006-04-171-3/+2
| | | | | | the builtin types. Two still remain: a check against ClassType and a check against MethodType. Also, fix some hinky type comparisons to use isinstance() consistently.
* - Convert all logging to Python's standard logging module. Get rid of allbwarsaw2006-04-178-65/+74
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | traces of our crufty old Syslog. Most of this work was purely mechanical, except for: 1) Initializing the loggers. For this, there's a new module Mailman/loginit.py (yes all modules from now on will use PEP 8 names). We can't call this 'logging.py' because that will interfere with importing the stdlib module of the same name (can you say Python 2.5 and absolute imports?). If you want to write log messages both to the log file and to stderr, pass True to loginit.initialize(). This will turn on propagation of log messages to the parent 'mailman' logger, which is set up to print to stderr. This is how bin/qrunner works when not running as a subprocess of mailmanctl. 2) The driver script. I had to untwist the StampedLogger stuff and implement differently printing exceptions and such to log/error because standard logging objects don't have a write() method. So we write to a cStringIO and then pass that to the logger. 3) SMTPDirect.py because of the configurability of the log messages. This required changing SafeDict into a dict subclass (which is better than using UserDicts anyway -- yay Python 2.3!). It's probably still possible to flummox things up if you change the name of the loggers in the SMTP_LOG_* variables in mm_cfg.py. However, the worst you can do is cause output to go to stderr and not go to a log file. Note too that all entry points into the Mailman system must call Mailman.loginit.initialize() or the log output will go to stderr (which may occasionally be what you want). Currently all CGIs and qrunners should be working properly. I wish I could have tested all code paths that touch the logger, but that's infeasible. I have tested this, but it's possible that there were some mistakes in the translation. - Mailman.Bouncers.BounceAPI.Stop is a singleton, but not a class instance any more. - True/False code cleanup, PEP 8 import restructuring, whitespace normalization, and copyright year updates, as appropriate.
* Now that Python 2.3 is the minimum requirement for Mailman 2.2:bwarsaw2006-04-157-91/+41
| | | | | | | | | | | - Remove True/False binding cruft - Remove __future__ statements for nested scopes - Remove ascii_letters import hack from Utils.py - Remove mimetypes.guess_all_extensions import hack from Scrubber.py - In Pending.py, set _missing to object() (better than using []) Also, update copyright years where appropriate, and re-order imports more to my PEP 8 tastes. Whitespace normalize.
* Now that we've converted from CVS to Subversion, remove CVS cruft frombwarsaw2006-04-151-1/+0
| | | | | the trunk. This serves also to test write permission to the svn repository, and tests email notifications.
* Add test in BounceRunner for non-fatal, VERPed bounce. Bug 1421285.msapiro2006-03-091-1/+5
|
* Changed BouncerAPI.py to return Stop from a Bouncer and changedmsapiro2006-03-011-0/+3
| | | | | | BounceRunner.py to discard the bounce when Stop returned. Changed DSN.py to recognize Action: headers with comments. Changed Qmail.py to recognize an observed different starting string.
* Set umask for creation of bounce-events queue file.msapiro2006-02-051-3/+8
|
* add OverflowError in exceptions. (CVE-2005-4153 related but not DoS)tkikuchi2006-02-031-1/+1
|
* back porting from 2.1.6tkikuchi2005-08-287-379/+307
|
* FSF office has moved. chdcking in for MAIN branch.tkikuchi2005-08-2714-14/+14
|
* MarshalSwitchboard._ext_write(), ASCIISwitchboard._ext_write():bwarsaw2003-10-101-6/+2
| | | | | Promote SYNC_AFTER_WRITE to a Defaults.py/mm_cfg.py variable after all.
* _oneloop(): Print the shunt log message after enqueing the file, sobwarsaw2003-09-121-4/+4
| | | | | | that we get the correct (new) filebase of the shunted message. This makes finding the correlation between log messages and the offending message files actually possible <wink>.
* SYNC_AFTER_WRITE: New flag which controls whether os.fsync() is calledbwarsaw2003-09-121-1/+23
| | | | | | | | | | on the file descriptor for data files after they've been written. We always flush the buffer, but sync'ing can be a huge performance hit. Still, some sites might want to enable this for additional data integrity. It's disabled by default. enqueue(): Return the newly calculated filebase. This let's the shunt message in Runner.py display the correct (new) filename.
generated by cgit v1.2.3-70-g09d2 (git 2.51.2) at 2025-11-08 18:28:11 +0000