summaryrefslogtreecommitdiff
path: root/Mailman/Queue/Runner.py
Commit message (Collapse)AuthorAgeFilesLines
* Reorganize the qrunner infrastructure. First, the package has been renamedBarry Warsaw2007-09-291-244/+0
| | | | | | | | | | | | | | | | | | | | | | from Mailman.Queue to Mailman.queue (note the case change to be more PEP 8 compliant). The Switchboard and Runner classes have been moved into the package __init__.py and the previous class modules have been removed. The switchboard cache is removed; I don't think it was ultimately buying us much. Now, just import the Switchboard class and instantiate it directly. Added an IRunner interface. Renamed the ArchRunner to ArchiveRunner. bin/qrunner and bin/mailmanctl are updated accordingly. For the former, it no long accepts -r=All to run all qrunners. You can still use the short name (e.g. --runner=incoming) to run the built-in queue runners, but this design will eventually allow for plugin qrunners by allowing them to be run specifying the full package path to the class. It also now accepts a leading dot to indicate a qrunner class relative to the Mailman.queue package.
* qrunner, mailmanctl and various other repairs.Barry Warsaw2007-09-291-2/+2
| | | | | | | | | | | | The convenience methods in Defaults for getting seconds, minutes, hours, and days now returns a subtype of timedelta, which provides conversion to float and int for compatibility with interfaces that require those values (e.g. signal.alarm() and time.sleep(). In bin/make_instance, the var_dir really needs to be an absolute path, otherwise it's possible to get a var dir nested inside the var dir. More MailList object eradication.
* Repair the test suite.Barry Warsaw2007-09-271-2/+1
|
* Implement a context manager for Python 2.5's with statement, which isBarry Warsaw2007-09-191-6/+4
| | | | | | | | used where we used to do a try/except to temporarily change the global translation language. This makes the code shorter and cleaner. E.g. with i18n.using_language(another_language): # do something
* Refactor IDatabase interface so that the user_manager, list_manager,Barry Warsaw2007-08-021-1/+1
| | | | | message_store, and pendings 'databases' hang off the IDatabase object attached to the config object. For IPending, no adaptation is necessary.
* Convert the rest of test_runners.py to doctests; even though incomplete, theyBarry Warsaw2007-06-281-23/+6
| | | | | | | | | | | | | | | | | | | | | | test everything the old unit tests tested. There are XXX's left in the doctests as reminders to flesh them out. Change the NNTP_REWRITE_DUPLICATE_HEADERS to use proper capitalization. Revert a change I made in the conversion of the Switchboard class: Switchboard.files is no longer a generator. The Runner implementation is cleaner if this returns a concrete list, so that's what it does now. Update the tests to reflect that. The Runner simplifies now too because it no longer needs _open_files() or the _listcache WeakValueDictionary. The standard list manager handles all this now, so just use it directly. Also change the way the Runner sets the language context in _onefile(). It still tries to set it to the preferred language of the sender, if the sender is a member of the list. Otherwise it sets it to the list's preferred language, not the system's preferred language. Removed a conditional that can't possibly happen.
* Convert the Switchboard test in test_runners.py to a doctest. Add anBarry Warsaw2007-06-271-1/+1
| | | | | | | | | | | | | | | ISwitchboard interface and modernize the Python code in the Switchboard.py implementation. The SAVE_MSGS_AS_PICKLES option is removed. Messages are always saved as pickles unless the metadata '_plaintext' key is present, though this should eventually go away too. In testall.py, put the entire VAR_PREFIX in a temporary directory. This helps the switchboard tests by not mixing their data with the installation's queue directories. The Configuration object now also ensures that all the queue and log directories exist -- one more step on the road to getting rid of the autoconf mess.
* Merge exp-elixir-branch to trunk. There is enough working to make me feelbwarsaw2007-05-281-12/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | confident the Elixir branch is ready to become mainline. Also, fewer branches makes for an easier migration to a dvcs. Don't expect much of the old test suite to work, or even for much of the old functionality to work. The changes here are disruptive enough to break higher level parts of Mailman. But that's okay because I am slowly building up a new and improved test suite, which will lead to a functional system again. For now, only the doctests in Mailman/docs (and their related test harnesses) will pass, but they all do pass. Note that Mailman/docs serve as system documentation first and unit tests second. You should be able to read the doctest files to understand the underlying data model. Other changes included in this merge: - Added the Mailman.ext extension package. - zope.interfaces uses to describe major components - SQLAlchemy/Elixir used as the database model - Top level doinstall target renamed to justinstall - 3rd-party packages are now installed in pythonlib/lib/python to be more compliant with distutils standards. This allows us to use just --home instead of all the --install-* options. - No longer need to include the email package or pysqlite, as Python 2.5 is required (and comes with both packages). - munepy package is included, for Python enums - IRosterSets are added as a way to manage a collection of IRosters. Roster sets are named so that we can maintain the indirection between mailing lists and rosters, where the two are maintained in different storages. - IMailingListRosters: remove_*_roster() -> delete_*_roster() - Remove IMember interface. - Utils.list_names() -> config.list_manager.names - fqdn_listname() takes an optional hostname argument. - Added a bunch of new exceptions used throughout the new interfaces. - Make LockFile a context manager for use with the 'with' statement.
* Update copyright years.bwarsaw2007-01-191-1/+1
|
* Delete the .bak file from the queue for an unparseable message.msapiro2006-11-171-0/+1
|
* Added robustness to Switchboards and Runners so that if a runner crashesbwarsaw2006-07-161-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | uncleanly (e.g. segfaults the Python interpreter), messages being processed will not be lost. The vulnerability, ideas, and patches are credited to Richard Barrett and Mark Sapiro. Their original work was modified by Barry for this commit and any bugs are his fault. The basic idea is that instead of unlinking a .pck file in dequeue(), the file is renamed to a .bak file. The Switchboard grows a finish() method which then unlinks the .bak file. That class's constructor also grows a 'restore' argument (defaulting to false), which when true moves all .bak files it finds in its hash space to .pck, thereby restoring a file lost while "in flight". This relies on the fact that even with multiple qrunners, exactly one process will be responsible for one hash space slice, so it's never possible (under normal operation) for a .bak file to be renamed to .pck by some other process. Test cases for both the new Switchboard behavior and the use of that by Runner subclasses has been added. There are two things to watch out for, either of which may require some additional changes. There is some small potential to duplicate messages in various queues, if say 'mailmanctl' were improperly started more than once by a site admin. This usually won't happen unless an admin is overly eager with the mailmanctl -s switch, so we can chalk this one up to operator error. I'm not sure what more we can do about that. There's also a possibility that if we're processing a message that continually causes the Python interpreter to crash, we could end up duplicating messages endlessly. This is especially troublesome for the Outgoing runner which could conceivably cause a mail flood. I consider this the more critical issue to defend against, probably by adding a numbering scheme to the .bak file names and refusing to restore a .bak file more than say 3 times without human intervention.
* First crack at real virtual domain support, i.e. mailing lists with the samebwarsaw2006-07-081-2/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | name in more than one domain. - Totally eradicate MAILMAN_SITE_LIST, and in fact the entire need for a site list. The functions that the site list previously performed are either removed or supported in other ways. For example, instead of forwarding owner bounces to the site list, we now have a SITE_OWNER_ADDRESS which should point to a human, and such bounces are sent there instead. There's also a "no reply" email address that should be set up to go to devnull. For any message that never expects a reply, the sender is set to this address. - Remove the Site.py module. It was an experimental approach to trying to support virtual domains, and we're going to do it so much better now that this module is no longer necessary. Site._makedirs() -> Utils.makedir(). - VIRTUAL_HOST_OVERVIEW is completely removed, since now virtual hosts are always enabled. Virtual domains should be added to mailman.cfg by using the new add_domain() function. add_virtualhost() is gone. If no virtual domains are added explicitly, we add the default one that configure guessed (but we never add that if domains are added explicitly). - Utils.get_domain() -> Utils.get_request_domain() - withlist code cleanup and make sure that we load etc/mailman.cfg - A new base exception called MailmanException is added, from which all exceptions defined in Errors.py ultimately derive. MailmanError is retained and derives from MailmanException. - BadDomainSpecificationError is added. - Remove the -V/--virtual-host-overview option from list_lists and add instead -d/--domain and -f/--full. - bin/update probably works but needs more testing. - bin/newlist and bin/rmlist take fqdn list names, but default to the default domain if @whatever isn't given. newlist's -u/--urlhost and -e/--emailhost options are removed. The domain that the list is being added to must already exist. - Minor code cleanup in Message.py - Bump version to 2.2.0a1 - The Configuration object grows a .domain dictionary which maps email hosts to url hosts. The reverse mapping is supported, but not directly; use Configuration.get_email_host() instead. - Mailman/Cgi/create is converted from mm_cfg to config, and some minor code cleanup is performed. Also, convert to __i18n_templates__ = True. - New MailList APIs: + property .fqdn_listname + GetNoReplyEmail() + Create() API changes and refactoring.
* Massive conversion process so that Mailman can be run from a user specifiedbwarsaw2006-07-081-7/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | configuration file. While the full conversion is not yet complete, everything that seems to be required to run mailmanctl, qrunner, rmlist, and newlist have been updated. Basically, modules should no longer import mm_cfg, but instead they should import Mailman.configuration.config. The latter is an object that's guaranteed to exist, but not guaranteed to be initialized until some top-level script calls config.load(). The latter should be called with the argument to -C/--config which is a new convention the above scripts have been given. In most cases, where mm_cfg.<variable> is used config.<variable> can be used, but the exceptions are where the default value must be available before config.load() is called. Sometimes you can import Mailman.Default and get the variable from there, but other times the code has to be changed to work around this limitation. Take each on a case-by-case basis. Note that the various directories calculated from VAR_PREFIX, EXEC_PREFIX, and PREFIX are now calculated in config.py, not in Defaults.py. This way a configuration file can override the base directories and everything should work correctly. Other changes here include: - mailmanctl, qrunner, and update are switched to optparse and $-strings, and changed to the mmshell architecture - An etc directory has been added to /usr/local/mailman and a mailman.cfg.sample file is installed there. Sites should now edit an etc/mailman.cfg file to do their configurations, although the mm_cfg file is still honored. The formats of the two files are identical. - list_lists is given the -C/--config option - Some coding style fixes in bin/update, but not extensive - Get rid of nested scope hacks in qrunner.py - A start on getting EmailBase tests working (specifically test_message), although not yet complete.
* - Convert all logging to Python's standard logging module. Get rid of allbwarsaw2006-04-171-11/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | traces of our crufty old Syslog. Most of this work was purely mechanical, except for: 1) Initializing the loggers. For this, there's a new module Mailman/loginit.py (yes all modules from now on will use PEP 8 names). We can't call this 'logging.py' because that will interfere with importing the stdlib module of the same name (can you say Python 2.5 and absolute imports?). If you want to write log messages both to the log file and to stderr, pass True to loginit.initialize(). This will turn on propagation of log messages to the parent 'mailman' logger, which is set up to print to stderr. This is how bin/qrunner works when not running as a subprocess of mailmanctl. 2) The driver script. I had to untwist the StampedLogger stuff and implement differently printing exceptions and such to log/error because standard logging objects don't have a write() method. So we write to a cStringIO and then pass that to the logger. 3) SMTPDirect.py because of the configurability of the log messages. This required changing SafeDict into a dict subclass (which is better than using UserDicts anyway -- yay Python 2.3!). It's probably still possible to flummox things up if you change the name of the loggers in the SMTP_LOG_* variables in mm_cfg.py. However, the worst you can do is cause output to go to stderr and not go to a log file. Note too that all entry points into the Mailman system must call Mailman.loginit.initialize() or the log output will go to stderr (which may occasionally be what you want). Currently all CGIs and qrunners should be working properly. I wish I could have tested all code paths that touch the logger, but that's infeasible. I have tested this, but it's possible that there were some mistakes in the translation. - Mailman.Bouncers.BounceAPI.Stop is a singleton, but not a class instance any more. - True/False code cleanup, PEP 8 import restructuring, whitespace normalization, and copyright year updates, as appropriate.
* Now that Python 2.3 is the minimum requirement for Mailman 2.2:bwarsaw2006-04-151-16/+8
| | | | | | | | | | | - Remove True/False binding cruft - Remove __future__ statements for nested scopes - Remove ascii_letters import hack from Utils.py - Remove mimetypes.guess_all_extensions import hack from Scrubber.py - In Pending.py, set _missing to object() (better than using []) Also, update copyright years where appropriate, and re-order imports more to my PEP 8 tastes. Whitespace normalize.
* back porting from 2.1.6tkikuchi2005-08-281-26/+33
|
* FSF office has moved. chdcking in for MAIN branch.tkikuchi2005-08-271-1/+1
|
* _oneloop(): Print the shunt log message after enqueing the file, sobwarsaw2003-09-121-4/+4
| | | | | | that we get the correct (new) filebase of the shunted message. This makes finding the correlation between log messages and the offending message files actually possible <wink>.
* run(): Always call _snooze() but pass the file count to the method sobwarsaw2003-05-031-17/+22
| | | | | | | | | | | it can decide whether to do more work or sleep for a while. Pushes the policy into the derived classes. _snooze(): Default is to sleep only if filecnt == 0. True/False where appropriate. Backport candidate (as is RetryRunner.py)
* _oneloop(): Include a clue as to which filebase is being shunted.bwarsaw2002-12-131-0/+1
|
* __oneloop(): Renamed to _oneloop() -- i.e. single leading underscore,bwarsaw2002-08-091-6/+6
| | | | so that it can be overridden in derived classes.
* Make Pychecker happy and do some whitespace normalization.bwarsaw2002-08-081-2/+2
|
* __onefile(): If there's no `listname' key in the message metadata, usebwarsaw2002-05-021-9/+8
| | | | | the site list to send the message. If there's no site list, you're in heap big trouble.
* _snooze(): Use self.SLEEPTIME instead of mm_cfg.QRUNNER_SLEEP_TIME forbwarsaw2002-03-261-2/+4
| | | | | additional indirection that's overridden by the BounceRunner. Defaults to QRUNNER_SLEEP_TIME for backwards compatibility.
* Dang, left some debugging syslog()'s in there.bwarsaw2002-03-131-2/+0
|
* We can get rid of the old list cache, and the freshen machinery. Nowbwarsaw2002-03-131-26/+11
| | | | | | | | | _open_list() caches the MailList instances in a WeakValueDictionary which don't contribute to the reference count, but ensure that each runner will get one and only one copy of the MailList data. This should help in the memory footprint of the runners, but I suspect we may still be leaking something.
* __init__(): Remove cachelists from the constructor arguments. We canbwarsaw2002-03-121-2/+4
| | | | | | more conveniently use a class attribute to specify this. On the BounceRunner, turn off MailList object caching so that it'll be more friendly to long-term memory use.
* __oneloop(): Stick the queue directory in the message metadata so thatbwarsaw2002-03-111-0/+2
| | | | we can unshunt it to the correct queue.
* Constructor simplification. Since most derived classes are justbwarsaw2002-03-021-3/+2
| | | | | | | specializing the queue directory, make QDIR a class attribute and have Runner's ctor use that instead of an __init__() argument. This lets us get rid of most of the __init__()'s in the derived classes. (OutgoingRunner still needs one though.)
* A slightly cleaner way to do the freshening of MailList objects.bwarsaw2002-02-121-7/+6
| | | | | | | | | run(): After once through the loop, look at the freshen flag and if true, clear the cache. Thus all subsequent MailList objects will get updated state, but we'll gain the performance benefits of the cache for each iteration the the big loop. _open_list(): Return this function to its original state.
* __init__(): Default the self._freshen attribute to zero.bwarsaw2002-02-121-2/+13
| | | | | _open_list(): If we've pulled the list object from the cache, then freshen the state by re-Load()-ing it.
* Update a comment.bwarsaw2001-10-041-1/+1
|
* Convert from mimelib to email.bwarsaw2001-10-011-1/+1
| | | | | Also, use cStringIO directly instead of our own hack-around StringIO module.
* run(): Add a note about a minor bogosity in the calls tobwarsaw2001-08-291-1/+3
| | | | _doperiodic().
* __onefile(): Another step towards conversion to MemberAdaptorbwarsaw2001-07-291-1/+1
| | | | API. GetPreferredLanguage() -> getMemberLanguage()
* 'UnimplementedError' -> 'NotImplementedError'twouters2001-07-101-1/+1
|
* _shortcircuit(): Return self._stop as a default. This allows abwarsaw2001-07-081-1/+1
| | | | | runner.stop() method call to break the __oneloop() loop after any single message is processed.
* _open_list(): The lockp argument was unused, so remove it.bwarsaw2001-07-061-1/+1
|
* Several changes which allow subclasses to do some calculation for eachbwarsaw2001-06-281-3/+48
| | | | | | | | | | | | | | | | | | | | | file processed. This would allow for example, a throttling algorithm which limited the number of files processed per time period, or when load averages were too high. Specifically, run(): Allow __oneloop() to return any false value, not just zero, to trigger the snooze alarm. __oneloop(): After each file is processed, call _shortcircuit() and break out of the file loop if that returns a true value. This way, _shortcircuit() is executed for each file processed, and gives subclasses a chance to do some per-file calculation. _doperiodic() is at a finer granularity, since it conflates per-file and per-processing-loop calculations. _cleanup(), _dispose(), _doperiodic(), _snooze(), _shortcircuit(): Add docstrings describing their interface. _shortcircuit(): Default is to return 0, meaning don't shortcircuit.
* Better syslog() calling conventions.bwarsaw2001-06-271-4/+4
|
* __onefile(): It is possible that there is no list name in thebwarsaw2001-05-221-12/+13
| | | | | | | message's metadata, which means that the message does not originate from a mailing list. In that case, don't try to open the list (since you can't ;), and set the global language to the server's default language instead of the list's preferred language.
* syslog() entries should go to logs/error not logs/qrunner. It'sbwarsaw2001-05-151-5/+5
| | | | highly annoying to have to tail more than one log file to find bugs.
* __oneloop(): Guarantee FIFO order on the processed messages. Webwarsaw2001-05-141-16/+23
| | | | | | | | | | assume that the list of files coming back from Switchboard.files() is sorted by received time, so we don't need to randomize this list. Also, it's now possible for Switchboard.dequeue() to return None for either or both of msg and msgdata (say if the .msg file for a .db file got lost somehow). Check that both are not None before proceeding (but if either is None, log an error).
* __onefile(): The qrunner framework now sets up the language contextbwarsaw2001-03-021-1/+18
| | | | | | | | | | | | | | | for each processed message. It does this by getting the preferred language of the sender of the message (defaulting to the list's preferred language if the sender isn't a member of the list). Next, it saves the previous global language context in a local variable, sets the context to the message's preferred language and disposes of the message. Then it restores the original language context. It also sets the message's metadata (key: `lang') to contain the language name for the language context in case any handler modules need access to it.
* intermediatebwarsaw2001-02-151-0/+1
|
* __init__(): Accept information for hash space slicing (slice numberbwarsaw2001-02-151-87/+112
| | | | | | | | | | | | | | | | | | | | | | | | | | | | and total number of slices). Create the runner-specific switchboard, the shunt switchboard (for messages that generate uncaught exceptions), and the mainloop stop flag. We also have no need for a runner-specific lock since the qrunner watchdog infrastructure handles all that for us now. _dequeue(): This is no longer part of the Runner interface (it's been moved to the Switchboard). run(), __oneloop(), __onefile(): Rework the mainloop for flexibility and robustness. run() is the main loop and handles KeyboardInterrupt exceptons, guaranteed cleanup at mainloop exit, checking the stop flag and calling _doperiod() after each cycle through the mainloop, and sleeping for a while if there are no files in the queue directory. __oneloop() handles cruising through the list of all the files in the queue (modulo any hash space slicing) exactly once. __onefile(): handles the work for one queue file. _snooze(): sleeps for a specified amount of time. _log(): A convenient logging facility for subclasses _start(), _onefile(): deleted.
* Multiple queue runner classes, in a new package Mailman.Queue.bwarsaw2000-12-201-0/+149
Runner is the base class providing the framework for other qrunner specializations. IncomingRunner runs through qfiles/in taking messages from the smtpd, sending them through the standard message handler pipeline and potentially writing the message to qfiles/out or qfiles/news. OutgoingRunner takes message from qfiles/out and hands them over to the MTA. NewsRunner takes message from qfiles/news and hands them over to the news server.
generated by cgit v1.2.3-70-g09d2 (git 2.51.2) at 2025-11-08 18:24:08 +0000