summaryrefslogtreecommitdiff
path: root/Mailman/Queue
Commit message (Collapse)AuthorAgeFilesLines
...
* prepare_message(): mimelib -> email.bwarsaw2001-10-091-1/+1
|
* Update a comment.bwarsaw2001-10-041-1/+1
|
* DumperSwitchboard: New class to provide a public interface for readingbwarsaw2001-10-011-0/+10
| | | | | .db files, and to allow argument-less construction. This is used by bin/dumpdb.
* prepare_message(): Remove X-Trace: and NNTP-Posting-Host: headers,bwarsaw2001-10-011-5/+5
| | | | | since some nntpd servers will complain about these if they exist, and will reject the message posting.
* Convert from mimelib to email.bwarsaw2001-10-016-20/+13
| | | | | Also, use cStringIO directly instead of our own hack-around StringIO module.
* _dispose(): Reorganize the big if/elif/else switch so that it neverbwarsaw2001-09-051-9/+7
| | | | | | just short-circuit returns. We need to do this after moving the Save() into the try clause (which may turn out to not be the right thing to do...)
* _dispose(): Factor out common code paths to _toadmins().bwarsaw2001-08-291-25/+36
| | | | | | | | | | | | | | | | _toadmins(): Run the message through SpamDetect to filter out known spam patterns in messages to the list owners/moderators. Log such detected spam to logs/spam and discard the message. Also, for every message to a list's -owner or -admin address, we always send it on to the list owners and moderators, even though the moderators can't access the admin pages (only the admindb pages). Note 1: this makes the `tomoderators' key in the message metadata obsolete. Note 2: this may not be the Right Thing To Do.
* run(): Add a note about a minor bogosity in the calls tobwarsaw2001-08-291-1/+3
| | | | _doperiodic().
* MarshalSwitchboard._ext_read(): If the .db file is of version 2, justbwarsaw2001-08-151-0/+3
| | | | | delete the `filebase' key. That should be enough to update it to version 3.
* __onefile(): Another step towards conversion to MemberAdaptorbwarsaw2001-07-291-1/+1
| | | | API. GetPreferredLanguage() -> getMemberLanguage()
* MarshalSwitchboard._ext_read(): Add a little comment to thebwarsaw2001-07-191-0/+3
| | | | `mysterious' eval() call.
* Only eval() a float's repr() back to a float if the marshal actuallytwouters2001-07-191-1/+2
| | | | contained it.
* dequeue(): Be a bit more robust about possible exceptions, ensuringbwarsaw2001-07-191-12/+14
| | | | that the msgfp file is definitely closed.
* _dopipeline(): We don't need to save the return value of __import__().bwarsaw2001-07-131-1/+1
|
* _dopipeline(): Do not include the blank Exception catch-all here, butbwarsaw2001-07-131-9/+0
| | | | | | | | | | instead let them percolate up to Runner.__oneloop(). Now, the proper action when an uncaught exception occurs is to shunt the message instead of trying forever to re-deliver it (possibly causing logs/error to fill up /very/ quickly). Shunted messages can always be resent by moving them back to qfiles/in.
* start_runner(): Changed indentation, since the parent always returns.bwarsaw2001-07-111-14/+20
| | | | | Also added a note why "mailmanctl restart" doesn't Do The Right Thing w.r.t. allowing the subrunners to re-import various Mailman modules.
* _dopipeline(): Use a better mechanism for digging out the process()bwarsaw2001-07-101-4/+5
| | | | | | | | | function from the dynamically imported module. Also, MessageHeld -> HoldMessage Also, catch RejectMessage exceptions, bouncing the rejected message via mlist.BounceMessage().
* 'UnimplementedError' -> 'NotImplementedError'twouters2001-07-102-3/+3
|
* import time so it can be used.twouters2001-07-101-0/+1
|
* Add back COMMASPACE to globals, as it apparently *is* used, just nottwouters2001-07-101-1/+3
| | | | | | correctly. prepare_message(): 'COMMA' -> 'COMMASPACE'
* Don't forget to import LockFile for its exception.twouters2001-07-101-0/+1
|
* Move COMMASPACE global from NewsRunner.py (where it isn't used) totwouters2001-07-101-2/+0
| | | | | | ToUsenet.py (where it is.) This is probably a glitch in the functionality split, anyway, and moving around the global saves on creation and destruction costs. Very important, that!
* _shortcircuit(): Return self._stop as a default. This allows abwarsaw2001-07-081-1/+1
| | | | | runner.stop() method call to break the __oneloop() loop after any single message is processed.
* Cleaner startup/restart/shutdown implementation. This allows thebwarsaw2001-07-081-30/+48
| | | | | | | | | | | | | | | | | | | | | | | | sub-qrunners to cleanly exit themselves instead of potentially stopping them in the middle of processing a message. Specifically, sighup_handler(): Send SIGHUPs to the child processes instead of SIGINTs. start_lock_refresher(): Install our own SIGHUP handler, which simply toggles off an instance-based loop flag. The signal kicks us out of the sleep, so the next while test will return false and break us out of the loop, existing the lock refresher. No need to deal with KeyboardInterrupts. start_runner(): In the parent, just return the pid of the child. In the child, do the syslogging, and also install our own process's SIGHUP handler, which will call the qrunner's stop() method. This lets the qrunner break out of its loop at the next convenient place (i.e. after processing one of its files). Again, no need to deal with KeyboardInterrupts. master(): More accurate syslog messages. Also, make sure the children receive SIGHUP instead of SIGINT.
* prepare_message(): Use Utils.unique_message_id() to generate thebwarsaw2001-07-061-6/+13
| | | | | | | | Message-ID header. mcre: Message-ID headers crafted by Mailman now include a serial number. Recognize this, and also use a verbose regular expression so it's not so cryptic.
* _open_list(): The lockp argument was unused, so remove it.bwarsaw2001-07-061-1/+1
|
* intermediatebwarsaw2001-07-051-0/+205
|
* DEAL_WITH_PERMFAILURES_EVERY: controls how often _doperiodic() willbwarsaw2001-06-281-1/+12
| | | | | | | | | | | | | try to deal with deferred permanent failures. _doperiodic(): It can be a bit expensive to actually deal with permanent failures every time through this loop, and we don't need that level of accuracy. So now we keep a counter and only process permanent failures every DEAL_WITH_PERMFAILURES_EVERY times. (Okay, so this variable defaults to 1, but you get the idea. :) Also, move the Save() inside the try clause. I've decided that finally clauses should only unlock the list, not try to save it.
* Several changes which allow subclasses to do some calculation for eachbwarsaw2001-06-281-3/+48
| | | | | | | | | | | | | | | | | | | | | file processed. This would allow for example, a throttling algorithm which limited the number of files processed per time period, or when load averages were too high. Specifically, run(): Allow __oneloop() to return any false value, not just zero, to trigger the snooze alarm. __oneloop(): After each file is processed, call _shortcircuit() and break out of the file loop if that returns a true value. This way, _shortcircuit() is executed for each file processed, and gives subclasses a chance to do some per-file calculation. _doperiodic() is at a finer granularity, since it conflates per-file and per-processing-loop calculations. _cleanup(), _dispose(), _doperiodic(), _snooze(), _shortcircuit(): Add docstrings describing their interface. _shortcircuit(): Default is to return 0, meaning don't shortcircuit.
* Performance enhancement. In order to save on the time it takes tobwarsaw2001-06-271-18/+35
| | | | | | | | | | | | | | | | | | | | | parse and generate the plain text message each time it's enqueued or dequeued, we now use a binary cpickle as the message representation file. This is controlled by a module global SAVE_MSGS_AS_PICKLES which defaults to 1. When a message is saved as a pickle, its extension will be .pck instead of .msg. dequeuing will automatically recognize the different message formats and load accordingly. Specific changes include: enqueue(): If SAVE_MSGS_AS_PICKLES is set and the message metadata does not have a "_plaintext" key, then the message object is stored as a binary cpickle dump for speed. The incoming scripts post, join, leave, etc. will set the _plaintext key because they funnel the text straight from stdin to the .msg file, and the message text is only parsed on the first dequeue from qfiles/in. dequeue(): First try to load and unpickle the .pck file, falling back to loading and parsing the .msg text file if the former is missing.
* Better syslog() calling conventions.bwarsaw2001-06-271-6/+2
| | | | | | | _dispose(): Remove the "except Exception" class which allows any exceptions to percolate up to Runner.__oneloop()'s catch-all exception handler. Runner properly implements the log-and-shunt procedure for any such exceptions.
* Better syslog() calling conventions.bwarsaw2001-06-275-12/+13
|
* _dispose(): When a message is destined "toowner" and the metadatabwarsaw2001-05-311-1/+12
| | | | | contains the "tomoderators" key, sent the message to both the list owners and the list moderators.
* __onefile(): It is possible that there is no list name in thebwarsaw2001-05-221-12/+13
| | | | | | | message's metadata, which means that the message does not originate from a mailing list. In that case, don't try to open the list (since you can't ;), and set the global language to the server's default language instead of the list's preferred language.
* dequeue(): Since IOError and OSError are both derived frombwarsaw2001-05-181-1/+1
| | | | | | EnvironmentError, it makes for slightly cleaner and more efficient code to catch the common base class rather than a tuple containing the two derived classes.
* syslog() entries should go to logs/error not logs/qrunner. It'sbwarsaw2001-05-151-5/+5
| | | | highly annoying to have to tail more than one log file to find bugs.
* This file was never published in yr 2000.bwarsaw2001-05-141-1/+1
|
* enqueue(): The `received_time' metadata (which is set here once, butbwarsaw2001-05-141-22/+71
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | only if it has no prior value), is encoded into the file name so that we can guarantee FIFO order on the processed files. We can't encode the received time in the file attributes because there isn't enough precision (and I suspect that stat'ing all those files will be too much of a disk I/O drain). Instead, the filebase is composed of the string representation of the current time in float seconds, the symbol `+', and the SHA1 hexdigest of a hash of the uniquifying data. This makes it easy and quick to decode received time for FIFO sorting, but retains the "random" digest for bitrange slicing. Note that the received_time metadata value is never changed once its set so the first part of the filebase will remain unchanged as it moves between queues (while the hexdigest will almost definitely change on each queue move). dequeue(): Be more robust about missing .msg or .db files when the other exists (usually, it'll be the .msg file that's missing). Return None for either the msg or data part of the 2-tuple return value, where None means "missing". files(): Utililize the new file naming convention to break apart the file name and sort the files in FIFO order, while still retaining the bitrange random hash feature. MarshalSwitchboard: All Python versions up to and including Python 2.1 have a bug in the marshal representation of binary floating point numbers. Specifically, it loses precision that Mailman requires. The solution in this class is to have a hardcoded list of known float attributes, convert them to strings via repr() before marshaling the dictionary, and convert them back to floats -- via a safe eval() -- when reading the marshal back from file.
* __oneloop(): Guarantee FIFO order on the processed messages. Webwarsaw2001-05-141-16/+23
| | | | | | | | | | assume that the list of files coming back from Switchboard.files() is sorted by received time, so we don't need to randomize this list. Also, it's now possible for Switchboard.dequeue() to return None for either or both of msg and msgdata (say if the .msg file for a .db file got lost somehow). Check that both are not None before proceeding (but if either is None, log an error).
* _dispose(): Add support for mylist-join and mylist-leave aliases. Thebwarsaw2001-05-111-0/+10
| | | | | | | | | | | | mail scripts simply add metadata `tojoin' or `toleave' respectively, which tell CommandRunner which operation to perform. This is a bit crufty since MailCommandHandler.ParseMailCommands() still handles the actual job of adding or deleting the member. CommandRunner hacks the Subject: line of the message to add just the specific desired command, and it empties the message's payload (effectively ignoring any additional commands or useless text in the email message).
* _dispose(): Oops! Remove an outdated comment.bwarsaw2001-05-091-3/+1
|
* _dispose(): Add a new header X-List-Received-Date: which contains thebwarsaw2001-05-091-1/+4
| | | | | | time that Mailman received the message. This header only goes into the archived copy of the message (header name and semantics were discussed on mailman-developers).
* Added missing import.bwarsaw2001-05-011-3/+22
| | | | | | | | | _dispose(): Support for new site-wide date clobbering policy. No longer is the Date: header clobbering a list-specific option, instead ARCHIVER_CLOBBER_DATE_POLICY and ARCHIVER_ALLOWABLE_SANE_DATE_SKEW control how Date: munging happens site-wide (this makes the most sense since the choice of Pipermail vs. external archiver is also made site-wide). See Defaults.py.in for more details.
* _dispose(): Fix a bug whereby if an uncaught exception occurred, thebwarsaw2001-04-021-1/+4
| | | | | unprocessed pipeline module list wouldn't get saved in the message metadata.
* _dispose(): Have to pass ParseMailCommands() the msgdata dictionary.bwarsaw2001-03-031-1/+1
|
* __onefile(): The qrunner framework now sets up the language contextbwarsaw2001-03-021-1/+18
| | | | | | | | | | | | | | | for each processed message. It does this by getting the preferred language of the sender of the message (defaulting to the list's preferred language if the sender isn't a member of the list). Next, it saves the previous global language context in a local variable, sets the context to the message's preferred language and disposes of the message. Then it restores the original language context. It also sets the message's metadata (key: `lang') to contain the language name for the language context in case any handler modules need access to it.
* _dispose(): If there's no appropriate key in the message metadata,bwarsaw2001-03-011-0/+12
| | | | | determine whether this message was destined for the -owner or -admin address and update the metadata.
* intermediatebwarsaw2001-02-281-1/+1
|
* CommandRunner which watches qfiles/commands for such things asbwarsaw2001-02-281-0/+123
| | | | bounces, confirm notices, and -request email commands.
* _dispose(): A bit of elaboration so that we can properlybwarsaw2001-02-161-7/+40
| | | | | | | | | | | | | RegisterBounce on known permanent failures. These are handed to us via the exception object when SomeRecipientsFailed is raised. Append any permanent failures to an instance variable to be handled in the _doperiodic() method. Temporary failures are still handled the same way, except we only requeue the message if we've actually got temp failures to retry. _doperiodic(): Periodically, we attempt to acquire the list lock and register all permanent failures we've got pending for that list.
generated by cgit v1.2.3-70-g09d2 (git 2.51.2) at 2025-11-09 19:40:39 +0000