| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
from Mailman.Queue to Mailman.queue (note the case change to be more PEP 8
compliant).
The Switchboard and Runner classes have been moved into the package
__init__.py and the previous class modules have been removed.
The switchboard cache is removed; I don't think it was ultimately buying us
much. Now, just import the Switchboard class and instantiate it directly.
Added an IRunner interface.
Renamed the ArchRunner to ArchiveRunner.
bin/qrunner and bin/mailmanctl are updated accordingly. For the former, it no
long accepts -r=All to run all qrunners. You can still use the short name
(e.g. --runner=incoming) to run the built-in queue runners, but this design
will eventually allow for plugin qrunners by allowing them to be run
specifying the full package path to the class. It also now accepts a leading
dot to indicate a qrunner class relative to the Mailman.queue package.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
The convenience methods in Defaults for getting seconds, minutes, hours, and
days now returns a subtype of timedelta, which provides conversion to float
and int for compatibility with interfaces that require those values
(e.g. signal.alarm() and time.sleep().
In bin/make_instance, the var_dir really needs to be an absolute path,
otherwise it's possible to get a var dir nested inside the var dir.
More MailList object eradication.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
during early initialization so that we're guaranteed to get the right value
regardless of the shell umask used to invoke the command line script. While
we're at it, we can remove almost all individual umask settings previously in
the code, and make file permissions consistently -rw-rw---- (IOW, files are no
longer other readable).
The only subsystem that wasn't changed was the archiver, because it uses its
own umask settings to ensure that private archives have the proper
permissions. Eventually we'll mess with this, but if it ain't broken...
Note that check_perms complains about directory permissions, but I think
check_perms can be fixed (or perhaps, even removed?!). If we decide to use
LMTPRunner and HTTPRunner exclusively then no outside process will be touching
our files potentially with the incorrect permissions, umask, owner, or group.
If we control all of our own touch points then I think we can lock out
'other'.
Another open question is whether Utils.set_global_password() can have its
umask setting removed. It locks permissions down so even the group can't
write to the site password file, but the default umask of 007 might be good
enough even for this file.
Utils.makedirs() now takes an optional mode argument, which defaults to 02775
for backward compatibility. First, the default mode can probably be changed
to 02770 (see above). Second, all code that was tweaking the umask in order
to do a platform compatible os.mkdir() has now been refactored to use
Utils.makedirs().
Another tricky thing was getting SQLite via SQLAlchemy to create its
data/mailman.db file with the proper permissions. From the comment in
dbcontext.py:
# XXX By design of SQLite, database file creation does not honor
# umask. See their ticket #1193:
# http://www.sqlite.org/cvstrac/tktview?tn=1193,31
More details in that file, but the work around is to essentially 'touch' the
database file if 'sqlite' is the scheme of the SQLAlchemy URL. This little
pre-touch sets the right umask honoring permission and won't hurt if the file
already exists. SQLite will happily keep the existing permissions, and in
fact that ticket referenced above recommends doing things this way.
In the Mailman.database.initialize(), create a global lock that prevents more
than one process from entering this init function at the same time. It's
probably not strictly necessary given that I believe all the operations in
dbcontext.connect() are multi-processing safe, but it also doesn't seem to
hurt and prevents race conditions regardless of the database's own
safeguards (or lack thereof).
Make sure nightly_gzip.py calls initialize().
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
name in more than one domain.
- Totally eradicate MAILMAN_SITE_LIST, and in fact the entire need for a site
list. The functions that the site list previously performed are either
removed or supported in other ways. For example, instead of forwarding
owner bounces to the site list, we now have a SITE_OWNER_ADDRESS which
should point to a human, and such bounces are sent there instead. There's
also a "no reply" email address that should be set up to go to devnull. For
any message that never expects a reply, the sender is set to this address.
- Remove the Site.py module. It was an experimental approach to trying to
support virtual domains, and we're going to do it so much better now that
this module is no longer necessary. Site._makedirs() -> Utils.makedir().
- VIRTUAL_HOST_OVERVIEW is completely removed, since now virtual hosts are
always enabled. Virtual domains should be added to mailman.cfg by using the
new add_domain() function. add_virtualhost() is gone. If no virtual
domains are added explicitly, we add the default one that configure guessed
(but we never add that if domains are added explicitly).
- Utils.get_domain() -> Utils.get_request_domain()
- withlist code cleanup and make sure that we load etc/mailman.cfg
- A new base exception called MailmanException is added, from which all
exceptions defined in Errors.py ultimately derive. MailmanError is retained
and derives from MailmanException.
- BadDomainSpecificationError is added.
- Remove the -V/--virtual-host-overview option from list_lists and add instead
-d/--domain and -f/--full.
- bin/update probably works but needs more testing.
- bin/newlist and bin/rmlist take fqdn list names, but default to the default
domain if @whatever isn't given. newlist's -u/--urlhost and -e/--emailhost
options are removed. The domain that the list is being added to must
already exist.
- Minor code cleanup in Message.py
- Bump version to 2.2.0a1
- The Configuration object grows a .domain dictionary which maps email hosts
to url hosts. The reverse mapping is supported, but not directly; use
Configuration.get_email_host() instead.
- Mailman/Cgi/create is converted from mm_cfg to config, and some minor code
cleanup is performed. Also, convert to __i18n_templates__ = True.
- New MailList APIs:
+ property .fqdn_listname
+ GetNoReplyEmail()
+ Create() API changes and refactoring.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
configuration file. While the full conversion is not yet complete, everything
that seems to be required to run mailmanctl, qrunner, rmlist, and newlist have
been updated.
Basically, modules should no longer import mm_cfg, but instead they should
import Mailman.configuration.config. The latter is an object that's
guaranteed to exist, but not guaranteed to be initialized until some top-level
script calls config.load(). The latter should be called with the argument to
-C/--config which is a new convention the above scripts have been given.
In most cases, where mm_cfg.<variable> is used config.<variable> can be used,
but the exceptions are where the default value must be available before
config.load() is called. Sometimes you can import Mailman.Default and get the
variable from there, but other times the code has to be changed to work around
this limitation. Take each on a case-by-case basis.
Note that the various directories calculated from VAR_PREFIX, EXEC_PREFIX, and
PREFIX are now calculated in config.py, not in Defaults.py. This way a
configuration file can override the base directories and everything should
work correctly.
Other changes here include:
- mailmanctl, qrunner, and update are switched to optparse and $-strings, and
changed to the mmshell architecture
- An etc directory has been added to /usr/local/mailman and a
mailman.cfg.sample file is installed there. Sites should now edit an
etc/mailman.cfg file to do their configurations, although the mm_cfg file is
still honored. The formats of the two files are identical.
- list_lists is given the -C/--config option
- Some coding style fixes in bin/update, but not extensive
- Get rid of nested scope hacks in qrunner.py
- A start on getting EmailBase tests working (specifically test_message),
although not yet complete.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
traces of our crufty old Syslog. Most of this work was purely mechanical,
except for:
1) Initializing the loggers. For this, there's a new module
Mailman/loginit.py (yes all modules from now on will use PEP 8
names). We can't call this 'logging.py' because that will
interfere with importing the stdlib module of the same name (can
you say Python 2.5 and absolute imports?).
If you want to write log messages both to the log file and to
stderr, pass True to loginit.initialize(). This will turn on
propagation of log messages to the parent 'mailman' logger, which
is set up to print to stderr. This is how bin/qrunner works when
not running as a subprocess of mailmanctl.
2) The driver script. I had to untwist the StampedLogger stuff and
implement differently printing exceptions and such to log/error
because standard logging objects don't have a write() method. So
we write to a cStringIO and then pass that to the logger.
3) SMTPDirect.py because of the configurability of the log messages.
This required changing SafeDict into a dict subclass (which is
better than using UserDicts anyway -- yay Python 2.3!). It's
probably still possible to flummox things up if you change the
name of the loggers in the SMTP_LOG_* variables in mm_cfg.py.
However, the worst you can do is cause output to go to stderr and
not go to a log file.
Note too that all entry points into the Mailman system must call
Mailman.loginit.initialize() or the log output will go to stderr
(which may occasionally be what you want). Currently all CGIs and
qrunners should be working properly.
I wish I could have tested all code paths that touch the logger, but
that's infeasible. I have tested this, but it's possible that there
were some mistakes in the translation.
- Mailman.Bouncers.BounceAPI.Stop is a singleton, but not a class
instance any more.
- True/False code cleanup, PEP 8 import restructuring, whitespace
normalization, and copyright year updates, as appropriate.
|
| |
|
|
|
|
|
|
|
|
|
| |
- Remove True/False binding cruft
- Remove __future__ statements for nested scopes
- Remove ascii_letters import hack from Utils.py
- Remove mimetypes.guess_all_extensions import hack from Scrubber.py
- In Pending.py, set _missing to object() (better than using [])
Also, update copyright years where appropriate, and re-order imports more to
my PEP 8 tastes. Whitespace normalize.
|
| | |
|
| |
|
|
|
|
| |
BounceRunner.py to discard the bounce when Stop returned. Changed
DSN.py to recognize Action: headers with comments. Changed Qmail.py
to recognize an observed different starting string.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
_register_bounces(): Return empty list as default for sitebounces so
we can concatenate lists easier later.
_cleanup(): Only call _register_bounces() if there are bounces to
register.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
efficient -- and more controllable -- bounce processing without
killing lock contention.
__init__(): Two new variables to collect bounce event information.
_bouncecnt keeps a count of all events since the last processing pass
(mostly for logging). _bounces is a dictionary mapping listnames to
sequence of bounce events. A bounce event is a tuple of
(address, day, msg)
where day is (YYYY, MM, DD).
_dispose(): Move all the actual registration of bounces out of this
method. Instead, we add bounce events to the queue, which gets
processed by a _doperiodic() pass. This allows us to never lock any
lists in _dispose().
_doperiodic(): Check periodically for queued bounces that need to be
processed, by default once every 15 minutes.
_cleanup(): Needs to call _register_bounces() to take care of any
queued bounces before the runner is shut down.
_register_bounces(): This does the actual bounce registration,
hopefully in a more efficient manner, by collating all queued bounce
events and locking each list only once. Also handles registering
bounces to the site list.
This stuff still needs a bit more testing.
|
| |
|
|
|
| |
addresses were found, but for which those addresses were not members.
This is just annoying.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
where if the list lock couldn't be obtained, we couldn't register the
bounce and the message would get forwarded to the list owners.
Instead, let's just requeue it and try again later.
Note: untested! I'm checking this into cvs so I can test it on
python.org... yes on a live system. ;/
|
| | |
|
| |
|
|
|
| |
a forwarded bounce message. Complained upon by Daniel Buchmann,
improved upon by Greg Ward. Further mungulated by Barry.
|
| | |
|
| | |
|
| |
|
|
|
| |
should go only to the list owners. I don't think the list moderators
need to be bothered.
|
| |
|
|
| |
method on the MailList instance.
|
| |
|
|
|
|
|
|
|
| |
message did not bounce match, or if no member addresses could be
extracted from it.
maybe_forward(): MIME Forward the given message to the list owner +
list moderator, if bounce_unrecognized_goes_to_list_owner is true.
Otherwise it discards the message. Write a log entry in either case.
|
| |
|
|
|
|
| |
addresses sucked out of the bounce message actually matched a member
of any mailing list. If not, we still don't forward it, but we do log
this situation.
|
| |
|
|
| |
header.
|
| |
|
|
| |
sure why that is, but let's filter them out anyway.
|
| |
|
|
|
| |
list's state via xlist.Load() -- conservative if the state hasn't
changed.
|
| |
|
|
|
| |
list, call mlist.Load() so we can be sure we've got the most current
state.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
perform better. In general there are two changes: 1) we never lock a
list until and unless absolutely necessary, 2) we only run the bounce
runner once every minute (as opposed to the normal runners'
once-per-second sleep time). Specifically,
SLEEPTIME: The base class now looks at this attribute to decide how
long to sleep between loop iterations, and it defines a default. We
override the bounce runner's value to lengthen the time it sleeps to 1
minute (we may eventually want to move this to Defaults.py).
__scanbounce(): Gone. All its functionality is subsumed in the new
ScanMessages() method and the new _dispose().
__verpbounce(): Renamed to the non-method verp_bounce(). Also, we rip
out the code that iterates through all the lists if the bounce came to
the site list's -bounces address.
_dispose(): Simply calls verp_bounce() to find bouncing addresses, and
if that returns a false value (empty list), calls ScanMessages() to
see if the bounce detector can find some bouncing addresses. If not,
we're done (after possibly forwarding the non-matching message).
Otherwise, we register the bounce either on the target mailing list,
or on all the mailing lists if this came to the site's -bounces
address.
|
| |
|
|
|
|
|
|
|
| |
_open_list() caches the MailList instances in a WeakValueDictionary
which don't contribute to the reference count, but ensure that each
runner will get one and only one copy of the MailList data.
This should help in the memory footprint of the runners, but I suspect
we may still be leaking something.
|
| |
|
|
|
|
| |
more conveniently use a class attribute to specify this. On the
BounceRunner, turn off MailList object caching so that it'll be more
friendly to long-term memory use.
|
| |
|
|
|
| |
sending the unrecognized message on to the list owner. In either
case, write a log entry to logs/bounce.
|
| |
|
|
|
|
| |
try/except for IndexError. If the site admin's configured VERP_REGEXP
incorrectly, it's better to provide a more useful error message in
log/errors and ignore the bounce than to let a traceback percolate up.
|
| |
|
|
|
|
|
| |
specializing the queue directory, make QDIR a class attribute and have
Runner's ctor use that instead of an __init__() argument. This lets
us get rid of most of the __init__()'s in the derived classes.
(OutgoingRunner still needs one though.)
|
| |
|
|
| |
MailList object instead of instantiating one explicitly.
|
| |
|
|
|
|
|
|
| |
list that's already locked, e.g. the site list that we know we're
processing for! Bug reported by Dan Mick.
BAW: The loop in these two methods should really be factored out, but
there's a bit of an impedance mismatch for the inner logic.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
least one in the wild (reported by Dan Mick) that leaves the To:
header empty or useless, while the useful VERP'd bounce information
ends up in a Delivered-To: header.
Note that there can be multiple Delivered-To: headers so we need to
search them all (and we don't worry about false positives for
forwarded email, because only one should match VERP_REGEXP).
Also try Apparently-To: if neither of the others yields a hit.
|
| | |
|
| |
|
|
| |
VERP decoding. Use mm_cfg.VERP_REGEXP instead.
|
| | |
|
|
|
listname-bounces alias.
|
