| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Pending.py module is removed. Added an interface to this functionality such
that any IPendable (essentially a key/value mapping) can be associated with a
token, and that token can be confirmed and has a lifetime. Any keys and
values can be stored, as long as both are unicodes.
Added a doctest.
Modified initialization of the database layer to support pluggability via
setuptools. No longer is this layer initialized from a module, but now it's
instantiated from a class that implements IDatabase. The StockDatabase class
implements the SQLAchemy/Elixir layer, but this can be overridden in a
setup.py. Bye bye MANAGERS_INIT_FUNCTION, we hardly knew ye.
Added a package Mailman.app which will contain certain application specific
functionality. Right now, the only there there is an IRegistar
implementation, which didn't seem to fit anywhere else.
Speaking of which, the IRegistrar interface implements all the logic related
to registration and verification of email addresses. Think the equivalent of
MailList.AddMember() except generalized out of a mailing list context. This
latter will eventually go away. The IRegistrar sends the confirmation email.
Added an IDomain interface, though the only implementation of this so far
lives in the registration.txt doctest. This defines the context necessary for
domain-level things, like address confirmation.
A bunch of other cleanups in modules that are necessary due to the refactoring
of Pending, but don't affect anything that's actually tested yet, so I won't
vouch for them (except that they don't throw errors on import!).
Clean up Defaults.py; also turn the functions seconds(), minutes(), hours()
and days() into their datetime.timedelta equivalents.
Consolidated the bogus email address exceptions.
In some places where appropriate, use email 4.0 module names instead of the
older brand.
Switch from Mailman.Utils.unique_message_id() to email.utils.make_msgid()
everywhere. This is because we need to allow sending not in the context of a
mailing list (i.e. domain-wide address confirmation message). So we can't use
a Message-ID generator that requires a mailing list. OTOH, this breaks
Message-ID collision detection in the mail->news gateway. I'll fix that
eventually.
Remove the 'verified' row on the Address table. Now verification is checked
by Address.verified_on not being None.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
confident the Elixir branch is ready to become mainline. Also, fewer branches
makes for an easier migration to a dvcs.
Don't expect much of the old test suite to work, or even for much of the old
functionality to work. The changes here are disruptive enough to break higher
level parts of Mailman. But that's okay because I am slowly building up a new
and improved test suite, which will lead to a functional system again.
For now, only the doctests in Mailman/docs (and their related test harnesses)
will pass, but they all do pass. Note that Mailman/docs serve as system
documentation first and unit tests second. You should be able to read the
doctest files to understand the underlying data model.
Other changes included in this merge:
- Added the Mailman.ext extension package.
- zope.interfaces uses to describe major components
- SQLAlchemy/Elixir used as the database model
- Top level doinstall target renamed to justinstall
- 3rd-party packages are now installed in pythonlib/lib/python to be more
compliant with distutils standards. This allows us to use just --home
instead of all the --install-* options.
- No longer need to include the email package or pysqlite, as Python 2.5 is
required (and comes with both packages).
- munepy package is included, for Python enums
- IRosterSets are added as a way to manage a collection of IRosters. Roster
sets are named so that we can maintain the indirection between mailing lists
and rosters, where the two are maintained in different storages.
- IMailingListRosters: remove_*_roster() -> delete_*_roster()
- Remove IMember interface.
- Utils.list_names() -> config.list_manager.names
- fqdn_listname() takes an optional hostname argument.
- Added a bunch of new exceptions used throughout the new interfaces.
- Make LockFile a context manager for use with the 'with' statement.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
during early initialization so that we're guaranteed to get the right value
regardless of the shell umask used to invoke the command line script. While
we're at it, we can remove almost all individual umask settings previously in
the code, and make file permissions consistently -rw-rw---- (IOW, files are no
longer other readable).
The only subsystem that wasn't changed was the archiver, because it uses its
own umask settings to ensure that private archives have the proper
permissions. Eventually we'll mess with this, but if it ain't broken...
Note that check_perms complains about directory permissions, but I think
check_perms can be fixed (or perhaps, even removed?!). If we decide to use
LMTPRunner and HTTPRunner exclusively then no outside process will be touching
our files potentially with the incorrect permissions, umask, owner, or group.
If we control all of our own touch points then I think we can lock out
'other'.
Another open question is whether Utils.set_global_password() can have its
umask setting removed. It locks permissions down so even the group can't
write to the site password file, but the default umask of 007 might be good
enough even for this file.
Utils.makedirs() now takes an optional mode argument, which defaults to 02775
for backward compatibility. First, the default mode can probably be changed
to 02770 (see above). Second, all code that was tweaking the umask in order
to do a platform compatible os.mkdir() has now been refactored to use
Utils.makedirs().
Another tricky thing was getting SQLite via SQLAlchemy to create its
data/mailman.db file with the proper permissions. From the comment in
dbcontext.py:
# XXX By design of SQLite, database file creation does not honor
# umask. See their ticket #1193:
# http://www.sqlite.org/cvstrac/tktview?tn=1193,31
More details in that file, but the work around is to essentially 'touch' the
database file if 'sqlite' is the scheme of the SQLAlchemy URL. This little
pre-touch sets the right umask honoring permission and won't hurt if the file
already exists. SQLite will happily keep the existing permissions, and in
fact that ticket referenced above recommends doing things this way.
In the Mailman.database.initialize(), create a global lock that prevents more
than one process from entering this init function at the same time. It's
probably not strictly necessary given that I believe all the operations in
dbcontext.connect() are multi-processing safe, but it also doesn't seem to
hurt and prevents race conditions regardless of the database's own
safeguards (or lack thereof).
Make sure nightly_gzip.py calls initialize().
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch
................
r8114 | bwarsaw | 2006-12-06 00:16:54 -0500 (Wed, 06 Dec 2006) | 44 lines
Initial take on using SQLAlchemy to store list data in lieu of Python pickles.
While all the list data (including OldStyleMemberships attributes) are stored
in the database, many attributes are stored as PickleTypes binary data. This
isn't idea but it gets things working until a more sophisticated schema can be
developed.
MailList class is now a new-style class, as is required by SQLAlchemy. This
makes several things, er, interesting. Rip out all the low-level pickle
reading and writing stuff. Hook SA transaction events into Lock() and
Unlock(). Move the hooking of the _memberadaptor into InitTempVars(), which
gets called by the SQLAlchemy hooks (MailList.__init__() never is).
Add an initialize.py module which centralizes all the initialization bits that
command line scripts have to do, including configuration, logging, and atabase
initialization.
This change also converts bin/withlist to mmshell wrapper.
Update to SQLAlchemy 0.3.1.
Revamp paths.py.in considerably. There were several problems with the old
way. We no longer disable default loading of site-packages so we don't need
to add Python's site-packages back to sys.path. Also, because
site.addsitedir() causes things like .pth paths to be /appended/ to sys.path,
they actually won't override any site-installed packages. E.g. if SQLAlchemy
is installed in the system Python, our version will not override. IIUC,
setuptools-based packages can be configured to work properly in the face of
package versions, however not all packages we currently depend on are
setuptools-based. So instead, we steal a bit of stuff from site.py but change
things so the prepend .pth stuff to sys.path.
Update several modules to use True/False and whitespace normalization.
Convert from mm_cfg to config object. Modernize a few coding constructs.
Add a couple of exceptions to handle database problems.
In the export script, include the widget type in the elements. This helped in
my stupid little throw away conversion script, but I think it will be more
generally useful.
Add an interact.py module which refactors interactive interpreter access.
Mostly this is used by withlist -i, but it lets us import Mailman.interact and
drop into a prompt just about anywhere (e.g. debugging).
................
r8115 | bwarsaw | 2006-12-07 09:13:56 -0500 (Thu, 07 Dec 2006) | 22 lines
Start to flesh out more of the SQLAlchemy mechanisms.
Added a MailList.__new__() which hooks instantiation to use a query on
dbcontext to get an existing mailing list. A 'no-args' call means we're doing
a Create(), though eventually that will change too.
For now, disable the CheckVersion() call. Eventually this will be folded into
schema migration.
list_exists(): Rewrite to use the dbcontext query to determine if the named
mailing list exists or not. Requires the fqdn_listname.
Eradicate two failed member adaptors: BDBMemberAdaptor and SAMemberships.
Change the way the DBContext holds onto tables. It now keeps a dictionary
mapping the table's name to the SA Table instance. This makes it easier to
look up and use the individual tables.
Add 'web_page_url' as an attribute managed by SA, and remove a debugging
print.
................
r8116 | bwarsaw | 2006-12-11 07:27:47 -0500 (Mon, 11 Dec 2006) | 29 lines
Rework the whole dbcontext and transaction framework. SA already handles
nested transactions so we don't have to worry about them. However, we do have
the weird situation where some transactions are tied to MailList
.Lock()/.Unlock()/.Save() and some are tied to non-mlist actions. So now we
use an @txn decorator to put methods in a session transaction, but then we
also hook into the above MailList methods as possibly sub-transactions. We
use a weakref subclass to manage the MailList interface, with a dictionary
mapping MailList fqdn_listnames against transactions. The weakrefs come in by
giving us a callback when a MailList gets derefed such that we're guaranteed
to rollback any outstanding transaction.
Also, we have one global DBContext instance but rather than force the rest of
Mailman to deal with context objects, instead we expose API methods on that
object into the Mailman.database module, which the rest of the code will use.
Such methods must be prepended with 'api_' to get exposed this way.
bin/rmlist now works with the SA-backend. I refactored the code here so that
other code (namely, the test suite) can more easily and consistently remove a
mailing list. This isn't the best place for it ultimately, but it's good
enough for now.
New convenience functions Utils.split_listname(), .fqdn_listname().
Convert testall to use Mailman.initialize.initialize(). Not all tests work,
but I'm down to only 8 failures and 7 errors. Also, do a better job of
recovering from failures in setUp().
MailList.__new__() now takes keyword arguments.
................
r8117 | bwarsaw | 2006-12-11 22:58:06 -0500 (Mon, 11 Dec 2006) | 7 lines
Unit test repairs; even though the unit tests are still pretty fragile,
everything now passes with the SQLAlchemy storage of list data.
Added missing 'personalize' column. Converted mailmanctl and qrunner to
initialize() interface. Fixed _cookie_path() to not fail if SCRIPT_NAME is
not in the environment.
................
r8118 | bwarsaw | 2006-12-27 18:45:41 -0500 (Wed, 27 Dec 2006) | 21 lines
Utils.list_names(): Use a database query to get all the list names.
dbcontext.py: Added api_get_list_names() to support Utils.list_names().
listdata.py: Added two additional MailList attributes which need to be stored
in the database. The first is 'admin_member_chunksize' which isn't modifiable
from the web. The second is 'password' which holds the list's password.
HTMLFormatObject: item strings can now be unicodes.
bin/list_lists.py: Must call initialize() to get the database properly
initialized, not just config.load(). This will be a common theme.
SecurityManager.py:
- Remove md5 and crypt support
- Added mailman.debug logger, though it will be only used during
debugging.
- The 'secret' can be a unicode now.
- A few coding style updates; repr() instead of backticks, 'key in dict'
instead of 'dict.has_key(key)'
................
r8119 | bwarsaw | 2006-12-27 19:13:09 -0500 (Wed, 27 Dec 2006) | 2 lines
genaliases.py: config.load() -> initialize()
................
r8120 | bwarsaw | 2006-12-27 19:17:26 -0500 (Wed, 27 Dec 2006) | 9 lines
Blocked revisions 8113 via svnmerge
........
r8113 | bwarsaw | 2006-12-05 23:54:30 -0500 (Tue, 05 Dec 2006) | 3 lines
Initialized merge tracking via "svnmerge" with revisions "1-8112" from
https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch
........
................
r8121 | bwarsaw | 2006-12-28 23:34:52 -0500 (Thu, 28 Dec 2006) | 20 lines
Remove SIGTERM handling from all the CGI scripts. This messes with HTTPRunner
because when you issue "mailmanctl stop" after the signal handler has been
installed, the process will get a SIGTERM, the signal handler will run, and
the process will exit with a normal zero code. This will cause mailmanctl to
try to restart the HTTPRunner.
I don't think we need that stuff at all when running under wsgi with a
SQLAlchemy backend. If mailmanctl kills the HTTPRunner in the middle of the
process, I believe (but have not tested) that the transaction should get
properly rolled back at process exit. We need to make sure about this, and
also we need to test the signal handling functionality under traditional CGI
environment (if we even still want to support that).
Also, make sure that we don't try to initialize the loggers twice in qrunner.
This was the cause of all the double entries in logs/qrunner.
Fix a coding style nit in mailmanctl.py.
De-DOS-ify line endings in loginit.py.
................
|
| |
|
|
|
|
|
|
|
|
|
| |
- Remove True/False binding cruft
- Remove __future__ statements for nested scopes
- Remove ascii_letters import hack from Utils.py
- Remove mimetypes.guess_all_extensions import hack from Scrubber.py
- In Pending.py, set _missing to object() (better than using [])
Also, update copyright years where appropriate, and re-order imports more to
my PEP 8 tastes. Whitespace normalize.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
shared among all lists, we need to be a lot more careful with locks -- with
thousands of busy (and/or moderated) lists, a lot of lock conflits can
happen. Each pending.pck manipulation is now done inside a loop, and the
loop gets retried in its entirety, starting with grabbing the lockfile and
loading the db, whenever we have a lock failure. Several places now check
more agressively for locks, as the .pck can get very large and take a while
to write, and each process now uses a different temp-filename so as not to
overwrite each other.
This should have no impact on 'average' Mailman installations, with only a
few lists, but should fix (or at least alleviate) Peer's problems with
TimeOutErrors, NotLockedErrors and corrupt pending.pck's, with his Mailman
installation with many, many lists.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
database, we need to use pickle instead of marshal. Pickle's more
reliable and portable anyway.
We can do auto-upgrade from marshal to pickle. If the .pck file isn't
found, we'll read the .db file. But we always write the .pck file and
if that's successful we can delete the .db file. We also do the
standard "write a .tmp file and rotate with os.rename()" dance.
|
| |
|
|
| |
memberships.
|
| |
|
|
|
| |
the valid Pending types. (They're 1-tuples so that the assert is more
robust).
|
| |
|
|
|
|
|
|
| |
the evictions dictionary too.
_save(): Go through and delete any evictions keys which don't appear
in the pending dictionary (i.e. we're cleaning up after ourselves
due to the above bug).
|
| | |
|
| |
|
|
|
|
| |
is a flag specifying whether pending entries should be removed from
the database or not, and will be used with the revamped confirmation
web ui.
|
| |
|
|
|
|
|
|
| |
Specifically,
CHANGE_OF_ADDRESS: new constant for this type of confirmable request.
new(): Add CHANGE_OF_ADDRESS to the assert.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Entry eviction times are stored as a sub-dictionary under a separate
key called `evictions'. This leaves the cookie data as whatever was
given to the new() function. The keys of the eviction dictionary
are the cookies and the values are the time at which the entry
should be removed.
- Another new key `version' for future backwards compatibility <wink>.
- Each entry for the cookie should have a op key as its first element,
which currently should be either Pending.SUBSCRIPTION or
Pending.UNSUBSCRIPTION.
new(): Make sure the `evictions' dictionary is present.
confirm(): Don't strip off the last element of the tuple.
_load(): If there's no pending.db file, return a dictionary containing
an empty `evictions' sub-dictionary.
_save(): Check the eviction times in the `evictions' sub-dictionary.
Also, add the schema `version' entry.
_update(): Fix the conversion from pending_subscriptions.db to
pending.db file format so that SUBSCRIPTION entries include a default
language. Also move the eviction times (which in the old format are
the time the request was made) to the `evictions' sub-dictionary and
add the PENDING_REQUEST_LIFE value.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
inside a class; use the module as if a singleton instance.
The pending database is now stored in data/pending.db.
new(): Stores a new cookie in the pending.db with the key an sha
hexdigest based on the current time, a random number and the content.
The timestamp is now the point in the future at which this entry can
be evicted.
confirm(): Given a cookie, return the data (with the timestamp
stripped off), or None if the cookie is missing from the database.
_load(), _save(): Rewritten and simplified low-level marshal/unmarshal
of the database. Assumes lock is acquired.
_update(): For use by the bin/update script to convert from
pending_subscription.db to pending.db (the old keys ought to be
compatible).
|
| | |
|
| |
|
|
| |
LockFile.AlreadyLockedError
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1. Eliminate an unlocked window, where another process could also load
the db, then save after the current process and stomp its changes.
All transactions are now atomically locked.
2. Implement periodic culling of the db, to remove old entries.
Otherwise (i assume) unclaimed entries would just accumulate
forever.
3. Simplified the interface so you can only put in new entries and
retrieve them. Cookie handling is implicit. All external
functionality is now all in two methods of the 'Pending' class object.
Details:
1. Atomicity: it used to be that the lock would only be set during the
write phase. Now, the db load and save handling is not exposed - it
is taken care of in the two exposed methods, which take care of the
locking around the load/save sequence.
Pending().new(stuff...) places an item's data in the db, returning its cookie.
Pending().confirmed(cookie) returns a tuple for the data, removing the item
from the db. It returns None if the cookie is not registered.
2. Periodically, on saves that occur longer than (module var)
CULL_INTERVAL after the last save, the db is checked for stale items,
and they're removed. Items are stale if they're older than the value
of the new default config var, PENDING_REQUEST_LIFE. (The timestamp
on the db is new, but old versions lacking it will just be culled on
the first save, and have it added.)
3. Two methods provide all the functionality:
Pending().new(stuff...) places an item in the db, returning a cookie
for it.
Pending().confirmed(cookie) returns the item, removing it from the
db - or it returns None if the item is not found.
|
| |
|
|
| |
try/finally, with unlocking being done in the finally clause.
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
in pending...
|
| |
|
