summaryrefslogtreecommitdiff
path: root/Mailman
Commit message (Collapse)AuthorAgeFilesLines
...
* | | senddigests.py - Changed to catch exceptions thrown by ↵msapiro2007-06-151-2/+13
| | | | | | | | | | | | | | | mlist.send_digest_now() and report them and continue processing the remaining lists.
* | | admindb.py - Changed to not show the "Discard all messages marked Defer" ↵msapiro2007-06-121-4/+7
| |/ |/| | | | | | | | | | | | | | | | | checkbox when there are only (un)subscribes and no held messages. - Added a separator and heading for "Held Messages" like the ones for "Subscribe Requests" and "Unsubscribe Requests". - Added additional test to not display "Database Updated ..." when coming from the login page.
* | Improved processing of an Approve(d): body line by decoding the body payloadmsapiro2007-06-091-5/+21
|/ | | before looking for/deleting the Approve(d): line.
* rename a few testsbwarsaw2007-06-024-2/+2
|
* Add doctest for Replybot handler. The test in test_handlers.py didn't reallybwarsaw2007-05-315-37/+277
| | | | | | | | | do anything. The doctest needs to have some way of testing the grace period, but it's still more tests of the module than there every was before. Update the Replyboty handler to use $-strings internally. Eliminate the use of %-strings in auto-response textsy. Only $-strings can be used, which allows us to get rid of another use of SafeDict.
* Remove leftover reference to removed Mailman.database.tables.languagesbwarsaw2007-05-303-2/+11
| | | | | | | | | module. Add in a temporary reference (with XXX comment) to Mailman.database.model.Language where necessary, although I will need to remove this eventually. Added clean up code to the mlist-rosters.txt doctest, otherwise it leaks the test mailing list.
* Go ahead and remove the Mailman/database/tables directory since all the Elixirbwarsaw2007-05-3013-783/+374
| | | | | | | | | | | | | | | | | | classes live in Mailman/databae/model now. Remove the TestDecorate test class from test_handlers.py and move them into a doctest called decorate.txt (with harness in test_decorate.py). Remove the dependence on SafeDict from the Decorate handler because I can now use string.Template object to safely fill in header and footer templates. Eventually I want to completely remove SafeDict from Mailman, but it's still used in a few other places. This also means that only $-strings will be supported in headers and footers, and the import script will have to convert %-strings to $-strings. Also, '_internal_name' is no longer a supported header/footer substitution variable. Use $real_name or $list_name now. Added $fqdn_listname as a substitution variable. Update the DEFAULT_MSG_FOOTER accordingly.
* Support resetting a user's password if the imported file does not have abwarsaw2007-05-291-6/+13
| | | | password for the user, or if it's "{NONE}".
* Minor style issue.bwarsaw2007-05-291-1/+1
|
* Merge exp-elixir-branch to trunk. There is enough working to make me feelbwarsaw2007-05-28101-940/+3753
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | confident the Elixir branch is ready to become mainline. Also, fewer branches makes for an easier migration to a dvcs. Don't expect much of the old test suite to work, or even for much of the old functionality to work. The changes here are disruptive enough to break higher level parts of Mailman. But that's okay because I am slowly building up a new and improved test suite, which will lead to a functional system again. For now, only the doctests in Mailman/docs (and their related test harnesses) will pass, but they all do pass. Note that Mailman/docs serve as system documentation first and unit tests second. You should be able to read the doctest files to understand the underlying data model. Other changes included in this merge: - Added the Mailman.ext extension package. - zope.interfaces uses to describe major components - SQLAlchemy/Elixir used as the database model - Top level doinstall target renamed to justinstall - 3rd-party packages are now installed in pythonlib/lib/python to be more compliant with distutils standards. This allows us to use just --home instead of all the --install-* options. - No longer need to include the email package or pysqlite, as Python 2.5 is required (and comes with both packages). - munepy package is included, for Python enums - IRosterSets are added as a way to manage a collection of IRosters. Roster sets are named so that we can maintain the indirection between mailing lists and rosters, where the two are maintained in different storages. - IMailingListRosters: remove_*_roster() -> delete_*_roster() - Remove IMember interface. - Utils.list_names() -> config.list_manager.names - fqdn_listname() takes an optional hostname argument. - Added a bunch of new exceptions used throughout the new interfaces. - Make LockFile a context manager for use with the 'with' statement.
* Another round of merges of my Pycon branch...bwarsaw2007-04-179-67/+575
| | | | | | | | | | | | | | | | | | | | bin/testall.py - Improvements to setting up the tests with temporary files for the db and configuration. Don't accept -C/--config because that will just confuse things. - Copy our new testing.cfg.in template file to a temp file, populate it with a few additional run-time calculate values, and use that in both the parent (bin/testall script) and children (bin/mailmanctl and friends). Mailman/initialize.py - Split up initialize() into two functions as required by the test suite reorg above. Almost everything else in Mailman will continue to use the initialize() function though. Mailman/configuration.py - Store the filename that was used to load the .cfg file from on the configuration object.
* Add a missing import.bwarsaw2007-04-101-0/+1
| | | | Python 2.5 is required.
* Improve the way we handle avoiding InitTempVars() multiple times on anbwarsaw2007-04-095-31/+55
| | | | | | | | | | | | | | | | | | | instantiated MailList object via the mapper extension's populate_instance() method. This based on information from the SQLAlchemy folks. Add more useful output for LockFile debugging. Add checks in loginit.py's emit() method (and .flush()) so that if the stream has been closed, log messages will go to stderr. This happens under the test suite with SQLAlchemy, because SA keeps references to the MailList objects which it doesn't seem like we can clear before Python exits. So if say the lock logger is at debug level, when the lock object gets cleared at Python shutdown, the stream will have been closed by the time LockFile.__del__() gets called. This change avoids the traceback at the expense of a little extra stderr output. MailList.__lock -> MailList._lock MailList.__timestamp -> MailList._timestamp
* r8187 from 2.1-maint branch; fix bin/withlist -r to work with package pathsbwarsaw2007-04-051-2/+3
| | | | more deeply nested than 1 level.
* Moved the session.expire() to the MailList.Load() method, viabwarsaw2007-04-034-2/+15
| | | | | | | | | | | | | | | | | DBContext.api_load(). This is where it really ought to be based on the internal semantics of .Load()/.Lock(). MailListMapperExtension.populate_instance(): Checking the state of the isnew flag is not sufficient to know whether the MailList object we're getting is brand spankin' new or not. It turns out that when we session.expire() the MailList object, the next time SA loads this from the db, the populate_instance() will get called with isnew=True, even though the object really isn't new. Instead, check to make sure InitTempVars() isn't incorrectly called twice. Note that I might move this test, but I wanted to check in something that works, and then see if this is what we expect from the SA guys (this flag appears underdocumented). LockFile.py: Add some additional debugging.
* Add a configuration variable called LOG_CONFIG_FILE. If set, this names abwarsaw2007-03-312-47/+79
| | | | | | | | | | | | | | | ConfigParser style ini file which can be used to override various default settings for just the named loggers. For example, if you want just the locks logger to print at DEBUG level, add this: etc/mailman.cfg: LOG_CONFIG_FILE = 'etc/mailman.log' etc/mailman.log: [locks] level = DEBUG
* api_lock(): When locking the MailList object, tell the SQLAlchemy session tobwarsaw2007-03-302-1/+1
| | | | | | | | | | | expire the object. This way, when the MailList attributes are next accessed, the ORM will reload them from the database, getting any new values possibly set in other processes. This works better than trying to use always_refresh=True on the mapper, or trying to do a reload() because both of those approaches blow away locks. I'm not sure why this, but I suspect that it's because the identity map is handing us back a different object, rather than invalidating the object's attributes.
* passwords.py: 'utf-8' normalization.tkikuchi2007-03-272-3/+3
| | | | | Cgi/create.py: password scheme lookup.
* Fix passwords in bin/newlist. Yes, this needs unit tests!bwarsaw2007-03-271-6/+6
|
* passwords.py: Looks like we still need unicode checking.tkikuchi2007-03-254-44/+199
| | | | | | | Mark Sapiro's patch for 'format' parameter. (Decorate.py, Scrubber.py) Scrubber.py: More brush up of code ... 'Content-Transfer-Encoding' is not updated by msg.set_payload(). 'Url:' to 'URL:' normalization. test_handlers.py: Test codes for Decorate.py and Scrubber.py.
* CookHeaders.py:tkikuchi2007-03-242-30/+77
| | | | | | | | | Subject munging code made unified for both i18n and ascii encodings. test_handlers.py Adding test code for i18n and numbering subject munging. (also tab normalization for the last commit)
* Brush up the fall-back-to-utf8 mechanizm in ToDigest.py. Neither cStringIOtkikuchi2007-03-242-16/+52
| | | | | | | | nor StringIO have 'encoding'. What we need was unicode string StringIO and python StringIO has it already. Also, use utf-8 for the table of contents in MIME digest if the subject has a different charset. Adding a test code for the multi-language digest.
* Fix of Decorate.py for the case (input_charset != output_charset) and failtkikuchi2007-03-232-5/+23
| | | | | | to encode by output_charset (language == 'ja' and cset == 'euc-jp'). Add a test code for list language is 'ja' and message is 'iso-2022-jp'; ('Japanese' in header/footer, 'French' in message).
* Restore the use of passwords.Schemes enum for selection of password hashingbwarsaw2007-03-229-24/+91
| | | | | | | | | | | | | | | | | | | | | | | | | scheme. Fix mmsitepass and test cases accordingly. Details: - set_global_password(): Instead of taking a string for 'scheme' argument, take None and then coerce that into passwords.Schemes.ssha - Add a base PasswordError and a BadPasswordSchemeError error that derives from that. For consistency, multiply inherit MMBadPasswordError and MMPasswordsMustMatch from PasswordError. - Add a passwords.lookup_scheme() method which turns scheme_names into scheme enum constants. It returns None if the lookup fails. - passwords.py: change the internal representation of _SCHEMES_BY_TAG dictionary to map scheme names to scheme enum values. Change internal uses of this dictionary to then turn those enum values into hash classes, or whatever else we need. - make_secret(): Raise BadPasswordSchemeErrorif the given schema (which should be an enum value) is invalid. - TestBase.tearDown(): Clear out any <site> locks that might hang around after a test case runs.
* Added a test case for the code that Tokio fixed for enums in r8164.bwarsaw2007-03-212-2/+9
| | | | | | However, I also changed the semantics a bit to be closer to what I wanted, namely that iteration returns the EnumValue objects, not the string names of the attributes.
* A few style fixes based on commit reviews.bwarsaw2007-03-213-19/+12
| | | | | | | | | | | | | | | | | | | | Switchboard.py - Use listname.encode('utf-8') to produce the necessary 8-bit string, instead of str(listname). Also update the preceding comment. senddigests.py - Remove an unnecessary import. Decorate.py - Remove a commented out section of code. - Remove some redundant local variables - Reorganize the section that's trying to find a usable encoding for the payload of the modified message. I don't think it really hurts much to try duplicate charsets when lcset == mcset, or when either == utf-8. Just go ahead and try them and let them fail. This simplifies the code. Also, try to get just the minimum necessary code under the UnicodeError. I think it's enough to catch the payload.encode() call.
* Test suite repair. All tests are now passing again.bwarsaw2007-03-218-30/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | | - In i18n.py, change this method so that everything it returns will be guaranteed to be a unicode. Mailman 2.2 will be unicode-safe, meaning all strings internally will be unicodes. The translation service is one boundary point were strings come from the outside, so ensure that they are unicodes and convert if necessary. This may break some things, but it's better to fix those situations than to continue to return 8-bit strings from _(). - In Mailman/testing/base.py, craft a fake module called Mailman.MTA.stub and stick no-op functions on stub.create() and stub.remove(). We really don't need the MTA modules for testing purposes (yet at least), and if you're using the default configuration, you'll get tons of cruft on stdout when the Manual MTA tries to add and remove mailing lists. Set up the test configuration environment to use this stub MTA module. - In test_handlers.py, remove an extraneous str(). - Convert ToDigest.py, Hold.py and Acknowledge.py to __i18n_templates__. (I'm pretty darn close to just making everything use $-strings by default.) - In CookHeaders.py, there's no need to unicode()-ify the subject since that should already be a unicode when passed from _(). - In MailList.py, we can use the str.capitalize() method.
* - Fixed a bug in OldStyleMemberships.addNewMember that allowed adding an addressmsapiro2007-03-201-1/+8
| | | | | | | with upper case in the domain if the local part was all lower case. - Changed the semantics of OldStyleMemberships.changeMemberAddress os that in the case of a straightforward address change, i.e. nodelete = 0, delivery status and time are preserved if BYUSER or BYADMIN.
* Scheme is selected by TAG. Use string not object reference.tkikuchi2007-03-201-5/+5
|
* csetstr is not set if lang==None. Use charset as string.tkikuchi2007-03-101-5/+4
|
* Some fixes for rev 8162 for mmsitepass to work.tkikuchi2007-03-033-5/+8
| | | | | | | | | | emum.py: 'cls' is used instead of 'self' mmsitepass.py: SCHEMES -> Schemes passwords.py: make_secret is called with 'scheme' in string. decode() of challenge string fails if it is unicode.
* Fixes for i18n digest to work.tkikuchi2007-03-026-14/+32
| | | | | | | | | | | | | | | | | | Mailman/Queue/Switchboard.py: listname is returned in unicode. ( '\x80' + 'a' is OK, '\x80' + u'a' is NG) Mailman/Utils.py: Utils.oneline() is extended for returning unicode string. Mailman/Digester.py: next_post_number is not used anywhere. Mailman/database/listdata.py: Attributes added (esp. for non web u/i) Mailman/bin/senddigests.py: Initialization Mailman/Handlers/ToDigest.py: Internal string calculation is done in unicode. So, several fixes. StringIO is used because cStringIO doesn't have encoding attribute.
* Backport the enum package from the abortive Mailman 3 branch. This lets usbwarsaw2007-03-024-22/+457
| | | | | | | | | | | | | | | | | | use much nicer identifiers for constants than plain ints or strings. New code using enumerating constants should use subclasses of enum.Enum. Along those lines, the passwords.py module has been rewritten to use enum constants instead of strings. So now e.g. the default password scheme is Mailman.passwords.Schemes.ssha and there are Scheme.pbkdf2 and Scheme.sha (etc) schemes as well. Also, rework the passwords.py implementation to better support unicode passwords. This elaborates on Tokio's r8160 by recognizing that the hash algorithms always operate on byte-strings not on unicodes. Thus if the secret or response are unicodes, encode them to byte-strings via utf-8 before hashing and comparing. Unit tests added for both enums and passwords.
* Restoreing i18n in 2.2tkikuchi2007-03-0110-50/+64
| | | | | | | | - As the default type of string in mailman-2.2 was set to 'unicode', i18n codes became need to be fixed. - Fixed: admin web interface. Other web interfaces needs more verification. - Fixed: non-digest delivery. Stil to go: digest and archive.
* Hashed password should be ascii string not unicode.tkikuchi2007-02-221-1/+1
|
* Where should etc directory belong, $prefix or $var-prefix?tkikuchi2007-02-032-4/+12
| | | | | | | | | | I put it in $var-prefix. mmsitepass.py ... parseargs() fails because config is not loaded before initialize(config). passwords.make_secret() should be called.
* - Changed to show hidden members when authorization is by list admin or ↵msapiro2007-01-211-14/+36
| | | | | moderator password. - Changed public roster syntax to accept optional password.
* Changed header_filter_rules processing to ignore blank patterns.msapiro2007-01-201-0/+3
|
* Update copyright years.bwarsaw2007-01-19163-309/+269
|
* Rework MailList.available_languages so that we don't need to use a PickleTypebwarsaw2007-01-1822-145/+245
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | column in the database for this list of strings. We use SQLAlchemy's many-to-many relationship, however because of this, you cannot simply append new unicodes to .available_languages. You need to wrap the language code in a Language instance and append that instance to the list. In order to handle this, I added a property MailList.language_codes which returns a list of the code strings (not Language instances). Also new are MailList.set_languages() for setting (i.e. overriding) the set of available languages for the list; and add_language() which takes a single language code, wraps it, and appends it. The code does not and should not use .available_languages directory any more. MailList.GetAvailableLanguages() is removed. The 'available_languages' column is removed from the Listdata table. Add a getValue() to Mailman.Gui.Language in order to unwrap the language codes stored in the database's association table. Modify _setValue() to do the wrapping. In dbcontext.py, don't import * from the sqlalchemy package. It contains a 'logging' name which is not the standard Python logging package. I also added essentially a bag of attributes class called Tables which will hold references to all the SA tables that are created. Update the make_table() API to take an instance of Tables. Added a close() method to DBContext. This is needed for the updated unit test suite. Changed bin/import.py so that when available_languages is being set, it calls MailList.set_languages() instead of trying to set that attribute directly. Updated some language idioms while I was at it. More eradication of mm_cfg in favor of the config object and the Defaults module. In testall.py, call initialize() instead of loginit.initialize(). Promote MAX_RESTARTS into a Defaults.py.in variable. This is because the unit tests will knock that value down to something not so annoying should one of the qrunner-required tests traceback. Several other important changes to the unit test suite (which now completely succeeds again!): - Set the uid and gid of the temporary mailman.cfg and tmp*.db files to the Mailman user and group as specified in the config object. - Make sure that all of the tests point to a SQLite database file that was created with the tempfile module. This way we don't pollute our main database with data that is getting created during the unit tests. - In the TestBase.setUp() method, be sure to close the existing dbcontext, clear out the mappers, and then reconnect the dbcontext with the new SQLALCHEMY_ENGINE_URL pointing to the tempfile. However, we don't need to reload the MailList instance any more. - Make all tests work, except for the tests that require crypt. That upgrade path will not be available in this version of Mailman.
* Passwords done right.bwarsaw2007-01-1416-147/+318
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | First off, there are several password hashing schemes added including SHA, salted-SHA, and RFC 2989 PBKDF2 (contributed by Bob Fleck). Then we encode the password using RFC 2307 style syntax. At least I think: specifically things like the PRF and iteration count for PBKDF2 are encoded the way I /think/ is intended for RFC 2307 but I could be wrong. Seems darn hard to find definitive information about that. In any event, even though CLEARTEXT passwords are supported, they are mostly deprecated, even for user passwords. It also allows us to easily update all passwords to a new hashing scheme when the existing schemes get cracked. The default scheme (specified in Defaults.py.in) is salted-SHA with a 20 byte salt (the salt length and PBKDF2 iteration counts can only be specified in the passwords.py file). These hashed passwords are used for user passwords, list owner and moderator passwords, and site and list creator passwords. Of course this means that user password reminders are impossible now. They've been ripped out of the code for a while, but now we'll need to implement password resets since user passwords cannot be recovered. bin/export has had several changes: - export no longer converts to dollar strings. Were assuming dollar strings are used by default for all new lists and any imported lists will already be converted to dollar strings. - Likewise, rip out the password scheme stuff, since cleartext passwords can never be exported, so we might as well always include the member's hashed password. - Fix exporting to stdout when that stream can only handle ascii by wrapping stdout in a utf-8 codec writer. Other changes: - add a missing import to HTTPRunner.py - Convert GUIBase.py to use Defaults.* for constants instead of mm_cfg.* - Remove pre-Python 2.4 compatibility from Utils.py. We've already said Python 2.4 will be a minimum requirement. - Change the permissions on the global password file. The default 007 umask is used and should be good enough. - bin/newlist adds the ability to specify the password scheme (or list the available schemes) for the list owner password. It is not possible to set the scheme on a per-list basis. bin/mmsitepass does the same, but for the site and list creator passwords. - Fix a nasty problem with bin/import. The comment in the code says it best: # XXX Here's what sucks. Some properties need to have # _setValue() called on the gui component, because those # methods do some pre-processing on the values before they're # applied to the MailList instance. But we don't have a good # way to find a category and sub-category that a particular # property belongs to. Plus this will probably change. So # for now, we'll just hard code the extra post-processing # here. The good news is that not all _setValue() munging # needs to be done -- for example, we've already converted # everything to dollar strings. - Set the 'debug' logger to logging.DEBUG level. It doesn't seem to make much sense for the debugging log to ignore debug messages.
* Change the semantics of the -f and -b flags. Now -f prints the fullybwarsaw2007-01-081-14/+8
| | | | | qualified list name, and it is no longer incompatible with -b (which just turns off the description).
* Fixed documentation for POSTFIX_STYLE_VIRTUAL_DOMAINS to clarify that themsapiro2007-01-071-7/+8
| | | empty list is the appropriate value if Postfix virtual domains aren't used.
* Add post_id in list data schema.tkikuchi2007-01-061-0/+1
|
* In HTTPRunner, when we see a KeyboardInterrupt during the serve_forever(),bwarsaw2007-01-055-53/+65
| | | | | | | | | | | | | | | | | don't re-raise the exception since that will show up in the log files. Instead just exit with a code equal to SIGTERM. Rework the way qrunners are specified in the mailman.cfg file. Always start the default number of the default set of qrunners, but allow mailman.cfg to delete some with the del_qrunner() function. Rename add_runner() to add_qrunner() and make this actually work <wink>. Both take the shortened qrunner name as the first argument (e.g. 'Arch' instead of 'ArchRunner'). Automatically start the MaildirRunner if USE_MAILDIR = Yes; same goes for LMTPRunner and USE_LMTP = Yes. In both cases, you do not need to also use add_qrunner() in your mailman.cfg file to enable them. You still do need to put "add_qrunner('HTTP')" in your mailman.cfg if you want to enable the wsgi server. This may end up being added to the default set.
* Clean up file permissions and umask settings. Now we set the umask to 007bwarsaw2007-01-0521-203/+138
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | during early initialization so that we're guaranteed to get the right value regardless of the shell umask used to invoke the command line script. While we're at it, we can remove almost all individual umask settings previously in the code, and make file permissions consistently -rw-rw---- (IOW, files are no longer other readable). The only subsystem that wasn't changed was the archiver, because it uses its own umask settings to ensure that private archives have the proper permissions. Eventually we'll mess with this, but if it ain't broken... Note that check_perms complains about directory permissions, but I think check_perms can be fixed (or perhaps, even removed?!). If we decide to use LMTPRunner and HTTPRunner exclusively then no outside process will be touching our files potentially with the incorrect permissions, umask, owner, or group. If we control all of our own touch points then I think we can lock out 'other'. Another open question is whether Utils.set_global_password() can have its umask setting removed. It locks permissions down so even the group can't write to the site password file, but the default umask of 007 might be good enough even for this file. Utils.makedirs() now takes an optional mode argument, which defaults to 02775 for backward compatibility. First, the default mode can probably be changed to 02770 (see above). Second, all code that was tweaking the umask in order to do a platform compatible os.mkdir() has now been refactored to use Utils.makedirs(). Another tricky thing was getting SQLite via SQLAlchemy to create its data/mailman.db file with the proper permissions. From the comment in dbcontext.py: # XXX By design of SQLite, database file creation does not honor # umask. See their ticket #1193: # http://www.sqlite.org/cvstrac/tktview?tn=1193,31 More details in that file, but the work around is to essentially 'touch' the database file if 'sqlite' is the scheme of the SQLAlchemy URL. This little pre-touch sets the right umask honoring permission and won't hurt if the file already exists. SQLite will happily keep the existing permissions, and in fact that ticket referenced above recommends doing things this way. In the Mailman.database.initialize(), create a global lock that prevents more than one process from entering this init function at the same time. It's probably not strictly necessary given that I believe all the operations in dbcontext.connect() are multi-processing safe, but it also doesn't seem to hurt and prevents race conditions regardless of the database's own safeguards (or lack thereof). Make sure nightly_gzip.py calls initialize().
* Ported from 2.1 branch:bwarsaw2007-01-023-17/+31
| | | | | | | | | | | | | Ensure that exported XML is written in utf-8, at least if we're writing to a file other than stdout. Fix a typo in getting the digest style. Update copyright years. In HTTPRunner.py, catch KeyboardInterrupt. In Python 2.5 this has been moved in the exception hierarchy so that it's no longer caught by "except Exception". import.py: Import user topic selections. Fix a typo in an error message. Catch BadDomainSpecificationErrors that can be raised in MailList.Create().
* Import list topics and header filters. Still to do: user topic selections.bwarsaw2006-12-311-5/+7
|
* A rudimentary import script. Several things are not yet imported, includingbwarsaw2006-12-304-18/+360
| | | | | | | | | | | | | | | | | | | header_filters and topics (both list topics and user topic selections). Everything else seems to work pretty well. dbcontext.py: Don't key the mlist transactions off of mlist.fqdn_listname because this can change. For example, if you "bin/withlist -l mylist" and then "m.host_name = 'new.example.com'" the fqdn_listname property will change and the commit machinery won't be able to find the correct transaction. Instead, store the fqdn_listname as it's seen during the api_lock() call back on the mailing list under the _txnkey attribute. Use that attribute in api_save() and api_unlock(). Upgrade to SQLAlchemy 0.3.3 Port from MM2.1 the support for multiple password schemes. Change the MailList's repr to use the fqdn_listname.
* Merged revisions 8113-8121 via svnmerge from bwarsaw2006-12-2949-1689/+1326
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch ................ r8114 | bwarsaw | 2006-12-06 00:16:54 -0500 (Wed, 06 Dec 2006) | 44 lines Initial take on using SQLAlchemy to store list data in lieu of Python pickles. While all the list data (including OldStyleMemberships attributes) are stored in the database, many attributes are stored as PickleTypes binary data. This isn't idea but it gets things working until a more sophisticated schema can be developed. MailList class is now a new-style class, as is required by SQLAlchemy. This makes several things, er, interesting. Rip out all the low-level pickle reading and writing stuff. Hook SA transaction events into Lock() and Unlock(). Move the hooking of the _memberadaptor into InitTempVars(), which gets called by the SQLAlchemy hooks (MailList.__init__() never is). Add an initialize.py module which centralizes all the initialization bits that command line scripts have to do, including configuration, logging, and atabase initialization. This change also converts bin/withlist to mmshell wrapper. Update to SQLAlchemy 0.3.1. Revamp paths.py.in considerably. There were several problems with the old way. We no longer disable default loading of site-packages so we don't need to add Python's site-packages back to sys.path. Also, because site.addsitedir() causes things like .pth paths to be /appended/ to sys.path, they actually won't override any site-installed packages. E.g. if SQLAlchemy is installed in the system Python, our version will not override. IIUC, setuptools-based packages can be configured to work properly in the face of package versions, however not all packages we currently depend on are setuptools-based. So instead, we steal a bit of stuff from site.py but change things so the prepend .pth stuff to sys.path. Update several modules to use True/False and whitespace normalization. Convert from mm_cfg to config object. Modernize a few coding constructs. Add a couple of exceptions to handle database problems. In the export script, include the widget type in the elements. This helped in my stupid little throw away conversion script, but I think it will be more generally useful. Add an interact.py module which refactors interactive interpreter access. Mostly this is used by withlist -i, but it lets us import Mailman.interact and drop into a prompt just about anywhere (e.g. debugging). ................ r8115 | bwarsaw | 2006-12-07 09:13:56 -0500 (Thu, 07 Dec 2006) | 22 lines Start to flesh out more of the SQLAlchemy mechanisms. Added a MailList.__new__() which hooks instantiation to use a query on dbcontext to get an existing mailing list. A 'no-args' call means we're doing a Create(), though eventually that will change too. For now, disable the CheckVersion() call. Eventually this will be folded into schema migration. list_exists(): Rewrite to use the dbcontext query to determine if the named mailing list exists or not. Requires the fqdn_listname. Eradicate two failed member adaptors: BDBMemberAdaptor and SAMemberships. Change the way the DBContext holds onto tables. It now keeps a dictionary mapping the table's name to the SA Table instance. This makes it easier to look up and use the individual tables. Add 'web_page_url' as an attribute managed by SA, and remove a debugging print. ................ r8116 | bwarsaw | 2006-12-11 07:27:47 -0500 (Mon, 11 Dec 2006) | 29 lines Rework the whole dbcontext and transaction framework. SA already handles nested transactions so we don't have to worry about them. However, we do have the weird situation where some transactions are tied to MailList .Lock()/.Unlock()/.Save() and some are tied to non-mlist actions. So now we use an @txn decorator to put methods in a session transaction, but then we also hook into the above MailList methods as possibly sub-transactions. We use a weakref subclass to manage the MailList interface, with a dictionary mapping MailList fqdn_listnames against transactions. The weakrefs come in by giving us a callback when a MailList gets derefed such that we're guaranteed to rollback any outstanding transaction. Also, we have one global DBContext instance but rather than force the rest of Mailman to deal with context objects, instead we expose API methods on that object into the Mailman.database module, which the rest of the code will use. Such methods must be prepended with 'api_' to get exposed this way. bin/rmlist now works with the SA-backend. I refactored the code here so that other code (namely, the test suite) can more easily and consistently remove a mailing list. This isn't the best place for it ultimately, but it's good enough for now. New convenience functions Utils.split_listname(), .fqdn_listname(). Convert testall to use Mailman.initialize.initialize(). Not all tests work, but I'm down to only 8 failures and 7 errors. Also, do a better job of recovering from failures in setUp(). MailList.__new__() now takes keyword arguments. ................ r8117 | bwarsaw | 2006-12-11 22:58:06 -0500 (Mon, 11 Dec 2006) | 7 lines Unit test repairs; even though the unit tests are still pretty fragile, everything now passes with the SQLAlchemy storage of list data. Added missing 'personalize' column. Converted mailmanctl and qrunner to initialize() interface. Fixed _cookie_path() to not fail if SCRIPT_NAME is not in the environment. ................ r8118 | bwarsaw | 2006-12-27 18:45:41 -0500 (Wed, 27 Dec 2006) | 21 lines Utils.list_names(): Use a database query to get all the list names. dbcontext.py: Added api_get_list_names() to support Utils.list_names(). listdata.py: Added two additional MailList attributes which need to be stored in the database. The first is 'admin_member_chunksize' which isn't modifiable from the web. The second is 'password' which holds the list's password. HTMLFormatObject: item strings can now be unicodes. bin/list_lists.py: Must call initialize() to get the database properly initialized, not just config.load(). This will be a common theme. SecurityManager.py: - Remove md5 and crypt support - Added mailman.debug logger, though it will be only used during debugging. - The 'secret' can be a unicode now. - A few coding style updates; repr() instead of backticks, 'key in dict' instead of 'dict.has_key(key)' ................ r8119 | bwarsaw | 2006-12-27 19:13:09 -0500 (Wed, 27 Dec 2006) | 2 lines genaliases.py: config.load() -> initialize() ................ r8120 | bwarsaw | 2006-12-27 19:17:26 -0500 (Wed, 27 Dec 2006) | 9 lines Blocked revisions 8113 via svnmerge ........ r8113 | bwarsaw | 2006-12-05 23:54:30 -0500 (Tue, 05 Dec 2006) | 3 lines Initialized merge tracking via "svnmerge" with revisions "1-8112" from https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch ........ ................ r8121 | bwarsaw | 2006-12-28 23:34:52 -0500 (Thu, 28 Dec 2006) | 20 lines Remove SIGTERM handling from all the CGI scripts. This messes with HTTPRunner because when you issue "mailmanctl stop" after the signal handler has been installed, the process will get a SIGTERM, the signal handler will run, and the process will exit with a normal zero code. This will cause mailmanctl to try to restart the HTTPRunner. I don't think we need that stuff at all when running under wsgi with a SQLAlchemy backend. If mailmanctl kills the HTTPRunner in the middle of the process, I believe (but have not tested) that the transaction should get properly rolled back at process exit. We need to make sure about this, and also we need to test the signal handling functionality under traditional CGI environment (if we even still want to support that). Also, make sure that we don't try to initialize the loggers twice in qrunner. This was the cause of all the double entries in logs/qrunner. Fix a coding style nit in mailmanctl.py. De-DOS-ify line endings in loginit.py. ................
generated by cgit v1.3.1 (git 2.54.0) at 2026-08-11 19:57:49 +0000