summaryrefslogtreecommitdiff
path: root/Mailman/testing
Commit message (Collapse)AuthorAgeFilesLines
* More work on completing the transition to setuptools.Barry Warsaw2007-07-1898-11846/+0
| | | | | | | | | | | | | | | | | * Mailman/testing -> Mailman/test * Removed Mailman/testing/base.py * Fix mailmanctl by using a different way of calculating where the qrunner script is. The configuration file no longer knows what BIN_DIR is, but the mailmanctl script knows where it lives via sys.argv[0]. Also, PREFIX_DIR -> VAR_DIR. Also, * Since the overwhelmingly predominant use of ILanguageManager is to get the description, and since .get_language_data(code)[0] is not very readable, split the interface into .get_description() and .get_charset(). * In the setup, automatically add all Mailman.bin modules as command line scripts.
* Major surgery to get the setuptools based installation passing all theBarry Warsaw2007-07-163-13/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | existing unit tests. Here's a summary of the changes. - Removed all dependent third party packages, since the setup.py file now claims all package dependencies such that they can be automatically installed from the cheeseshop. - Moved the misc directory into the Mailman package as Mailman/data. Moved templates and messages to Mailman subpackages. - Added an ILanguageManager interface, plus an implementation, so that we don't use Defaults.LC_DESCRIPTIONS directly anymore. Added a doctest for this interface and implementation. Defaults.LANGUAGES is moved into mailman.cfg. Defaults.LANGUAGE_DICT is moved to _DEFAULT_LANGUAGE_DATA, and LC_DESCRIPTIONS is removed. The calculation of the available and enabled languages is moved to the Configuration class, but this will probably still need work. Utils.GetLanguageDescr() and Utils.IsLanguage() are removed. I'd like to remove GetCharSet() eventually too, but there are too many uses of this currently, so I'm deferring it. - Utils.findtext(): Hacks added so that templates can be retrieved from the language catalog. The hack is that the template contents are used to find the translation, but in the one test case where this is actually flexed, the trailing newline in the file contents has to be trimmed. This is probably not right. - No more Defaults.py.in or mm_cfg.py! Defaults.py.in is moved to Defaults.py and is no longer created from a template file. The script called make_instance is added which creates an etc/mailman.cfg file from mailman.cfg.in (previously, mailman.cfg.sample) and /that/ file now has the small number of calculated values. In general, make_instance will not touch mailman.cfg if it exists, unless the --force option is given. CGIEXT is made the empty string by default (i.e. not generated). make_instance grows a --var-dir option. Fleshed out the --languages opton. - Defaults.py grows a DEFAULT_VAR_DIRECTORY variable, which is the default location of the 'var' directory. The Configuration class uses this as one of the directories it searches for its landmark, i.e. etc/mailman.cfg. RUNTIME_DIR is gone, as is VAR_PREFIX. - testall needs to write MAILMAN_USER, MAILMAN_UID, MAILMAN_GROUP, MAILMAN_GID, and LANGUAGES run time variables. - bin/withlist no longer needs to add config.BIN_DIR to sys.path, because in fact that variable doesn't exist any more. - Tweak the French catalog to make a test work. This is needed because of the conversion from %-strings to $-strings. - The setup.py now generates the .mo files before it does its thing. This will have to be fixed, but for now we must generate these files on setup build time instead of installation time. - Removed an unused interface.
* The start of a setuptools conversion. All the Makefile.in and autoconfBarry Warsaw2007-07-132-148/+0
| | | | artifacts are removed, as is the C files which we will no longer need.
* Last bit of existing test suite cleanup for now. I've removed the inmemory.pyBarry Warsaw2007-07-123-400/+4
| | | | | | | module since we're not using that any more. I've also disabled the remaining failing tests in test_handlers and test_security_mgr. Because of the unified user database, both of these modules will change significantly, but I don't want to remove them just yet.
* Convert the Scrubber test to a doctest, and fix Scrubber.py, but otherwiseBarry Warsaw2007-07-121-106/+0
| | | | | | | | don't modernize the Scrubber handler. The is the last of the handler test conversions until we figure out what to do with the Approve handler. In a unified user database the semantics of this are unclear.
* Convert ToArchive tests to doctests and do a minimal amount of handler moduleBarry Warsaw2007-07-111-57/+0
| | | | cleanup (really, not much was necessary).
* Conversion of the ToDigest tests to doctests, along with just enough handlerBarry Warsaw2007-07-102-120/+8
| | | | | | | | | module update to make the thing pass. Much more coverage is still necessary, but this at least recreates the existing tests. Changes to the test infrastructure to make REPORT_ONLY_FIRST_FAILURE conditional on lower (default) verbosity. Increase the verbosity via 'bin/testall -v' and you get all the failure output.
* Convert ToOutgoing handler to a doctest. Minor cleanup of the handler code toBarry Warsaw2007-07-081-42/+0
| | | | a more modern Python style.
* Convert the ToUsenet handlerto a doctest. No other handler clean up isBarry Warsaw2007-07-061-50/+0
| | | | included.
* Convert MimeDel tests to doctest. No other changes to the handler module areBarry Warsaw2007-07-051-176/+0
| | | | made.
* Convert the SpamDetect handler tests (what there was of them anyway) to aBarry Warsaw2007-07-051-38/+0
| | | | doctest, but don't otherwise clean up the handler module.
* Convert the TestTagger to a doctest. No other cleaning up of the handlerBarry Warsaw2007-07-041-117/+0
| | | | module is done.
* Convert TestHold to a doctest and update code as necessary. No generalBarry Warsaw2007-07-031-160/+0
| | | | modernization of Hold.py was performed.
* Convert TestFileRecips to a doctest, and update the handler to more modernBarry Warsaw2007-07-021-85/+0
| | | | | | | | Python idioms. The recipients are now returned as a set instead of a list, so duplicates are quashed. In MailList.InitTempVars() we need to create the list's data directory when the list is initialized. If the directory already exists, this does nothing.
* Convert failing test_message.py to doctests bounces.txt and message.txt, whichBarry Warsaw2007-07-011-98/+0
| | | | | | | | | | | | | of course now succeed. Rename Bouncer.py's BounceMessage() method to bounce_message() and remove the 'msgdata' parameter, which wasn't being used. Change the RejectNotice exception class to expose .notice directly, as there's no reason for this to be an accessor or property. Move the coverage.py installation to the install-packages target instead of the install-other target, so that it only gets installed once the pythonlib directory is created.
* Support for case-preserving addresses. When an Address is given an emailBarry Warsaw2007-07-011-1/+2
| | | | | | | | | | | | | | | | | | address that is not lower cased, the original, case-preserved version is store on the '_original' attribute. The lower-cased version is always used as the key and thus always stored on the 'address' attribute. The IAddress interface is given a new 'original_address' property which returns the case-preserved version. Address's __str__() and __repr__() are similarly modified. The former always includes the case-preserved address; the latter does too, but now also includes the lower-cased 'key' email address (along with the object's id). Searching for an address always does so on the lower-cased version. Test suite is updated as necessary. Also, I'm adding the REPORT_ONLY_FIRST_FAILURE doctest flag so that it's easier to debug doctest failures without having pages of problems to scroll through.
* Repair docfiles broken with the new ISwitchboard interface. Mostly these wereBarry Warsaw2007-06-271-0/+5
| | | | | | | | | | due to the change from the method Switchboard.files() to the generator property Switchboard.files. Add support in cleaning_teardown() for the doctests for removing all queue files leftover after a test is complete. This way, failures in one test that queues files won't break all subsequent such tests. Update the 2.1.9 section of the NEWS.txt file from the Mailman 2.1 branch.
* Simplify doctests by having just a single test_documentation.py module inBarry Warsaw2007-06-2316-393/+69
| | | | | Mailman/testing. This introspects the Mailman/docs directory and adds DocFileSuites for all .txt files found there.
* Refactor tests to get consistent clean up behavior. The clean up sectionsBarry Warsaw2007-06-2215-81/+62
| | | | | have been removed from the .txt files and instead added to the test harness, which all doctests now use.
* Convert the AvoidDuplicates handler tests to a doctest. Well,Barry Warsaw2007-06-211-0/+33
| | | | | | | actually there /was/ no separate AvoidDuplicates test, but now there is, even though it may not be full coverage. Update the AvoidDuplicates handler to use more modern Python idioms.
* Convert the CookHeaders tests in test_handlers to using doctests, split upBarry Warsaw2007-06-212-348/+35
| | | | | | | | | | | | | | | | | into several sub-documents. Defaults.py.in: Removed OLD_STYLE_PREFIXING. So-called 'new style' prefixing is the default and only option now. CookHeaders.py is updated to the new API and some (but not all) of the code has been updated to more modern Python idioms. reply_goes_to_list attribute has been changed from a strict integer to a munepy enum called ReplyToMunging. RFC 2369 headers List-Subscribe and List-Unsubscribe now use the preferred -join and -leave addresses instead of the -request address with a subject value.
* Convert the Cleanse handler tests to doctest style.Barry Warsaw2007-06-192-65/+32
|
* Convert the tests for the CalcRecips handler to doc tests. There areBarry Warsaw2007-06-193-99/+33
| | | | | | | | | | | | | | | | | some XXX's in the doc test because digest recipients aren't tested (though those may go in a different doctest), and neither are urgent messages. This latter is for the same reason that the Approved handler is not yet tested; which password do you use in that header? The CalcRecips tests would also seem the natural place to test the receive_list_copy preference, but that actually gets processed in the AvoidDuplicates handler, so it isn't tested here. Add delivery_status (of type enum DeliveryStatus) to preferences. I'm not entirely sure that's the right place for it, but it lets me finish converting the test for now. Expose the rest of the preferences through the IMember interface.
* Convert the AfterDelivery handler test to a doctest. Also, change theBarry Warsaw2007-06-182-14/+32
| | | | | MailingList.last_post_time column to a DateTime (i.e. Python datetime object).
* Remove both the model and interface for RosterSets. These are no longer usedBarry Warsaw2007-06-182-129/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | or necessary in the current data model. Convert the test_handlers.py Python test to an acknowledge.txt doctest, and make the Acknowledge.py handler work with the new data model. There are a few XXX comments left in here due to the fact that the web stuff is a total hack in the current branch currently. Added IMailingListWeb methods and properties to the MailingList model class: web_host and script_url(). Work out how IMembers will expose the lookup-order based preferences. By getting the attribute IMember.preferences you can see exactly the preferences overridden by this member. To use the lookup order, use IMember.delivery_mode, IMember.acknowledge_posts, etc. IOW, the IMember interface now provides the properties directly and access through this mechanism supports lookup order with definitive preference values. Also added IMember.unsubscribe() which does the obvious, and IMember.options_url() which is a total hack for providing a url (but not the ultimately right one) for the user's option page. Refactor the model's roster classes. Also added IRoster.get_member() method with efficient queries to return the right results. Make AdministratorRoster.members more efficient due to a better query. Update the membership.txt doctest to eliminate a chance ordering effect, and also to test finding members with .get_member(). The clean up section uses the new .unsubscribe() method.
* Update the IUser interface and tests, specifically as it relates toBarry Warsaw2007-06-151-1/+3
| | | | | | | | | | | | | | | | | | | | preferences. IAddresses, IUsers, and IMembers all get preferences by default, althoughthe attributes of these preferences are None by default. IMailingLists don't get preferences by default though; because these live in the user database, we can't cross-polinate them in the mailing lists. We'll figure something out later for these. IUser.register(): Add this method which registers and links an address to the user. Allow EnumType database columns to accept and return Nones. This is useful for when the columns are not defined NOT NULL. Update doctests. Removed teh hide_address preference. I can't think of a reason not to want to hide addresses for everyone.
* More test updates based on simplified data model. Rosters are gone so theBarry Warsaw2007-06-101-30/+0
| | | | | | | | | | | | | | | | | | | | mlist-rosters.txt tests are removed. Give MailingList entities a Preferences foreign key, so that they now have preferences. Member.preferences(): Watch out for unlinked addresses, i.e. which have no user and thus no user preferences. Address.controls() takes a text email address, not an IAddress. UserManager: add create_address(), delete_address(), get_address(), and 'addresses' property. Updated the address.txt and user.txt doctests as needed. Added User.register() to register an email address and immediately link it to a user.
* Implement the new, simplified membership model. Rosters and RosterSets asBarry Warsaw2007-06-091-2/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | they were previously known are now gone. Rosters, rather than being a database entity that collects users, is now just a filter on the member database. This way, we can use generic rosters to search for regular members, digest members, owners, or moderators. More advanced rosters can do all kinds of other membership queries. But rosters no longer need to be a database entity. Users have a name, password, optional preferences, and a set of addresses, but users are not subscribed to mailing lists. Addresses have the email address, some verification information, and optional preferences. Members tie an address to a mailing list, through a role, with optional preferences. Other changes here include: MailList.fqdn_listname() moved to the MailingList model entity. Added MemberRole enum and SystemDefaultPreferences to Mailman.constants. Profiles are renamed to Preferences (same with the interface), but the files are not yet moved. This happens later. We mostly don't need has_*() relationships on the entity classes, because we generally don't need the reverse relationship. Use belongs_to() because that creates the foreign key, even though the wording seems counter intuitive. IAddress.subscribe() added. Tell Elixir to use shortnames for all tables. Remove the OldStyleMembership fields from MailingList. Remove all the interface elements and database fields that talk about rosters and rostersets. Convert Version entity to has_field().
* rename a few testsbwarsaw2007-06-022-2/+2
|
* Add doctest for Replybot handler. The test in test_handlers.py didn't reallybwarsaw2007-05-312-7/+32
| | | | | | | | | do anything. The doctest needs to have some way of testing the grace period, but it's still more tests of the module than there every was before. Update the Replyboty handler to use $-strings internally. Eliminate the use of %-strings in auto-response textsy. Only $-strings can be used, which allows us to get rid of another use of SafeDict.
* Go ahead and remove the Mailman/database/tables directory since all the Elixirbwarsaw2007-05-302-223/+32
| | | | | | | | | | | | | | | | | | classes live in Mailman/databae/model now. Remove the TestDecorate test class from test_handlers.py and move them into a doctest called decorate.txt (with harness in test_decorate.py). Remove the dependence on SafeDict from the Decorate handler because I can now use string.Template object to safely fill in header and footer templates. Eventually I want to completely remove SafeDict from Mailman, but it's still used in a few other places. This also means that only $-strings will be supported in headers and footers, and the import script will have to convert %-strings to $-strings. Also, '_internal_name' is no longer a supported header/footer substitution variable. Use $real_name or $list_name now. Added $fqdn_listname as a substitution variable. Update the DEFAULT_MSG_FOOTER accordingly.
* Minor style issue.bwarsaw2007-05-291-1/+1
|
* Merge exp-elixir-branch to trunk. There is enough working to make me feelbwarsaw2007-05-289-218/+174
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | confident the Elixir branch is ready to become mainline. Also, fewer branches makes for an easier migration to a dvcs. Don't expect much of the old test suite to work, or even for much of the old functionality to work. The changes here are disruptive enough to break higher level parts of Mailman. But that's okay because I am slowly building up a new and improved test suite, which will lead to a functional system again. For now, only the doctests in Mailman/docs (and their related test harnesses) will pass, but they all do pass. Note that Mailman/docs serve as system documentation first and unit tests second. You should be able to read the doctest files to understand the underlying data model. Other changes included in this merge: - Added the Mailman.ext extension package. - zope.interfaces uses to describe major components - SQLAlchemy/Elixir used as the database model - Top level doinstall target renamed to justinstall - 3rd-party packages are now installed in pythonlib/lib/python to be more compliant with distutils standards. This allows us to use just --home instead of all the --install-* options. - No longer need to include the email package or pysqlite, as Python 2.5 is required (and comes with both packages). - munepy package is included, for Python enums - IRosterSets are added as a way to manage a collection of IRosters. Roster sets are named so that we can maintain the indirection between mailing lists and rosters, where the two are maintained in different storages. - IMailingListRosters: remove_*_roster() -> delete_*_roster() - Remove IMember interface. - Utils.list_names() -> config.list_manager.names - fqdn_listname() takes an optional hostname argument. - Added a bunch of new exceptions used throughout the new interfaces. - Make LockFile a context manager for use with the 'with' statement.
* Another round of merges of my Pycon branch...bwarsaw2007-04-176-58/+508
| | | | | | | | | | | | | | | | | | | | bin/testall.py - Improvements to setting up the tests with temporary files for the db and configuration. Don't accept -C/--config because that will just confuse things. - Copy our new testing.cfg.in template file to a temp file, populate it with a few additional run-time calculate values, and use that in both the parent (bin/testall script) and children (bin/mailmanctl and friends). Mailman/initialize.py - Split up initialize() into two functions as required by the test suite reorg above. Almost everything else in Mailman will continue to use the initialize() function though. Mailman/configuration.py - Store the filename that was used to load the .cfg file from on the configuration object.
* api_lock(): When locking the MailList object, tell the SQLAlchemy session tobwarsaw2007-03-301-1/+0
| | | | | | | | | | | expire the object. This way, when the MailList attributes are next accessed, the ORM will reload them from the database, getting any new values possibly set in other processes. This works better than trying to use always_refresh=True on the mapper, or trying to do a reload() because both of those approaches blow away locks. I'm not sure why this, but I suspect that it's because the identity map is handing us back a different object, rather than invalidating the object's attributes.
* passwords.py: Looks like we still need unicode checking.tkikuchi2007-03-251-1/+135
| | | | | | | Mark Sapiro's patch for 'format' parameter. (Decorate.py, Scrubber.py) Scrubber.py: More brush up of code ... 'Content-Transfer-Encoding' is not updated by msg.set_payload(). 'Url:' to 'URL:' normalization. test_handlers.py: Test codes for Decorate.py and Scrubber.py.
* CookHeaders.py:tkikuchi2007-03-241-10/+74
| | | | | | | | | Subject munging code made unified for both i18n and ascii encodings. test_handlers.py Adding test code for i18n and numbering subject munging. (also tab normalization for the last commit)
* Brush up the fall-back-to-utf8 mechanizm in ToDigest.py. Neither cStringIOtkikuchi2007-03-241-0/+35
| | | | | | | | nor StringIO have 'encoding'. What we need was unicode string StringIO and python StringIO has it already. Also, use utf-8 for the table of contents in MIME digest if the subject has a different charset. Adding a test code for the multi-language digest.
* Fix of Decorate.py for the case (input_charset != output_charset) and failtkikuchi2007-03-231-0/+17
| | | | | | to encode by output_charset (language == 'ja' and cset == 'euc-jp'). Add a test code for list language is 'ja' and message is 'iso-2022-jp'; ('Japanese' in header/footer, 'French' in message).
* Restore the use of passwords.Schemes enum for selection of password hashingbwarsaw2007-03-225-11/+49
| | | | | | | | | | | | | | | | | | | | | | | | | scheme. Fix mmsitepass and test cases accordingly. Details: - set_global_password(): Instead of taking a string for 'scheme' argument, take None and then coerce that into passwords.Schemes.ssha - Add a base PasswordError and a BadPasswordSchemeError error that derives from that. For consistency, multiply inherit MMBadPasswordError and MMPasswordsMustMatch from PasswordError. - Add a passwords.lookup_scheme() method which turns scheme_names into scheme enum constants. It returns None if the lookup fails. - passwords.py: change the internal representation of _SCHEMES_BY_TAG dictionary to map scheme names to scheme enum values. Change internal uses of this dictionary to then turn those enum values into hash classes, or whatever else we need. - make_secret(): Raise BadPasswordSchemeErrorif the given schema (which should be an enum value) is invalid. - TestBase.tearDown(): Clear out any <site> locks that might hang around after a test case runs.
* Added a test case for the code that Tokio fixed for enums in r8164.bwarsaw2007-03-211-1/+8
| | | | | | However, I also changed the semantics a bit to be closer to what I wanted, namely that iteration returns the EnumValue objects, not the string names of the attributes.
* Test suite repair. All tests are now passing again.bwarsaw2007-03-212-1/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | - In i18n.py, change this method so that everything it returns will be guaranteed to be a unicode. Mailman 2.2 will be unicode-safe, meaning all strings internally will be unicodes. The translation service is one boundary point were strings come from the outside, so ensure that they are unicodes and convert if necessary. This may break some things, but it's better to fix those situations than to continue to return 8-bit strings from _(). - In Mailman/testing/base.py, craft a fake module called Mailman.MTA.stub and stick no-op functions on stub.create() and stub.remove(). We really don't need the MTA modules for testing purposes (yet at least), and if you're using the default configuration, you'll get tons of cruft on stdout when the Manual MTA tries to add and remove mailing lists. Set up the test configuration environment to use this stub MTA module. - In test_handlers.py, remove an extraneous str(). - Convert ToDigest.py, Hold.py and Acknowledge.py to __i18n_templates__. (I'm pretty darn close to just making everything use $-strings by default.) - In CookHeaders.py, there's no need to unicode()-ify the subject since that should already be a unicode when passed from _(). - In MailList.py, we can use the str.capitalize() method.
* Scheme is selected by TAG. Use string not object reference.tkikuchi2007-03-201-5/+5
|
* Backport the enum package from the abortive Mailman 3 branch. This lets usbwarsaw2007-03-022-0/+246
| | | | | | | | | | | | | | | | | | use much nicer identifiers for constants than plain ints or strings. New code using enumerating constants should use subclasses of enum.Enum. Along those lines, the passwords.py module has been rewritten to use enum constants instead of strings. So now e.g. the default password scheme is Mailman.passwords.Schemes.ssha and there are Scheme.pbkdf2 and Scheme.sha (etc) schemes as well. Also, rework the passwords.py implementation to better support unicode passwords. This elaborates on Tokio's r8160 by recognizing that the hash algorithms always operate on byte-strings not on unicodes. Thus if the secret or response are unicodes, encode them to byte-strings via utf-8 before hashing and comparing. Unit tests added for both enums and passwords.
* Update copyright years.bwarsaw2007-01-196-6/+6
|
* Rework MailList.available_languages so that we don't need to use a PickleTypebwarsaw2007-01-185-61/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | column in the database for this list of strings. We use SQLAlchemy's many-to-many relationship, however because of this, you cannot simply append new unicodes to .available_languages. You need to wrap the language code in a Language instance and append that instance to the list. In order to handle this, I added a property MailList.language_codes which returns a list of the code strings (not Language instances). Also new are MailList.set_languages() for setting (i.e. overriding) the set of available languages for the list; and add_language() which takes a single language code, wraps it, and appends it. The code does not and should not use .available_languages directory any more. MailList.GetAvailableLanguages() is removed. The 'available_languages' column is removed from the Listdata table. Add a getValue() to Mailman.Gui.Language in order to unwrap the language codes stored in the database's association table. Modify _setValue() to do the wrapping. In dbcontext.py, don't import * from the sqlalchemy package. It contains a 'logging' name which is not the standard Python logging package. I also added essentially a bag of attributes class called Tables which will hold references to all the SA tables that are created. Update the make_table() API to take an instance of Tables. Added a close() method to DBContext. This is needed for the updated unit test suite. Changed bin/import.py so that when available_languages is being set, it calls MailList.set_languages() instead of trying to set that attribute directly. Updated some language idioms while I was at it. More eradication of mm_cfg in favor of the config object and the Defaults module. In testall.py, call initialize() instead of loginit.initialize(). Promote MAX_RESTARTS into a Defaults.py.in variable. This is because the unit tests will knock that value down to something not so annoying should one of the qrunner-required tests traceback. Several other important changes to the unit test suite (which now completely succeeds again!): - Set the uid and gid of the temporary mailman.cfg and tmp*.db files to the Mailman user and group as specified in the config object. - Make sure that all of the tests point to a SQLite database file that was created with the tempfile module. This way we don't pollute our main database with data that is getting created during the unit tests. - In the TestBase.setUp() method, be sure to close the existing dbcontext, clear out the mappers, and then reconnect the dbcontext with the new SQLALCHEMY_ENGINE_URL pointing to the tempfile. However, we don't need to reload the MailList instance any more. - Make all tests work, except for the tests that require crypt. That upgrade path will not be available in this version of Mailman.
* Merged revisions 8113-8121 via svnmerge from bwarsaw2006-12-295-32/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch ................ r8114 | bwarsaw | 2006-12-06 00:16:54 -0500 (Wed, 06 Dec 2006) | 44 lines Initial take on using SQLAlchemy to store list data in lieu of Python pickles. While all the list data (including OldStyleMemberships attributes) are stored in the database, many attributes are stored as PickleTypes binary data. This isn't idea but it gets things working until a more sophisticated schema can be developed. MailList class is now a new-style class, as is required by SQLAlchemy. This makes several things, er, interesting. Rip out all the low-level pickle reading and writing stuff. Hook SA transaction events into Lock() and Unlock(). Move the hooking of the _memberadaptor into InitTempVars(), which gets called by the SQLAlchemy hooks (MailList.__init__() never is). Add an initialize.py module which centralizes all the initialization bits that command line scripts have to do, including configuration, logging, and atabase initialization. This change also converts bin/withlist to mmshell wrapper. Update to SQLAlchemy 0.3.1. Revamp paths.py.in considerably. There were several problems with the old way. We no longer disable default loading of site-packages so we don't need to add Python's site-packages back to sys.path. Also, because site.addsitedir() causes things like .pth paths to be /appended/ to sys.path, they actually won't override any site-installed packages. E.g. if SQLAlchemy is installed in the system Python, our version will not override. IIUC, setuptools-based packages can be configured to work properly in the face of package versions, however not all packages we currently depend on are setuptools-based. So instead, we steal a bit of stuff from site.py but change things so the prepend .pth stuff to sys.path. Update several modules to use True/False and whitespace normalization. Convert from mm_cfg to config object. Modernize a few coding constructs. Add a couple of exceptions to handle database problems. In the export script, include the widget type in the elements. This helped in my stupid little throw away conversion script, but I think it will be more generally useful. Add an interact.py module which refactors interactive interpreter access. Mostly this is used by withlist -i, but it lets us import Mailman.interact and drop into a prompt just about anywhere (e.g. debugging). ................ r8115 | bwarsaw | 2006-12-07 09:13:56 -0500 (Thu, 07 Dec 2006) | 22 lines Start to flesh out more of the SQLAlchemy mechanisms. Added a MailList.__new__() which hooks instantiation to use a query on dbcontext to get an existing mailing list. A 'no-args' call means we're doing a Create(), though eventually that will change too. For now, disable the CheckVersion() call. Eventually this will be folded into schema migration. list_exists(): Rewrite to use the dbcontext query to determine if the named mailing list exists or not. Requires the fqdn_listname. Eradicate two failed member adaptors: BDBMemberAdaptor and SAMemberships. Change the way the DBContext holds onto tables. It now keeps a dictionary mapping the table's name to the SA Table instance. This makes it easier to look up and use the individual tables. Add 'web_page_url' as an attribute managed by SA, and remove a debugging print. ................ r8116 | bwarsaw | 2006-12-11 07:27:47 -0500 (Mon, 11 Dec 2006) | 29 lines Rework the whole dbcontext and transaction framework. SA already handles nested transactions so we don't have to worry about them. However, we do have the weird situation where some transactions are tied to MailList .Lock()/.Unlock()/.Save() and some are tied to non-mlist actions. So now we use an @txn decorator to put methods in a session transaction, but then we also hook into the above MailList methods as possibly sub-transactions. We use a weakref subclass to manage the MailList interface, with a dictionary mapping MailList fqdn_listnames against transactions. The weakrefs come in by giving us a callback when a MailList gets derefed such that we're guaranteed to rollback any outstanding transaction. Also, we have one global DBContext instance but rather than force the rest of Mailman to deal with context objects, instead we expose API methods on that object into the Mailman.database module, which the rest of the code will use. Such methods must be prepended with 'api_' to get exposed this way. bin/rmlist now works with the SA-backend. I refactored the code here so that other code (namely, the test suite) can more easily and consistently remove a mailing list. This isn't the best place for it ultimately, but it's good enough for now. New convenience functions Utils.split_listname(), .fqdn_listname(). Convert testall to use Mailman.initialize.initialize(). Not all tests work, but I'm down to only 8 failures and 7 errors. Also, do a better job of recovering from failures in setUp(). MailList.__new__() now takes keyword arguments. ................ r8117 | bwarsaw | 2006-12-11 22:58:06 -0500 (Mon, 11 Dec 2006) | 7 lines Unit test repairs; even though the unit tests are still pretty fragile, everything now passes with the SQLAlchemy storage of list data. Added missing 'personalize' column. Converted mailmanctl and qrunner to initialize() interface. Fixed _cookie_path() to not fail if SCRIPT_NAME is not in the environment. ................ r8118 | bwarsaw | 2006-12-27 18:45:41 -0500 (Wed, 27 Dec 2006) | 21 lines Utils.list_names(): Use a database query to get all the list names. dbcontext.py: Added api_get_list_names() to support Utils.list_names(). listdata.py: Added two additional MailList attributes which need to be stored in the database. The first is 'admin_member_chunksize' which isn't modifiable from the web. The second is 'password' which holds the list's password. HTMLFormatObject: item strings can now be unicodes. bin/list_lists.py: Must call initialize() to get the database properly initialized, not just config.load(). This will be a common theme. SecurityManager.py: - Remove md5 and crypt support - Added mailman.debug logger, though it will be only used during debugging. - The 'secret' can be a unicode now. - A few coding style updates; repr() instead of backticks, 'key in dict' instead of 'dict.has_key(key)' ................ r8119 | bwarsaw | 2006-12-27 19:13:09 -0500 (Wed, 27 Dec 2006) | 2 lines genaliases.py: config.load() -> initialize() ................ r8120 | bwarsaw | 2006-12-27 19:17:26 -0500 (Wed, 27 Dec 2006) | 9 lines Blocked revisions 8113 via svnmerge ........ r8113 | bwarsaw | 2006-12-05 23:54:30 -0500 (Tue, 05 Dec 2006) | 3 lines Initialized merge tracking via "svnmerge" with revisions "1-8112" from https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch ........ ................ r8121 | bwarsaw | 2006-12-28 23:34:52 -0500 (Thu, 28 Dec 2006) | 20 lines Remove SIGTERM handling from all the CGI scripts. This messes with HTTPRunner because when you issue "mailmanctl stop" after the signal handler has been installed, the process will get a SIGTERM, the signal handler will run, and the process will exit with a normal zero code. This will cause mailmanctl to try to restart the HTTPRunner. I don't think we need that stuff at all when running under wsgi with a SQLAlchemy backend. If mailmanctl kills the HTTPRunner in the middle of the process, I believe (but have not tested) that the transaction should get properly rolled back at process exit. We need to make sure about this, and also we need to test the signal handling functionality under traditional CGI environment (if we even still want to support that). Also, make sure that we don't try to initialize the loggers twice in qrunner. This was the cause of all the double entries in logs/qrunner. Fix a coding style nit in mailmanctl.py. De-DOS-ify line endings in loginit.py. ................
* Upgrade to email package version 4.0.1. Because email 4.0.1 is onlybwarsaw2006-07-303-27/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | compatible back to Python 2.3, this change should not get back ported to Mailman 2.1. Port to Python 2.5. The non-test suite changes should get back ported to Mailman 2.1 (which I will do next), but don't worry about the test suite ones because MM2.1's test suite is hopeless. Specifically: - In SecurityManager.py, fix the parsecookie() code to work with Python 2.5 generated cookie text. The latter was changed to be more RFC compliant so it does not output training semicolons for each line of cookie text. This broke the splitting rules, so now first split on newlines, then on ';\s*'. This should work across all Python versions. - In Python 2.5, exceptions are new-style, and thus are no longer of ClassType. The instantiation type test in hold_for_approval() was too naive. - Raising strings generates deprecation warnings in Python 2.5. Switch the one weird use of this in Utils.py to use a class exception. Don't call it "quick exit" though because it's probably not. - In the tests, use True/False instead of 1/0 - Use failUnless/failIf instead of assertEqual against True/False. - In the tests, use Message.get_content_type() instead of Message.get_type() since the latter is gone in email 4.0.1. Same with get_content_maintype() and get_main_type().
* Add a framework for easier use of alternative MemberAdaptor implementations.bwarsaw2006-07-162-1/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | Also add an experimental (and currently non-functioning) SQLAlchemy implementation. The MemberAdaptor.py interface has been updated in a couple of ways. First, a "transaction interface" has been added so that Mailman can properly sync with the member adaptor. Newly supported methods are load(), lock(), save(), and unlock() and these correspond to methods in the MailList object. Second, __init__() is officially documented to take a MailList instance and nothing else. Third, some of the existing docstrings were incorrect w.r.t. the OldStyleMemberships implementation (such as rasing BadPasswordError in some cases). Most of these should not be the responsibility of the MemberAdaptor, so the docstrings have been updated. Test cases have been added and a new Defaults.py.in variable called MEMBER_ADAPTOR_CLASS has been added which names the class to use. Of course OldStyleMemberships are named by default. There's also a SQLALCHEMY_ENGINE_URL variable for use with the experimental member adaptor. Fix a bug in Configuration where if the etc/mailman.cfg file wasn't found and the mm_cfg.py file was used as a fallback, it would blow away the original namespace copied from Defaults.py.in. This wasn't a problem until we started adding additional names to that namespace, such as 'add_domain'.
* Added robustness to Switchboards and Runners so that if a runner crashesbwarsaw2006-07-161-114/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | uncleanly (e.g. segfaults the Python interpreter), messages being processed will not be lost. The vulnerability, ideas, and patches are credited to Richard Barrett and Mark Sapiro. Their original work was modified by Barry for this commit and any bugs are his fault. The basic idea is that instead of unlinking a .pck file in dequeue(), the file is renamed to a .bak file. The Switchboard grows a finish() method which then unlinks the .bak file. That class's constructor also grows a 'restore' argument (defaulting to false), which when true moves all .bak files it finds in its hash space to .pck, thereby restoring a file lost while "in flight". This relies on the fact that even with multiple qrunners, exactly one process will be responsible for one hash space slice, so it's never possible (under normal operation) for a .bak file to be renamed to .pck by some other process. Test cases for both the new Switchboard behavior and the use of that by Runner subclasses has been added. There are two things to watch out for, either of which may require some additional changes. There is some small potential to duplicate messages in various queues, if say 'mailmanctl' were improperly started more than once by a site admin. This usually won't happen unless an admin is overly eager with the mailmanctl -s switch, so we can chalk this one up to operator error. I'm not sure what more we can do about that. There's also a possibility that if we're processing a message that continually causes the Python interpreter to crash, we could end up duplicating messages endlessly. This is especially troublesome for the Outgoing runner which could conceivably cause a mail flood. I consider this the more critical issue to defend against, probably by adding a numbering scheme to the .bak file names and refusing to restore a .bak file more than say 3 times without human intervention.
generated by cgit v1.2.3-70-g09d2 (git 2.51.2) at 2025-11-09 06:09:15 +0000