Merged revisions 8113-8121 via svnmerge from

https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch

................
  r8114 | bwarsaw | 2006-12-06 00:16:54 -0500 (Wed, 06 Dec 2006) | 44 lines
  
  Initial take on using SQLAlchemy to store list data in lieu of Python pickles.
  While all the list data (including OldStyleMemberships attributes) are stored
  in the database, many attributes are stored as PickleTypes binary data.  This
  isn't idea but it gets things working until a more sophisticated schema can be
  developed.
  
  MailList class is now a new-style class, as is required by SQLAlchemy.  This
  makes several things, er, interesting.  Rip out all the low-level pickle
  reading and writing stuff.  Hook SA transaction events into Lock() and
  Unlock().  Move the hooking of the _memberadaptor into InitTempVars(), which
  gets called by the SQLAlchemy hooks (MailList.__init__() never is).
  
  Add an initialize.py module which centralizes all the initialization bits that
  command line scripts have to do, including configuration, logging, and atabase
  initialization.
  
  This change also converts bin/withlist to mmshell wrapper.
  
  Update to SQLAlchemy 0.3.1.
  
  Revamp paths.py.in considerably.  There were several problems with the old
  way.  We no longer disable default loading of site-packages so we don't need
  to add Python's site-packages back to sys.path.  Also, because
  site.addsitedir() causes things like .pth paths to be /appended/ to sys.path,
  they actually won't override any site-installed packages.  E.g. if SQLAlchemy
  is installed in the system Python, our version will not override.  IIUC,
  setuptools-based packages can be configured to work properly in the face of
  package versions, however not all packages we currently depend on are
  setuptools-based.  So instead, we steal a bit of stuff from site.py but change
  things so the prepend .pth stuff to sys.path.
  
  Update several modules to use True/False and whitespace normalization.
  Convert from mm_cfg to config object.  Modernize a few coding constructs.
  
  Add a couple of exceptions to handle database problems.
  
  In the export script, include the widget type in the elements.  This helped in
  my stupid little throw away conversion script, but I think it will be more
  generally useful.
  
  Add an interact.py module which refactors interactive interpreter access.
  Mostly this is used by withlist -i, but it lets us import Mailman.interact and
  drop into a prompt just about anywhere (e.g. debugging).
................
  r8115 | bwarsaw | 2006-12-07 09:13:56 -0500 (Thu, 07 Dec 2006) | 22 lines
  
  Start to flesh out more of the SQLAlchemy mechanisms.
  
  Added a MailList.__new__() which hooks instantiation to use a query on
  dbcontext to get an existing mailing list.  A 'no-args' call means we're doing
  a Create(), though eventually that will change too.
  
  For now, disable the CheckVersion() call.  Eventually this will be folded into
  schema migration.
  
  list_exists(): Rewrite to use the dbcontext query to determine if the named
  mailing list exists or not.  Requires the fqdn_listname.
  
  Eradicate two failed member adaptors: BDBMemberAdaptor and SAMemberships.
  
  Change the way the DBContext holds onto tables.  It now keeps a dictionary
  mapping the table's name to the SA Table instance.  This makes it easier to
  look up and use the individual tables.
  
  Add 'web_page_url' as an attribute managed by SA, and remove a debugging
  print.
................
  r8116 | bwarsaw | 2006-12-11 07:27:47 -0500 (Mon, 11 Dec 2006) | 29 lines
  
  Rework the whole dbcontext and transaction framework.  SA already handles
  nested transactions so we don't have to worry about them.  However, we do have
  the weird situation where some transactions are tied to MailList
  .Lock()/.Unlock()/.Save() and some are tied to non-mlist actions.  So now we
  use an @txn decorator to put methods in a session transaction, but then we
  also hook into the above MailList methods as possibly sub-transactions.  We
  use a weakref subclass to manage the MailList interface, with a dictionary
  mapping MailList fqdn_listnames against transactions.  The weakrefs come in by
  giving us a callback when a MailList gets derefed such that we're guaranteed
  to rollback any outstanding transaction.
  
  Also, we have one global DBContext instance but rather than force the rest of
  Mailman to deal with context objects, instead we expose API methods on that
  object into the Mailman.database module, which the rest of the code will use.
  Such methods must be prepended with 'api_' to get exposed this way.
  
  bin/rmlist now works with the SA-backend.  I refactored the code here so that
  other code (namely, the test suite) can more easily and consistently remove a
  mailing list.  This isn't the best place for it ultimately, but it's good
  enough for now.
  
  New convenience functions Utils.split_listname(), .fqdn_listname().
  
  Convert testall to use Mailman.initialize.initialize().  Not all tests work,
  but I'm down to only 8 failures and 7 errors.  Also, do a better job of
  recovering from failures in setUp().
  
  MailList.__new__() now takes keyword arguments.
................
  r8117 | bwarsaw | 2006-12-11 22:58:06 -0500 (Mon, 11 Dec 2006) | 7 lines
  
  Unit test repairs; even though the unit tests are still pretty fragile,
  everything now passes with the SQLAlchemy storage of list data.
  
  Added missing 'personalize' column.  Converted mailmanctl and qrunner to
  initialize() interface.  Fixed _cookie_path() to not fail if SCRIPT_NAME is
  not in the environment.
................
  r8118 | bwarsaw | 2006-12-27 18:45:41 -0500 (Wed, 27 Dec 2006) | 21 lines
  
  Utils.list_names(): Use a database query to get all the list names.
  
  dbcontext.py: Added api_get_list_names() to support Utils.list_names().
  
  listdata.py: Added two additional MailList attributes which need to be stored
  in the database.  The first is 'admin_member_chunksize' which isn't modifiable
  from the web.  The second is 'password' which holds the list's password.
  
  HTMLFormatObject: item strings can now be unicodes.
  
  bin/list_lists.py: Must call initialize() to get the database properly
  initialized, not just config.load().  This will be a common theme.
  
  SecurityManager.py:
      - Remove md5 and crypt support
      - Added mailman.debug logger, though it will be only used during
        debugging.
      - The 'secret' can be a unicode now.
      - A few coding style updates; repr() instead of backticks, 'key in dict'
        instead of 'dict.has_key(key)'
................
  r8119 | bwarsaw | 2006-12-27 19:13:09 -0500 (Wed, 27 Dec 2006) | 2 lines
  
  genaliases.py: config.load() -> initialize()
................
  r8120 | bwarsaw | 2006-12-27 19:17:26 -0500 (Wed, 27 Dec 2006) | 9 lines
  
  Blocked revisions 8113 via svnmerge
  
  ........
    r8113 | bwarsaw | 2006-12-05 23:54:30 -0500 (Tue, 05 Dec 2006) | 3 lines
    
    Initialized merge tracking via "svnmerge" with revisions "1-8112" from 
    https://mailman.svn.sourceforge.net/svnroot/mailman/branches/tmp-sqlalchemy-branch
  ........
................
  r8121 | bwarsaw | 2006-12-28 23:34:52 -0500 (Thu, 28 Dec 2006) | 20 lines
  
  Remove SIGTERM handling from all the CGI scripts.  This messes with HTTPRunner
  because when you issue "mailmanctl stop" after the signal handler has been
  installed, the process will get a SIGTERM, the signal handler will run, and
  the process will exit with a normal zero code.  This will cause mailmanctl to
  try to restart the HTTPRunner.
  
  I don't think we need that stuff at all when running under wsgi with a
  SQLAlchemy backend.  If mailmanctl kills the HTTPRunner in the middle of the
  process, I believe (but have not tested) that the transaction should get
  properly rolled back at process exit.  We need to make sure about this, and
  also we need to test the signal handling functionality under traditional CGI
  environment (if we even still want to support that).
  
  Also, make sure that we don't try to initialize the loggers twice in qrunner.
  This was the cause of all the double entries in logs/qrunner.
  
  Fix a coding style nit in mailmanctl.py.
  
  De-DOS-ify line endings in loginit.py.
................

1 files changed, 249 insertions, 0 deletions

diff --git a/Mailman/bin/withlist.py b/Mailman/bin/withlist.py
new file mode 100644
index 000000000..63513d70b
--- /dev/null
+++ b/Mailman/bin/withlist.py
@@ -0,0 +1,249 @@
+# Copyright (C) 1998-2006 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,
+# USA.
+
+import os
+import sys
+import atexit
+import optparse
+
+from Mailman import Errors
+from Mailman import MailList
+from Mailman import Utils
+from Mailman import Version
+from Mailman import interact
+from Mailman.configuration import config
+from Mailman.i18n import _
+from Mailman.initialize import initialize
+
+__i18n_templates__ = True
+
+LAST_MLIST  = None
+VERBOSE     = True
+LOCK        = False
+
+
+
+def exitfunc(mlist):
+    """Unlock a locked list, but do not implicitly Save() it."""
+    if mlist.Locked():
+        if VERBOSE:
+            listname = mlist.fqdn_listname
+            print >> sys.stderr, _(
+                'Unlocking (but not saving) list: $listname')
+        mlist.Unlock()
+    if VERBOSE:
+        print >> sys.stderr, _('Finalizing')
+
+
+
+def do_list(listname, args, func):
+    global LAST_MLIST
+
+    if '@' not in listname:
+        listname += '@' + config.DEFAULT_EMAIL_HOST
+
+    if VERBOSE:
+        print >> sys.stderr, _('Loading list $listname'),
+        if LOCK:
+            print >> sys.stderr, _('(locked)')
+        else:
+            print >> sys.stderr, _('(unlocked)')
+
+    try:
+        mlist = MailList.MailList(listname, lock=LOCK)
+        atexit.register(exitfunc, mlist)
+        LAST_MLIST = mlist
+    except Errors.MMUnknownListError:
+        print >> sys.stderr, _('Unknown list: $listname')
+
+    # try to import the module and run the callable
+    if func:
+        return func(mlist, *args)
+    return None
+
+
+
+def parseargs():
+    parser = optparse.OptionParser(version=Version.MAILMAN_VERSION,
+                                   usage=_("""\
+%prog [options] listname [args ...]
+
+General framework for interacting with a mailing list object.
+
+There are two ways to use this script: interactively or programmatically.
+Using it interactively allows you to play with, examine and modify a MailList
+object from Python's interactive interpreter.  When running interactively, a
+MailList object called 'm' will be available in the global namespace.  It also
+loads the class MailList into the global namespace.
+
+Programmatically, you can write a function to operate on a MailList object,
+and this script will take care of the housekeeping (see below for examples).
+In that case, the general usage syntax is:
+
+    % bin/withlist [options] listname [args ...]
+
+Here's an example of how to use the -r option.  Say you have a file in the
+Mailman installation directory called 'listaddr.py', with the following
+two functions:
+
+    def listaddr(mlist):
+        print mlist.GetListEmail()
+
+    def requestaddr(mlist):
+        print mlist.GetRequestEmail()
+
+Now, from the command line you can print the list's posting address by running
+the following from the command line:
+
+    % bin/withlist -r listaddr mylist
+    Loading list: mylist (unlocked)
+    Importing listaddr ...
+    Running listaddr.listaddr() ...
+    mylist@myhost.com
+
+And you can print the list's request address by running:
+
+    % bin/withlist -r listaddr.requestaddr mylist
+    Loading list: mylist (unlocked)
+    Importing listaddr ...
+    Running listaddr.requestaddr() ...
+    mylist-request@myhost.com
+
+As another example, say you wanted to change the password for a particular
+user on a particular list.  You could put the following function in a file
+called 'changepw.py':
+
+    from Mailman.Errors import NotAMemberError
+
+    def changepw(mlist, addr, newpasswd):
+        try:
+            mlist.setMemberPassword(addr, newpasswd)
+            mlist.Save()
+        except NotAMemberError:
+            print 'No address matched:', addr
+
+and run this from the command line:
+
+    % bin/withlist -l -r changepw mylist somebody@somewhere.org foobar"""))
+    parser.add_option('-l', '--lock',
+                      default=False, action='store_true', help=_("""\
+Lock the list when opening.  Normally the list is opened unlocked (e.g. for
+read-only operations).  You can always lock the file after the fact by typing
+'m.Lock()'
+
+Note that if you use this option, you should explicitly call m.Save() before
+exiting, since the interpreter's clean up procedure will not automatically
+save changes to the MailList object (but it will unlock the list)."""))
+    parser.add_option('-i', '--interactive',
+                      default=None, action='store_true', help=_("""\
+Leaves you at an interactive prompt after all other processing is complete.
+This is the default unless the -r option is given."""))
+    parser.add_option('-r', '--run',
+                      type='string', help=_("""\
+This can be used to run a script with the opened MailList object.  This works
+by attempting to import 'module' (which must be in the directory containing
+withlist, or already be accessible on your sys.path), and then calling
+'callable' from the module.  callable can be a class or function; it is called
+with the MailList object as the first argument.  If additional args are given
+on the command line, they are passed as subsequent positional args to the
+callable.
+
+Note that 'module.' is optional; if it is omitted then a module with the name
+'callable' will be imported.
+
+The global variable 'r' will be set to the results of this call."""))
+    parser.add_option('-a', '--all',
+                      default=False, action='store_true', help=_("""\
+This option only works with the -r option.  Use this if you want to execute
+the script on all mailing lists.  When you use -a you should not include a
+listname argument on the command line.  The variable 'r' will be a list of all
+the results."""))
+    parser.add_option('-q', '--quiet',
+                      default=False, action='store_true',
+                      help=_('Suppress all status messages.'))
+    parser.add_option('-C', '--config',
+                      help=_('Alternative configuration file to use'))
+    opts, args = parser.parse_args()
+    return parser, opts, args
+
+
+
+def main():
+    global LAST_MLIST, LOCK, VERBOSE
+
+    parser, opts, args = parseargs()
+    initialize(opts.config, not opts.quiet)
+
+    VERBOSE = not opts.quiet
+    LOCK = opts.lock
+
+    # Append our bin directory to sys.path so that any withlist scripts living
+    # their can be simply imported.
+    sys.path.append(config.BIN_DIR)
+
+    # The default for interact is true unless -r was given
+    if opts.interactive is None:
+        if not opts.run:
+            opts.interactive = True
+        else:
+            opts.interactive = False
+
+    dolist = True
+    if len(args) < 1 and not opts.all:
+        warning = _('No list name supplied.')
+        if opts.interactive:
+            # Let them keep going
+            print >> sys.stderr, warning
+            dolist = False
+        else:
+            parser.error(warning)
+
+    if opts.all and not opts.run:
+        parser.error(_('--all requires --run'))
+
+    # Try to import the module for the callable
+    func = None
+    if opts.run:
+        i = opts.run.find('.')
+        if i < 0:
+            module = opts.run
+            callable = opts.run
+        else:
+            module = opts.run[:i]
+            callable = opts.run[i+1:]
+        if VERBOSE:
+            print >> sys.stderr, _('Importing $module ...')
+        mod = __import__(module)
+        if VERBOSE:
+            print >> sys.stderr, _('Running ${module}.${callable}() ...')
+        func = getattr(mod, callable)
+
+    r = None
+    if opts.all:
+        r = [do_list(listname, args, func) for listname in Utils.list_names()]
+    elif dolist:
+        listname = args.pop(0).lower().strip()
+        r = do_list(listname, args, func)
+
+    # Now go to interactive mode, perhaps
+    if opts.interactive:
+        if dolist:
+            banner = _("The variable 'm' is the $listname MailList instance")
+        else:
+            banner = interact.DEFAULT_BANNER
+        overrides = dict(m=LAST_MLIST, r=r)
+        interact.interact(upframe=False, banner=banner, overrides=overrides)