Passwords done right.

First off, there are several password hashing schemes added including SHA,
salted-SHA, and RFC 2989 PBKDF2 (contributed by Bob Fleck).  Then we encode
the password using RFC 2307 style syntax.  At least I think: specifically
things like the PRF and iteration count for PBKDF2 are encoded the way I
/think/ is intended for RFC 2307 but I could be wrong.  Seems darn hard to
find definitive information about that.

In any event, even though CLEARTEXT passwords are supported, they are mostly
deprecated, even for user passwords.  It also allows us to easily update all
passwords to a new hashing scheme when the existing schemes get cracked.  The
default scheme (specified in Defaults.py.in) is salted-SHA with a 20 byte salt
(the salt length and PBKDF2 iteration counts can only be specified in the
passwords.py file).

These hashed passwords are used for user passwords, list owner and moderator
passwords, and site and list creator passwords.

Of course this means that user password reminders are impossible now.  They've
been ripped out of the code for a while, but now we'll need to implement
password resets since user passwords cannot be recovered.

bin/export has had several changes:

- export no longer converts to dollar strings.  Were assuming dollar strings
  are used by default for all new lists and any imported lists will already be
  converted to dollar strings.
- Likewise, rip out the password scheme stuff, since cleartext passwords can
  never be exported, so we might as well always include the member's hashed
  password.
- Fix exporting to stdout when that stream can only handle ascii by wrapping
  stdout in a utf-8 codec writer.

Other changes:

- add a missing import to HTTPRunner.py

- Convert GUIBase.py to use Defaults.* for constants instead of mm_cfg.*

- Remove pre-Python 2.4 compatibility from Utils.py.  We've already said
  Python 2.4 will be a minimum requirement.

- Change the permissions on the global password file.  The default 007 umask
  is used and should be good enough.

- bin/newlist adds the ability to specify the password scheme (or list the
  available schemes) for the list owner password.  It is not possible to set
  the scheme on a per-list basis.  bin/mmsitepass does the same, but for the
  site and list creator passwords.

- Fix a nasty problem with bin/import.  The comment in the code says it best:

                # XXX Here's what sucks.  Some properties need to have
                # _setValue() called on the gui component, because those
                # methods do some pre-processing on the values before they're
                # applied to the MailList instance.  But we don't have a good
                # way to find a category and sub-category that a particular
                # property belongs to.  Plus this will probably change.  So
                # for now, we'll just hard code the extra post-processing
                # here.  The good news is that not all _setValue() munging
                # needs to be done -- for example, we've already converted
                # everything to dollar strings.

- Set the 'debug' logger to logging.DEBUG level.  It doesn't seem to make much
  sense for the debugging log to ignore debug messages.

4 files changed, 68 insertions, 84 deletions

diff --git a/Mailman/bin/export.py b/Mailman/bin/export.py
index d83959279..b0b5519ef 100644
--- a/Mailman/bin/export.py
+++ b/Mailman/bin/export.py
@@ -18,8 +18,9 @@
 """Export an XML representation of a mailing list."""
 
 import os
-import sys
+import re
 import sha
+import sys
 import base64
 import codecs
 import datetime
@@ -35,16 +36,11 @@ from Mailman import Version
 from Mailman.MailList import MailList
 from Mailman.configuration import config
 from Mailman.i18n import _
+from Mailman.initialize import initialize
 
 __i18n_templates__ = True
 
 SPACE           = ' '
-DOLLAR_STRINGS  = ('msg_header', 'msg_footer',
-                   'digest_header', 'digest_footer',
-                   'autoresponse_postings_text',
-                   'autoresponse_admin_text',
-                   'autoresponse_request_text')
-SALT_LENGTH     = 4 # bytes
 
 TYPES = {
     Defaults.Toggle         : 'bool',
@@ -148,7 +144,6 @@ class XMLDumper(object):
             print >> self._fp, '<%s%s>%s</%s>' % (_name, attrs, value, _name)
 
     def _do_list_categories(self, mlist, k, subcat=None):
-        is_converted = bool(getattr(mlist, 'use_dollar_strings', False))
         info = mlist.GetConfigInfo(k, subcat)
         label, gui = mlist.GetConfigCategories()[k]
         if info is None:
@@ -167,12 +162,6 @@ class XMLDumper(object):
                 value = gui.getValue(mlist, vtype, varname, data[2])
             if value is None:
                 value = getattr(mlist, varname)
-            # Do %-string to $-string conversions if the list hasn't already
-            # been converted.
-            if varname == 'use_dollar_strings':
-                continue
-            if not is_converted and varname in DOLLAR_STRINGS:
-                value = Utils.to_dollar(value)
             widget_type = TYPES[vtype]
             if isinstance(value, list):
                 self._push_element('option', name=varname, type=widget_type)
@@ -182,7 +171,7 @@ class XMLDumper(object):
             else:
                 self._element('option', value, name=varname, type=widget_type)
 
-    def _dump_list(self, mlist, password_scheme):
+    def _dump_list(self, mlist):
         # Write list configuration values
         self._push_element('list', name=mlist.fqdn_listname)
         self._push_element('configuration')
@@ -207,8 +196,7 @@ class XMLDumper(object):
                 attrs['original'] = cased
             self._push_element('member', **attrs)
             self._element('realname', mlist.getMemberName(member))
-            self._element('password',
-                          password_scheme(mlist.getMemberPassword(member)))
+            self._element('password', mlist.getMemberPassword(member))
             self._element('language', mlist.getMemberLanguage(member))
             # Delivery status, combined with the type of delivery
             attrs = {}
@@ -252,7 +240,7 @@ class XMLDumper(object):
         self._pop_element('roster')
         self._pop_element('list')
 
-    def dump(self, listnames, password_scheme):
+    def dump(self, listnames):
         print >> self._fp, '<?xml version="1.0" encoding="UTF-8"?>'
         self._push_element('mailman', **{
             'xmlns:xsi': 'http://www.w3.org/2001/XMLSchema-instance',
@@ -264,7 +252,7 @@ class XMLDumper(object):
             except Errors.MMUnknownListError:
                 print >> sys.stderr, _('No such list: $listname')
                 continue
-            self._dump_list(mlist, password_scheme)
+            self._dump_list(mlist)
         self._pop_element('mailman')
 
     def close(self):
@@ -273,41 +261,6 @@ class XMLDumper(object):
 
 
 
-def no_password(password):
-    return '{NONE}'
-
-
-def plaintext_password(password):
-    return '{PLAIN}' + password
-
-
-def sha_password(password):
-    h = sha.new(password)
-    return '{SHA}' + base64.b64encode(h.digest())
-
-
-def ssha_password(password):
-    salt = os.urandom(SALT_LENGTH)
-    h = sha.new(password)
-    h.update(salt)
-    return '{SSHA}' + base64.b64encode(h.digest() + salt)
-
-
-SCHEMES = {
-    'none'  : no_password,
-    'plain' : plaintext_password,
-    'sha'   : sha_password,
-    }
-
-try:
-    os.urandom(1)
-except NotImplementedError:
-    pass
-else:
-    SCHEMES['ssha'] = ssha_password
-
-
-
 def parseargs():
     parser = optparse.OptionParser(version=Version.MAILMAN_VERSION,
                                    usage=_("""\
@@ -319,15 +272,6 @@ Export the configuration and members of a mailing list in XML format."""))
                       help=_("""\
 Output XML to FILENAME.  If not given, or if FILENAME is '-', standard out is
 used."""))
-    parser.add_option('-p', '--password-scheme',
-                      default='none', type='string', help=_("""\
-Specify the RFC 2307 style hashing scheme for passwords included in the
-output.  Use -P to get a list of supported schemes, which are
-case-insensitive."""))
-    parser.add_option('-P', '--list-hash-schemes',
-                      default=False, action='store_true', help=_("""\
-List the supported password hashing schemes and exit.  The scheme labels are
-case-insensitive."""))
     parser.add_option('-l', '--listname',
                       default=[], action='append', type='string',
                       metavar='LISTNAME', dest='listnames', help=_("""\
@@ -339,26 +283,21 @@ included in the XML output.  Multiple -l flags may be given."""))
     if args:
         parser.print_help()
         parser.error(_('Unexpected arguments'))
-    if opts.list_hash_schemes:
-        for label in SCHEMES:
-            print label.upper()
-        sys.exit(0)
-    if opts.password_scheme.lower() not in SCHEMES:
-        parser.error(_('Invalid password scheme'))
     return parser, opts, args
 
 
 
 def main():
     parser, opts, args = parseargs()
-    config.load(opts.config)
+    initialize(opts.config)
 
+    close = False
     if opts.outputfile in (None, '-'):
-        # This will fail if there are characters in the output incompatible
-        # with system encoding.
-        fp = sys.stdout
+        writer = codecs.getwriter('utf-8')
+        fp = writer(sys.stdout)
     else:
         fp = codecs.open(opts.outputfile, 'w', 'utf-8')
+        close = True
 
     try:
         dumper = XMLDumper(fp)
@@ -370,8 +309,8 @@ def main():
                 listnames.append(listname)
         else:
             listnames = Utils.list_names()
-        dumper.dump(listnames, SCHEMES[opts.password_scheme])
+        dumper.dump(listnames)
         dumper.close()
     finally:
-        if fp is not sys.stdout:
+        if close:
             fp.close()
diff --git a/Mailman/bin/import.py b/Mailman/bin/import.py
index b643de389..e9171f73a 100644
--- a/Mailman/bin/import.py
+++ b/Mailman/bin/import.py
@@ -34,11 +34,9 @@ from Mailman.MailList import MailList
 from Mailman.i18n import _
 from Mailman.initialize import initialize
 
-
 __i18n_templates__ = True
 
 
-
 
 def nodetext(node):
     # Expect only one TEXT_NODE in the list of children
@@ -211,6 +209,20 @@ def create(all_listdata):
         mlist.Lock()
         try:
             for option, value in list_config.items():
+                # XXX Here's what sucks.  Some properties need to have
+                # _setValue() called on the gui component, because those
+                # methods do some pre-processing on the values before they're
+                # applied to the MailList instance.  But we don't have a good
+                # way to find a category and sub-category that a particular
+                # property belongs to.  Plus this will probably change.  So
+                # for now, we'll just hard code the extra post-processing
+                # here.  The good news is that not all _setValue() munging
+                # needs to be done -- for example, we've already converted
+                # everything to dollar strings.
+                if option in ('filter_mime_types', 'pass_mime_types',
+                              'filter_filename_extensions',
+                              'pass_filename_extensions'):
+                    value = value.splitlines()
                 setattr(mlist, option, value)
             for member in list_roster:
                 mid = member['id']
diff --git a/Mailman/bin/mmsitepass.py b/Mailman/bin/mmsitepass.py
index 3bd2c77d7..0d986bf6a 100644
--- a/Mailman/bin/mmsitepass.py
+++ b/Mailman/bin/mmsitepass.py
@@ -1,4 +1,4 @@
-# Copyright (C) 1998-2006 by the Free Software Foundation, Inc.
+# Copyright (C) 1998-2007 by the Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -21,8 +21,10 @@ import optparse
 
 from Mailman import Utils
 from Mailman import Version
+from Mailman import passwords
 from Mailman.configuration import config
 from Mailman.i18n import _
+from Mailman.initialize import initialize
 
 __i18n_templates__ = True
 
@@ -49,20 +51,34 @@ If password is not given on the command line, it will be prompted for.
 Set the list creator password instead of the site password.  The list
 creator is authorized to create and remove lists, but does not have
 the total power of the site administrator."""))
+    parser.add_option('-p', '--password-scheme',
+                      default=config.PASSWORD_SCHEME, type='string',
+                      help=_("""\
+Specify the RFC 2307 style hashing scheme for passwords included in the
+output.  Use -P to get a list of supported schemes, which are
+case-insensitive."""))
+    parser.add_option('-P', '--list-hash-schemes',
+                      default=False, action='store_true', help=_("""\
+List the supported password hashing schemes and exit.  The scheme labels are
+case-insensitive."""))
     parser.add_option('-C', '--config',
                       help=_('Alternative configuration file to use'))
     opts, args = parser.parse_args()
     if len(args) > 1:
-        parser.print_help()
-        print >> sys.stderr, _('Unexpected arguments')
-        sys.exit(1)
+        parser.error(_('Unexpected arguments'))
+    if opts.list_hash_schemes:
+        for label in passwords.SCHEMES:
+            print label.upper()
+        sys.exit(0)
+    if opts.password_scheme.lower() not in passwords.SCHEMES:
+        parser.error(_('Invalid password scheme'))
     return parser, opts, args
 
 
 
 def main():
     parser, opts, args = parseargs()
-    config.load(opts.config)
+    initialize(opts.config)
     if args:
         password = args[0]
     else:
@@ -77,7 +93,8 @@ def main():
             print _('Passwords do not match; no changes made.')
             sys.exit(1)
         password = pw1
-    Utils.set_global_password(password, not opts.listcreator)
+    Utils.set_global_password(password,
+                              not opts.listcreator, opts.password_scheme)
     if Utils.check_global_password(password, not opts.listcreator):
         print _('Password changed.')
     else:
diff --git a/Mailman/bin/newlist.py b/Mailman/bin/newlist.py
index a127b25e1..bde615974 100644
--- a/Mailman/bin/newlist.py
+++ b/Mailman/bin/newlist.py
@@ -26,6 +26,7 @@ from Mailman import Message
 from Mailman import Utils
 from Mailman import Version
 from Mailman import i18n
+from Mailman import passwords
 from Mailman.configuration import config
 from Mailman.initialize import initialize
 
@@ -72,9 +73,24 @@ This option suppresses the prompt prior to administrator notification but
 still sends the notification.  It can be used to make newlist totally
 non-interactive but still send the notification, assuming listname,
 listadmin-addr and admin-password are all specified on the command line."""))
+    parser.add_option('-p', '--password-scheme',
+                      default='ssha', type='string', help=_("""\
+Specify the RFC 2307 style hashing scheme for passwords included in the
+output.  Use -P to get a list of supported schemes, which are
+case-insensitive."""))
+    parser.add_option('-P', '--list-hash-schemes',
+                      default=False, action='store_true', help=_("""\
+List the supported password hashing schemes and exit.  The scheme labels are
+case-insensitive."""))
     parser.add_option('-C', '--config',
                       help=_('Alternative configuration file to use'))
     opts, args = parser.parse_args()
+    if opts.list_hash_schemes:
+        for label in passwords.SCHEMES:
+            print label.upper()
+        sys.exit(0)
+    if opts.password_scheme.lower() not in passwords.SCHEMES:
+        parser.error(_('Invalid password scheme'))
     # Can't verify opts.language here because the configuration isn't loaded
     # yet.
     return parser, opts, args
@@ -146,7 +162,7 @@ def main():
     # set available_languages.
     mlist.preferred_language = opts.language
     try:
-        pw = sha.new(listpasswd).hexdigest()
+        pw = passwords.make_secret(listpasswd, config.PASSWORD_SCHEME)
         try:
             mlist.Create(fqdn_listname, owner_mail, pw)
         except Errors.BadListNameError, s: