summaryrefslogtreecommitdiff
path: root/cron/gate_news
Commit message (Collapse)AuthorAgeFilesLines
* Move all cron scripts to the new Mailman.bin package layout and complete thebwarsaw2006-05-131-269/+0
| | | | | conversion to optparse style option parsing. Remove mailpasswds as password reminders will go away for MM2.2.
* - Fixes to further remove the old Syslog from bin and cron scripts. Note thatbwarsaw2006-04-241-46/+31
| | | | | | | | | | | I didn't update cron/mailpasswds because that is going away. cron/disabled and cron/gate_news are only minimally tested. - Instead of using the RotatingFileHandler for our logs, use our own ReopenableFileHandler, which only minimally derives from FileHandler and implements a reopen() method. I think it's generally better to leave file rotation to external tools such as logrotate. - Remove some Python 2.1 compatibility stuff. - Ignore .mo files in messages/vi/LC_MESSAGES
* FSF office has moved. chdcking in for MAIN branch.tkikuchi2005-08-271-1/+1
|
* open_newsgroup(): mlist.nntp_host may now be a host:port pair.bwarsaw2003-05-121-2/+12
|
* Jon Parise's patch to improve the usage() output for the command linebwarsaw2002-10-211-3/+7
| | | | | | and cron scripts. When code/status == 0, there's no error (it's likely --help output) so send that to stdout. Otherwise, it's an error and the output goes to stderr.
* In the LogStdErr() call, we don't need to pass in the tee argument,bwarsaw2002-10-211-1/+1
| | | | since the default is fine.
* open_newsgroup(): SF bug # 620033 indicates that an IOError can bebwarsaw2002-10-151-5/+5
| | | | | | raised in the NNTP() constructor. Just catch it, log it, and move on. Also, whitespace normalization.
* poll_newsgroup(): Eek! Typo! The intent was definitely to remove anybwarsaw2002-09-061-1/+1
| | | | existing X-Originally-To header before we place our own there.
* Share a single nntp connection amongst all lists with the samebwarsaw2002-04-281-9/+29
| | | | | | | | | | | | | | | | nntp_host. Also, clean up some exception handling. Specifically, open_newsgroup(): Cache the connection object. clearcache(): New function to clean things up at script exit. poll_newsgroup(): Catch all NNTPErrors, not just (obsolete) error_temp. process_lists(): Catch all socket and nntplib errors when opening the newsgroup. main(): Be sure to clear the nntp connection cache when done.
* poll_newsgroup(): GetAdminEmail() is gone so replace this with morebwarsaw2002-03-271-2/+5
| | | | appropriate addresses.
* Update imports for the new rules on shadowing standard librarybwarsaw2002-02-231-7/+5
| | | | | modules. Specifically, import the email package /after/ we've imported paths.
* mimelib -> emailbwarsaw2001-10-221-4/+4
|
* Ben Gertzfield's patch to hardcode the path to the python executablebwarsaw2001-09-071-1/+1
| | | | | | | | | | | | | into the script's #! line -- based on --with-python settings, without losing CVS revision history. #! /usr/bin/env python becomes #! @PYTHON@ which gets substituted for in configure.
* Reorder the imports so mm_cfg is always the first module importedbwarsaw2001-07-271-3/+6
| | | | | | | | | | | | | | | after paths. Because these scripts are run from cron with "python -S" they must import mm_cfg before attempting to import mimelib; mm_cfg hacks sys.path to include site-packages which -S inhibits. For those scripts that already import mm_cfg, add a comment describing the import order dependencies. For those that didn't, add a comment about why it's being imported. Also, gate_news: Removed the _ContinueLoop docstring so translators have one less non-translatable docstring to worry about. Closes Thomas's SF patch #443803.
* poll_newsgroup(): work around Python 2.0 restriction on continue inside trytwouters2001-07-261-1/+8
| | | | clauses.
* poll_newsgroup(): Untested patch so that newsgroup messages withbwarsaw2001-07-251-1/+8
| | | | broken MIME boundaries are ignored.
* Duh, don't try to import mimelib.Parser until after the paths havebwarsaw2001-02-161-2/+2
| | | | | | been hacked (thanks Ron Jarrell). I hadn't been trying this as a cron job, and the Python on my path has mimelib installed (and I wasn't using -S).
* De-string-module-ify.bwarsaw2001-02-151-17/+23
| | | | | | | | References to HandlerAPI module removed. Use the new mimelib and switchboard interfaces. Add some i18n markup.
* process_lists(): Wrap the open_newsgroup() call in a try/except tobwarsaw2000-12-201-2/+6
| | | | | catch any socket errors coming back from the nntp connection attempts.
* Minor clean up. Also turn off incessant logging on every execution.bwarsaw2000-09-141-4/+0
|
* process_lists(): Rewritten for efficiency. The list object is neverbwarsaw2000-06-281-30/+35
| | | | locked until absolutely necessary.
* Get rid of the os.fork() for gating each newsgroup. All gating nowbwarsaw2000-06-271-78/+49
| | | | | | | | | | | | happens in the parent process. poll_newsgroup(): Take a `lock' argument to keep the gate_news lock fresh while gating the newsgroup. Some changes to the log messages. gate_news(): Removed -- folded into process_lists(). process_lists(): Take a `lock' argument to pass to poll_newsgroup(). Perform all work in the parent process. Better logging.
* Use syslog() interface instead of writing to stderr. All syslogbwarsaw2000-06-231-24/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | errors go to the logs/gate_news file, while any exceptions that percolate to the top would go to logs/error. poll_newsgroup(): Always update the watermark, even if the message has been seen before (i.e. we found an X-BeenThere: header) or if an nntp.error_temp occurred (which can happen if the message has expired). Any other exceptions will not be caught and thus will not update the watermark. process_lists(): Be extra paranoid about the forks and child processes. First of all, the fork could fail and if so, we stop cruising through the mailing lists and return to the parent the dictionary of child pids gathered so far. Second, make absolutely sure that the child process exits without returning control to the caller. If we complete successfully or if a TimeOutError occurred, we exit with status 0, otherwise we print a traceback and exit with status 1. main(): First off, we added a keepalive function to geep the gate_news lock fresh while we're waiting for our child processes to exit. Second, we added a failsafe around process_list() -- which does a fork -- so that in the unlikely situation where the child process could have gotten returned here, we exit immediately.
* Get rid of the -s/--stderrs and --quiet/-q options. Use syslog() forbwarsaw2000-06-201-123/+80
| | | | | | | | | | | | | | | | | | | | all logging activities except stderr output from child tracebacks. Get rid of unused imports. poll_newsgroup(): Reorganized so as not to use the crufty QuickEscape. Also, be sure to update mlist.usenet_watermark even if we see a message that's already been through the list. When delivering the message to the list, enqueue it immediately (by adding the _enqueue_immediate flag to msgdata) instead of trying to deliver it. This lets qrunner deliver the message as appropriate. reap(): Move to Utils.py process_lists(): Don't try to be fancy about catching exceptions in the child. Just let them percolate up and Python will do the right thing by logging it and exiting with a non-zero status. main(): Use standard LockFile module for acquiring the gate_news lock.
* open_newsgroup(): Use NNTP_USERNAME and NNTP_PASSWORD for optionalbwarsaw2000-06-051-1/+3
| | | | site-wide NNTP authentication values.
* poll_newsgroup(): MailList objects don't have a Post() methodbwarsaw2000-06-031-3/+4
| | | | anymore. Use HandlerAPI.DeliverToList().
* poll_newsgroup(): Use time.ctime() as a more succinct way of craftingbwarsaw2000-05-311-1/+1
| | | | the unixfrom line.
* Significant reworking based on the comments of Harald Meland and Jimbwarsaw2000-04-071-207/+165
| | | | | | | | | | | | | | | | | | Tittsler. In summary, each list maintains its own watermark as a config.db attribute, and the separate gate_watermarks is no longer used. This greatly simplifies the logic in this script. Some specifics, main(): The command line options have changed, and no longer supports invocation as "gate_news listname first last". Added --stderrs/-s, --quiet/-q, and --help/-h flags. Removed unnecessary imports. WATERMARK_FILE, LIST_LOCK_FILE both are gone. There's a global block file and each list locks itself when actually doing the gating of articles.
* cosmeticbwarsaw2000-03-281-1/+1
|
* Update the copyright lines to include the years 1999 & 2000.bwarsaw2000-03-211-1/+1
|
* Import nntplib from Mailman.pythonlib package to pick up the 1.5.2+bwarsaw2000-03-131-17/+11
| | | | | | version of this module. main(): simplified connection creation, set readermode in constructor.
* poll_newsgroup(): If a To: header is found in the original message, itbwarsaw2000-02-091-2/+3
| | | | | is moved to X-Originally-To:. In all cases, To: is set to the list's email address. This resolves PR#172.
* main(): Watch out for error 500 when using the group command on thebwarsaw2000-02-071-1/+10
| | | | | | | | NNTP connection. If the Mailman gateway is on the same machine as the news server, you probably need to set the news server to reader mode first. Patch given by Thomas Wouters inspired by a patch from Jim Tittsler.
* poll_newsgroup(): slightly better debuggingbwarsaw1999-12-251-2/+2
|
* Added "tee_to_stdout=0" argument to the LogStdErr() call, to avoidhmeland1999-12-161-1/+1
| | | | cron mailing me the output sent to stdout on every run.
* Close the NNTP connection in a better way.bwarsaw1999-12-131-7/+5
|
* Be sure to close the NNTP connection so we don't leak file descriptorsbwarsaw1999-12-131-13/+6
|
* Don't include process ids in messages since these are now implicitlybwarsaw1999-12-131-11/+9
| | | | added by StampedLogger.
* Pre-load the cgi module from our special one, similar to what happensbwarsaw1999-12-111-0/+12
| | | | | in scripts/driver. Maybe this will fix the problem with the weird Archiver import errors??
* Increase the logging, because something weird is going on!bwarsaw1999-12-111-5/+18
| | | | | Also, catch socket.error's that can occur during the nntplib.NNTP() call.
* More logging so we can find errors in the gating module better.bwarsaw1999-12-091-7/+9
|
* A serious rewrite to make it more robust, and to add a somewhat usefulbwarsaw1999-11-241-85/+200
| | | | | | | | | | | | | | | | command line interface. This latter allows you to manually poll a specific list's newsgroup for a range of message numbers, sending them to the list. Using this interface does /not/ update the watermark file. We also don't update the watermark entry for a specific list until that list's child process (which does the actually gating of the messages from Usenet to mailing list) exits with a zero status. If it exits with non-zero, it is taken as a complete failure of the gating process and the watermark isn't updated. This could cause duplicates, so a really anal approach would be to create a pipe between child and parent and write the new watermark after each message is successfully gated.
* typobwarsaw1999-11-121-1/+1
|
* Move all the newsgroup polling logic to here from GatewayManager.bwarsaw1999-11-121-1/+63
|
* cosmeticbwarsaw1999-11-121-0/+1
|
* flock => LockFilebwarsaw1999-08-201-3/+3
| | | | hung_timeout => lifetime
* On some RedHat Linuxes, there appears to be a problem with some cronsbwarsaw1999-01-131-0/+5
| | | | | | | | | | | | | | | | | | | | | | messing up their SIGCHLD handling. This would manifest as an IOError getting raised in the cmdproc.close() call of DeliverToList() in Deliverer.py. Thanks go Harald Meland for describing the real problem and giving the real fix. Thanks go to Greg Stein for testing this patch and verifying that it works. Deliverer.DeliverToList(): Undo previously checked in patch, which simply wrapped a try/except around the cmdproc.close() call and ignored errno=10 IOErrors. This was treating the symptoms, not the cause. All files in cron/: Added Harald's work around of the Linux bug by including these lines in every file: # Work around known problems with some RedHat cron daemons import signal signal.signal(signal.SIGCHLD, signal.SIG_DFL) It helps RedHat and doesn't appear to hurt on e.g. Solaris.
* Significantly rewritten to handle many lurking bugs.bwarsaw1998-12-181-32/+85
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - reorganized into functions with a small amount of __main__ code - re-ordered the way the work gets done between the parent and the child. it was possible the gate_watermarks file would be continuously trashed because it was written by every child process, which run asynchronously. now only the parent writes this file, but it does so after each list's gating child process starts. This also means that the parent is what makes the initial NNTP connection, and gets the stats of the list (e.g. first and last). These it passes to the child process, along with the current watermark. The child just does the actual gating. - Added a per-list locking file so that only one gate_news process will be gating a list at any one time. What happens is: the parent acquires the per-list lock. If that fails, it just moves on to the next list; gate_news runs again in 5 minutes anyway, and the hung_timeout on the lock is set to 4 minutes. If the parent gets the per-list lock, it forks a child to handle gating that list. The child then steals the lock from the parent (this should be safe), gates all the new messages, and then releases the per-list lock. This new code should be forking-safe, should not have run away processes, and lets you run gate_news from the command line without interfering with any cron spawned gate_news process.
* added os._exit(0) at end of for loop, to prevent forking too manycotton1998-11-091-0/+6
| | | | | | | times. XXX - this should probably be made to fork only for lists that actually need to gate news. scott
* Whoops, forgot the packagization had happened, should have been:viega1998-07-071-1/+1
| | | | from Mailman import mm_cfg
generated by cgit v1.3.1 (git 2.54.0) at 2026-08-13 19:21:31 +0000