Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!usc!samsung!know!ladcgw.ladc.bull.com!melb.bull.oz.au!sjg From: sjg@melb.bull.oz.au (Simon J. Gerraty) Newsgroups: news.software.nntp Subject: XBATCH - transfer compressed batches via NNTP Message-ID: <1991Apr2.030527.22507@melb.bull.oz.au> Date: 2 Apr 91 03:05:27 GMT Sender: news@melb.bull.oz.au (USENET News Account) Organization: Bull HN Information Systems Australia. Lines: 716 Nntp-Posting-Host: sun0 Sorry for the long delay, this article contains the patches to nntpd to add the XBATCH command. We use this to receive compressed batches from the US. Your mileage may vary but we have seen our news transfer times drop from 12+ hours to 2-3 hours using XBATCH. As indicated in the README.XBATCH file the NNTPD version 2 draft describes a BATCH command that should obsolete XBATCH, but you can have XBATCH now anyway. The following article (I may split it in two) contains the XBATCH client software developed by Frank Mayhar fmayhar@hermes.ladc.bull.com We would of course like to be kept informed of changes, improvements bug fixes etc. Hopefully you will find this simple to install (as we have real jobs to do, so would hope not to have to many "it won't go in" qestions :-) Enjoy! #! /bin/sh # This is a shell archive. Remove anything before this line, then unpack # it by saving it into a file and typing "sh file". To overwrite existing # files, type "sh file -c". You can also feed this as standard input via # unshar, or by typing "sh README.XBATCH <<'END_OF_README.XBATCH' XXBATCH README April 2, 1991 (NNTP 1.5.10) X XThe file xbatch.diffs contains patches to X./common/nntp.h X./server/help.c X./server/access.c X./server/batch.c X./server/serve.c X Xthat implement a new NNTP server command XBATCH. X XXBATCH allows transfer of complete batches via NNTP. XThe batches are not intererated by nntp in any way at all. They Xare simply received and enqueued for injection into news. X XThe aim is of course to be able to transfer compressed batches Xvia NNTP. XBATCH achieves this admirably. There has been Xdiscussion in the news about the benefits of compressed batches Xvs ihave/sendme etc. The concensus seemed to be that compressed Xbatches were not needed. However it seemed to me that most folk Xholding that opinion were probably at sites with nice fast Xreliable IP links. X Xtaureau.melb.bull.oz.au receives news from ladcgw.ladc.bull.com X(LA) via a NNTP. The link at times is very slow and sometimes Xunreliable. At one stage we had to severely restrict the feed Xto be able to transfer a days worth of news within 24 hours! XUsing compressed batches via XBATCH, the same amount of news Xis typically transferred in 2-3 hours. X XThe server patches were done by myself. I have allowed for the Xaddition of extra permission checks for xbatch, but have not Ximplemented them myself. The other major change is that NNTPD Xdoes not fork newsrun each time in enqueues a batch (using Xxbatch the normal NNTPD behavior remains for normal usage). X XThe xbatch client software was developed by Frank Mayhar Xfmayhar@hermes.ladc.bull.com, it is contained in a separate Xshell archive complete with Manual pages etc. See the README Xfile and man pages for details of the client side. X XThe functionality provided by XBATCH will (hopefully) soon be Xreplaced by the BATCH command of the NNTP v2 specification. XXBATCH is simple to install and is dedicated to the Public XDomain in the hope it should prove a useful interim solution to Xthose with slow or unreliable IP links. X XBelow is the description of xbatch from the source: X XNAME: X xbatch - accept complete batches X XSYNOPSIS: X xbatch size X XDESCRIPTION: X This function receives complete batches from the client X and injects them into the Cnews input queue. It uses X the same enqueue() routine as the normal X batch_input_article(). However, for efficiency this X routine disables the exec'ing of "newsrun". X X xbatch takes a single argument "nbytes" which is the X size in bytes of the batch to be transfered. NNTP is X expected to simpy copy and enqueue this many bytes from X stdin. X X Using xbatch, the time taken to transfer news from X ladcgw.ladc.bull.com to taureau.melb.bull.oz.au X is reduced by a number of hours. X X The possible? client/server conversations look like: X C: XBATCH nbytes X S: 339 Ok X C: sends "nbytes" of data X S: 239 Batch transfered successfully. X X The client has the option of sending another batch. X X C: XBATCH nbytes X S: 339 Ok X C: sends "nbytes" of data X S: 436 xbatch failed: [extra data] X X The client can try to re-send the current batch. X The following can happen any time, and may well occur X after a failed transfer. X X C: XBATCH nbytes X S: 400 xbatch failed: [extra data], goodbye. X X The client should call back later. X X C: XBATCH nbytes X S: 449 Sorry, you are not allowed to transfer batches. X X The client should not bother trying again. X X Please send copies of updates to this code to: X X XRETURN VALUE: X None X X X END_OF_README.XBATCH if test 3622 -ne `wc -c xbatch.diffs <<'END_OF_xbatch.diffs' X*** ./common/nntp.h.orig Tue Apr 2 10:44:06 1991 X--- ./common/nntp.h Mon Feb 11 11:45:39 1991 X*************** X*** 2,7 **** X--- 2,8 ---- X * Response codes for NNTP server X * X * @(#)$Header: nntp.h,v 1.8 90/07/05 02:08:31 sob Exp $ X+ * @(#)nntp.h 1.2 91/02/11 12:45:37 (root) X * X * First digit: X * X*************** X*** 45,55 **** X--- 46,58 ---- X #define OK_NEWNEWS 230 /* New articles by message-id follow */ X #define OK_NEWGROUPS 231 /* New newsgroups follow */ X #define OK_XFERED 235 /* Article transferred successfully */ X+ #define OK_XBATCHED 239 /* Batch transferred successfully */ X #define OK_POSTED 240 /* Article posted successfully */ X #define OK_AUTHSYS 280 /* Authorization system ok */ X #define OK_AUTH 281 /* Authorization (user/pass) ok */ X X #define CONT_XFER 335 /* Continue to send article */ X+ #define CONT_XBATCH 339 /* Continue to send batch */ X #define CONT_POST 340 /* Continue to post article */ X #define NEED_AUTHINFO 380 /* authorization is required */ X #define NEED_AUTHDATA 381 /* authorization data required */ X*************** X*** 67,72 **** X--- 70,76 ---- X #define ERR_XFERRJCT 437 /* Article rejected, don't resend */ X #define ERR_NOPOST 440 /* Posting not allowed */ X #define ERR_POSTFAIL 441 /* Posting failed */ X+ #define ERR_NOXBATCH 449 /* Batch transfer not allowed */ X #define ERR_NOAUTH 480 /* authorization required for command */ X #define ERR_AUTHSYS 481 /* Authorization system invalid */ X #define ERR_AUTHREJ 482 /* Authorization data rejected */ X*** ./server/help.c.orig Tue Apr 2 10:32:30 1991 X--- ./server/help.c Mon Feb 11 11:20:51 1991 X*************** X*** 21,28 **** X printf("NEXT POST QUIT\r\n"); X printf("STAT NEWGROUPS HELP\r\n"); X printf("IHAVE NEWNEWS SLAVE\r\n"); X! printf("\r\nAdditionally, the following extention is supported:\r\n\r\n"); X printf("XHDR Retrieve a single header line from a range of articles.\r\n"); X printf("\r\n"); X printf("Bugs to Stan Barber (Internet: nntp@tmc.edu; UUCP: ...!bcm!nntp)\r\n"); X printf(".\r\n"); X--- 21,29 ---- X printf("NEXT POST QUIT\r\n"); X printf("STAT NEWGROUPS HELP\r\n"); X printf("IHAVE NEWNEWS SLAVE\r\n"); X! printf("\r\nAdditionally, the following extentions are supported:\r\n\r\n"); X printf("XHDR Retrieve a single header line from a range of articles.\r\n"); X+ printf("XBATCH Transfer prepared news batches.\r\n"); X printf("\r\n"); X printf("Bugs to Stan Barber (Internet: nntp@tmc.edu; UUCP: ...!bcm!nntp)\r\n"); X printf(".\r\n"); X*** ./server/access.c.orig Tue Apr 2 10:32:28 1991 X--- ./server/access.c Mon Feb 11 11:21:16 1991 X*************** X*** 1,5 **** X #ifndef lint X! static char *sccsid = "@(#)$Header: access.c,v 1.23 90/08/10 22:58:39 sob Exp $"; X #endif X X #include "common.h" X--- 1,8 ---- X #ifndef lint X! static char *sccsid[] ={ X! "@(#)$Header: access.c,v 1.23 90/08/10 22:58:39 sob Exp $", X! "@(#)access.c 1.2 91/02/11 12:18:54 (root)" X! }; X #endif X X #include "common.h" X*************** X*** 26,32 **** X * X * "canxfer" is a pointer to storage for X * an integer,which we set to 1 if the X! * client can transfer news, 0 otherwise. X * X * "gdlist" is a comma separated list of X * newsgroups/distributions which the client X--- 29,37 ---- X * X * "canxfer" is a pointer to storage for X * an integer,which we set to 1 if the X! * client can transfer news, 2 if they can X! * send batches for passing straight to X! * rnews, 0 otherwise. X * X * "gdlist" is a comma separated list of X * newsgroups/distributions which the client X*************** X*** 70,75 **** X--- 75,83 ---- X X #ifdef DEBUG X *canread = *canpost = *canxfer = 1; X+ #ifdef XBATCH X+ *canxfer = 2; X+ #endif X return; X #endif X X*************** X*** 208,213 **** X--- 216,225 ---- X (void) strcpy(gdlist, groups); X } X } X+ #ifdef XBATCH X+ if (*canxfer) X+ *canxfer = 2; X+ #endif X /* X * The access check expects there to be spaces between the group names. X * In the access file, there are commas between the groupnames. X*** ./server/batch.c.orig Tue Apr 2 10:32:29 1991 X--- ./server/batch.c Mon Feb 11 11:44:51 1991 X*************** X*** 1,5 **** X #ifndef lint X! static char *rcsid = "@(#)Header: batch.c,v 1.5 90/08/02 13:32:10 sob Exp $"; X #endif X /* X * Batch subroutine for Cnews. X--- 1,8 ---- X #ifndef lint X! static char *rcsid[] = { X! "@(#)Header: batch.c,v 1.5 90/08/02 13:32:10 sob Exp $", X! "@(#)batch.c 1.3 91/02/11 12:44:48 (root)" X! }; X #endif X /* X * Batch subroutine for Cnews. X*************** X*** 55,60 **** X--- 58,65 ---- X off_t size; /* current size */ X } btch = { NULL, NULL, NO, 0, 0 }; X X+ static int do_newsrun=0; /* don't do it */ X+ X /* X * stash stdin (up to ".") on the end of the batch input file. X * kick newsrun if the batch is non-empty and too big or too old. X*************** X*** 85,90 **** X--- 90,96 ---- X } X if (btch.name == NULL) X return 0; X+ X tempfile[0] = '\0'; X #ifdef UMASK X (void) umask(UMASK); X*************** X*** 99,105 **** X if (tempfile[0] != '\0') X (void) unlink(tempfile); X if (status == 1 && oktorunbatch()) X! status = enqueue(cont_code, err_code, errbuf); X return status; X } X X--- 105,114 ---- X if (tempfile[0] != '\0') X (void) unlink(tempfile); X if (status == 1 && oktorunbatch()) X! { X! do_newsrun++; X! status = enqueue(cont_code, err_code, errbuf); X! } X return status; X } X X*************** X*** 314,320 **** X X if (btch.isopen && fstat(fileno(btch.file), &stbuf) >= 0) { X if (btch.size > 0) X! enqueue(cont_code, err_code, errbuf); X else { X (void) fclose(btch.file); X btch.file = NULL; X--- 323,331 ---- X X if (btch.isopen && fstat(fileno(btch.file), &stbuf) >= 0) { X if (btch.size > 0) X! { X! enqueue(cont_code, err_code, errbuf); X! } X else { X (void) fclose(btch.file); X btch.file = NULL; X*************** X*** 408,413 **** X--- 419,426 ---- X signal(SIGHUP, SIG_IGN); X (void) fflush(stdout); X (void) fflush(stderr); X+ if (!do_newsrun) X+ exit(0); /* that's all */ X newsrun = strsave(NEWSRUN); X if (newsrun == NULL) X newsrun = "/usr/lib/newsbin/input/newsrun"; X*************** X*** 438,440 **** X--- 451,703 ---- X #endif X X X+ #ifdef XBATCH X+ #ifndef DFUNIT X+ # define DFUNIT 1024 X+ #endif X+ X+ /* NAME: X+ * xbatch - accept complete batches X+ * X+ * SYNOPSIS: X+ * xbatch size X+ * X+ * DESCRIPTION: X+ * This function receives complete batches from the client X+ * and injects them into the Cnews input queue. It uses X+ * the same enqueue() routine as the normal X+ * batch_input_article(). However, for efficiency this X+ * routine disables the exec'ing of "newsrun". X+ * X+ * xbatch takes a single argument "nbytes" which is the X+ * size in bytes of the batch to be transfered. NNTP is X+ * expected to simpy copy and enqueue this many bytes from X+ * stdin. X+ * X+ * Using xbatch, the time taken to transfer news from X+ * ladcgw.ladc.bull.com to taureau.melb.bull.oz.au X+ * is reduced by a number of hours. X+ * X+ * The possible? client/server conversations look like: X+ * C: XBATCH nbytes X+ * S: 339 Ok X+ * C: sends "nbytes" of data X+ * S: 239 Batch transfered successfully. X+ * X+ * The client has the option of sending another batch. X+ * X+ * C: XBATCH nbytes X+ * S: 339 Ok X+ * C: sends "nbytes" of data X+ * S: 436 xbatch failed: [extra data] X+ * X+ * The client can try to re-send the current batch. X+ * The following can happen any time, and may well occur X+ * after a failed transfer. X+ * X+ * C: XBATCH nbytes X+ * S: 400 xbatch failed: [extra data], goodbye. X+ * X+ * The client should call back later. X+ * X+ * C: XBATCH nbytes X+ * S: 449 Sorry, you are not allowed to transfer batches. X+ * X+ * The client should not bother trying again. X+ * X+ * Please send copies of updates to this code to: X+ * X+ * X+ * RETURN VALUE: X+ * None X+ */ X+ void X+ xbatch(argc, argv) X+ int argc; X+ char *argv[]; X+ { X+ char errbuf[2 * NNTP_STRLEN]; X+ int retcode = 1; X+ size_t nbytes; X+ X+ if (canxfer < 2) X+ { X+ printf("%d Sorry, you are not allowed to transfer batches.\r\n", X+ ERR_NOXBATCH); X+ #ifdef LOG X+ syslog(LOG_INFO, "%s xbatch rejected", hostname); X+ #endif X+ (void) fflush(stdout); X+ return ; X+ } X+ if (argc > 1) X+ nbytes = atoi(argv[1]); X+ else X+ nbytes = 0; X+ if (nbytes <= 0) X+ { X+ printf("%d Invalid or missing size for xbatch.\r\n", X+ ERR_CMDSYN); X+ (void) fflush(stdout); X+ return ; X+ } X+ if (!space(MINFREE + (nbytes / DFUNIT))) X+ { X+ /* force error reporting code into sending */ X+ /* an out-of-space error message */ X+ if (gethostname(errbuf, MAXHOSTNAMELEN) < 0) X+ (void) strcpy(errbuf, "Amnesiac"); X+ X+ (void) strcat(errbuf, " NNTP server out of space. Try later."); X+ X+ retcode = 0; /* indicates that an error occurred */ X+ } X+ do_newsrun=0; /* don't do newsrun */ X+ if (btch.isopen) X+ { X+ enqpartbatch(INF_DEBUG, INF_DEBUG, errbuf); X+ if (btch.isopen) X+ { X+ if (gethostname(errbuf, MAXHOSTNAMELEN) < 0) X+ (void) strcpy(errbuf, "Amnesiac"); X+ (void) strcat(errbuf, " NNTP server can't prepare batch. Try Later."); X+ retcode = -1; /* a problem */ X+ } X+ } X+ if (retcode == 1) X+ { X+ /* X+ * Ok, we can now try and receive the batch. X+ */ X+ retcode = get_xbatch(CONT_XBATCH, errbuf, nbytes); X+ } X+ switch (retcode) X+ { X+ case -1: X+ printf("%d xbatch failed: %s, goodbye\r\n", ERR_GOODBYE, errbuf); X+ break; X+ case 0: X+ printf("%d xbatch failed: %s\r\n", ERR_XFERFAIL, errbuf); X+ break; X+ default: X+ printf("%d Batch transfered successfully.\r\n", OK_XBATCHED); X+ break; X+ } X+ (void) fflush(stdout); X+ X+ #ifdef LOG X+ syslog(LOG_INFO, "%s xbatch %s", hostname, X+ retcode == > 0 ? "succeeded" : "failed"); X+ #endif X+ return ; X+ } X+ X+ int X+ get_xbatch(cont_code, errbuf, nbytes) X+ int cont_code; X+ char *errbuf; X+ size_t nbytes; X+ { X+ char buf[512]; X+ int status = 1; /* okay status */ X+ size_t count=0; X+ register int icnt, ocnt; X+ X+ /* protect locking */ X+ signal(SIGINT, SIG_IGN); X+ signal(SIGQUIT, SIG_IGN); X+ signal(SIGHUP, SIG_IGN); X+ X+ if (btch.name == NULL) X+ { X+ /* BATCH_FILE may trigger unprivileged() */ X+ btch.name = mktemp(strsave(BATCH_FILE)); X+ } X+ if (btch.name == NULL) X+ return 0; X+ #ifdef UMASK X+ (void) umask(UMASK); X+ #endif X+ if (btch.file == NULL) X+ { X+ btch.file = fopen(btch.name, "a"); X+ if (btch.file == NULL) X+ { X+ #ifdef SYSLOG X+ syslog(LOG_ERR,"xbatch(): %s: %m", btch.name); X+ #endif X+ return 0; X+ } X+ btch.isopen = YES; X+ btch.size = 0; X+ btch.start = time(&btch.start); X+ } X+ printf("%d Ok\r\n", cont_code); X+ (void) fflush(stdout); X+ #ifdef XFER_TIMEOUT X+ signal(SIGALRM, xfer_timeout); X+ (void) alarm(XFER_TIMEOUT); X+ #endif X+ while (count < nbytes) X+ { X+ if ((ocnt = nbytes - count) > sizeof (buf)) X+ ocnt = sizeof (buf); X+ if ((icnt = fread(buf, sizeof (char), ocnt, stdin)) > 0) X+ { X+ if ((ocnt = fwrite(buf, sizeof (char), icnt, btch.file)) == icnt) X+ { X+ count += ocnt; X+ } X+ else X+ { X+ #ifdef DEBUG X+ printf("%d xbatch: ocnt == %d != icnt == %d\r\n", X+ INF_DEBUG, ocnt, icnt); X+ (void) fflush(stdout); X+ #endif /* DEBUG */ X+ break; X+ } X+ } X+ else X+ { X+ #ifdef DEBUG X+ printf("%d xbatch: looking for %d, got icnt == %d\r\n", X+ INF_DEBUG, ocnt, icnt); X+ (void) fflush(stdout); X+ #endif /* DEBUG */ X+ break; X+ } X+ } X+ (void) fflush(btch.file); X+ (void) fclose(btch.file); X+ #ifdef XFER_TIMEOUT X+ (void) alarm(0); X+ (void) signal(SIGALRM, SIG_DFL); X+ #endif X+ X+ if (count < nbytes) X+ { X+ sprintf(errbuf, "xbatch: short by %d bytes", X+ nbytes - count); X+ #ifdef SYSLOG X+ #ifdef LOG X+ syslog(LOG_ERR, "%s %s", hostname, errbuf); X+ #else X+ syslog(LOG_ERR, errbuf); X+ #endif X+ #endif X+ unlink(btch.name); /* toss it. */ X+ btch.file = NULL; X+ btch.isopen = NO; X+ btch.start = 0; X+ btch.size = 0; X+ return 0; X+ } X+ #ifdef DEBUG X+ printf("%d Got that, please wait a tick while I pass it on.\r\n", X+ INF_DEBUG); X+ (void) fflush(stdout); X+ #endif /* DEBUG */ X+ return (enqueue(INF_DEBUG, INF_DEBUG, errbuf) == 1); X+ } X+ #endif /* XBATCH */ X*** ./server/serve.c.orig Tue Apr 2 10:32:31 1991 X--- ./server/serve.c Mon Feb 11 11:21:03 1991 X*************** X*** 1,5 **** X #ifndef lint X! static char *sccsid = "@(#)$Header: serve.c,v 1.35 90/08/11 21:33:10 sob Exp $"; X #endif X X /* X--- 1,8 ---- X #ifndef lint X! static char *sccsid[] = { X! "@(#)$Header: serve.c,v 1.35 90/08/11 21:33:10 sob Exp $", X! "@(#)serve.c 1.2 91/02/11 12:21:01 (root)" X! }; X #endif X X /* X*************** X*** 40,45 **** X--- 43,51 ---- X extern int ahbs(), group(), help(), ihave(); X extern int list(), newgroups(), newnews(), nextlast(), post(); X extern int slave(), stat(), xhdr(); X+ #ifdef XBATCH X+ extern void xbatch(); X+ #endif X X extern int errno; X X*************** X*** 74,79 **** X--- 80,88 ---- X #ifdef XHDR X "xhdr", 0, xhdr, X #endif /* XHDR */ X+ #ifdef XBATCH X+ "xbatch", 0, xbatch, X+ #endif X }; X #define NUMCMDS (sizeof(cmdtbl) / sizeof(struct cmdent)) X X*************** X*** 239,251 **** X *cp = '\0'; X else X timeptr = "Unknown date"; X! #ifdef AUTH X! printf("%d %s NNTP[auth] server version %s ready at %s (%s).\r\n", X #else X! printf("%d %s NNTP server version %s ready at %s (%s).\r\n", X #endif X canpost ? OK_CANPOST : OK_NOPOST, X! host, nntp_version, X timeptr, X canpost ? "posting ok" : "no posting"); X (void) fflush(stdout); X--- 248,270 ---- X *cp = '\0'; X else X timeptr = "Unknown date"; X! X! line[0] = '\0'; X! #if defined(AUTH) && defined(XBATCH) X! strcpy(line, "[xbatch,auth]"); X #else X! # ifdef XBATCH X! strcpy(line, "[xbatch]"); X! # else X! # ifdef AUTH X! strcpy(line, "[auth]"); X! # endif X! # endif X #endif X+ X+ printf("%d %s NNTP%s server version %s ready at %s (%s).\r\n", X canpost ? OK_CANPOST : OK_NOPOST, X! host, line, nntp_version, X timeptr, X canpost ? "posting ok" : "no posting"); X (void) fflush(stdout); END_OF_xbatch.diffs if test 14658 -ne `wc -c (work) (home) #include /* imagine something *very* witty here */