C-News software can be obtained from ftp://ftp.uu.net/networking/news/transport/cnews/cnews.tar.Z and will need to be uncompressed using the BSD uncompress utility or a compatible program. The tarball is about 650 KBytes in size. It has its own highly intelligent configuration and installation processes, which are very well documented. The version that is available is Cleanup Release revision G, on which our own version is based.
NNTPd (the NNTP Reference Implementation) is available from ftp://ftp.uu.net/networking/news/nntp/nntp.220.127.116.11.tar.Z. It has no automatic scripts and processes to configure itself. After fetching the sources, you will have to follow a set of directions given in the documentation and configure some C header files. These configuration settings must be done keeping in mind what you have specified when you build the C-News sources, because NNTPd and C-News must work together. Therefore, some key file formats, directory paths, etc., will have to be specified identically in both software systems.
The third software system we use is Nestor. This too is to be found in the same place where the NNTPd software is kept, at ftp://ftp.uu.net/networking/news/nntp/nestor.tar.Z. This software compiles to one binary program, which must be run periodically to process the logs of nntpd, the NNTP server which is part of NNTPd, and report usage statistics to the administrator. We have integrated Nestor into our source base.
The fourth piece of the system, without which no Usenet server administrator dares venture out into the wild world of public Internet newsfeeds, is pgpverify.
We have been working with C-News and NNTPd for many years now, and have fixed a few bugs in both packages. We have also integrated the four software systems listed above, and added a few features here and there to make things work more smoothly. We offer our entire source base to anyone for free download from http://www.starcomsoftware.com/proj/usenet/src/news.tar.gz. There are no licensing restrictions on our sources; they are as freely redistributable as the original components we started with.
When you download our software distribution, you will extract it to find a directory tree with the following subdirectories and files:
c-news: the source tree of the CR.G software release, with our additions like pgpverify integration, our scripts like mail2news, and pre-created configuration files.
nntp-18.104.22.168: the source tree of the original NNTPd release, with header files pre-configured to fit in with our configuration of C-News, and our addition of bits and pieces like Nestor, the log analysis program.
howto: this document, and its SGML sources and Makefile.
build.sh: a shellscript you can run to compile the entire combined source tree and install binaries in the right places, if you are lucky and all goes well.
Needless to say, we believe that our source tree is a better place to start with than the original components, specially if you are installing a Usenet server on a Linux box and for the first time. We will be available on email to provide technical assistance should you run into trouble.
Once you get the sources, you will need some key configuration files to seed your C-News system. These configuration files are actually database tables, and are changing frequently, whenever newsgroups are created, modified or deleted. These files specify the list of active newsgroups in the ``public'' Usenet. You can, and should, add your organisation's internal newsgroups to this list when you set up your own server, but you will need to know the list of public standard newsgroups to begin with. This list can be obtained from the same FTP server by downloading the files active.gz and newsgroups.gz from ftp://ftp.uu.net/networking/news/config/. You can create your own active and newsgroups files by retaining a subset of the entries in these two files. Both these are ASCII text files.
Getting the sources from our server will not obviate the need to get the latest versions of these files from ftp.uu.net. We do not (yet) maintain an up-to-date copy of these files on our server, and we will add no value to the original by just mirroring them.
For installing, first make sure you have an entry for a user called news in your /etc/password file. This is setting the news-database owner to news. Now download the source from us and untar it in the home directory of news. This creates two main directories viz. c-news and nntp. To install and compile, run the script build.sh as root in the directory that contains the script. It is important that the script run as root as it sets ownerships, installs and compiles the source as user news. This is a one-step process that puts in place both the C-News and the NNTP software, setting correct permissions and paths. Following is a brief description of what build.sh does:
Checks for the OS platform and exits if it is not Linux.
Again, exits if you are not running as root.
Looks for and exits if cannot find the above two directories.
Compiles C-News and performs regression tests if the compilation was successfull. Sends out a warning to read the error file make.out.r and to fix 'em. Compilation erros are written to a file called make.out.
Performs the above operation in the nntp directory, too.
Checks for the presence of the three key directories: $NEWSARTS - (/var/spool/news) that houses the artciles, $NEWSCTL -(/var/lib/news) that contain configuration, log and status files and $NEWSBIN - (/usr/lib/newsbin) that contain binaries and executables for the working of the Usenet News system. Tries to create them if non-existent and exits if it results in failure.
Changes the ownership of these directories to news.news. This is important since the entire Usenet News System runs as user news. It will not function properly as any other user.
Then starts the installation process of C News. It runs make install to install binaries at the right locations; make setup to set the correct paths and umask, create directories for newsgroups, determine who will receive reports; make ui to set up inews and injnews and make readpostcheck to use readnews, postnews and checknews scripts provided by C News. The errors, if any are to be found in the respective make.out files. e.g. make.setup will write errors to make.out.setup
Newsspool, which queues incoming batches in $NEWSARTS/in.coming directory should run as set-userid and set-groupid. This is done.
A softlink is made to /var/lib/news from /usr/lib/news.
The NNTP software is installed.
Sets up the manpages for C News and makes it world readable. The NNTP manpages get installed when the software is installed. Compiles the C News documentation guide.ps and makes it readable and available in /usr/doc/packages/news or /usr/doc/news.
Checks for the PGP binary and asks the administrator to get it, if not found.
Once installed, you have to now configure the system to accept feeds and batch them for your neighbours. You will have to do the following:
nntpd: Copy the compiled nntpd into a directory where executables are kept and activate it. It runs on port 119 as a daemon through inetd unless you have compiled it as stand-alone. An entry in the /etc/services file for nntp would look like this:
nntp 119/tcp \# Network News Transfer Protocol
nntp stream tcp nowait news path-to-tcpd path-to-nntpd
Configuring control files: There are plenty of control files in $NEWSCTL that will need to be configured before you can start using the news system. The files mentioned here are also discussed in the first section of the section titled "Components of a running system>". These control files are dealt in detail in the following below.
sys: One line per system/NDN listing all the newsgroup hierarchies each system subscribes to. Each line is prefixed with the system name and the one beginning with
explist: This file has entries indicating which articles expire and when and whether they have to be archived. The order in which the newsgroups are listed is important. An example follows:
comp.lang.java.3d x 60 /var/spool/news/Archive
batchparms: sendbatches is a program that administers batched transmission of news articles to other sites. To do this it consults the batchparms file. Each line in the file specifies the behaviour for each of your NDN mentioned in the sys file. There are five fields for each site to be specified.
server u 100000 100 batcher | gzip -9 | viauux -d gunzip
The first field is the site name which matches the entry in the sys file and has a corresponding directory in $NEWSARTS/out.going by that name.
The second field is the class of the site,u for UUCP and n for NNTP feeds. A "!" in this field means that batching for this site has been disabled.
The third field is the size of batches to be prepared in bytes.
The fourth field is the maximum length of the output queue for transmission to that site.
The fifth field is the command line to be used to build, compress and transmit batches to that site. The contents of the togo file are made available on standard input.
controlperm: This file controls how the news system responds to control messages. Each line consists of 4-5 fields separated by white space. Control messages has been discussed in "Section 2.4>".
comp,sci firstname.lastname@example.org nrc pv news.announce.newsgroups
The first field is a newsgroup pattern to which the line applies.
The second field is either the keyword "any" or an e-mail address. The latter specifies that the line applies to control messages from only that author.
The third field is a set of opcode letters indicating what control operations need to be performed on messages emanating from the e-mail address mentioned in the second field. n stands for creating a newgroup, r stands for deleting a newsgroup and c stands for checkgroup.
The fourth field is a set of flag letters indicating how to respond to a control message that meets all the applicability tests:
y Do it. n Don't do it. v Report it and include the entire control message in the report. q Don't report it. p Do it iff the control message carries a valid PGP signature.
The fifth field, which is optional, will be used if the fourth field contains a p. It must contain the PGP key ID of the public key to be used for signature verification.
mailpaths: This file describes how to reach the moderators of various hierarchies of newsgroups by mail. Each line consists of two fields: a news group pattern and an e-mail address. The first line whose group pattern matches the newsgroup is used. As an example:
comp.lang.java.3d email@example.com all %firstname.lastname@example.org
Miscellaneous files: The other files to be modified are:
mailname: Contains the Internet domain name of the news system. Consider getting one if you don't have it.
organization: Contains the default value for the Organization: header for postings originating locally.
whoami: Contains the name of the news system. This is the site name used in the Path: headers and hence should concur with the names your neighbours use in their sys files.
active file: This file specifies one line for each newsgroup (not just the hierarchy) to be found on your news system. You will have to get the most recent copy of the active file from ftp://ftp.isc.org/usenet/CONFIG/active and prune it to delete newsgroups that you have not subscribed to. Run the script addgroup for each newsgroup in this file which will create relevant directories in the $NEWSARTS area. The addgroup script takes two paramters: the newsgroup name being created and a flag. The flag can be any one of the following:
y local postings are allowed n no local postings, only remote ones m postings to this group must be approved by the moderator j articles in this group are only passed and not kept x posting to this newsgroup is disallowed =foo.bar articles are locally filed in "foo.bar" group
comp.lang.java.3d 0000003716 01346 m
newsgroups file: This contains a one-line description of each newsgroup to be found in the active file. You will have to get the most recent file from ftp://ftp.isc.org/usenet/CONFIG/newsgroups and prune it to remove unwanted information. As an example:
comp.lang.java.3d 3D Graphics APIs for the Java language
Aliases: These aliases are required for trouble reporting. Once the system is in place and scripts are run, anomalies/problems are reported to addresses in the /etc/aliases file. These entries include email addresses for newsmaster, newscrisis, news, usenet, newsmap. They should ideally point to an email address that will be accessed at regularly. Arrange the emails for newsmap to be discarded to minimize the effect of sendsys bombing by practical jokers.
Cron jobs: Certain scripts like newsrun that picks up incoming batches and maintenance scripts, should run through news-database owner's cron which is news. The cron entries ideally will be for the following: A more detailed report can be found in "Section 9.4>"
newsrun: This script processes incoming batches of article. Run this as frequently as you want them to get digested.
sendbatches: This script transmit batches to the NDNs. Set the frequency according to your requirements.
newsdaily: This should be run ideally once a day since it reports errors and anomalies in the news system.
newswatch: This looks for errors/anomalies at a more detailed level and hence should be run atleast once every hour
doexpire: This script expires old articles as determined by the explist file. Run this once a day.
newslog: Make an entry in the system's syslog.conf file for logging messages spewed out by nntpd in newslog . It should be located in $NEWSCTL. The entry will look like this:
Newsboot: Have this run (as news the news-database owner) when the system boots to clear out debris left around by crashes.
Add a Usenet mailer in sendmail: The mail2news program provided as part of the source code is a handy tool to send an e-mail to a newsgroup which gets digested as an article. You will have to add the following ruleset and mailer definition in your sendmail.cf file:
Under SParse1, add the following:
R$+ . USENET < @ $=w . > $#usenet $: $1
Under mailer definitions, define the mailer Usenet as:
MUsenet P=/usr/lib/newsbin/mail2news/m2nmailer, F=lsDFMmn, S=10, R=0, M=2000000, T=X-Usenet/X-Usenet/X-Unix, A=m2nmailer $u
In order to send a mail to a newsgroup you will now have to suffix the newsgroup name with usenet i.e. your To: header will look like this:
This, more or less, completes the configuration part.
To locally test the system, follow the steps given below:
post an article: Create a local newsgroup
cnewsdo addgroup mysite.test y
Has it arrived in $NEWSARTS/in.coming?: The article should show up in the directory mentioned. Note the nomenclature of the article.
When newsrun runs: When newsrun runs from cron , the article disappears from in.coming directory and appears in $NEWSARTS/mysite/test. Look how the newsgroup, active, log and history (not the errorlog) files and .overview file in $NEWSARTS/mysite/test reflect the digestion of the file into the news system.
reading the article: Try to read the article through readnews or any news client. If you are able to, then you have set most everything right.
As mentioned in "Section 2.4>", it becomes necessary to authenticate control messages to protect yourself from being attacked by pranksters. For this, you will have to configure the $NEWSCTL/controlperm file to declare whose control messages you are willing to honour and for what newsgroups alongwith their public key ID. The controlperm manpage shall give you details on the format.
This will work only in association with pgpverify which verifies the Usenet control messages that have been signed using the signcontrol process. The script can be found at ftp://ftp.isc.org/pub/pgpcontrol/pgpverify. pgpverify internally uses the PGP binary which will have to be made available in the default executables directory. If you wish to send control messages for your local news system, you will have to digitally sign them using the above mentioned signcontrol program which is available at ftp://ftp.isc.org/pub/pgpcontrol/signcontrol. You will also have to configure the signcontrol program accordingly.
For external feeds, commercial customers will have to buy them from a regular News Provider like dejanews.com or newsfeeds.com. You will have to specify to them what hierarchies you want and decide on the mode of transmission, i.e. UUCP or NNTP, based on your requirements. Once that is done, you will have to ask them to initiate feeds, and check $NEWSARTS/in.coming directory to see if feeds are coming in.
If your organisation belongs to the academic community or is otherwise lucky enough to have an NDN server somewhere which is willing to provide you a free newsfeed, then the payment issue goes out of the picture, but the rest of the technical requirements remain the same.
One problem with incoming NNTP feeds is that it is far easier to use (relatively) efficient NNTP inflows if you have a server with a permanent Internet connection and a fixed IP address. If you are a small office with a dialup Internet connection, this may not be possible. In that case, the only way to get incoming newsfeeds by NNTP may be by using a highly inefficient pull feed.
If you are a leaf node, you will only have to send feeds back to your news provider for your postings in public newsgroups to propagate to the outside world. To enable this, you need one line in the sys and batchparms files and one directory in $NEWSARTS/out.going. If you are willing to transmit articles to your neighbouring sites, you will have to configure sys and batchparms with more entries. The number of directories in $NEWSARTS/out.going shall increase, too. Refer to first two sections of the chapter titled "Components of a running system>"for a better understanding of outgoing feeds. Again, you will have to determine how you wish to transmit the feed: UUCP or NNTP.
For outgoing feeds by UUCP, we recommend that you start with Taylor UUCP. In fact, this is the UUCP version which forms part of the GNU Project and is the default UUCP on Linux systems.
A full treatment of UUCP configuration is beyond the scope of this document. However, the basic steps will be as follows. First, you will have to define a "system" in your Usenet server for the NDN (next door neighbour) host. This definition will include various parameters, including the manner in which your server will call the remote server, the protocol it will use, etc. Then an identical process will have to be followed on the NDN server's UUCP configuration, for your server, so that that server can recognize your Usenet server.
Finally, you will need to set up appropriate cron jobs for the user uucp to run uucico periodically. Taylor UUCP comes with a script called uusched which may be modified to your requirements; this script calls uucico. One uucico connection will both upload and download news batches. Smaller sites can run uusched even once or twice a day.
Later versions of this document will include the uusched scripts that we use in Starcom. We use UUCP over TCP/IP, and we run the uucico connection through an SSH tunnel, to prevent transmission of UUCP passwords in plain text over the Internet, and our SSH tunnel is established using public-key cryptography, without passwords being used anywhere.
For NNTP feeds, you will have to decide whether your server will be the connection initiator or connection recipient. If you are the connection initiator, you can send outgoing NNTP feeds more easily. If you are the connection recipient, then outgoing feeds will have to be pulled out of your server using the NNTP NEWNEWS command, which will place heavy loads on your server. This is not recommended.
Connecting to your NDN server for pushing out outgoing feeds will require the use of the nntpsend.sh script, which is part of the NNTPd source tree. This script will perform some housekeeping, and internally call the nntpxmit binary to actually send the queued set of articles out. You may have to provide authentication information like usernames and passwords to nntpxmit to allow it to connect to your NDN server, in case that server insists on checking the identity of incoming connections. (You can't be too careful in today's world.) nntpsend.sh will clean up after an nntpxmit connection finishes, and will requeue any unsent articles for the next session. Thus, even if there is a network problem, typically nothing is lost and all pending articles are transmitted next time.
Thus, pushing feeds out via may mean setting up nntpsend.sh properly, and then invoking it periodically from cron. If your Usenet server connects to the Internet only intermittently, then the process which sets up the Internet connection should be extended or modified to fire nntpsend.sh whenever the Internet link is established. For instance, if you are using the Linux pppd, you can add statements to the /etc/ppp/ip-up script to change user to news and run nntpsend.sh