Setting up CNews + NNTPd

Usenet News HOWTO
Prev |  | Next

# 4. Setting up CNews + NNTPd

## 4.1. Getting the sources and stuff

### 4.1.1. The sources

C-News software can be obtained from `ftp://ftp.uu.net/networking/news/transport/cnews/cnews.tar.Z` and will need to be uncompressed using the BSD `uncompress` utility or a compatible program. The tarball is about 650 KBytes in size. It has its own highly intelligent configuration and installation processes, which are very well documented. The version that is available is Cleanup Release revision G, on which our own version is based.

NNTPd (the NNTP Reference Implementation) is available from `ftp://ftp.uu.net/networking/news/nntp/nntp.1.5.12.1.tar.Z`. It has no automatic scripts and processes to configure itself. After fetching the sources, you will have to follow a set of directions given in the documentation and configure some C header files. These configuration settings must be done keeping in mind what you have specified when you build the C-News sources, because NNTPd and C-News must work together. Therefore, some key file formats, directory paths, etc., will have to be specified identically in both software systems.

The third software system we use is Nestor. This too is to be found in the same place where the NNTPd software is kept, at `ftp://ftp.uu.net/networking/news/nntp/nestor.tar.Z`. This software compiles to one binary program, which must be run periodically to process the logs of `nntpd`, the NNTP server which is part of NNTPd, and report usage statistics to the administrator. We have integrated Nestor into our source base.

The fourth piece of the system, without which no Usenet server administrator dares venture out into the wild world of public Internet newsfeeds, is `pgpverify`.

We have been working with C-News and NNTPd for many years now, and have fixed a few bugs in both packages. We have also integrated the four software systems listed above, and added a few features here and there to make things work more smoothly. We offer our entire source base to anyone for free download from `http://www.starcomsoftware.com/proj/usenet/src/news.tar.gz`. There are no licensing restrictions on our sources; they are as freely redistributable as the original components we started with.

When you download our software distribution, you will extract it to find a directory tree with the following subdirectories and files:

- `c-news`: the source tree of the CR.G software release, with our additions like `pgpverify` integration, our scripts like `mail2news`, and pre-created configuration files.

- `nntp-1.5.12.1`: the source tree of the original NNTPd release, with header files pre-configured to fit in with our configuration of C-News, and our addition of bits and pieces like Nestor, the log analysis program.

- `howto`: this document, and its SGML sources and Makefile.

- `build.sh`: a shellscript you can run to compile the entire combined source tree and install binaries in the right places, if you are lucky and all goes well.

Needless to say, we believe that our source tree is a better place to start with than the original components, specially if you are installing a Usenet server on a Linux box and for the first time. We will be available on email to provide technical assistance should you run into trouble.

### 4.1.2. The key configuration files

Once you get the sources, you will need some key configuration files to seed your C-News system. These configuration files are actually database tables, and are changing frequently, whenever newsgroups are created, modified or deleted. These files specify the list of active newsgroups in the ``public'' Usenet. You can, and should, add your organisation's internal newsgroups to this list when you set up your own server, but you will need to know the list of public standard newsgroups to begin with. This list can be obtained from the same FTP server by downloading the files `active.gz` and `newsgroups.gz` from `ftp://ftp.uu.net/networking/news/config/`. You can create your own `active` and `newsgroups` files by retaining a subset of the entries in these two files. Both these are ASCII text files.

Getting the sources from our server will not obviate the need to get the latest versions of these files from `ftp.uu.net`. We do not (yet) maintain an up-to-date copy of these files on our server, and we will add no value to the original by just mirroring them.

## 4.2. Compiling and installing

For installing, first make sure you have an entry for a user called `news` in your `/etc/password` file. This is setting the news-database owner to `news`. Now download the source from us and untar it in the home directory of news. This creates two main directories viz. `c-news` and `nntp`. To install and compile, run the script `build.sh` as root in the directory that contains the script. It is important that the script run as `root` as it sets ownerships, installs and compiles the source as user `news`. This is a one-step process that puts in place both the C-News and the NNTP software, setting correct permissions and paths. Following is a brief description of what build.sh does:

- Checks for the `OS` platform and exits if it is not `Linux`.

- Again, exits if you are not running as `root`.

- Looks for and exits if cannot find the above two directories.

- Compiles `C-News` and performs regression tests if the compilation was successfull. Sends out a warning to read the error file `make.out.r` and to fix 'em. Compilation erros are written to a file called `make.out`.

- Performs the above operation in the `nntp` directory, too.

- Checks for the presence of the three key directories: `$NEWSARTS - (/var/spool/news)` that houses the artciles, `$NEWSCTL -(/var/lib/news)` that contain configuration, log and status files and `$NEWSBIN - (/usr/lib/newsbin)` that contain binaries and executables for the working of the Usenet News system. Tries to create them if non-existent and exits if it results in failure.

- Changes the ownership of these directories to `news.news`. This is important since the entire Usenet News System runs as user `news.` It will not function properly as any other user.

- Then starts the installation process of C News. It runs `make install` to install binaries at the right locations; `make setup` to set the correct paths and umask, create directories for newsgroups, determine who will receive reports; `make ui` to set up inews and injnews and `make readpostcheck` to use readnews, postnews and checknews scripts provided by C News. The errors, if any are to be found in the respective `make.out` files. e.g. make.setup will write errors to `make.out.setup`

- `Newsspool`, which queues incoming batches in `$NEWSARTS/in.coming` directory should run as set-userid and set-groupid. This is done.

- A softlink is made to `/var/lib/news` from `/usr/lib/news.`

- The NNTP software is installed.

- Sets up the manpages for C News and makes it world readable. The NNTP manpages get installed when the software is installed. Compiles the C News documentation `guide.ps` and makes it readable and available in `/usr/doc/packages/news` or `/usr/doc/news`.

- Checks for the PGP binary and asks the administrator to get it, if not found.

## 4.3. Configuring the system: What and how to configure files?

Once installed, you have to now configure the system to accept feeds and batch them for your neighbours. You will have to do the following:

- `nntpd`: Copy the compiled `nntpd` into a directory where executables are kept and activate it. It runs on port 119 as a daemon through `inetd` unless you have compiled it as stand-alone. An entry in the `/etc/services` file for nntp would look like this:

      nntp	119/tcp    \# Network News Transfer Protocol

  An entry in the `inetd.conf` file will be:

       nntp    stream    tcp    nowait   news    path-to-tcpd  path-to-nntpd

  The last two fields in the `inetd.conf` file are paths to binaries of the `tcp` and the `nntp` daemon respectively.

- Configuring control files: There are plenty of control files in `$NEWSCTL` that will need to be configured before you can start using the news system. The files mentioned here are also discussed in the first section of the section titled "Components of a running system>". These control files are dealt in detail in the following below.

  - `sys`: One line per system/NDN listing all the newsgroup hierarchies each system subscribes to. Each line is prefixed with the system name and the one beginning with

        ME:

    indicates what your server is willing to receive. Following are typical entries that go into this file:

        ME:comp,news,misc,netscape

    This line indicates what newsgroups your server has subscribed to.

        server/server.starcomsoftware.com:all,!general/all:f

    This is a list of newsgroups your server will pass on to your NDN. The newsgroups specified should be a comma separated list and the entire line should contain no spaces. The f flag indicates that the newsgroup name and the article number alongwith its size will make up one entry in the `togo` file in the `$NEWSARTS/out.going` directory.

  - `explist`: This file has entries indicating which articles expire and when and whether they have to be archived. The order in which the newsgroups are listed is important. An example follows:

        comp.lang.java.3d    x    60    /var/spool/news/Archive

    This means that the articles of comp.lang.java expire after 60 days and shall be archived in the directory mentioned in the fourth field. Archiving is an option. The second field indicates that this line applies to both moderated and unmoderted newsgroups. m would specify moderated and u would specify unmoderated groups. If you want to specify an extremely large no. as the expiry period you can use the keyword "never".

  - `batchparms`: `sendbatches` is a program that administers batched transmission of news articles to other sites. To do this it consults the `batchparms` file. Each line in the file specifies the behaviour for each of your NDN mentioned in the `sys` file. There are five fields for each site to be specified.

         server   u     100000    100    batcher | gzip -9 | viauux -d gunzip

    The first field is the site name which matches the entry in the `sys` file and has a corresponding directory in `$NEWSARTS/out.going` by that name.

    The second field is the class of the site,u for UUCP and n for NNTP feeds. A "!" in this field means that batching for this site has been disabled.

    The third field is the size of batches to be prepared in bytes.

    The fourth field is the maximum length of the output queue for transmission to that site.

    The fifth field is the command line to be used to build, compress and transmit batches to that site. The contents of the `togo` file are made available on standard input.

  - `controlperm`: This file controls how the news system responds to control messages. Each line consists of 4-5 fields separated by white space. Control messages has been discussed in "Section 2.4>".

        comp,sci    tale@uunet.uu.net   nrc    pv   news.announce.newsgroups

    The first field is a newsgroup pattern to which the line applies.

    The second field is either the keyword "any" or an e-mail address. The latter specifies that the line applies to control messages from only that author.

    The third field is a set of opcode letters indicating what control operations need to be performed on messages emanating from the e-mail address mentioned in the second field. n stands for creating a newgroup, r stands for deleting a newsgroup and c stands for checkgroup.

    The fourth field is a set of flag letters indicating how to respond to a control message that meets all the applicability tests:

        	     y 	Do it.
        	     n	Don't do it.
        	     v 	Report it and include the entire control
        	        message in the report.
        	     q 	Don't report it.
        	     p	Do it iff the control message carries a valid PGP signature.

    Exactly one of y, n or p must be present.

    The fifth field, which is optional, will be used if the fourth field contains a p. It must contain the PGP key ID of the public key to be used for signature verification.

  - `mailpaths`: This file describes how to reach the moderators of various hierarchies of newsgroups by mail. Each line consists of two fields: a news group pattern and an e-mail address. The first line whose group pattern matches the newsgroup is used. As an example:

        	   comp.lang.java.3d		somebody@mydomain.com
        	   all				%s@moderators.uu.net

    In the second example, the `%s` gets replaced with the groupname and all dots appearing in the newsgroup name are substituted with dashes.

  - Miscellaneous files: The other files to be modified are:

    - `mailname:` Contains the Internet domain name of the news system. Consider getting one if you don't have it.

    - `organization:` Contains the default value for the `Organization:` header for postings originating locally.

    - `whoami:` Contains the name of the news system. This is the site name used in the `Path:` headers and hence should concur with the names your neighbours use in their `sys` files.

  - `active` file: This file specifies one line for each newsgroup (not just the hierarchy) to be found on your news system. You will have to get the most recent copy of the active file from `ftp://ftp.isc.org/usenet/CONFIG/active` and prune it to delete newsgroups that you have not subscribed to. Run the script `addgroup` for each newsgroup in this file which will create relevant directories in the `$NEWSARTS` area. The `addgroup` script takes two paramters: the newsgroup name being created and a flag. The flag can be any one of the following:

        	    y		local postings are allowed
        	    n 		no local postings, only remote ones
        	    m		postings to this group must be approved
        	                by the moderator
        	    j		articles in this group are only passed and not kept
        	    x		posting to this newsgroup is disallowed
        	    =foo.bar	articles are locally filed in
        	                "foo.bar" group

    An entry in this file looks like this:

        comp.lang.java.3d	0000003716	01346	m

    The first field is the name of the newsgroup. The second field is the highest article number that has been used in that newsgroup. The third field is the lowest article number in the group. The fourth field is a flag as explained above.

  - `newsgroups` file: This contains a one-line description of each newsgroup to be found in the active file. You will have to get the most recent file from `ftp://ftp.isc.org/usenet/CONFIG/newsgroups` and prune it to remove unwanted information. As an example:

        comp.lang.java.3d 	3D Graphics APIs for the Java language

  - Aliases: These aliases are required for trouble reporting. Once the system is in place and scripts are run, anomalies/problems are reported to addresses in the `/etc/aliases` file. These entries include email addresses for `newsmaster, newscrisis, news, usenet, newsmap`. They should ideally point to an email address that will be accessed at regularly. Arrange the emails for `newsmap` to be discarded to minimize the effect of `sendsys bombing` by practical jokers.

  - Cron jobs: Certain scripts like `newsrun` that picks up incoming batches and maintenance scripts, should run through news-database owner's cron which is `news`. The cron entries ideally will be for the following: A more detailed report can be found in "Section 9.4>"

    1. `newsrun:` This script processes incoming batches of article. Run this as frequently as you want them to get digested.

    2. `sendbatches:` This script transmit batches to the NDNs. Set the frequency according to your requirements.

    3. `newsdaily:` This should be run ideally once a day since it reports errors and anomalies in the news system.

    4. `newswatch:` This looks for errors/anomalies at a more detailed level and hence should be run atleast once every hour

    5. `doexpire:` This script expires old articles as determined by the explist file. Run this once a day.

  - `newslog:` Make an entry in the system's `syslog.conf` file for logging messages spewed out by `nntpd` in `newslog` . It should be located in `$NEWSCTL`. The entry will look like this:

        news.debug		-/var/lib/news/newslog

  - `Newsboot:` Have this run (as `news` the news-database owner) when the system boots to clear out debris left around by crashes.

  - Add a Usenet mailer in sendmail: The `mail2news` program provided as part of the source code is a handy tool to send an e-mail to a newsgroup which gets digested as an article. You will have to add the following ruleset and mailer definition in your `sendmail.cf` file:

    - Under SParse1, add the following:

          	    R$+ . USENET < @ $=w . >      $#usenet     $: $1

    - Under mailer definitions, define the mailer Usenet as:

          	    MUsenet 	 P=/usr/lib/newsbin/mail2news/m2nmailer, F=lsDFMmn,
          		S=10, R=0, M=2000000, T=X-Usenet/X-Usenet/X-Unix, A=m2nmailer $u

    In order to send a mail to a newsgroup you will now have to suffix the newsgroup name with usenet i.e. your `To:` header will look like this:

        To: misc.test.usenet@yourdomain.

    The mailer definition of usenet will intercept this mail and post it to the respective newsgroup, in this case, `misc.test`

  This, more or less, completes the configuration part.

## 4.4. Testing the system

To locally test the system, follow the steps given below:

- post an article: Create a local newsgroup

          cnewsdo addgroup mysite.test y

  and using `postnews` post an article to it.

- Has it arrived in `$NEWSARTS/in.coming`?: The article should show up in the directory mentioned. Note the nomenclature of the article.

- When newsrun runs: When newsrun runs from `cron` , the article disappears from `in.coming` directory and appears in `$NEWSARTS/mysite/test`. Look how the `newsgroup, active, log and history` (not the errorlog) files and `.overview` file in `$NEWSARTS/mysite/test` reflect the digestion of the file into the news system.

- reading the article: Try to read the article through `readnews` or any news client. If you are able to, then you have set most everything right.

## 4.5. `pgpverify` and `controlperms`

As mentioned in "Section 2.4>", it becomes necessary to authenticate control messages to protect yourself from being attacked by pranksters. For this, you will have to configure the `$NEWSCTL/controlperm` file to declare whose control messages you are willing to honour and for what newsgroups alongwith their public key ID. The `controlperm` manpage shall give you details on the format.

This will work only in association with `pgpverify` which verifies the Usenet control messages that have been signed using the `signcontrol` process. The script can be found at `ftp://ftp.isc.org/pub/pgpcontrol/pgpverify`. `pgpverify` internally uses the PGP binary which will have to be made available in the default executables directory. If you wish to send control messages for your local news system, you will have to digitally sign them using the above mentioned `signcontrol` program which is available at `ftp://ftp.isc.org/pub/pgpcontrol/signcontrol`. You will also have to configure the `signcontrol` program accordingly.

## 4.6. Feeding off an upstream neighbour

For external feeds, commercial customers will have to buy them from a regular News Provider like `dejanews.com` or `newsfeeds.com`. You will have to specify to them what hierarchies you want and decide on the mode of transmission, i.e. UUCP or NNTP, based on your requirements. Once that is done, you will have to ask them to initiate feeds, and check `$NEWSARTS/in.coming` directory to see if feeds are coming in.

If your organisation belongs to the academic community or is otherwise lucky enough to have an NDN server somewhere which is willing to provide you a free newsfeed, then the payment issue goes out of the picture, but the rest of the technical requirements remain the same.

One problem with incoming NNTP feeds is that it is far easier to use (relatively) efficient NNTP inflows if you have a server with a permanent Internet connection and a fixed IP address. If you are a small office with a dialup Internet connection, this may not be possible. In that case, the only way to get incoming newsfeeds by NNTP may be by using a highly inefficient pull feed.

## 4.7. Configuring outgoing feeds

If you are a leaf node, you will only have to send feeds back to your news provider for your postings in public newsgroups to propagate to the outside world. To enable this, you need one line in the `sys` and `batchparms` files and one directory in `$NEWSARTS/out.going`. If you are willing to transmit articles to your neighbouring sites, you will have to configure `sys` and `batchparms` with more entries. The number of directories in `$NEWSARTS/out.going` shall increase, too. Refer to first two sections of the chapter titled "Components of a running system>"for a better understanding of outgoing feeds. Again, you will have to determine how you wish to transmit the feed: UUCP or NNTP.

### 4.7.1. By UUCP

For outgoing feeds by UUCP, we recommend that you start with Taylor UUCP. In fact, this is the UUCP version which forms part of the GNU Project and is the default UUCP on Linux systems.

A full treatment of UUCP configuration is beyond the scope of this document. However, the basic steps will be as follows. First, you will have to define a "system" in your Usenet server for the NDN (next door neighbour) host. This definition will include various parameters, including the manner in which your server will call the remote server, the protocol it will use, etc. Then an identical process will have to be followed on the NDN server's UUCP configuration, for your server, so that that server can recognize your Usenet server.

Finally, you will need to set up appropriate `cron` jobs for the user `uucp` to run `uucico` periodically. Taylor UUCP comes with a script called `uusched` which may be modified to your requirements; this script calls `uucico`. One `uucico` connection will both upload and download news batches. Smaller sites can run `uusched` even once or twice a day.

Later versions of this document will include the `uusched` scripts that we use in Starcom. We use UUCP over TCP/IP, and we run the `uucico` connection through an SSH tunnel, to prevent transmission of UUCP passwords in plain text over the Internet, and our SSH tunnel is established using public-key cryptography, without passwords being used anywhere.

### 4.7.2. By NNTP

For NNTP feeds, you will have to decide whether your server will be the connection initiator or connection recipient. If you are the connection initiator, you can send outgoing NNTP feeds more easily. If you are the connection recipient, then outgoing feeds will have to be pulled out of your server using the NNTP `NEWNEWS` command, which will place heavy loads on your server. This is not recommended.

Connecting to your NDN server for pushing out outgoing feeds will require the use of the `nntpsend.sh` script, which is part of the NNTPd source tree. This script will perform some housekeeping, and internally call the `nntpxmit` binary to actually send the queued set of articles out. You may have to provide authentication information like usernames and passwords to `nntpxmit` to allow it to connect to your NDN server, in case that server insists on checking the identity of incoming connections. (You can't be too careful in today's world.) `nntpsend.sh` will clean up after an `nntpxmit` connection finishes, and will requeue any unsent articles for the next session. Thus, even if there is a network problem, typically nothing is lost and all pending articles are transmitted next time.

Thus, pushing feeds out via may mean setting up `nntpsend.sh` properly, and then invoking it periodically from `cron`. If your Usenet server connects to the Internet only intermittently, then the process which sets up the Internet connection should be extended or modified to fire `nntpsend.sh` whenever the Internet link is established. For instance, if you are using the Linux `pppd`, you can add statements to the `/etc/ppp/ip-up` script to change user to `news` and run `nntpsend.sh`

Prev | Home | Next
Usenet news software |  | Setting up INN