Products Downloads Prices Support Company
Index FAQ Configs Feeds In Feeds Out Feeds Out  

Guide to USENET News and DNews

The following is a brief guide to explain how Usenet news works. We suggest you read thru this in a linear fashion to gain a good basic understanding of News.

How UseNet News works?

The first question you should ask is how does news work, how does a news message get from your computer to every other news server in the world, and vice versa, how does a message get from somewhere in Japan to your machine?

Many people imagine the news servers are connected in a tree-like pattern, with the message going from your machine to the 'top' of the tree and then propagate down to all the other news servers.

This is entirely incorrect. Although simple in concept this would not work well in reality, and because the internet is more of an anarchy than an organized system so the news topology is much more random.

The real topology of news servers   

Note that systems are more or less randomly connected together, there is no ‘top’. This structure results in three conditions that a server can find itself in.

  1. Tails, machines connected to only one server
  2. Feeders, servers connected to 2 or more servers
  3. Loops, servers which are part of a loop, so the same message could arrive from two directions at once.

Imagine a message posted on “Goofy”. The message is stamped with a 'PATH' header which is added to as it passes thru every machine. If a host name is in the path then the server knows not to send it to that host again.

Here is what happens to the 'path' header as a message travels from machine to machine:

On Goofy: Path: Goofy

On Mickey: Path: Mickey:Goofy
On Mars: Path: Mars:Mickey:Goofy
On Pluto: Path: Pluto:Mars:Mickey:Goofy

On Venus: Path: Venus:Mars:Mickey:Goofy
On Sun: Path: Sun:Venus:Mars:Mickey:Goofy

It may sound dumb but a news server will use this logic even to decide about sending a message straight back to the host it just got the message from. And because machines have multiple names it's possible for this logic to fail, when this happens your news server will try and send every single message back up stream. (A waste of network bandwidth)

Above the message is about to be delivered to 'Earth' from two directions. Here even with the path header working correctly both machines will try and send to 'Earth' (because neither has ‘Earth’ in it’s path header). How does Earth know when it has received this message once? Enter the history database...

Each message is guaranteed to contain a unique message ID (The unique message ID is created by the news client when the message is generated and reflects the machine on which the message is generated and the time it is generated). The news server stores these unique message ID's in a database, and with every incoming message it first checks if it currently has that message ID before it stores the new message.

This database which could be 100MB’s, has to be very fast, DNews probably has the fastest history database of any news server.

Pseudo code for processing an incoming message

NNTP Protocol / commands

Network News Transfer Protocol (NNTP) is a protocol that enables news server and client software to "communicate" with each other to; distribute, inquire, retrieve, and post news articles. NNTP is designed so that news articles are stored in a central database allowing a subscriber to select those items he wishes to read. Indexing, cross-referencing, and expiration of aged messages are also provided.The description of the protocol is embodied in RFC's 977 and 1036 and maintained by the Internet Engineering Task Force (IETF), the standards body that maintains Internet protocols.

NNTP works with simple TEXT commands, using telnet you can run NNTP by hand. On some computers (NT 3.5) you need to type ^J after each command instead of just ‘enter’ There will be no ‘echo’ so you have to type carefully :-), here is a list of some of the NNTP commands you might use by hand:

200 161.29.2.2 DNews Version 2.4d, posting OK
help
mode reader
list
group rec.humor
article,head,body number|<message id>
ihave,post
 

CommandDescription
helpGives a list of valid nntp commands for this server
mode readerTells it you are a news reader, not a machine trying to send a feed in.
listGives a full list of news groups, this could take 20 minutes!
group x.ySets the current group, and shows you the number of articles and the first and last item in each group
article nnnShows you an entire article, you can specify it by item number or by message
ihaveOffers the server a news message, it responds with please send the message, or no thank you.
postOffers the server a NEW article.

NNTP example

telnet news.server.name 119
200 161.29.2.2 DNews Version 2.4d, posting OK
mode reader
200 DNews, Posting OK.
group rec.humor
211 299 101760 104058 rec.humor selected
article 104058

NNTP Protocol & extensions

Here are some important NNTP commands that you should understand:

ihave <message-ID>
check, takethis
post
xover
xhdr

When a news server wants to send a feed at your news server it would traditionally use the 'ihave' NNTP command. Here is how the traditional IHAVE protocol works:

Send: ihave <messageid>
Waitfor: Wanted or Not Wanted
Send Message (if wanted)

This protocol doesn’t allow streaming, as it is necessary to wait until the server responds before sending the message, or skipping.

The new NNTP commands 'check' and 'takethis' are basically the same as ihave, but the operation is split in half so that a feed can send 100 check’s and then send ‘takethis’ followed by the articles for the ones that the server wanted.

The 'XOVER' command was added to allow a directory of articles in one group to be shown without downloading the header information from every article in the group. Example:
xover 1200-1400
The server responds with 200 lines, each line contains the interesting header fields for one message separated by tabs.

A raw NNTP article

From: bn946@FreeNet.Carleton.CA (Les Griswold)
Newsgroups: soc.couples.intercultural,alt.politics.white-power,alt.fan.oj-simpson
Subject: Re: Beaver's pride
Date: 23 Apr 1996 09:06:26 GMT
Message-ID: <4li6ii$fj8@freenet-news.carleton.ca>
NNTP-Posting-Host: freenet3.carleton.ca

This is the message here
which I’ve actually just made up.

Problems

Traditional news servers have the following problems:

  • 20,000 News Groups
  • 128Kbps 24 hours / day
  • 146000 articles / day
  • INN, installation
  • INN, management
  • Disk Use
  • Memory Use

For many organizations these problems have made it either not practical to run a local news server or difficult to provide a good news service. DNews was developed to overcome these problems: DNews is easy to install, exceptionally stable and designed to run unattended. It was also designed to be very fast and much more efficient than earlier servers.

In addition to being able to take a full news feed several times faster than earlier news servers, DNews introduces the unique dynamic sucking feed option. In many enviroments where less than about 3000 groups are actively read the Dynamic Sucking Feed can provide substantial performance advantages and savings in network bandwidth and diskspace.

DNews Dynamic Sucking Feed Option

Characteristics
  • Only groups which users read are sucked.
  • All groups are available for the user to choose from (within your organizations policy) without having to ask
  • Standard NNTP ‘reader’ protocol is used, so a feed doesn’t have to be set up
  • Low impact, history lookups avoided, newnews not used

DNews dynamic sucking results

Advantages
1-10% of disk space used
1-10% of network bandwidth used
Messages can be stored much longer
Nightly expire run down from 24 hours to 5 minutes
Much easier to setup at both ends.