RumorMill 1.3 Advanced Commands
Copyright © 1996-2002, Jim Calvin.
Updated to reflect RumorMill version 1.3.4
This documentation describes, in great detail, the RumorMill NNTP server. This documentation is not an introduction or overview text. This text is intended for users who require additional insight into how RumorMill functions, or those users wishing to configure RumorMill without using the RumorMill Setup application.
Detailed information Thanks!
RumorMill has support for authentication. While configuring users is best done from RumorMill Setup, here are the details on setting up user names and passwords for authentication.
You can set a preference called Require Login. When this is set, clients can connect (assuming they aren't blocked for other reasons, like Site Restrictions), but cannot get group lists until they have issued:authinfo user <username> authinfo pass <password>(Same format for authinfo as INN.)
The commands:XAUTHINFO ADD <username> READONLY | POSTOK <read user-group index> <post user-group index> <password> XAUTHINFO DELETE <username> XAUTHINFO LIST XAUTHINFO MODIFY <username> READONLY | POSTOK <read user-group index> <post user-group index> <password>... are available as soon as you issue XPASS, and their function is described elsewhere in this document.
MT-Newswatcher and YA-Newswatcher were used to try out the authentication stuff and it worked well with 'Always Authenticate'.
Below are three examples of change RumorMill preferences. A full listing of RumorMill preferences appears below.
xpref set 389 Send preference data; end with <crlf>.<crlf> Article Expiration Period 12 . 288 Preference updated xpref set 389 Send preference data; end with <crlf>.<crlf> Flood Interval 5 .The exceptions, which must also be retrieved specially are the Site Restrictions Masks, the list of Newsfeeds and the list of Newsgroups.
Get:xpref get Site Address Restrictions 288 Preference data follows Name:Site Address Restrictions Type:restrictions LogicalGroup:1 Description:Site address restrictions Default:203.8.112.0 255.255.255.0 Value:!203.8.112.12 255.255.255.255 Value:203.8.112.0 255.255.255.0 Value:130.95.156.0 255.255.255.0 .Set:xpref set 389 Send preference data; end with <crlf>.<crlf> Site Address Restrictions !203.8.112.12 255.255.255.255 203.8.112.0 255.255.255.0 130.95.156.0 255.255.255.0 . 288 Preference updatedResults and masks are separated by tabs and each entry should be on a separate line.
Get :xpref get Remote Servers 288 Preference data follows Name:Remote Servers Type:rmsv LogicalGroup:1 Description:Remote Server description record hostName:zany.com hostType:DownStream Pull groupsToSend:* groupsToExclude: groupsToPull:* authinfo: pullPeriod:0 floodPeriod:0 .Set :xpref set 389 Send preference data; end with <crlf>.<crlf> Remote Servers zimble.bunny.org zimblebunnyhostpathalias Pull DownStream *,mixed,small *,mixed,small 0 0 . 288 Preference updatedhost Name: should be a full host name or IP address. An optional host path alias may follow the host name, but it must be separated from the host name by a tab. This is useful if an upstream (or pull) newsfeed inserts something other than the hostname into the "Path:" header.
host Types: (UpStream or Pull) and/or DownStream.
groupsTo: (Send/Exclude /Pull): This is a list of groups to Send, Exclude or Pull from this host. You can use *, ? and ! to pattern match. (See the RumorMill Documentation for more explanation of this pattern matching.) Groups in a list should be comma separated. Note that blank lines are significant when setting groups.
It is possible to provide a user name and password for authentication for remote servers. This is particularly useful when pulling articles from a server that requires authentication.
In addition, one can specify the pull and flood intervals for each remote server. If these intervals are not specified (or set to zero), the defaults ("Pull interval" and "Flood Interval") are used.
Removing a host:xpref set 389 Send preference data; end with <crlf>.<crlf> Remote Servers zimble.bunny.org . 288 Preference updated
RumorMill has support for moderated groups. A group can be labeled as moderated when you create it simply by appending a trailing ' m' e.g.:
xaddlist 340 send groups followed by <CR-LF>.<CR-LF> my.moderated.group m . 241 groups added list 215 list of newsgroups follows my.moderated.group 0 0 m .Normally when a newsgroup is moderated no one can post to it unless there is an Approved: field in the article. If someone posts and there is not an approved field, the newsserver e-mails the article to the moderator (via the EMail submission address).
The EMail submission address can be set using for a group as follows.
xgroup set moderator my.moderated.group mmg@email.mycom.com 211 group moderator setFor more details, see the text on setting/getting a group's moderator.
Remember that the SMTP host preference must be set to point to an SMTP server in order for articles to be sent to the moderator. Queued mail is kept in the "Queued EMail" folder within the "Databases" folder.
Groups can be managed remotely. The following information describes the basics for managing news groups with RumorMill.
The news group list
list 215 list of newsgroups follows alt.answers 0 0 y alt.ascii-art 0 0 y alt.binaries.pictures.anime 0 0 y .Adding groups via XADDLIST
xaddlist 340 send groups followed by <CR-LF>.<CR-LF> alt.answers alt.ascii-art alt.binaries.pictures.anime sci.physics.accelerators m sci.physics.particle m sci.physics.research m . 241 groups addedRemoving groups via XRMGROUP
xrmgroup alt.ascii-art 281 Group deletedCreate Groups and Save Groups menu commands
There are two menu commands in RumorMill that also deal with maintaining news groups. These are the Create Groups and Save Groups commands. Create Groups reads a file of group data (as written by Save Groups, or an ASCII text file of group names, one per line. When Create Groups is processing a file, any group name appearing in the file that already exists as a group in the database is ignored.
Save Groups writes a text file in a particular format that allows Create Groups to restore all information about a group, including:
- Moderated flag
- Moderator EMail submission address
- The current article number indices (high and low water marks)
- EMail lists (news -> EMail gateway)
- Group description
RumorMill can act as a simple News to EMail and EMail to News gateway. To configure the News -> EMail half of the gateway, use the XGROUP SET EXPLODER command to indicate the EMail list for each group that you would like forwarded via EMail. A user defined header may also be added to outgoing EMail articles. (Note that the SMTP Host and Sys AdminEMail Addr preferences must also be configured properly.)
To configure the EMail -> News half of the gateway, one must configure RumorMill with the following preferences set to valid values:Optionally, the EMail Map preference may also be necessary if you are receiving EMail that does not have newsgroups specified as a header.
When the POP parameters are properly configured, and the EMail Add Reply-To preference is enabled, the News -> EMail part of the gateway can automatically add a header such as:Reply-To: <POP Account>@<POP Host>One can set the period at which RumorMill will check for incoming EMail.
The News <-> EMail gateway feature has some limitations. They include:
- Forwarding articles via EMail may require additional RAM, especially if large articles are being EMailed.
- Articles that are heavily cross-posted may not send EMail to all EMail lists. There is currently a limit of 10 EMail lists per article.
- RumorMill attempts to truncate EMail loops (articles posted into the database, EMailed, received via EMail, reposted into the database with a new message-id, EMailed, ...). It does this by adding two headers to EMailed articles, "X-NNTP-Message-ID" and "X-Comment". Some EMail clients may delete these headers. If that happens, and you have a single POP account that receives replies and is also on a group exploder list, you will likely encounter EMail looping.
- RumorMill attempts to send as few EMails as possible per posted article. Usually it is one EMail per posted article. However, when an article is heavily cross-posted (and each group has an EMail list), multiple EMailings per article can occur.
Virtual groups is a new capability that allows a system administrator to configure RumorMill to have thousands of groups with only a subset of those groups active. This feature provides the capability of appearing to support many groups while maintaining articles for only those in active use.
When the virtual group capability is enabled (see the Enable Virtual Groups preference), groups are either active or virtual. Only active groups will pull or accept new articles from a news feed. Groups which are virtual may have articles in them, but no new articles are added while the group is virtual.
Groups transition from virtual to active and back based on references to each group. Every time a client attempts to look at a group (the GROUP command is issued), a counter associated with the group is incremented. When that counter equals or exceeds the value of the Virtual Group Threshold preference, the group becomes active. Each evening RumorMill will decrement the counter associated with the group. If the counter drops below the Virtual Group Threshold, the group will become virtual.
The system administrator may also mark selected groups as permanently active. This can be done using the XGROUP SET PERM command described elsewhere.
The XINITVIRTUAL command may also be used by the system administrator to automatically create a large number of groups. Since there is no way to remove large numbers of groups, this command should be used with great care, and only after the Virtual Group Filter preference is setup.
Please note: The RumorMill Setup application has not yet been updated to support virtual groups. Also note that the Setup application will not support more than about 1,300 groups in the database.
In some cases, system administrators may wish to user a finer level of control over which users may read from and post to groups. Requiring login for any use, or to be able to post provides one level of control.
Version 1.3 of RumorMill provides capabilities to limit which groups users may read from and/or post to. The sysadmin may create a set of "User Group descriptions." Each user group simply contains the following:A unique user-group index (small integer) A name; provided for the user and is not used by RumorMill A list of group names, e.g., "comp.sys.mac.*,rec.bicycle.*" A flag saying whether the specified group list is to be EXCLUDEd or PERMITtedEach user has two such associated descriptions, one for read limitations, and one for posting limitations. So if a user's read limitations were set to "permit" "local.*", then that reader could only read groups that began with "local."
However, if the user's group description was set to "exclude" "local.*," then the user would be able to read all news groups other than those starting with "local." The posting limitations work exactly the same way.
Users not logged in (authenticated) are automatically defaulted to user group description with an index 0 (zero) for reading and 1 (one) for posting. By using these "special" user group descriptions, guest, or non-authenticated users can also be restricted.
User groups can be accessed using the "User Groups" and "User" dialogs in the Setup Application
The following commands (via Telnet) can be used to configure user group descriptions.
XUSERGROUP ADD <user-group index> EXCLUDE | PERMIT <user-group name><cr> <group list> XUSERGROUP DELETE <user-group index> XUSERGROUP LIST returns the following for each user-group <user-group index> PERMIT | EXCLUDE <user-group name> <group list><cr> The list ends with <cr>.<cr>The commands for assigning these user-groups to users are located elsewhere in this document.
There may be times, such as recovering from a mis-configuration, that the system administrator will want to add articles to the database from articles already on the local disk. RumorMill has a feature to allow this.
Periodically (see this preference) RumorMill will scan the folder "Articles to Inject" for visible files. Any visible file found in that folder will be opened and treated as a single article. The article will be subjected to the standard checks for incoming articles and if the article is valid and newsgroups match, it will be inserted into the local database. The article will be forewarded to remote servers as prescribed by preference settings.
After each file is processed, it is moved to the trash can.
RumorMill has a number of features which are not currently accessible from RumorMillSetup. These features can be accessed via a telnet application such as Nifty Telnet.
To connect, open your telnet application and telnet to your host on port 119. In Nifty Telnet enter:
my.host.computer:119When you connect there are three modes you can be in: normal client, unauthorized client or administrator. When Require Login is true (non-zero) a client which connects and has not yet issued the AUTHINFO instructions can only issue the following commands:
AUTHINFO USER <username> | PASS <password> HELP NOOP QUIT VERSION XPASS <password>An authorized client can use the following commands:
ARTICLE [MessageID | Number] AUTHINFO USER <username> | PASS <password> BODY [MessageID | Number] DATE HEAD [MessageID | Number] GROUP newsgroup HELP LAST LIST [ACTIVE [wildmat] | EXTENSIONS | GROUPDATA [wildmat] | JUSTGROUPS | NEWSGROUPS [wildmat] | OVERVIEW.FMT | SUBSCRIPTIONS | MOTD] LISTGROUP [newsgroup] MODE READER NEWGROUPS [yy]yymmdd hhmmss ["GMT"] NEWNEWS newsgroups [yy]yymmdd hhmmss ["GMT"] NEXT NOOP PAT header range | MessageID [pat] POST QUIT SLAVE STAT [MessageID | Number] VERSION XGTITLE [wildmat] XHDR header [range | MessageID] XOVER | OVER [range] XROVER [range] XPASS <password> XPAT header range | MessageID [pat]Most of these commands are used by newsreaders in the course of reading articles. Special commands that are used to configure RumorMill are listed below.
There are a set of commands that are only available to clients that are configured at peers, that is as a site that can upload articles to your server. Of the command listed below, CHECKTHIS and TAKETHIS are only available if you allow streaming and the client has issued the MODE STREAM command.
CHECKTHIS <Message ID> IHAVE <Message ID> MODE READER | STREAM TAKETHIS <Message ID>If you issue the command:
xpass <adminpassword>...then you can switch into the administration mode and the administration commands will be available. (Note, the admin password can be configured in the Security window of RumorMill Setup. You do not need to issue the admin password if you are connecting to a copy RumorMill which is running on the same from which you are telneting.) The additional Administration commands you can use are:
XADDLIST <group names> XDELETEARTICLE <message ID> | [range] XDELOLDARTICLEFILE XAUTHINFO ADD <username> READONLY | POSTOK <read user-group index> <post user-group index> <password> XAUTHINFO DELETE <username> XAUTHINFO LIST XAUTHINFO MODIFY <username> READONLY | POSTOK <new password> <read user-group index> <post user-group index> <new password> XCOMPACTDB <db flags> XCTL EXECUTE <message-id> XCTL FORGET <message-id> XCTL LIST XGROUP GET DESCRIPTION <group name> XGROUP SET DESCRIPTION <group name> <description> XGROUP GET EXPIRY <group name> XGROUP SET EXPIRY <group name> <expiration period in days> XGROUP GET EXPLODER <group name> XGROUP SET EXPLODER <group name> <EMail address(es) to mail postings> XGROUP GET MODERATOR <group name> XGROUP SET MODERATOR <group name> <Group submission EMail address> XKILL connection_index | -1 XMAX XPREF LIST XPREF GET <preference name> XPREF SET XPREF PREFGRP XPULL XRMGROUP <group> XSTATS XLOG XUSERGROUP ADD <user-group inde> EXCLUDE | PERMIT <name><cr><group list><cr>.<cr> XUSERGROUP DELETE <user-group index> XUSERGROUP LIST XMAKEFWD <remote host name> QUITNOWEach of these are explained below.
To restrict access beyond simple Site Restrictions Masks you can require users to have passwords by setting Require Login to non-zero. See Require Login for a description of the valid settings for this preference. Use XPREF SET as described above to change the setting. When Users connect they will not be able to access group lists and retrieve commands until they have issued authinfo commands:
xpref set 389 Send preference data; end with <crlf>.<crlf> Require Login 1 . 288 Preference updated
This command causes to RumorMill to unconditionally exit. No effort is made to gracefully disconnect connections or complete in-progress operations before exiting.
This command simply adds users that are valid for the AUTHINFO command. The READONLY or POSTOK token states whether the user may post articles (at all). The user-group indices specify the set of groups this user may read from and post to.xauthinfo add jem readonly 0 1 jempass 281 User added authinfo user jem 381 User name accepted, follow with PASS authinfo pass jempass 281 User authenticated
Deletes existing users from the system.xauthinfo delete jem 281 User deleted
Lists the set of valid users in the system. Note that the passwords are returned in a scrambled form.
Use MODIFY to change either the password or posting capability for an existing user. If you don't wish to change the password, send the password received via XAUTHINFO LIST and pre-pend a chr(1) on the front of the password.
xauthinfo modify jem readonly 0 1 jemnewpass 281 User password changed
These commands are used to list and manage control articles that create and delete newsgroups. Disposition of control articles is managed by the preferences Admin OKs Group Deletes and Admin OKs New Groups.
To process the list, send XCTL LIST and hang on to the reply. Each line is the article ID of an article that has a control field in it. You can then send ARTICLE <id> to retrieve the article. If it isn't returned, send XCTL FORGET <id>. If it is returned, read the article (it's supposed to have the reasons etc. for creating/deleting the group). Then issue:
XCTL EXECUTE <id> to do the add/delete, or
XCTL FORGET <id> to remove the article from the queue of deferred msgs without processing the control header of the article.
Also, if the EMail Control Msgs preference is set to non-zero, the system administrator will receive an EMail for each such deferred or processed article. The message-ID can be copied from these messages and used in the XCTL EXECUTE or XCTL FORGET commands. (Remember that the SMTP Host and Sys Admin EMail Addr must also be set for this feature to work.)
The xlog command toggles logging to the telnet connection.
xlog 285 Logging to this connection 1997/ 6/28 15:20:30 [3] Pulling from 128.101.83.4 1997/ 6/28 15:20:32 [3] Pull completed, msgs = 0, rejects = 0 1997/ 6/28 15:23:39 [4] Pulling from 128.101.83.4 1997/ 6/28 15:23:41 [4] Pull completed, msgs = 0, rejects = 0 1997/ 6/28 15:24:27 [5] Client (127.0.0.1:2207) connect 1997/ 6/28 15:24:41 *** Password accepted *** 1997/ 6/28 15:24:42 [6] Pulling from 128.101.83.4 1997/ 6/28 15:25:48 [6] Pull completed, msgs = 0, rejects = 0 1997/ 6/28 15:26:16 group "brian.fan.janet-jackson" added via XADDLIST 1997/ 6/28 15:26:16 group "walker.fan.kathy" added via XADDLIST 1997/ 6/28 15:26:51 [7] Pulling from 128.101.83.4 1997/ 6/28 15:26:51 [7] Pull completed, msgs = 0, rejects = 0 1997/ 6/28 15:27:21 group "planet.janet" added via XADDLIST 1997/ 6/28 15:27:52 [8] Pulling from 128.101.83.4 1997/ 6/28 15:27:54 [8] Pull completed, msgs = 0, rejects = 0 xlog 284 No logging to this connection
These commands are used to get and set the description for a group. Normally the group description is set when the group is created (a description line is included in every NEWGROUP control article). However, if you're just setting up a server, you'll need use these commands to set the descriptions for long existing newsgroups.
xgroup set description my.moderated.group This is an example description. 211 group description set xgroup get description my.moderated.group 212 This is an example description.
RumorMill normally expires articles when they reach a certain age. An article can have an explicit expiration date declared in the article, or it can be expired using one of two expiration "defaults." Each group in RumorMill can have a default expiration period. If this value is non-zero and an article has no explicit expiration date, RumorMill sets the expiration date to be the date of receipt plus the group expiration period. If the group expiration period is zero, the "Article Expiration Period" preference is used.
xgroup set expiry my.group 7 211 group EXPIRY set xgroup get expiry my.group 212 7
RumorMill can forward articles added to the database to specific groups to a list of EMail addresses. This feature should be used carefully since setting up an EMail list for an active group can create quite a bit of EMail. Also be aware of setting up EMail address lists for groups that are often cross posted. RumorMill attempts to eliminate sending duplicate EMails for a single article, but in some circumstances, it cannot avoid it.
xgroup set exploder comp.sys.mac.announce fred@flintstone.net 211 group email set xgroup get exploder comp.sys.mac.announce 212 fred@flintstone.net
Newsgroups can be moderated. A moderated group does not allow direct posting to the group. Instead the server EMails the posting to a group submission EMail address (a moderator). If the moderator approves the article, the moderator will repost the article to the group with an "Approved" field added. The commands above are used to set and get the EMail submission address for a given group. Note that the set command also will set (or clear) the moderated flag for a news group. If the <Group submission EMail address> is empty, the group is marked as unmoderated.
xgroup set moderator my.moderated.group mmg@myEmail.mycom.com 211 group moderator (un)set xgroup get moderator my.moderated.group 212 mmg@myEmail.mycom.com
This command can be used to see if a given group is permanent or not. When virtual groups are enabled, a permanent group cannot become virtual due to lack of use. Please note: This command is not implemented in the RumorMill Setup application.
This command is used to set (or unset) whether a given group is permanent or not. Permanent groups do not become virtual due to lack of use. Please note: This command is not implemented in the RumorMill Setup application.
This command is used to determine if a given group is virtual or active. Please note: This command is not implemented in the RumorMill Setup application.
Caution: Use this command with care. It will likely add thousands of groups to your database! Please note: This command is not implemented in the RumorMill Setup application.
This command will:
- initiate connections to all remote servers marked as PULL or upstream
- retrieve the total list of groups from that site
- add each of those groups to the local database, subject to the VirtualGroupFilter preference
This command can be used to construct a forwarding file for remote server that is already configured. When the command is received by RumorMill, the "<remote server>" is verifed and then the entire database is scanned for articles that should be forwarded to that remote server. The groups specified in the "groupsToSend" list is honored when this command is executed.
This command can be used to kill connections to the RumorMill server. The XSTATS command can used to get a list of active connections. The connection numbers in the XSTATS response can be used to selectively kill connections. A value of -1 can be used to kill all connections except the connection issuing the XKILL command.
XKILL 23 299 Did it
This command will report the maximum number of remote servers and site restrictions this version of RumorMill will support. The final value is the number of active groups.
xmax 280 maxAddrMasks:20,maxremoteServers:16,ngroups:637
This command is a RumorMill extension that will delete articles from the message-id and article index databases without adding the deleted articles to the canceled database. It is a convenient way remove articles from your database without going through the overhead of creating articles to cancel the articles.
This command will delete the article file that would next be deleted for expiration. If you're tight on disk space, this command is the best way to quickly free up some disk space. Successive invocations will continue to delete the article file nearest to reaching expiration until no article files remain.
These commands set the majority of the preferences in RumorMill. Most Preferences and their values are revealed with xpref list:
xpref list Admin OKs Group Deletes longint 1 0 2 Admin OKs New Groups longint 1 0 2 Allow Old Articles longint 0 0 1 Article Expiration Period longint 180 1 3650 Article File Window longint 15 1 365 Auto delete article files longint 0 0 2 Automatic Expiration longint 0 0 1 Background Quantum longint 4 1 50 CHECK Threshold longint 10000 1 1000000000 Compact Period By Group Index longint 3 1 365 Compact Period Canceled longint 5 1 365 Compact Period Deferred longint 2 1 365 Compact Period Group longint 5 1 365 Compact Period MessageID longint 5 1 365 Compact Period User longint 11 1 365 Connection retry count longint 0 0 10 Database synch count longint 30 1 500 Database synch interval longint 30 1 600 Date Bounds longint 180 1 3660 Detailed Logging longint 1 0 1 Disk space threshold longint 10 2 100000000 EMail Add Reply-to longint 0 0 1 EMail Control Msgs longint 1 0 1 EMail Interval longint 480 1 1439 EMail Map map EMail Server Group Changes longint 1 0 1 EMail Statistics longint 1 0 1 Enable group cache longint 1 0 1 Enable streaming receive longint 1 0 1 Enable streaming send longint 1 0 1 Enable virtual groups longint 0 0 1 Flood Interval longint 30 1 1439 Foreground Quantum longint 6 1 50 Garbage Collect Quantum longint 3 1 50 Garbage Collect Time longint 12900 0 235900 Help note Str255 Test system. Go away. Hide control group longint 1 0 1 Host path name Str255 testing123 Inject Interval longint 5 1 1439 Limit IHAVE longint 1 0 1 Log File Name Str255 RumorMill log Log Rejected Msgs longint 1 0 1 Log unknown groups longint 1 0 1 Max Article File Size longint 15 1 2000 Max Article Gap longint 300 1 100000 Max Article Size longint 250000 2000 10000000 Max Clients longint 10 1 100 Max Expiration Delta longint 180 1 3650 Max Feeds longint 3 0 20 Max Idle Period longint 0 0 240 Max Logfile Size longint 128 10 20000 Max Msgs Per Connection longint 10000 1 100000 Max Newsgroups Per Msg longint 50 1 40000 Max Open Article Files longint 30 5 1000 NAT host name Str255 NNSP Server port longint 433 0 100000 No NEXT longint 1 0 1 Only propagate local postings longint 0 0 1 PAT permitted longint 1 0 1 POP Interval longint 30 1 1439 POP account Str255 test POP host Str255 POP password Str255 **** Pull Interval longint 30 1 1439 Read Only longint 0 0 1 Remote Servers rmsv Require Login longint 2 0 2 SMTP host Str255 127.0.0.1 SPAM if chrh SPAM if not chrh Server mode longint 1 0 1 Server port longint 119 1 100000 Site Address Restrictions restrictions Site Password Str255 **** Status window update rate longint 5 1 10000 Sys Admin EMail Addr Str255 support@swaystairs.com SysLog IP Addr Str255 SysLog Port longint 514 1 32000 SysLog Priority longint -1 -1 7 Text PWD longint 0 0 1 Use alternate AUTHINFO codes longint 0 0 1 User EMail Header Str255 User Header Str255 Virtual Group Filter chrh Virtual group threshold longint 2 1 255 .Each line in the response from XPREF LIST has five (5) fields. The fields are: the preference name, the preference data type, the current value, the minimum value, and the maximum value. More information for each preference can be obtained by using the XPREF GET <preference name> command.
Gives you some statistics on what has been going on within RumorMill.
xstats 277 Current information follows RumorMill version 1.3b10 RumorMill started: 18 Jun 2001 21:26:34 -0400 Current local time: 18 Jun 2001 21:33:06 -0400 Last expiration at: 1 Dec 2000 1:29:00 -0400 Group RAM Cache is active, Free mem = 3959600 bytes Statistics for today/total IHAVEs: 0/0 Pulled: 0/0 TakeThis: 0/0 POSTs: 0/0 Connects: 1/1 Forwarded: 0/0 Line Count: 3/3 Msgs not entered into DB: 0/0 EMail sent: 0/0 EMails posted: 0/0 Active connections: 3 [4] EMail: post.local.com 3s [1] Server: 123.29.3.7:3147 30m 3s Idle: 18m 6s [0] Server: 127.0.0.1:2048 1h 10m 3s Disk space health: Good .Each active connection lists a connection type (EMail and Server in the example above). The possible connection types are as follows:
Type Function Client/push Pushing news to a downstream feed. Forwarding queued EMail to an SMTP server. Get group info Retrieving information from a pull server to advise the System Administrator of changes in groups at that server. nnsp Serving an nnsp client POP EMail retreival Retrieving EMail for the EMail -> News gateway Server Acting as a server for a client or an upstream news feed. Slave/pull Pulling news from an upstream feed.
This command is used to remotely initiate database compaction. It is equivalent to using the Compact Databases... command on RumorMill's File menu. The database flags are used to indicate which databases to attempt to compact. They are encoded as follows:Articles = 1 Canceled = 2 Deferred = 4 GroupIndices = 8 Groups = 16 Users = 32 Article IDs = 64So if one wanted to compact the Article and the Groups databases, the following command should be issued.xcompactdb 17 299 Compaction to start shortly ** Server Closed Connection **Note that your connection is immediately terminated after issuing the command. All other users will also be disconnected from the server after about one minute.
This is equivalent to using the Expire Database command on RumorMill's File menu. Note that your connection to the server is immediately terminated. All other users will be disconnected within 5 minutes of issuing the command.xexpiredb 290 Expiration now pending
Article Expiration Period
If a article doesn't have an Expires field, and the relevant group(s) don't have an expiration period set, this is the default period of time it is allowed to exist in the article database. The value is expressed in days. RumorMill remembers the message-id of expired articles for an equal interval of time. This helps handle articles that arrive from multiple sources, some of which are much slower than others.
Automatic Expiration
This preference enables or disables (set to zero) the automatic expiration feature of RumorMill. When enabled, RumorMill will examine articles in the database on a periodic basis to determine which articles should be expired (removed from the database). See also Garbage Collect Time.
Date Bounds
For a POSTed article to be considered valid, the date field must be within this delta of the current time. This is used as an anti-spam measure. Specified value is in days.
EMail Interval
Occasionally when RumorMill attempts to send EMail, the specified SMTP server may not be available. In these cases, RumorMill will periodically check to see if the SMTP server is available to forward queued EMail. This preference controls how often this check is performed. Value is specified in minutes.
Flood Interval
The default time between attempts to send new articles to "downstream" servers. This value is in minutes. 2 hours or above (120 minutes) seems reasonable, although some UNIX INN systems seem to use 1 hour. Each downstream server specified can have an individually specified flood interval that can override this default.
Garbage Collect Time
This is the time of day that RumorMill will peform expiration processing each day.
Help note
This string is appended to the result of the "help" command. Normally the string should contain something like "Report problems to usenet@somewhere.com."
Hide control group (via Telnet only)
Many users found the control group confusing when it appeared in their news reader. This preference determines whether or not the group named control is visible to clients of RumorMill. The Hide control group is defaulted on (set to 1).
Inject Interval
The time in minutes between attempts to take articles from the folder "Articles to Inject" and add them to the local database.
Limit IHAVE
When non-zero, the IHAVE command is valid only for clients with a specified newsfeed server address. This should be set to non-zero to prevent inappropriate addition of articles to the database.
Max Newsgroups Per Msg
Specifies the maximum number of newsgroups to recognize within a single received article. When the article is being processed, if this number is exceeded, the article is rejected. This prevents massive cross-posting into your local database.
Max Article Size
Any article received that is larger that this setting (in bytes) is accepted (provider is given a positive response), but the article is discarded. The message-id is entered into the database to prevent additional downloads of this article.
Max Logfile Size
This preference limits the size of the RumorMill log file. It is expressed in K bytes. When detailed logging is enabled, RumorMill interprets this limit as 4 times the specified size.
NNSP Server Port (via Telnet only)
NNSP is a news server to news server protocol. This preference specifies the IP port for NNSP. The RFC specifies port 433. Setting this value to zero (0) will disable listening for NNSP connections.
Only propagate local postings
When set to non-zero, this preference prevents articles received from upstream servers (either active pull or receive modes) from being forwarded to any downstream servers. However, locally posted articles are forwarded to downstream servers. Locally posted articles are those delivered to the running RumorMill server by way of a POST command (which is what news reader clients like Netscape or Newsreader use).
Pull Interval
The default time between attempts to retrieve articles from "pull" servers. This value is in minutes. Each specified server can have an individually specified pull period that will override this default.
Read Only
If this has a non-zero value, posting is not permitted. readOnly currently means that POST is disallowed, but IHAVE is allowed (subject to Limit IHAVE setting).
Require Login
You can require users to have passwords by setting Require Login to non-zero. Require Login has three valid settings as follows:
Value Meaning 0 No authentication is required for a client to use the services of RumorMill. 1 Authentication required for any use of the server. 2 Authentication required to post to the server. Said another way, unless authenticated, the client has read-only access to the server. Server Mode
Certain conditions may cause RumorMill to notify the system administrator of error or anomalous behavior. When this preference is set to non-zero, RumorMill will not display such conditions using modal dialogs. Normally this preference is enabled after RumorMill has been running in a stable manner for a period of time.
Site Password
This is the password for the extended commands available in RumorMill. The extended commands include those for getting and setting preferences.
SysLog IP Addr
This is the IP address of a host running a syslog deamon. RumorMill will send logging data to this host as specified by the SysLog Priority preference. The IP address should be specified as a dotted quad, e.g., 18.1.6.77.
SysLog Port
This is the port number that the syslog deamon is listening on. The default is port 514.
SysLog Priority
This specifies the message priority that should be sent to the syslog deamon. The value of -1 disables syslog logging. Syslog protocol uses 0 as the most serious of log items, while 7 is considered debugging information. Setting it to 6 will log all but debugging information.
A number of RumorMill's preferences can be used to help limit SPAM. These preferences are:
EMail Add Reply-to
This preference controls whether or not RumorMill adds a Reply-to header in EMail that it sends (News -> EMail gateway). Setting it to non-zero enables the feature. However, this preference has no effect if the POP Account and POP Host preferences are not set.
Host path name
If this preference has a value, it is used instead of the local host name in various places in the article headers (particularly in the Path: header). Provided for servers that do not have an assigned name either because of dynamically assigned IP addresses, or do not have a DNS entry.
Inject Interval
This preferences specifies how often RumorMill should check the "Articles to Inject" folder for articles to be added to the database. Files placed in this folder must have one article per file and must be properly formated for a valid USENET article.
Remote Servers
A description of remote NNTP servers. Currently RumorMill can support definitions of up to sixteen (16) remote servers. [Note that XMAX command can be used to determine what this value is for your version of RumorMill.] Each server can be denoted as an upstream server (one that provides articles to your copy of RumorMill via IHAVE), a downstream server (a remote server that your RumorMill provides articles to via IHAVE), or both. Pull can also be specified to indicate that RumorMill is to act like a client (or slave) when interacting with the remote server. To have articles pushed back to the remote server, downstream should be specified with pull.
Each entry can also have a list of news groups to forward to the downstream server and groups to exclude from the upstream server. If the groups to send list is empty, all groups known to your RumorMill are forwarded. If the groups to exclude list is empty, no group articles are discarded.
Also see the example.
Site Address Restrictions
This resource controls who can use your news server. The resource should be configured as pairs that describe a mask to be applied to the address of a connecting client, and the acceptable result of masking the incoming IP address. An example: assume your site has addresses in the 9.3.x.x and the 10.12.4.x ranges.
By specifying the Site Restriction resource as 9.3.0.0, 255.255.0.0, 10.12.4.0, 255.255.255.0, only connections from these address ranges will be accepted by RumorMill. (At present there can be at most 20 result/mask pairs in this resource). N.B. creating a single pair of 0.0.0.0, 0.0.0.0 a client from any site can connect and is strongly discouraged. By default, RumorMill will create an entry that will allow any machine with a class C IP address like RumorMill's to connect (effectively X.Y.Z.0,255.255.255.0, where X.Y.Z are exactly the upper three octets of the host running RumorMill).
A "!" may preface the result IP address to negate the sense of the test; that is, disallow connections from any IP address matching the resulting IP address (after masking).
Note that an implicit entry is exists for any Remote Server (see below) that is marked as upstream (this entry is of the form W.X.Y.Z,255.255.255.255 where W.X.Y.Z is the IP address of the upstream remote server). This means that one need not create a site address restriction entry for each news feed.
Also see the example.
SMTP Host
The host name or IP address of the SMTP server RumorMill is to use to send EMail. This is primarily used to send article submissions to the group moderator.
Sys Admin EMail Addr
The EMail address of the person maintaining this particular RumorMill server. The EMail address used by RumorMill to send information to the Sys Admin. At present, such EMailings include:
- articles that are posted to a moderated group when there is no group submission EMail address (moderator) available.
- statistics reports (EMail Statistics preference).
- changes in groups at a remote server marked as pull (EMail Server Group Changes preference).
- automatically handled and deferred control articles (EMail Control Msgs preference).
User EMail Header
If supplied, this line is added to outgoing articles that are being EMailed. In general, the user supplied header should start with "X-" if the header is not one of the EMail defined header. However, if the user uses a standard header, such as "Subject:", then RumorMill will rename the existing "Subject:" header to "X-NNTP-Subject:" and insert the user defined "Subject:" header.
If the article about to be EMailed has the following headers:Subject: RumorMill Header Mapping Newsgroups: local.testingand the User EMail Header is defined as follows:Subject: [^2] ^1The resulting headers in the EMail sent are:Subject: [local.testing] RumorMill Header Mapping X-NNTP-Subject: RumorMill Header Mapping Newsgroups: local.testingThe following fields can be substituted into a User EMail Header:
^1 Previous header contents ^2 Newsgroup(s) for which this article had EMail exploders defined ^3 Host name of the machine running RumorMill
Admin OKs New Groups
This preference controls the processing of newgroup control articles. The permissible values and actions of this perference are as follows.
Value Action 0 NEWGROUP articles will automatically be processed 1 NEWGROUP articles will be deferred for review by the system administrator 2 NEWGROUP articles will be ignored Admin OKs Group Deletes
This preference controls the processing of rmgroup control articles. The permissible values and actions of this perference are as follows.
Value Action 0 RMGROUP articles will automatically be processed 1 RMGROUP articles will be deferred for review by the system administrator 2 RMGROUP articles will be ignored Allow Old Articles
Normally the "Date Bounds" preference is checked to see if articles being received (either forwarded to the server or the server is pulling) are "old." If the articles are old, or outside the date bounds, they are discarded upon receipt. If the "Allow Old Articles" preference is set to non-zero, then the "Date bounds" setting is ignored and the articles are added regardless of their age. This preference is normally set to non-zero if you're spooling a full database (or something like that) to a new RumorMill server. This preference is not used for POSTed articles (articles that a user adds from a newsreader). The recommended setting is zero.
Article File Window
Article files are organized such that all articles with expiration dates within a certain period of time (a window) are put into the same article file. This prefeference specifies the number of days that an article file may span. Smaller numbers allows more rapid expiration of article files, but create larger numbers of article files. Having more article files may decrease efficiency if the Max Open Article Files is smaller than the number of active article files.
Auto delete article files
This preference controls how RumorMill deals with article files once the available disk space drops below the "Disk space threshold" level. The status window will report this condition as disk health "bad". The preference has three possible settings as follows:
Value Action 0 No automatic deletion of article files 1 Delete article files about 1 per minute until disk health is no longer "bad" 2 Delete article files until disk health is no longer "bad" Background Quantum
The number of ticks (1/60 of a second) that RumorMill may consume will running as a background task. Making this larger improves RumorMill performance when running as a background application at the expense of other applications.
CHECK Threshold
The streaming protocol allows the sender to check with the downstream server to see if it wants an article or not. Alternatively, the sender can just send the article. This preference controls the threshold (maximum article size) of articles that are sent without using the CHECK command. For slow connections, this probably shouldn't be larger than 1200 or so. For faster connections, 10,000 or so may be more appropriate. N.B.Some news servers do not function properly if the CHECK command is not used. If you are streaming to such a server, set this parameter to 1 to assure that the CHECK command will always be used.
Compact Period By Group Index
Specifies the number of days between attempts to compact the By Group Index database.
Compact Period Canceled
Specifies the number of days between attempts to compact the canceled message-id database.
Compact Period Deferred
Specifies the number of days between attempts to deferred control items database.
Compact Period Group
Specifies the number of days between attempts to compact the Groups database.
Compact Period MessageID
Specifies the number of days between attempts to compact the message-id database.
Compact Period User
Specifies the number of days between attempts to compact the a users database.
Connection Retry Count (via Telnet only)
If your copy of RumorMill is connecting to other servers via an on-demand PPP (or similar) connection, you may find that RumorMill is not reliably connecting to other servers (for news or email). This can happen when an on-demand PPP connection takes longer to establish than RumorMill's connection timeout. Setting the Connection Retry Count to greater than 0 may fix this situation. Try a value like 5 to start and use detailed logging to determine the actual number of retries required for reliable connections.
Database synch count (via Telnet only)
When the number of changes to a database file exceeds this threshold, the changes are forced to disk.
Database synch interval (via Telnet only)
When the age of changed data in a database file exceeds this threshold (in seconds), the changes are forced to disk.
Date Bounds (via Telnet only)
RumorMill requires that incoming articles be within a small window of the current time and date to be accepted. This preference sets +/- the number of days of the current date and time that is accepted.
Detailed Logging
If this has a non-zero value, a more detailed log file is written as commands are received from clients and servers. This can be temporarily changed with the Detailed Logging command on the file menu.
Disk Space Threshold
When the free disk space goes below this threshold setting, RumorMill begins to limit what actions it will perform. If the free disk space goes below one-half the disk space threshold, RumorMill will no longer allow posting.
EMail Control Msgs
Some articles contain the "Control:" or "Also-control:" headers. Among other things, such articles can direct that groups be added or removed. RumorMill provides preferences to select whether these control articles are immediately acted upon, deferred, or ignored (see Admin OKs New Groups and Admin OKs Group Deletes). When control articles are acted upon or deferred, RumorMill can send EMail to the system administrator containing the control article. RumorMill will send EMail when this preference is set to non-zero.
EMail Server Group Changes
When this preference is enabled (non-zero), each night after garbage collection is completed, RumorMill will query each remote server marked as a "pull" server. RumorMill checks for groups that are to be pulled from that server and notes if that group does not exist at the server. RumorMill also retrieves (via NEWGROUPS) a list of groups added to the server since the previous query. If any new or missing groups are found, an EMail is sent to the Sys Admin EMail Addr. Potentially one EMail will be sent for each remote server marked as "pull."
EMail Statistics
If this preference is set to non-zero, an email containing statistics will be sent to the Sys Admin EMail Addr whenever RumorMill is quit (from menu or QUITNOW), or when the nightly garbage collection completes.
Enable group cache
If sufficient RAM is available, RumorMill with cache all of the group data in RAM to improve performance. This preference can be used to enable/disable this capability.
Enable Streaming Receive
This preference controls whether or not RumorMill will accept streaming feeds from upstream servers. Setting the preference to 1 enables receipt of streamed articles. When set to zero, RumorMill will not accept the "MODE STREAM" command.
Enable Streaming Send
When set to one (1), RumorMill will attempt to forward articles using streaming. If the downstream server refuses the "MODE STREAM", RumorMill will automatically revert to using non-streaming forwarding.
Enable virtual groups
This preference enables/disables the virtual group feature. Please note: This preference is not implemented in the RumorMill Setup application.
Foreground Quantum
The number of ticks (1/60 of a second) that RumorMill may consume will running as a foreground task. Making this larger improves RumorMill performance when running as the front (selected) application at the expense of other applications.
Garbage Collect Quantum (via Telnet only)
This preference sets the amount of time (in ticks, or 1/60th of a seconds) that garbage collection (or database maintainence tasks) can consume in a single burst of execution. This value should be kept small so that your server remains responsive while these tasks are underway.
Log File Name (via Telnet only)
Name of the log file.
Log Rejected Msgs
If this has a non-zero value, all articles that RumorMill rejects for some kind of formatting error will be appended to a file called "Rejected Article".
Max Article File Size
This preference sets (approximately) the maximum size of each article file. A large maximum is appropriate for installations receiving large volumes of articles. N.B.: This is specified in megabytes. RumorMill does not support article files greater than 2GB in size, nor does it have any need to since there are now multiple article files.
Max Article Gap (via Telnet only)
This preference can be adjusted to optimize performance when pulling articles from a server. RumorMill maintains a notion of which article number to next pull from a each group. When RumorMill is initially installed, or restarting after a long period of inactivity, RumorMill must decide which article to pull next. Often servers do not keep a particularly accurate notion of which articles are in the database.
When RumorMill issues a GROUP command to the remote server, the reply includes the following information for the group: the article low water mark the article high water mark and the count. In ideal situations, the high water mark minus the low water mark would be equal to the count. But that never happens. In actual practice, the count can be much smaller than the difference between the high water mark and the low water mark. When this happens, RumorMill is stuck stepping one article by one article trying to find the next valid article at the server. In some situations, this can take a very long time.
To help with this situation, RumorMill uses the Max Article Gap preference to limit how many articles it actually tries to get from given group. Basically, if RumorMill thinks it has more than Max Article Gap articles to pull for a single group, RumorMill adjusts the article number it asks for to essentially be the high water mark minus Max Article Gap.
Max Clients
Maximum number of clients that RumorMill will service (simultaneously).
Max Expiration Delta
Some articles have unreasonable expiration dates specified. This preference can be used to limit how far into the future an Expires header may specify. The preference is specified in days. Please note: This preference is not implemented in the RumorMill Setup application.
Max Feeds (via Telnet only)
Maximum number of downstream newservers than RumorMill will forward and/or pull articles to/from simultaneously.
Max Idle Period
The maximum time a connection (from a reader or a downstream feed) can be inactive before it is closed by RumorMill. The value is specified in seconds. If zero (0) is supplied, connections will never time out. [Note that setting this to a small value may cause connections to time out before some time consuming operations can complete.]
Max Msgs Per Connection (via Telnet only)
Specifies the maximum number of articles to forward to a downstream server on a single connection.
Max Open Article Files
This preference controls the number of article files RumorMill may have open at any one time. Under MacOS 9 and later systems this setting is not so critical as the maximum number of open files is dramatically increased. Please note: This preference is not implemented in the RumorMill Setup application.
NAT host name
Sometimes a system administrator may wish to run a RumorMill server behind a firewall or Network Address Translator (NAT). This allows the server to run on a machine that is not fully exposed to the Internet. Normally the firewall or NAT will port map news traffic (nntp protocol on port 119) to a machine behind the firewall.
However, if this is done, RumorMill will report the name of the machine behind the firewall in the greeting and in article postings. THe NAT host name can be used so that RumorMill will report the name of the firewall or NAT host instead. For this to work, the specified name must be a valid host name - that is, RumorMill must be able to look up the name and get an IP address for it.
No NEXT (via Telnet only)
Normally when pulling articles from a group RumorMill issues a NEXT command to determine which article it should next attempt to pull for a given group. The server reply to NEXT includes the message-id for the next article. RumorMill checks to make sure that this message-id isn't in the database, if it is, it issues another NEXT command, if not, an ARTICLE command is issued to retrieve the article.
In some cases this can be slow because there are two round trips to the server (the NEXT and its response and the ARTICLE and its response) to complete a transaction. The advantage is not pulling articles already in the database. However, based on the network capabilities between the computer running RumorMill and the remote server, it may, in some circumstances, be faster to forgo the NEXT command. The No NEXT when set to one will eliminate sending the NEXT command.
Performance with No NEXT on or off will vary widely from installation to installation. For instance, if RumorMill is pulling groups that are heavily cross-posted, using No Next means that articles will be pulled from the server multiple times.
PAT Permitted
This preference allows/disallows the use of the XPAT command. Since these commands can consume arbitrarily large amounts of processor time, some system administrators may elect to disable the use of the XPAT command.
SPAM If
The SPAM If preference considers any article SPAM if it contains any of the groups specified in this list. Any article with a matching group name will be discarded and not entered into the article database. Please note: This preference is not implemented in the RumorMill Setup application.
SPAM If Not
The SPAM If Not preference considers any article SPAM if the incoming article does not contain at least one of the groups specified in this list. Any article without a matching group name will be discarded and not entered into the article database. Please note: This preference is not implemented in the RumorMill Setup application.
Status window update rate
This preference specifies how often (in seconds) the status window is updated. Please note: This preference is not implemented in the RumorMill Setup application.
Text PWD
This preference is meant for temporary operation only. When adding a large number of users to RumorMill, it may be convient to type up the user list in a file rather than entering them one by one via RumorMill Setup. The problem is that the "Create Users..." command in RumorMill expects the passwords to be encoded rather than clear text and there is no way to create such a file by hand. Setting this preference to one (1) instructs the "Create Users..." command to interpret the passwords as clear text and convert them to encoded form. This preference should normally be set to zero (0). Please note: This preference is not implemented in the RumorMill Setup application.
Use alternate AUTHINFO codes (via Telnet only)
The IETF NNTP protocol group is revising the current practices document. Client software should accept the older AUTHINFO response codes, but may also accept the newer response codes. In the odd circumstance that you user's client software only accepts the newer AUTHINFO codes (x5x rather than x8x), setting this preference to non-zero will switch to using the newer response codes.
User Header
A system administrator may wish to add a header to every article posted at her server. The "User Header" may be used for this purpose. The header must start with "X-", and must contain a ": " with at least one trailing character. In other words, a minimally acceptable user header is something like "X-: Z". Something more descriptive like "X-complaints-to: admin@yoursite.com" is recommended.
Virtual Group Filter
This preference is used when initializing the virtual group capability by downloading the group list from a remote server (pull). Only groups matching this specification will be added to the group database.
Specifications of the form *,!*microsoft*,!*hipcrime* are permitted. This specification would allow any group except ones containing microsoft or hipcrime in the group name. Please note: This preference is not implemented in the RumorMill Setup application.
Virtual Group Threshold
For a virtual group to become active (articles will either be pulled or accepted from a feed), the number of recent (within 24 hours) references to the group (a client attempted to read it) must exceed this threshold. Please note: This preference is not implemented in the RumorMill Setup application.
EMail Map
The EMail Map preference is used when EMail is received by RumorMill that has no Newsgroups header specified. When this occurs, RumorMill looks through the EMail Map for the first header that can be found in the received EMail. If a header from the map is found, the matching newsgroups are inserted into the EMail as a header. RumorMill looks to find the header tag (the part to the left of the colon) and if found, the sees if the part to the right of the colon appears anywhere in the header line (a "contains" match).
xpref set 389 Send preference data; end with <crlf>.<crlf> email map Sender: macpascal@list.stairways.com local.programming.pascal X-Sender: tandem@hobbes local.bicycles . xpref get email map 288 Preference data follows Name:EMail Map Type:map LogicalGroup:1 Description:EMail header to newsgroup map table Header:Sender: macpascal@list.stairways.com Newsgroups:local.programming.pascal Header:Sender: tandem@hobbes Newsgroups:local.bicycles .The example above sets up two EMail headers to look for, a "Sender:" header that contains the string "macpascal@list.stairways.com", or an "X-Sender:" that contains "tandem@hobbes". If one of the entries is found, the corresponding "Newsgroups: " header will be added to the incoming article before attempting to add the article to the article database.
Specifying an empty EMail Map deletes the current information.
POP account
A POP account is required when configuring RumorMill to post articles via EMail. This account will recieve all EMail messages that are to be posted as articles.
POP host
A POP server is required when configuring RumorMill to post articles via EMail. This preference specifies the name of the host running a POP3 server. The name may be suffixed with a port number if the POP server is running on an alternate port (the default is 110).
POP Interval
Sets the frequency that RumorMill will check a POP account for incoming EMail postings.
POP password
Specifies the password to the POP account.
There are a number of factors that influence the size of RumorMill's database. The most important of these is how many groups are active and the number of articles posted to each group. Without any intervention, the various files in the RumorMill database would grow until the hard disk(s) is (are) completely full.
RumorMill has features to help the adminstrator avoid filling up disks with the RumorMill database. The primary mechanism is that of expiration - deciding after some period of time that articles are no longer relevant and deleting them from the database. See Automatic Expiration and Article Expiration Period for details.
However, deleting articles from RumorMill's database does not automatically make disk space available for other applications. That is, RumorMill's database files, in general, never shrink in size. For some users this is undesirable.
When sufficient disk space is available, RumorMill will periodically attempt to shrink its data files by copying active entries to a new copy of the database file and then deleting the old copy. For this process to be successful, RumorMill requires enough space to completely copy the existing database file. Space for a full copy is required because RumorMill does not know in advance how much shrinkage will be achieved - and it is possible that no shrinkage will occur.
It is useful to note that various files of the RumorMill database can be spread across multiple volumes. Simply put an alias to the real database file in the database folder. RumorMill honors the alias for all functions, including attempting to leave some space free on each volume (see Disk Space Threshold) and rebuilding database files on the indicated (by the alias) volume.
Expiration in RumorMill is made of of several parts. Rather than doing all steps of the expiration every day, RumorMill spreads this process out over several days. The table below describes this process.
Preference Processing performed Compact Period By Group Index Number of days between compacting the "By Group Index" file Compact Period Canceled Number of days between compacting the "Canceled" file Compact Period Deferred Number of days between compacting the "Deferred Control IDs" file Compact Period Group Number of days between compacting the "Groups" file Compact Period MessageID Number of days between compacting the "MessageIDs" file Compact Period User Number of days between compacting the "Users" file
Some newservers (including RumorMill) object to having certain headers in existence when the POST command is used. RumorMill can run afoul of this when substituting POST for IHAVE to a downstream server. Code was added to delete headers in this situation based on a resource description rather than being hard coded. STR# 506 resource contains the list of headers to be deleted when using POST to relay articles to a downstream server. RumorMill ships with the list set to "Xref", "Sender", and "NNTP-Posting-Host"
The overview format can be changed in RumorMill. At startup RumorMill looks for a STR# resource named OVERVIEW.FMT in the Preferences file. If found, it will use the contents (each header is one entry in the STR# resource) for the fields of the OVERVIEW format. Each string should include the trailing colon for the header, e.g., 'Date:'. If an entry ends with an asterix (*), then the overview field for that header will include the header name. N.B.: RumorMill will always force 'Message-ID:' to be in the OVERVIEW format.
Why would you change the overview format? It's possible that a special header (perhaps even non-standard) used at your site would be helpful to have in the overview. (For example, this header might be used for automatic filtering by your news reader.) In this case you might add that header to the OVERVIEW.FMT. In another case, one or more headers might create problems for some software package you are using. In this case you could remove headers from the OVERVIEW.FMT.
RumorMill uses a large number of files to implement the functionality required of a NNTP server. Most of these files are kept in the Databases folder within the same folder as the RumorMill application. The names and purposes of these files and folders within the Databases folder are as follows.
- Articles folder
- This folder contains all of the article files. Each file contains a collection of articles spanning a fixed period of time. Article expiration occurs by deleting these files.
The format of these files appears to be simple text. It is not simple text, and no attempt should be made to manipulate these files. The files are highly structured and other files described below contain offset indices into these article files.- Articles to Inject folder
- RumorMill will periodically check for the existance of files in this folder (if the folder exists). Any visible file will be read and handled as if it had been recieved via a normal connection. This feature is provided to allow users to insert articles into the database that were previously rejected or discard for some reason - usually because RumorMill had previously been configured in a manner that would not allow the article's inclusion in the database.
- Canceled
- Contains message IDs for canceled articles. This database is used to insure that previously canceled articles are not recreated by tardy delivery from another newsfeed.
- Deferred Control IDs
- This file contains the IDs of all articles that contain Control: or Also-control: headers that were not processed at the time the article was received.
- Downstream spool
- This folder contains files of the message IDs that are to be forwarded to downstream servers. There are three forms the files that may be found within the folder. Each file name is constructed from the downstream host's name and prefixed by either "q-" "noq-" or "unq-". These are text files that can be inspected with a text editor. The files may also be deleted - the result will be that those articles will never be forwarded to that server.
Several things should be noted about these files.
- You should not be looking at them with an editor or other tool while RumorMill is running. This may cause the file to be not writable when RumorMill attempts to update the file. This will cause articles not to be forwarded!
- Data is written to these files in batches (that is it's cached within RumorMill to keep from constantly opening and closing the files). If you run a simple test, don't expect the file(s) to instantly show up.
- q-
- This is the primary file that contains the queue of message IDs to be forwarded.
- noq-
- When forwarding articles to a downstream server, that server may reject (temporarily) some articles. When this happens, those message IDs are placed in this file. These article IDs are later merged back into the "q-" file.
- unq-
- When the forwarding process is in progress, the "q-" file is renamed to this (another q- may get created while the forwarding is in progress).
- Group History
- This file contains the creation date and time for each group. It is used to support NEWGROUPS command.
- By Group Index
- This file is used to map article indexes within a group to the message ID. (When a client asks for "ARTICLE 7235" RumorMill combines the group name and the article index and looks up the message ID to get the article from the Article file.)
- Groups
- This file contains all of the groups that RumorMill knows about. It also has entries for the group description and moderator information for each group.
- Message IDs
- This file is a database of all the known message IDs that occur in articles. This database is contains information telling RumorMill which article file contains each article.
- Queued EMail
- This folder holds all of the EMail queued by RumorMill (one file per EMail), but not yet sent to an SMTP server. In general there won't be much in this folder unless the SMTP server isn't available.
- Slave Group Indexes
- This file is used to remember the group article indexes for remote servers which are "pull" servers. This file can be deleted and RumorMill will recreate it. The effect will be to make RumorMill start at article 1 (one) for each group at each remote server the next time it pulls articles from the server. This isn't recommended because it will often take quite a while to get back in step with the remote server.
- Subscriptions
- This is an optional file. If present, the contents of the Subscriptions file will be returned by the "LIST SUBSCRIPTIONS" command. This feature is used by some sites to provide a list of groups new users should subscribe to by default.
- Users
- This file contains all the information about the users you've created for your copy of RumorMill.
There are two additional files that are kept in the Preferences folder within the System Folder. These are:
- RumorMill Log
- This is a text file that is a running log of RumorMill activity. Each connection is recorded as are errors and other events. Detailed information can also be logged to this file. See Detailed Logging.
- RumorMill Preferences
- This is the preference file for RumorMill. It contains all of the settings configured by RumorMill Setup or via a Telnet connection. RumorMill will also look for this file in the same folder as the application. If available, that will be used instead of the copy in the Preferences folder. Note well: it's a resource file and attempting to edit it directly will usually cause problems.
Occasionally when starting, RumorMill will fail and report an error. The message codes for these errors and some possible reasons for them are listed below.
10 A database consistency error. The code is checking to see that if databases were created, the number created is consistent with the number of databases opened (initial creation of the databases versus an error). This usually happens if one of the main databases was renamed, deleted, or otherwise damaged.
11 A database internal consistency error. This usually results from running a new version of RumorMill on an old database. It can also occur when the database has become corrupted.
12 An error occurred initializing the overview (XOVER) database.
13 An error occurred initializing the client (pull) side of RumorMill.
14 An error occurred initializing the preferences. Usually caused by an old version of a preference file.
15 An error occurred getting the local host address/name. This usually occurs when the system has no IP address assigned (either statically or dynamically).
17 An error occurred initializing the expiration code.
19 An error occurred initializing the threads code. Usually this occurs when the thread manager isn't installed.
20 An error occurred setting up the IP listener on port 119. This can occur if another application is already running that has port 119 (e.g., another copy of RumorMill), or if the system is in a damaged state due to some kind of application or system fault.
21 An error occurred initializing the common component of the database system.
Originally, large chunks of this text were lifted from Brian Aslakson's Wonder Cheat Sheet. Many thanks to Brian for his enthusiasm, encouragement and criticism. And thanks to Bill Stewart-Cole for making me get my [JN] act together. :)
A special thanks to Peter Lewis of Stairways Software whose help and encouragement was essential to RumorMill happening at all. Jeremy Nelson also played an essential role by writing RumorMill Setup. And finally, Andrew Tomazos for his comments and insights.
And finally, thanks to the RumorMill users, whose suggestions, comments, bug reports, and criticisms have helped to make RumorMill better.
2nd July 1997, Jeremy Nelson.