RumorMill 1.3 Changes

Copyright ©1996-2002, Jim Calvin.

Updated to reflect RumorMill version 1.3.4

Introduction

This document describes changes made to the 1.3 version of RumorMill. It is an interim document meant to provide information to beta testers of version 1.3. This information will eventually be folded into the standard documentation for RumorMill.

Note that RumorMill Setup 1.3 does not implement a way of accessing all of the features listed below. If you wish to use features not supported by RumorMill Setup, you must use Telnet (normally on port 119) to configure and or use those features.

Important Notices

Running RumorMill 1.3 in the same folder as your 1.2 installation will irreversably modify your databases. That is, version 1.3 will automatically update your files to be used with version 1.3 and there is no way to go back to 1.2. At a minimum you should use the following "save" commands before starting to use RumorMill 1.3

• Save Groups...
• Save Users...

Also be aware that the conversion process can take awhile and require some (temporarily) additional disk space.

RumorMill now requires a registration number for operation. When first installed, RumorMill will operation normally for about one week. After that time, a registration number is required for operation.

System requirements

The system requirements for RumorMill 1.3 are currently unspecified. Development and testing was done on MacOS 8.6 and 9.0.4 on PowerPC based Macs. RumorMill 1.3 is built for both PowerPC (but not as a Carbon application) and 68K machines.

RumorMill 1.3 will run properly on OS X as a classic application. An OS X native version of RumorMill is currently in development.

Changes summary

The list below summarizes the changes to RumorMill 1.3. Details on each of these can be found later in this document.

• A status window was added
• A registration ID is now required
• A new database structure was implemented
  • 2GB article file limit is gone
  • Much faster expiration processing
  • Faster forwarding
  • Somewhat faster article retrieval
• Automatic backups of the RumorMill preference file (1.3.2)
• Ability to add articles from files (1.3.2)
• User limitations for group reading and posting
• SYSLOGD logging
• Virutal groups (but, RumorMill Setup doesn't support this feature)
• Support for the X-Trace header
• Many new preferences and minor features

Contents

Status Window

Virtual Groups

User-Group Access Control

New Database Structure

New Preferences

Logging and syslog Support

New server commands

Changes in Behavior

Known limitations

Status Window

This is the most visible change in RumorMill 1.3 - a status window. When open this status window displays the following things:

1. RumorMill version and host name (address)
2. Date and time RumorMill was started
3. Date and time of last expiration
4. Free memory
5. Disk space health and group cache status
6. Connections, active and total handled
7. Articles, added and forwarded
8. Errors logged (count) and number threads active
9. Current status: Ready, Pull, Forwarding, GC (Expiration), No clients, ...

RumorMill will remember the position and open/closed-ness of the status window.

Virtual Groups

Virtual groups is a new capability that allows a system administrator to configure RumorMill to have thousands of groups with only a subset of those groups active. This feature provides the capability of appearing to support many groups while maintaining articles for only those in active use.

When the virtual group capability is enabled (see the Enable Virtual Groups preference), groups are either active or virtual. Only active groups will pull or accept new articles from a news feed. Groups which are virtual may have articles in them, but no new articles will be added while the group is virtual.

Groups transition from virtual to active and back based on references to each group. Every time a client attempts to look at a group (the GROUP command is issued), a counter associated with the group is incremented. When that counter equals or exceeds the value of the Virtual Group Threshold preference, the group becomes active. Each evening RumorMill will decrement the counter associated with the group. If the counter drops below the Virtual Group Threshold, the group will become virtual.

The system administrator may also mark selected groups as permanently active. This can be done using the XGROUP SET PERM command described elsewhere.

The XINITVIRTUAL command may also be used by the system administrator to automatically create a large number of groups. Since there is no way to remove large numbers of groups, this command should be used with great care, and only after the Virtual Group Filter preference is setup.

Please note: The RumorMill Setup application has not yet been updated to support virtual groups. Also note that the Setup application will not support more than about 1,300 groups in the database.

User-Group Access Control

In some cases, system administrators may wish to user a finer level of control over which users may read from and post to groups. Requiring login for any use, or to be able to post provides one level of control.

Version 1.3 of RumorMill provides capabilities to limit which groups users may read from and/or post to. The sysadmin may create a set of "User Group descriptions." Each user group simply contains the following:

  A unique user-group index (small integer)
  A name; provided for the user and is not used by RumorMill
  A list of group names, e.g., "comp.sys.mac.*,rec.bicycle.*"
  A flag saying whether the specified group list is to be EXCLUDEd or PERMITted

Each user has two such associated descriptions, one for read limitations, and one for posting limitations. So if a user's read limitations were set to "permit" "local.*", then that reader could only read groups that began with "local."

However, if the user's group description was set to "exclude" "local.*," then the user would be able to read all news groups other than those starting with "local." The posting limitations work exactly the same way.

Users not logged in (authenticated) are automatically defaulted to user group description with an index 0 (zero) for reading and 1 (one) for posting. By using these "special" user group descriptions, guest, or non-authenticated users can also be restricted.

User groups can be accessed using the "User Groups" and "User" dialogs in the Setup Application

The following commands (via Telnet) can be used to configure user group descriptions.

XUSERGROUP ADD <user-group index> EXCLUDE | PERMIT <user-group name><cr>
<group list>

XUSERGROUP DELETE <user-group index>

XUSERGROUP LIST
returns the following for each user-group
<user-group index> PERMIT | EXCLUDE <user-group name>
<group list><cr>

The list ends with <cr>.<cr>

In addition, the commands for manipulating users are modified to allow specifying user-group descriptions. See the changes below.

New Database Structure

The database structure has changed in RumorMill 1.3. After running version 1.3 the following new entries will appear in the Database folder:

Name	Purpose and notes
Articles folder	This folder contains all of the files comprising the articles and overview databases. If one looks at an article file with an editor like BBEdit, the files may look like text files. However, they are NOT text files and should not be modified in any way.
Articles to Inject	This folder is not created by RumorMill. The system administrator must create it. If the folder exists, then RumorMill will periodically check the folder for files. If files exist, each file will be handled as if it is a single article that had been received from a remote server (or posted). Once the file is processed, the file is moved to the trash.
By Group Index	This file contains the data required to find article <n> of group <name> in the articles files. This file is a database and cannot be read via an editor or other application.
Message IDs	This file contains the message IDs of all articles in the database. Associated with each message ID is sufficient data to find the article and overview in the articles files. This file is a database and cannot be read via an editor or other application.
By Group Index	This file contains the data required to find article <n> of group <name> in the articles files. This file is a database and cannot be read via an editor or other application.
User Group Restrictions	This file contains the user-group access information. The file is a database and cannot be read via an editor or other application.
Articles	This is the old article database and is normally moved to the trash after its contents are converted to the new article format.
Group Indexes	This is a old database that contained overviews for articles as well as the mapping from article <n> of group <name> into the article database. This file is moved to trash after the conversion to the new database format is completed.

As a result of these file changes, RumorMill is no longer limited to an article database that will fit into a single file (about 2GB). The limiting factor is now the 2GB limit on the files By Group Index and Message IDs.

The database files Cancel History and Article History are no longer used.

New Preferences

Normally the "Date Bounds" preference is checked to see if articles being received (either forwarded to the server or the server is pulling) are "old." If the articles are old, or outside the date bounds, they are discarded upon receipt. If the "Allow Old Articles" preference is set to non-zero, then the "Date bounds" setting is ignored and the articles are added regardless of their age. This preference is normally set to non-zero if you're spooling a full database (or something like that) to a new RumorMill server. This preference is not used for POSTed articles (articles that a user adds from a newsreader). The recommended setting is zero.

Article files are organized such that all articles with expiration dates within a certain period of time (a window) are put into the same article file. This prefeference specifies the number of days that an article file may span. Smaller numbers allows more rapid expiration of article files, but create larger numbers of article files. Having more article files may decrease efficiency if the Max Open Article Files is smaller than the number of active article files.

This preference controls how RumorMill deals with article files once the available disk space drops below the "Disk space threshold" level. The status window will report this condition as disk health "bad". The preference has three possible settings as follows:

Value	Action
0	No automatic deletion of article files
1	Delete article files about 1 per minute until disk health is no longer "bad"
2	Delete article files until disk health is no longer "bad"

The number of ticks (1/60 of a second) that RumorMill may consume will running as a background task. Making this larger improves RumorMill performance when running as a background application at the expense of other applications.

The streaming protocol allows the sender to check with the downstream server to see if it wants an article or not. Alternatively, the sender can just send the article. This preference controls the threshold (maximum article size) of articles that are sent without using the CHECK command. For slow connections, this probably shouldn't be larger than 1200 or so. For faster connections, 10,000 or so may be more appropriate. N.B.Some news servers do not function properly if the CHECK command is not used. If you are streaming to such a server, set this parameter to 1 to assure that the CHECK command will always be used.

Specifies the number of days between attempts to compact the By Group Index database.

Specifies the number of days between attempts to compact the canceled message-id database.

Specifies the number of days between attempts to deferred control items database.

Specifies the number of days between attempts to compact the Groups database.

Specifies the number of days between attempts to compact the message-id database.

Specifies the number of days between attempts to compact the a users database.

If sufficient RAM is available, RumorMill with cache all of the group data in RAM to improve performance. This preference can be used to enable/disable this capability.

This preference enables/disables the virtual group feature. Please note: This preference is not implemented in the RumorMill Setup application.

The number of ticks (1/60 of a second) that RumorMill may consume will running as a foreground task. Making this larger improves RumorMill performance when running as the front (selected) application at the expense of other applications.

This preference controls how often RumorMill checks the Articles to Inject folder for files. Please note: This preference is not implemented in the RumorMill Setup application.

When set to non-zero, this preference causes RumorMill to log any newsgroup name found in a article to a log file. Please note: This preference is not implemented in the RumorMill Setup application.

This preference sets (approximately) the maximum size of each article file. A large maximum is appropriate for installations receiving large volumes of articles. N.B.: This is specified in megabytes. RumorMill does not support article files greater than 2GB in size, nor does it have any need to since there are now multiple article files.

Some articles have unreasonable expiration dates specified. This preference can be used to limit how far into the future an Expires header may specify. The preference is specified in days. Please note: This preference is not implemented in the RumorMill Setup application.

This preference controls the number of article files RumorMill may have open at any one time. Under MacOS 9 and later systems this setting is not so critical as the maximum number of open files is dramatically increased. Please note: This preference is not implemented in the RumorMill Setup application.

Sometimes a system administrator may wish to run a RumorMill server behind a firewall or Network Address Translator (NAT). This allows the server to run on a machine that is not fully exposed to the Internet. Normally the firewall or NAT will port map news traffic (nntp protocol on port 119) to a machine behind the firewall.

However, if this is done, RumorMill will report the name of the machine behind the firewall in the greeting and in article postings. THe NAT host name can be used so that RumorMill will report the name of the firewall or NAT host instead. For this to work, the specified name must be a valid host name - that is, RumorMill must be able to look up the name and get an IP address for it.

This preference allows/disallows the use of the PAT and XPAT commands. Since these commands can consume arbitrarily large amounts of processor time, some system administrators may elect to disable the use of the PAT and XPAT commands.

The SPAM If preference considers any article SPAM if it contains any of the groups specified in this list. Any article with a matching group name will be discarded and not entered into the article database. Please note: This preference is not implemented in the RumorMill Setup application.

The SPAM If Not preference considers any article SPAM if the incoming article does not contain at least one of the groups specified in this list. Any article without a matching group name will be discarded and not entered into the article database. Please note: This preference is not implemented in the RumorMill Setup application.

This preference specifies how often (in seconds) the status window is updated. Please note: This preference is not implemented in the RumorMill Setup application.

This is the IP address of a host running a syslog deamon. RumorMill will send logging data to this host as specified by the SysLog Priority preference. The IP address should be specified as a dotted quad, e.g., 18.1.6.77.

This is the port number that the syslog deamon is listening on. The default is port 514.

This specifies the message priority that should be sent to the syslog deamon. The value of -1 disables syslog logging. Syslog protocol uses 0 as the most serious of log items, while 7 is considered debugging information. Setting it to 6 will log all but debugging information.

This preference is meant for temporary operation only. When adding a large number of users to RumorMill, it may be convient to type up the user list in a file rather than entering them one by one via RumorMill Setup. The problem is that the "Create Users..." command in RumorMill expects the passwords to be encoded rather than clear text and there is no way to create such a file by hand. Setting this preference to one (1) instructs the "Create Users..." command to interpret the passwords as clear text and convert them to encoded form. This preference should normally be set to zero (0). Please note: This preference is not implemented in the RumorMill Setup application.

A system administrator may wish to add a header to every article posted at her server. The "User Header" may be used for this purpose. The header must start with "X-", and must contain a ": " with at least one trailing character. In other words, a minimally acceptable user header is something like "X-: Z". Something more descriptive like "X-complaints-to: admin@yoursite.com" is recommended.

This preference is used when initializing the virtual group capability by downloading the group list from a remote server (pull). Only groups matching this specification will be added to the group database.

Specifications of the form *,!*microsoft*,!*hipcrime* are permitted. This specification would allow any group except ones containing microsoft or hipcrime in the group name. Please note: This preference is not implemented in the RumorMill Setup application.

For a virtual group to become active (articles will either be pulled or accepted from a feed), the number of recent (within 24 hours) references to the group (a client attempted to read it) must exceed this threshold. Please note: This preference is not implemented in the RumorMill Setup application.

Preferences removed from 1.3

The Thread Quantum preference was replaced by the Background quantum and Foreground Quantum preferences.

New server commands

There are several new commands (via a Telnet connection on port 119) in version 1.3. A summary of these follows:

This variant of the LIST command returns the message of the day. The text is taken from the file "Message of the day" in the "Databases" folder. If no file is provided by the SysAdmin, the command returns an empty message of the day. Please note: This command is not implemented in the RumorMill Setup application.

This version of the LIST command was added to return just the list of groups. No information about the number of articles is returned. This command is intended for use by RumorMill Setup.

This command will delete the article file that would next be deleted for expiration. If you're tight on disk space, this command is the best way to quickly free up some disk space. Successive invocations will continue to delete the article file nearest to reaching expiration until no article files remain.

This command can be used to see if a given group is permanent or not. When virtual groups are enabled, a permanent group cannot become virtual due to lack of use. Please note: This command is not implemented in the RumorMill Setup application.

This command is used to set (or unset) whether a given group is permanent or not. Permanent groups do not become virtual due to lack of use. Please note: This command is not implemented in the RumorMill Setup application.

This command is used to determine if a given group is virtual or active. Please note: This command is not implemented in the RumorMill Setup application.

Caution: Use this command with care. It will likely add thousands of groups to your database! Please note: This command is not implemented in the RumorMill Setup application.

This command will:

1. initiate connections to all remote servers marked as PULL or upstream
2. retrieve the total list of groups from that site
3. add each of those groups to the local database, subject to the VirtualGroupFilter preference

This command can be used to construct a forwarding file for remote server that is already configured. When the command is received by RumorMill, the "<remote server>" will be verifed and then the entire database will be scanned for articles that should be forwarded to that remote server. The groups specified in the "groupsToSend" list will be honored when this command is executed.

now returns the form:

280 maxAddrMasks:20,maxremoteServers:16,ngroups:66

where ngroups is the count of active groups at the server.

Logging and syslog Support

The logging system in RumorMill is expanded in version 1.3. First, summaries of all error conditions are kept and reported in the nightly EMail reports. The summaries list each error, the number of times that error occurred, and the first and most recent time that error occurred. These summaries are reset each day at midnight.

Additionally, RumorMill can now log via syslog. Many systems and devices (e.g., printers) use syslog to report error conditions etc. Such systems or devices are configured to send log entries to a syslog deamon running on a computer. A set of preferences (see preferences for details) has been added for configuring RumorMill to use syslog logging.

There are two implementations of a syslog deamon for the Mac. Check the HyperArchive at MIT for pointers to these.

Changes in Behavior

RumorMill 1.3 supports larger article databases. This was done by breaking the Articles file into multiple files. The limiting factor is now the size of the files Message IDs and By Group Index. Faster expiration is a welcome side effect of this change.

When login is required, the can post setting for each user is now verified before the user is permitted to post articles. Previously, any valid logged in user could post if the site permitted it as a whole.

Multiple control groups are now permitted. If the groups control.cancel, control.rmgroup, and control.newgroup exist, that is, the groups were created by the system administrator, these groups will be used as respositories for control articles of these types. These groups are hidden from users in the same way that the control group is.

A preference, PAT Permitted was added to allow this processor intensive operation to be disabled.

RumorMill now prefers to create log and preference files in the same folder as the RumorMill application. However, previous placements are still supported. If RumorMill finds a log or preference file in the system's preference folder, RumorMill will continue to use those files. However, if one does not exist there, the files will be created in the local folder.

Log and preference files currently in the preference folder may be moved to the local folder if desired. As always, aliases are supported.

RumorMill now supports the definition of an alias for a remote server's host path. This alias is useful when the name inserted into the Path header is different than the IP host name for the remote server. To specify a host path alias, via a Telnet connection, simply append the alias to the host name separated by a tab when specifying a remote server entry. For example:

news.macconnect.com newsguy        { host name and alias }
pull                               { a pull from this host }
                                   { empty groups to send list }
                                   { empty groups to exclude list }
comp.sys.mac.*                     { just pull the comp.sys.mac tree of groups }
foouser foopass                    { authentication data for this host }
0                                  { use default pull period }
0                                  { use default flood period }

Prior to version 1.3, RumorMill did not have a mechanism to detect when two newsfeeds were offering the same article. In the past RumorMill would tell both feeds to forward the article and then tell the slower of the two feeds that the article was a duplicate. While this did not affect the end result, it was a sub-optimal behavior. This situation is now properly handled in RumorMill 1.3.

RumorMill used to hang (in some cases, the entire machine) for a period of time while host name lookups were done. This process is now done asynchronously so that these hangs are eliminated.

Not all phases of expiration processing require that no clients be connected to RumorMill. In the past, all connections to RumorMill were terminated with expiration processing started. Connections are now only terminated when reconstruction of certain database files is in progress.

The number of outstanding articles when streaming was increased from five to ten. This should provide slightly better performance when forwarding articles to a downstream remote server.

The CHECK Threshold now specifies the size of articles for which the CHECK step of the streaming protocol may be skipped. This can be used to optimize streaming of articles. The threshold should be set depending upon the available bandwidth to the downstream servers. A value like 1200 is reasonable for slower links.

Changes were made to the user commands (XAUTHINFO ...) to support user-group descriptions. These commands now look like:

XAUTHINFO ADD <username> READONLY|POSTOK <read user-group index> <post user-group index> <password>

XAUTHINFO MODIFY <username> READONLY | POSTOK <read user-group index> <post user-group index> <new password>

XAUTHINFO LIST
returns:
<user name> READONLY | POSTOK <read user-group index> <post user-group index> <password> <last login date and time>

RumorMill (as of 1.3.2) automatically backs up its preference file. Each night the preference file is copied to a file of the same name with ".bk" appended to the file name. Once a week, the ".bk" file is renamed to a file ending in ".wkbk" before the new ".bk" file is created.

Known limitations

The RumorMill Setup application cannot handle more than about 1,300 groups. If you have more than 1,300 groups, RumorMill Setup will not allow you to manipulate groups using a listing of all groups.