MHonArc FAQ: Archives

[Prev] | [TOC][Manual][Home] | [Next]

## Archives

- Why does a message get split into mulitple messages with no headers?
- Can I move a message from one archive to another?
- Can I reconstruct a database from the HTML messages?
- Is it safe to add messages to an archive as they are received?
- So it is safe. How do I do it?
- How can I do it with Majordomo lists?
- Can I get MHonArc to filter messages to different archives?
- Does MHonArc support the "no archive" flag in messages?
- Is it safe to specify `-add` when no archive exists?
- Why are there "jumps" in message numbers?
- Why do some messages get re-added each time MHonArc processes a mail folder?
- How do I remove messages from an archive?
- Can I convert an archive back to mailbox format?

| ### Why does a message get split into mulitple messages with no headers?

If you are processing UUCP mailbox files, messages are separated by a line starting with "`From` " (ie. The word "`From`" followed by a space). Some mail software will prefix lines in message bodies with a ``>`' to avoid MUA's from incorrectly treating the line as a message separator. However, some mail software doesn't.

To avoid incorrect separator detection, many MUAs perform a more stricter detection of separators beyond "`From` ". MHonArc, by default, will treat lines starting with "`From` " as a message separator, which can lead to incorrect message termination if the From line has not been escaped with a ``>`'.

To fix the problem, use the MSGSEP resource to instruct MHonArc to use a stricter test detecting a message separator. The following MSGSEP resource setting is known to work well:

    <MsgSep>
    ^From \S+\s+\S+\s+\S+\s+\d+\s+\d+:\d+:\d+\s+\d+
    </MsgSep>

In case you have message separates with quoted local parts in the address part of the separator, you can use the following:

    <MsgSep>
    ^From\s+(?:"[^"]+"@\S+|\S+)\s+\S+\s+\S+\s+\d+\s+\d+:\d+:\d+\s+\d+
    </MsgSep>

Make sure to test things out before using in production environments.

If this fails, you can try the CONLEN resource available in v2.0 and later. The CONLEN resource, when set, tells MHonArc to utilize the `Content-Length` fields in the message head. If your MTA defines this field accurately, then you can utilize this feature. Sun Solaris' delivery agent will define the `Content-Length` field for messages delivered to local users.

If you use Procmail to filter your mail, you can try the following Procmail recipe (contributed by Christopher Lindsey):

> However, one can add a Content-Length: header with everyone's favorite tool, procmail. :) Here's a recipe borrowed from David Tamkin about 9 moons ago:

>       :0fhw # B won't help; size conditions ignore H and B flags on the :0 line
>       * ! ^Content-Length:.*[0-9]
>       * 1^1 B ?? >1
>       | formail -a "Content-Length:  $="

> So if you want to count on Content-Length, the message could be piped into procmail with a specific procmailrc file which would do this counting and then call MHonArc.

If sendmail is your system's MTA and you use Procmail as your local delivery agent, you can configure things to have `Content-Length` defined for all locally delivered mail. The following technique is contributed by Jason L Tibbitts III:

> I use Procmail as my local delivery agent. I have the following extremely disgusting settings in my .mc (M4 config) file to add in a Content-Length: header, but I'm not sure if I would recommend that anyone actually think of using this:

>     define(`LOCAL_MAILER_FLAGS',`SPfhn9Z')
>     define(`LOCAL_MAILER_ARGS',`procmail -a $h -d $u')

>     LOCAL_CONFIG
>     #
>     # Add fake Content-Length Header for local mailer
>     # This is corrected by Procmail
>     # Note that the Z flag is used here; if Z is ever defined, this
>     #  will break something
>     H?Z?Content-Length: 0000000000

If possible, try to avoid relying on the use of `Content-Length` since it is hard to guarantee that it is set properly. When CONLEN is specified, MHonArc will read at least the number of bytes specified by `Content-Length` before checking for the message separator, as defined by the MSGSEP resource. Therefore, the value of `Content-Length` can be less than the actual message size, and message extraction will work as expected. However, if `Content-Length` for a message has a value that is larger than the actual size, MHonArc may include the content of the next message as part of the current message.

| ### Can I move a message from one archive to another?

No. In order to achieve the same effect, you must add the original, unprocessed, message to the destination archive, then remove the appropriate HTML version of the message from the source archive.

| ### Can I reconstruct a database from the HTML messages?

Yes. v2.3 of MHonArc introduced a utility program called mha-dbrecover. It gets installed with the other MHonArc files during the installation process. See the documentation for usage information.

| ### Is it safe to add messages to an archive as they are received?

Yes. MHonArc performs archive locking to protect from multiple MHonArc process attempting to write to an archive at the same time. This locking allows MHonArc to safely be used to add messages as they are received.

NOTE: | As an archive increases in size, performing updates as a message is received takes more processing time. Therefore, for large archives, you may need to do updates through a periodic batch process (like via cron(8)) to avoid time-out problems.

| ### So it is safe. How do I do it?

Many users use Procmail <http://www.procmail.org/> to call MHonArc to archive messages. Procmail provides the ability to preprocess mail as it arrives to do selective processing and automated tasks with your mail.

For illustrative purposes, the following simple example shows a possible way of archiving messages as it arrives w/o using a tool like Procmail. This example assumes you are on a Unix-based system using sendmail as the mail transfer agent. Please refer to documentation about sendmail if you are not familiar with it (sendmail, 2ed, from O'Reilly is an excellent source).

The approach shown here uses a `.forward` file in the home directory of the account you want mailed archived. For this example, let's assume it is my account. Here is how to set up the `.forward` file to invoke MHonArc on incoming mail:

    \ehood, "|/home/ehood/bin/webnewmail #ehood"

NOTE: | The "`\ehood`" tells `sendmail` to still deposit the incoming message to my mail spool file. The "`#ehood`" Bourne shell comment is needed to insure the command is unique from another user. Otherwise, `sendmail` may not invoke the program for you or the other user.

`webnewmail` is a Perl program that calls MHonArc with the appropriate arguments. A wrapper program is used instead of calling MHonArc directly to keep the `.forward` file simple, but you can call MHonArc directly if you want. Here is the code to the `webnewmail` program:

    #!/usr/local/bin/perl
    # Edit above path to point to where perl is on your system.

    ##	Specify a package to protect names from MHonArc.

    package WebNewMail;

    ##	Edit to point to installed mhonarc.

    $MHonArc = "/home/ehood/bin/mhonarc";

    ##	Define ARGV (ARGV is same across all packages).
    ##	Edit options as required/desired.

    @ARGV = ("-add",
    	 "-quiet",
    	 "-outdir", "/home/ehood/public_html/newmail");

    ##	Just require mhonarc, this prevents the overhead of a
    ##	fork/exec.

    require $MHonArc;

The `webnewmail` program has to have the executable bit set. This is achieved by using "`chmod a+x webnewmail`".

NOTE: | For better scalability and resource usage, the author recommends calling MHonArc from a facility like `cron`, which is provided on Unix-based operating systems. For those unfamiliar cron, it is a daemon that allows the execution of commands on a scheduled basis.

| ### How can I do it with Majordomo lists?

Here is a template for archiving messages as they arrive for a Majordomo list to include in sendmail's `aliases` file:

    xxxx:                "|/usr/lib/majordomo/wrapper resend -l xxxx xxxx-outgoing"
    xxxx-outgoing:       :include:/var/lib/majordomo/lists/xxxx, xxxx-mhonarc
    xxxx-request:        list-admin-address
    owner-xxxx:          list-admin-address
    xxxx-owner:          list-admin-address

    xxxx-mhonarc:        "|/usr/lib/majordomo/wrapper mhonarc -add -quiet -outdir /home/httpd/html/yyyyyyy -rcfile rcs.mrc -stderr /var/log/mhonarc"

Replace text that `is rendered like this` with what is appropriate for your configuration.

In order to run MHonArc with Majordomo's wrapper, the program has to be in the same directory where the Majordomo programs are located. An easy way to insure this is to create a symbolic link to in Majordomo's program directory to where MHonArc is installed. For example:

    prompt> ln -s /usr/bin/mhonarc /usr/lib/majordomo/mhonarc

Make sure `/usr/bin/mhonarc` is readable and executable by the majordomo user. Something like the following can be done to insure this:

    prompt> chmod 755 /usr/bin/mhonarc

If you redirect stderr to a logfile, the logfile must be owned by the majordomo user and be writable by the majordomo user. The directory for that logfile must exist.

The MHonArc-archive directory must be owned by the majordomo user and must have the minimum access permission 755. Group ownership does not matter.

| ### Can I get MHonArc to filter messages to different archives?

No. This is outside of the MHonArc's scope. You can grow your own filter, using the method described in the previous question, to scan the message header an invoke MHonArc with the proper arguments. Or. you can use a tool like Procmail <http://www.procmail.org/>.

NOTE: | You may want to check out the following: mharc at <http://www.mhonarc.org/release/mharc/>: Mharc is a web-based mail archiving system for multiple mailing lists using Procmail, MHonArc, and Namazu.

Here are a some messages from users about using Procmail:

> - From: Achim Bohnet <ach@rosat.mpe-garching.mpg.de>
> - Date: Wed, 13 Nov 1996 13:56:08 +0100

>     ... some text deleted ...

>     Here is what I use in .procmailrc to archive the mhonarc list:

>     NEWDATE="`/usr/bin/date +%Y-%m`"
>     MHONARC_MBOX="/local/mail/lists/mhonarc/$NEWDATE.mbox"
>     :0: $MHONARC_MBOX$LOCKEXT
>     * ^Sender:.*owner-mhonarc@
>     {
>             :0 c
>             $MHONARC_MBOX

>             :0 c
>             | /local/mail/mhonarc-1.2.2/mailarchive -add mhonarc "$NEWDATE"
>     }

>     Mailarchive is nothing more than a wrapper around mhonarc with my long.
>     list of options.

>     Achim
>     P.S. Procmail itself comes with an example manual page. It's worth
>          looking into it.

> - From: "Eric D. Friedman" <friedman@hydra.acs.uci.edu>
> - Date: Wed, 13 Nov 1996 06:38:42 -0800

>     You can actually dispense with the wrapper if you use environment
>     variables to pass options to MHonArc, but I'm sure Achim has a good
>     reason for doing it his way.  Just for the purposes of comparion,
>     here's how I do it:

>     eeeweb% cat .procmailrc
>     #Set on when debugging
>     VERBOSE=off
>     #Replace `mail' with your mail directory (Pine uses mail, Elm uses Mail)
>     MAILDIR=$HOME/Mail
>     #Directory for storing procmail log and rc files
>     PMDIR=$HOME/.procmail
>     #Path and options for mhonarc
>     MHONARC='/dcs/packages/infosys/bin/mhonarc -add -quiet -umask 022 -idxfname inde
>     x.html'
>     :0
>     * ^Originator:.*@classes.uci.edu
>     {
>       MHHOME=$HOME/classarc
>       LOGFILE=$PMDIR/classlists.log
>       INCLUDERC=$PMDIR/rc.classlists
>     }
>     :0 E
>     {
>       MHHOME=$HOME/mail-arc
>       LOGFILE=$PMDIR/otherlists.log
>       INCLUDERC=$PMDIR/rc.otherlists
>     }

>     and then in the file .procmail/rc.classlists or rc.otherlists (depending
>     on the Originator: of the message), lots of the following:

>     # Procmail Entry for uci-www
>     :0 E
>     * ^TOuci-www
>     {
>       :0 c
>       uci-www/.

>       :0
>       |$MHONARC -rcfile $MHHOME/uci-www/0-rcfile.html -outdir $MHHOME/uci-www
>     }

>     Eric D. Friedman
>     friedman@uci.edu

> - From: Paul McKinley <mckinley@austin.asc.slb.com>
> - Date: Mon, 21 Apr 1997 15:29:08 -0500

>     ... some text deleted ...

>     I use procmail to drive mhonarc archives from Majordomo.  I set up a
>     single pseudouser and drive several archives from the one pseudouser.

>     Here's a sample .forward file:

>     "|/usr/ucb/rsh cappuccino \"set IFS=' '; exec
>     /usr/local/procmail/bin/procmail #widget\""

>     Another example is:

>     "|/bin/csh -c \"set IFS=' '; exec /usr/local/procmail/bin/procmail
>     #widget\""

>     Two reasons to use the "rsh cappuccino":
>     1. doesn't require the user to be able to login to server, although
>        the username must still be valid
>     2. gets the processing load off the mail server

>     Here's an example .procmail recipe:

>     LOGFILE=$HOME/procmail_errors
>     LOGABSTRACT=all
>     LOCKEXT=.lock
>     VERBOSE=on
>     UMASK=003

>     # widget: list short description
>     :0 H
>     * ^List-Name: widget
>     {
>       # The rotate call (under construction) does archive rotation
>       # leave commented!
>       #:0c i
>       #| /home/web-arch/bin/rotate /usr/local/web/webarchive/widget

>       # Put the mail in the mailbox, which is used by archiver to re-generate
>       # the html indexes
>       :0 cA
>       /usr/local/web/webarchive/widget/current/mbox

>       # The mhonarc call examines mbox, turns the mail messages into .html
>       # documents, and compiles the indexes.
>       # -reverse -treverse\
>       :0 ia
>       | /usr/local/mhonarc/bin/mhonarc \
>         -idxfname index.shtml \
>         -tidxfname threads.shtml \
>         -rcfile widget.rc\
>         -outdir /usr/local/web/webarchive/widget/current \
>         /usr/local/web/webarchive/widget/current/mbox

>     }

>     I have a directory per archive, and put the current period in directory
>     "current".  Then I have an index page per archive that indexes the
>     periods, plus gives information about the list and how to
>     subscribe/unsubscribe.  The widget.rc file resides in the pseudouser's
>     home directory.

>     Note the
>     * ^List-Name: widget
>     I put the following in the majordomo list's config file:

>     message_headers   <<  END
>     List-Name: widget
>     END

>     This adds the "List-Name" header to messages, which is what procmail
>     filters for.

>     Hope this helps

>     Paul McKinley
>     Unix SysAdmin Contractor

| ### Does MHonArc support the "no archive" flag in messages?

Version 2.4, or later, does via the CHECKNOARCHIVE resource.

If using an earlier version, or if you are already doing some preprocessing, you can use a pre-processor like Procmail to do the filtering. Here is a message sent to the MHonArc mailing list:

> - Subject: Re: No archive
> - From: Christopher Lindsey <lindsey@ncsa.uiuc.edu>
> - Date: Thu, 1 Oct 1998 11:33:07 -0500 (CDT)

>     > Subscribers who don't want their messages to be archived
>     > could add a "no archive" flag within their mail.

>     The most common way to do this is by checking for the existence
>     of an 'X-no-archive: yes' or 'Restrict: no-external-archive' header.

>     > As I'm invoking MHonArc through a procmail recipe I guess
>     > it's possible to do this within the recipe.

>     Very easy:

>        # If people don't want to be archived, then remove their
>        # message
>        :0
>        * ^(X-no-archive: yes|Restrict: no-external-archive)
>        /dev/null

>     Chris

| ### Is it safe to specify `-add` when no archive exists?

Yes. If MHonArc sees no archive exists when perform an add, it will automatically create the archive.

WARNING: | If using MHonArc versions 2.4, or earlier, make sure the file `maillist.html` (or the value of the IDXFNAME resource) does not exist if no archive exists and `-add` has been specified. Otherwise, unpredictable output of the `maillist.html` file may result if `maillist.html` is not in the proper format.

| ### Why are there "jumps" in message numbers?

Big gaps in the message number sequence may occur if you defined the MAXSIZE resource and you have MHonArc rescanning a mail folder for adding new messages. The problem occurs when MHonArc reads in messages that will automatically get deleted due to MAXSIZE. Ie. Messages subject to automatic deletion are the oldest ones. If the input contains old messages that will get deleted at the end of processing, the old messages will still use up message numbers since messages to be deleted are not determined until all input is read. Since MHonArc does not keep information about deleted messages, if the messages are fed into MHonArc again, the "jumping" will occur again (and the jump will get larger for each additional update).

To avoid the problem, try to pass only new, never processed, messages to MHonArc instead of having MHonArc rescanning the same mail folder for new messages. Another approach is to set either the EXPIREAGE or EXPIREDATE resources (available in v2.0 beta 2, or later). These work as an alternative to MAXSIZE and will help in preventing message number jumping since expiration of a message is checked when it is initially read (bypassing the assignment of a message number).

| ### Why do some messages get re-added each time MHonArc processes a mail folder?

This condition may occur when you have MHonArc examine the same folder periodically to add any new message. If there are messages in the folder without message-ids, then those messages will be re-added each time MHonArc runs.

Why? Well, MHonArc uses message-ids for determining if a message has been archived, or not. Therefore, if a message-id is missing for a message, then MHonArc believes it is new.

In general, mail has message-ids. They get assigned by MTAs. However, if messages are generated by a CGI program, or other non-mail specific software, then the program in question should create a message-id. Else, you will need to move already-processed messages into a different area so MHonArc does not read them again.

NOTE: | In MHonArc v2.4 and later, and if you have the Digest::MD5 module installed, MHonArc will compute the MD5 digest of message headers without message-ids. This allows MHonArc to skip the message in subsequent add operations.

A related problem is messages showing up again in the archive after you deleted them with RMM. MHonArc does not keep track of delete message-ids. Therefore, if want to make sure that a message will not appear in the archive after explicitly deleted via RMM, make sure to remove the message from input source.

| ### How do I remove messages from an archive?

Automatic removal can be done via the EXPIREAGE or EXPIREDATE resources (available in v2.0 beta 2, or later) or the MAXSIZE resource.

Explicit message removal can be done with the RMM resource. Please read the RMM resource page for more information and examples.

| ### Can I convert an archive back to mailbox format?

Anthony W developed a Perl program called `mhn2mbox` for converting archives back into mailbox format. A copy of the script is included in the contrib directory of the MHonArc distribution.

[Prev] | [TOC][Manual][Home] | [Next]