Log_*, separate_connection_log, sndopr_priority MTA options

From MsgServerDocWiki

Logging options (log_connection, log_connections_syslog, log_envelope_id, log_diagnostics, log_filename, log_filter, log_format, log_header, log_intermediate, log_local, log_message_id, log_messages_syslog, log_node, log_notary, log_priority, log_process, log_queue_time, log_reason, log_sensitivity, log_sndopr, log_username, separate_connection_log, sndopr_priority)

The options listed in this section affect MTA logging (and thus to some extent monitoring) and control the fields that appear in LOG_ACTION mapping probes. The log_sndopr option may be set by sites that wish to have syslog messages in cases of logging or counter updating problems. The rest of these logging options affect the formatting of the MTA message transaction log file and connection transaction log file, the logging of optional additional information, and the format of LOG_ACTION mapping probes.

The new XML format for MTA message transaction and MTA connection transaction logging is much more tolerant of extensions; plus it is a new format in any case. In the past, as new logging options were added, the default was that such options were disabled, so that upgrades would not unilaterally change the MTA's transaction logging format and thereby "break" existing transaction log parsers used by sites. However, with XML format which is a new format, and where parsers should to be written to ignore non-understood attributes, this is not a concern. Therefore, as of JS MS 6.3p1, when XML format is enabled (log_format is 4), many of the optional logging options default to being enabled, rather than defaulting to being disabled (as with any format other than XML format). This includes log_filename, log_filter, log_message_id, log_notary, log_priority, log_process, log_queue_time, log_reason, and log_username.

log_connection (0-1023)

The log_connection option controls whether or not connection information, e.g., the domain name of the SMTP client sending the message, is saved in the mail.log file (or the connection.log file if separate_connection_log is set to 1). This value is a decimal integer representing a bit-encoded integer, the interpretation of which is given in the table below.

TCP|MTA-IP|MTA-port|remote-IP|remote-port       in message log enqueue entries (E* entries): . This information        appears after the SMTP delivery-status/diagnostic information. In        XML-compatible format (log_format set to 4), this is the       tr attribute.     

Thus for instance setting log_connection to 3 will result both in additional sorts of log file entries---entries showing when an SMTP connection is opened or closed---and also additional information in regular log file entries showing the name of the system connecting (or being connected to), or the channel hostname of the enqueuing channel when the enqueuing channel is not an SMTP channel. (This is a change from PMDF V5.1 and earlier, where the value was simply 0 or 1, with 1 enabling all the then-available connection logging.) TCP/IP channels have a channel level option that can override this setting for particular channels.

log_connections_syslog (integer; UNIX and NT only)

Send MTA connection log file entries to syslog (UNIX) or event log (NT). 0 is the default and means no syslog/event logging is performed. Setting the option to a non-zero value causes syslog/event log logging. The absolute value sets the syslog facility/severity mask. Negative values disable the generation of the regular connection.log* entries.

Note that in JES MS 6.2 and earlier, the length of the MTA output line sent to syslog is limited to 256 characters. For JES MS 6.3 and later, the limit is 4096 characters.

log_diagnostics (0-3)

(New in Cubanelle.) Bit 0 (value 1), if set, causes diagnostics information to appear in certain log entries. Bit 1 (value 2), if set, causes diagnostics information to be included in LOG_ACTION mapping probes. For instance, in the case of "B" records (bad commands received by the SMTP server), the diagnostic field will show the SMTP server error response. In the case of connection close "C" records, the diagnostic file will show the reason why the connection was closed, e.g., reaching some session disconnect limit. In the case of authentication "U" entries (which note are generated as connection transaction entries, rather than message transaction entries), the result of an authentication attempt is shown in the diagnostic field. Appears just after the reason field (see the log_reason MTA option) and before the time-in-queue field (see the log_queue_time MTA option). In XML-compatible format (log_format set to 4), diagnostic information, if enabled, appears as the di attribute. This option defaults to 1 in order to maintain compatibility with previous releases, where diagnostics information was always logged.

log_envelope_id (0-3)

(New in JES MS 6.1.) The log_envelope_id option controls whether or not the envelope ID (ENVID parameter of the MAIL FROM ESMTP command) is included in mail.log records and LOG_ACTION mapping probes. This information may be useful when tracking a message across multiple systems. The default is 0, meaning that the field is not logged. Setting bit 0 (value 1) causes the envelope ID to be logged, immediately after the message filename field and immediately before the message-id field. In XML-compatible format (log_format set to 4), envelope ID logging, if enabled, appears as the ei attribute. New in Cubanelle, bit 1 (value 2), if set, causes the information to be included in LOG_ACTION mapping probes.

log_filename (0-3)

The log_filename option controls whether or not the names of the filesin which messages are stored are saved in the mail.log file or included in LOG_ACTION mapping probes. Setting bit 0 (value 1) enables file name logging. When file name logging is enabled, the file name will appear as the first field after the final form envelope To: address. In XML-compatible format (log_format set to 4), file name logging, if enabled, appears as the fi attribute. As of JS MS 6.3p1, enabling XML format (log_format set to 4) causes the default for log_filename to be 1 (filename logging enabled); with any other format, the default is 0 (file name logging disabled), as in earlier versions. New in Cubanelle, bit 1 (value 2), if set, causes filename information to be included in LOG_ACTION mapping probes.

log_filter (0-3)

The log_filter option controls whether or not any mailbox filter actions (Sieve filter actions) applicable to the message are logged in enqueue "E" records or included in LOG_ACTION mapping probes. Bit 0 (value 1), if set, causes filter information to appear in log entries. New in Cubanelle, bit 1 (value 2), if set, causes filter information to be included in LOG_ACTION mapping probes. This information appears after the optional "intermediate" and "original" forms of the destination address (see the log_intermediate MTA option), before the SMTP diagnostic field (which itself only appears for SMTP messages). In XML-compatible format (log_format set to 4), Sieve filter action(s) logging, if enabled, appears as the fl attribute. The filter action(s) will be enclosed within single quote characters. (In "D" records, one sees merely the two single quote characters, as no filter action is applicable during dequeue.) As of JS MS 6.3p1, enabling XML format (log_format set to 4) causes the default for log_filter to be 1 (Sieve filter action logging enabled); with any other format, the default is 0, (Sieve filter actions are not logged), as in previous versions. With log_filter set to 1, one might see, for instance

'fileinto "SPAM"' 

or

'redirect "user@domain.com"' 

Note that the case of a reject action is special, due to the inherent nature of the reject action. In this case, what occurs is the enqueue of a new message (a Message Disposition Notification) by the original enqueuing channel to the process channel, and that new message has an implicit keep occurring. As there is no enqueue of the rejected message, the reject does not show up in the filter action field of any log record.

log_format (1, 2, 3, or 4)

The log_format option controls formatting options for the MTA message transaction log file, mail.log. A value of 1 (the default) is the standard format. A value of 2 requests non-null formatting: empty address fields are converted to the string "<>". A value of 3 requests counted formatting: all variable length fields are preceded by "N:", where "N" is a count of the number of characters in the field. That is, log_format set to 3 causes length-count tagging of the following fields: The envelope-from address, the original-recipient address, the current-recipient address, the filename, the envelope-message-id, the message-id, the username, the connection information, the intermediate address(es), the Sieve filter action(s), the rejection reason text, the SMTP diagnostic, the transport information, the application information. A value of 4 (new in JS MS 6.3) requests XML-compatible format. As of JS MS 6.3p1, note that setting log_format to 4 also changes the defaults for a number of other log_* options. In general, it enables much of the useful (but in other formats disabled by default) optional logging options, including: log_filename, log_filter, log_message_id, log_notary, log_priority, log_process, log_queue_time, log_reason, and log_username.

In the (new in JES MS 6.3) XML-compatible format (log_format set to 4), each message log entry appears as a single XML element containing multiple attributes and no subelements. Three elements are currently defined: en for message transaction (e.g., enqueue/dequeue entries), co for connection transaction entries, and he for header entries (the optional additional records in the message transaction log file resulting from setting log_header to 1). Enqueue/dequeue (en) elements can have the following attributes:

• ts - time stamp (always present)
• no - node name (present if log_node is 1)
• pi - process id (present if log_process is 1)
• sc - source channel (always present)
• dc - destination channel (field always present, though it will be empty for types of entries other than an enqueue "E" entry)
• ac - entry type or action (always present)
• sz - message size (field always present, though potentially empty for types of entries such as J or V entries)
• so - source address (always present)
• od - original destination address (field always present, though potentially empty for types of entries such as J entries)
• de - destination address (field always present, though potentially empty)
• rf - recipient flags (present if bit 0/value 1 of log_notary is set)
• fi - filename (present if bit 0/value 1 of log_filename is set)
• ei - envelope ID (present if bit 0/value 1 of log_envelope_id is set)
• mi - message ID (present if bit 0/value 1 of log_message_id is set)
• us - username (present if bit 0/value 1 of log_username is set)
• ss - source system (present if bit 0/value 1 of log_connection is set and source system information is available)
• se - sensitivity (present if bit 0/value 1 of log_sensitivity is set)
• pr - priority (present if bit 0/value 1 of log_priority is set)
• in - intermediate address (present if bit 0/value 1 of log_intermediate is set and intermediate address information is available)
• ia - initial (RCPT TO) address (present if bit 1/value 2 of log_intermediate is set and initial address information is available)
• fl - filter (present if bit 0/value 1 of log_filter is set and filter information is available; in particular, will never be present in dequeue "D" records since there is not filtering performed (hence no filter information) at that point)
• re - reason (present if bit 0/value 1 of log_reason is set and a reason string is available)
• di - diagnostic (present if bit 0/value 1 of log_diagnostic is set and diagnostic information is available)
• tr - transport information (present if bit 5/value 32 of log_connection is set and transport information is available)
• ap - application information (present if bit 6/value 64 of log_connection is set and application information is available)
• qt---time in queue (present if bit 0/value 1 of log_queue_time is set)

Here is a sample en entry (wrapped for printing clarity; the actual log file entries always appear in reality on a single line):

<en ts="2004-12-08T00:40:26.70" pi="0d3730.10.43" sc="tcp_local" 
dc="l" ac="E" sz="12" so="info-E8944AE8D033CB92C2241E@whittlesong.com" 
od="rfc822;ned+2Bcharsets@mauve.sun.com" 
de="ned+charsets@mauve.sun.com" rf="22" 
fi="/path/ZZ01LI4XPX0DTM00IKA8.00" ei="01LI4XPQR2EU00IKA8@mauve.sun.com" 
mi="<11a3b401c4dd01$7c1c1ee0$1906fad0@elara>" us="" 
ss="elara.whittlesong.com ([208.250.6.25])" 
in="ned+charsets@mauve.sun.com" ia="ietf-charsets@innosoft.com" 
fl="spamfilter1:rvLiXh158xWdQKa9iJ0d7Q==, addheader, keep"/> 

Connection transaction (co) entries can have the following attributes:

• ts - time stamp (always present; also appears in en entries)
• no - node name (present if log_node is 1; also appears in en entries)
• pi - process id (present if log_process is 1; also appears in en entries)
• sc - source channel (always present; also appears in en entries)
• dr - direction (always present)
• ac - entry type or action (always present; also appears in en entries)
• tr - transport information (always present; also appears in en entries);
• ap - application information (always present; also appears in en entries)
• mi - the name presented on the ETRN command line (present only if bit 0/value 1 of log_message_id is set, and ETRN was used with an argument; note that setting bit 0/value 1 of log_message_id also causes the logging of message ID information, if available, in message transaction (en) log records)
• us - username (present only if bit 0/value 1 of log_username is set and username information is available; also appears in en entries)
• di - diagnostic (present only if bit 0/value 1 of log_diagnostic is set and diagnostic information is available, also appears in en entries)
• ct---time to connect/fail to connect (present only if bit 0/value 1 of log_queue_time is set)

Here is a sample co entry (wrapped purely for printing clarity; in reality, such entries always appear on a single line):

<co ts="2004-12-08T00:38:28.41" pi="1074b3.61.281" sc="tcp_local" dr="+" 
ac="O" tr="TCP|209.55.107.55|25|209.55.107.104|33469" ap="SMTP"/> 

Header (he) entries have the following attributes:

• ts - time stamp (always present; also used in en entries)
• no - node name (present if log_node is 1; also used in en entries)
• pi - process id (present if log_process is 1; also used in en entries)
• va - header line value (always present)

Here is a sample he entry:

<he ts="2004-12-08T00:38:31.41" pi="1074b3.61.281" va="Subject: foo"/> 

Note that the maximum length of MTA transaction record, both for message transaction records and connection transaction records, is 4096 characters.

log_header (0 or 1)

The log_header option controls whether the MTA writes message headers to the message log file, mail.log. Or, as of JES MS 6.0, see also the channel keyword logheader, which may be used to enable this facility on a per-channel basis. A value of 1 enables message header logging. Note that only outermost message headers are available for logging purposes; the "inner" headers on inner MIME parts, if any, are not available. The specific headers written to the log file are controlled by a site-supplied log_header.opt file. The format of this file is that of other MTA header option files. For instance, a log_header.opt file containing

To: MAXIMUM=1 
From: MAXIMUM=1 
Defaults: MAXIMUM=-1 

would result in writing the first To: and the first From: header per message to the log file. A value of 0 (the default) disables message header logging. The logheader channel option can override this MTA option on a per-source-channel basis. When header field logging is enabled and log_format is 1, 2, or 3, the header fields are logged in the format: dd-mmm-yyyy'hh:mm:ss.ss > header-line or up to two additional fields may be present (if log_node is 1 and log_process is 1 -- the header line logging makes no display of any other fields): dd-mmm-yyyy'hh:mm:ss.ss'node'process-field > header-line with one such line per header line to be logged. In XML-compatible format (log_format set to 4), the header fields are he elements.

log_intermediate(0-15)

(New in JES MS 6.2.) The log_intermediate option controls the inclusion of the "intermediate" form of the destination address, and the inclusion of the "original" (RCPT TO) form of the destination address, in MTA message transaction log (mail.log) records. The option takes a bit-encoded integer value.

log_local (0 or 1)

The log_local option controls whether or not the domain name for the local host is appended to logged addresses that don't already contain a domain name. A value of 1 enables this feature, which is useful when logs from multiple systems are concatenated and processed. A value of 0, the default, disables this feature.

log_message_id (0-3)

The log_message_id option controls whether or not message IDs are saved in the mail.log file or included in LOG_ACTION mapping probes. Bit 0 (value 1), if set, enables message ID logging. When message ID logging is enabled, the message ID will be logged after the final form envelope To: address entry---and after the message file name, if log_filename is also enabled. In XML-compatible format (log_format set to 4), message ID logging, if enabled, appears as the mi attribute. As of JS MS 6.3p1, enabling XML format (log_format set to 4) causes the default for log_message_id to be 1 (message ID logging enabled); with any other format, the default is 0 (message ID logging disabled), as in previous versions. New in Cubanelle, bit 1 (value 2), if set, causes message ID information to be included in LOG_ACTION mapping probes.

log_messages_syslog (integer; UNIX and NT only)

Send MTA message log file entries to syslog (UNIX) or event log (NT). 0 is the default and means no syslog/event logging is performed; setting the option to a non-zero values causes MTA message log file entries to be written to syslog. The absolute value of the non-zero value sets the syslog priority and facility mask. Negative values will disable the generation of the regular mail.log* entries; positive values mean that the syslog entries are generated in addition to the regular mail.log* entries.

Note that in JES MS 6.2 and earlier, the length of the MTA output line sent to syslog is limited to 256 characters. For JES MS 6.3 and later, the limit is 4096 characters.

log_node (0 or 1; OpenVMS only until JES 5)

The log_node option controls whether or not the node associated with the process generating an entry is saved in the mail.log file (and connection.log file, if a separate connection log file is used). This may be useful information when the MTA is running in a multi-node cluster. A value of 1 enables node name logging. When the node name is logged, it will appear as the first field following the date and time stamps in log entries. In XML-compatible format (log_format set to 4), node logging, if enabled, appears as the no attribute. A value of 0 (the default) disables node name logging. (Note that on UNIX and Windows, prior to JES 5, the node name was always considered to be null, so on UNIX and Windows, attempting to set log_node to 1 merely resulted in an empty node field; i.e., an extra space after the time stamp in formats other than XML format. As of JES 5, if log_node is 1, then the MTA on UNIX will log the result of a gethostname call when that call found a value, or the "L" channel official host name, otherwise.)

log_notary (0-3)

The log_notary option controls whether the MTA includes an indicator of NOTARY (delivery receipt) flags in the mail.log file entries or in LOG_ACTION probes. Bit 0/value 1, if set, enables NOTARY flag logging. As of JS MS 6.3P1, enabling XML format (log_format set to 4) causes the default for log_notary to be 1; with any other format, the default is 0, as in previous versions. The NOTARY flags will be logged as a bit encoded integer after the current form of the envelope To: address. In XML-compatible format (log_format set to 4), notification flag logging, if enabled, appears in the rf attribute.

New in Cubanelle, bit 1 (value 2), if set, causes NOTARY information to be included in LOG_ACTION mapping probes.

log_priority (0-3)

The log_priority option controls whether or not the effective processing priority for the message is included in the mail.log file entries or LOG_ACTION mapping probes. Bit 0/value 1, if set, enables priority logging. As of JS MS 6.3P1, enabling XML format (log_format set to 4) causes the default for log_priority to be 1; with any other format, the default is 0, as in previous versions. If logging is enabled, the priority will be logged after the message sensitivity value, before the transport information. In XML-compatible format (log_format set to 4), priority logging, if enabled, appears as the pr attribute. New in Cubanelle, bit 1 (value 2), if set, causes priority information to be included in LOG_ACTION mapping probes.

log_process (0 or 1)

The log_process option controls whether or not the ID of the process is saved in the mail.log file (and the connection.log file, if a separate connection log is used). A value of 1 enables process ID logging. A value of 0 disables it. As of JS MS 6.3p1, enabling XML format (log_format set to 4) causes the default for log_process to be 1; with any other format, the default is 0, as in previous versions. The process ID will be logged after the date and time stamps in log entries---and after the node name, if log_node is also enabled. In XML-compatible format (log_format set to 4), process ID logging, if enabled, appears is the pi attribute. The process ID field itself will consist of the process ID in a hexadecimal representation followed by a period, next in the case of a multithreaded channel the thread ID followed by a period, followed by a (per process) counter for logging operations. That is, in the case of a single threaded channel process-id.counter or in the case of a multithreaded channel process-id.thread-id.counter Note in particular that via the process id and thread id, TCP/IP channel message enqueue/dequeue (E/D) records may be correlated with SMTP connection open/close (O/C) records.

log_queue_time (0-3)

(New in JES MS 6.3.) The log_queue_time option controls whether "time in queue" information, measured in seconds, is included in MTA message transaction log records or LOG_ACTION mapping probes. Bit 0/value 1, if set, enables such logging; having the bit clear disables such logging. As of JS MS 6.3P1, enabling XML format (log_format set to 4) causes the default for log_queue_time to be 1; with any other format, the default is 0, as in previous versions. If logging is enabled, then the time that the message has been in this channel queue (the current system time minus the creation time at which this message file was generated as stored in the envelope information in the message file) will be logged after the optional log_connection bit 6 (value 64) application information; that is, it will be the very last field logged. Note that the final ordering is: log_filter field, log_reason field, diagnostic field, transport field (log_connection bit 5/value 32), application field (log_connection bit 6/value 64), queue time field. (It is not a counted-length field; log_format set to 3 has no effect upon this field.) When log_format is set to 4 so that XML format is output, queue time logging, if enabled, appears in the qt attribute. New in 7.0, this field also contains meaningful (though slightly different in meaning) information on LMTP server back end hosts: there it represents how long the message transaction took (the time from start of the message transaction until message deposit into the store and acknowledgement back to the LMTP client) Also new in 7.0, setting log_queue_time to 1 will cause inclusion of a final field in MTA connection log O and Y records showing the time it took to open the connection, or fail to open a connection, respectively. (This is the time that it took from just before the MTA begins attempting any appropriate DNS lookups, until writing its connection log entry.)

New in Cubanelle, bit 1 (value 2), if set, causes priority information to be included in LOG_ACTION mapping probes.

log_reason (0-3)

(New in JES MS 6.3.) The log_reason option controls whether message rejection reason text is included in MTA message transaction log entries and LOG_ACTION mapping probes that correspond to a message rejection ("R" or "K" records). Setting bit 0/value 1 enables such logging; clearing the bit disables such logging. As of JS MS 6.3P1, enabling XML format (log_format set to 4) causes the default for log_reason to be 1; with any other format, the default is 0, as in previous versions. If logging is enabled, the reason text string will be logged just after the filter field (log_filter set to 1) and just before the SMTP delivery status (SMTP diagnostic) field. In XML-compatible format (log_format set to 4), the reason logging, if enabled, appears as the re attribute. New in Cubanelle, bit 1 (value 2), if set, causes reason information to be included in LOG_ACTION mapping probes.

log_sensitivity (0-3)

The log_sensitivity option controls whether message Sensitivity: header values are included in log entries or LOG_ACTION mapping probes. Bit 0/value 1, if set, enables such logging; clearing the bit disables such logging. If logging is enabled, the sensitivity value will be logged in an integer representation after the connection information, before the priority field. In XML-compatible format (log_format set to 4), sensitivity logging, if enabled, appears in the se attribute. The values are (with "Normal" being the default, intended for less sensitive material, and "Company-Confidential" corresponding to the most sensitive material):

New in Cubanelle, bit 1 (value 2), if set, causes sensitivity information to be included in LOG_ACTION mapping probes.

log_sndopr (0 or 1)

The log_sndopr option controls the production of syslog messages (UNIX) or event log entries (NT) by the MTA message transaction and connection logging facility. If this feature is enabled by specifying a value of 1, the logging facility will produce a message if it encounters any difficulty writing to its log file. A value of 0 (the default) turns off these messages. The sndopr_priority option controls the syslog level (facility and severity) syslog messages generated.

log_username (0-3)

The log_username option controls whether or not the username associated with a process that enqueues mail is saved in the mail.log file or included in LOG_ACTION mapping probes. Note that messages submitted via SMTP with authentication (SMTP AUTH) will be considered to be owned by the username that authenticated, prefixed with the asterisk, *, character. Bit 0/value 1, if set, enables username logging. When username logging is enabled, the username will be logged after the final form envelope To: address field in log entries---and after the message ID, if log_message_id is enabled also, before the connection information, if bit 0/value 1 of log_connection is enabled also. In XML-compatible format (log_format set to 4), the username logging, if enabled, appears in the us attribute. A value of 0 disables username logging. As of JS MS 6.3p1, enabling XML format (log_format set to 4) causes the default for log_username to be 1; with any other format, the default is 0, as in previous versions. New in Cubanelle, bit 1 (value 2), if set, causes username information to be included in LOG_ACTION mapping probes.

separate_connection_log (0 or 1)

The separate_connection_log option controls whether the connection log information generated by setting log_connection to 1 is stored in the usual PMDF message logging files, mail.log*, or stored separately in connection.log* files. A separate_connection_log value of 0, the default, causes connection logging to be stored in the regular message log files; a value of 1 causes the connection logging to be stored separately.

sndopr_priority (0-127)

Set the priority of operator broadcast or the syslog level (facility and severity) of syslog messages or the severity of the NT event log entry. The default is 5 (that is, LOG_NOTICE on UNIX).