Difference between revisions of "Log format MTA option"

From Messaging Server Technical Reference Wiki
Jump to: navigation, search
m (Bulk update)
m (Bulk update)
 
Line 332: Line 332:
 
* [[Release Notes#log_header|log_header MTA Option]]
 
* [[Release Notes#log_header|log_header MTA Option]]
 
* [[Release Notes#log_header_options|log_header_options MTA Option]]
 
* [[Release Notes#log_header_options|log_header_options MTA Option]]
 +
* [[log_headers_maxchars MTA option#log_headers_maxchars|log_headers_maxchars MTA Option]]
 
* [[log_node MTA option#log_node|log_node MTA Option]]
 
* [[log_node MTA option#log_node|log_node MTA Option]]
 
* [[log_process MTA option#log_process|log_process MTA Option]]
 
* [[log_process MTA option#log_process|log_process MTA Option]]

Latest revision as of 13:40, 10 October 2019



Transaction logging MTA options: log_format 1, 2, 3, 4, or 5)

The log_format option controls formatting options for the MTA message transaction log file, mail.log, (as well as the MTA connection transaction log file, connection.log, if separate_connection_log=1 has been set so that connection entries are being recorded separately rather than in mail.log).

A value of 1 (the default) is the standard format. A value of 2 requests non-null formatting: empty address fields are converted to the string "<>". A value of 3 requests counted formatting: all variable length fields are preceded by "N:", where "N" is a count of the number of characters in the field. That is, log_format set to 3 causes length-count tagging of the following fields: The envelope-from address, the original-recipient address, the current-recipient address, the filename, the envelope-message-id, the message-id, the username, the connection information, the intermediate address(es), the Sieve filter action(s), the rejection reason text, the SMTP diagnostic, the transport information, the application information. A value of 4 (new in MS 6.3) requests XML-compatible format. And finally, a value of 5 (new in MS 8.0.2.3) requests JSON-compatiable format.

As of MS 6.3p1, note that setting log_format to a value greater than 3 (XML or JSON format) also changes the defaults for a number of other log_* options. In general, it enables much of the useful (but in other formats disabled by default) optional logging options, including: log_filename, log_filter, log_message_id, log_notary, log_priority, log_process, log_queue_time, log_reason, log_username, and (new options in 7.0.5) log_auth, log_delivery_flags, and log_imap_flags.

In the (new in MS 6.3) XML-compatible format (log_format set to 4), each message log entry appears as a single XML element containing multiple attributes and no subelements. Three elements are currently defined: en for message transaction (e.g., enqueue/dequeue entries), co for connection transaction entries, and he for header entries (the optional additional records in the message transaction log file resulting from setting log_header to 1).

In the (new in MS 8.0.2.3) JSON-compatible format (log_format set to 5), each message log entry appears as a single JSON object containing multiple name-value pairs. All values are strings or integers; at present there are no arrays or nested objects. The first name-value pair always has a name of "ty" and its value specifies the object type. Three object types are currently defined: "en" for message transaction (e.g., enqueue/dequeue entries), "co" for connection transaction entries, and "he" for header entries (the optional additional records in the message transaction log file resulting from setting log_header to 1).

Note that the maximum length of MTA transaction record, both for message transaction records and connection transaction records and regardless of format, is 4096 characters.

Transaction entries

Message transaction (en) elements/objects can have the following attributes (in XML) or name-value pairs (in JSON):

  • ts - time stamp (always present)
  • no - node name (present if log_node is 1)
  • pi - process id (present if log_process is 1)
  • sc - source channel (always present)
  • dc - destination channel (field always present, though it will be empty for types of entries other than an enqueue "E" entry)
  • ac - entry type or action (always present)
  • sz - message size, reported in units of MTA blocks (field always present, though potentially empty for types of entries such as J or V entries)
  • so - source address (always present)
  • od - original destination address (field always present, though potentially empty for types of entries such as J entries)
  • de - destination address (field always present, though potentially empty)
  • rf - recipient flags (present if bit 0/value 1 of log_notary is set)
  • fi - filename (present if bit 0/value 1 of log_filename is set)
  • ei - envelope ID (present if bit 0/value 1 of log_envelope_id is set)
  • mt - (new in 8.0) message tracking id and timeout, in the format tracking-id:timeout (present if bit 0/value 1 of log_tracking is set and the message has a tracking ID and/or timeout)
  • dd - (new in 8.0) deferred delivery time (present if bit 0/value 1 of log_timesis set and the message has a deferred delivery time request)
  • ex - (new in 8.0) expiry time (present if bit 2/value 4 of log_times is set and the message has an expiry time)
  • mi - message ID and optionally the message IDs of related messages, separated by commas (present if bit 0/value 1 of log_message_id is set)
  • us - username (present if bit 0/value 1 of log_username is set and a username is available)
  • as - (new in 8.0) Authenticated user's primary mail address, i.e., normally the value of the user's mail attribute (present if bit 2/value 4 of log_username is set and a primary address is available)
  • au - (new in 7.0.5) SMTP AUTH parameter (present if bit 0/value 1 of log_auth is set and an AUTH parameter was present)
  • ss - source system (present if bit 0/value 1 of log_connection is set and source system information is available)
  • se - sensitivity (present if bit 0/value 1 of log_sensitivity is set)
  • mp - (new in 8.0) MT-PRIORITY (present if bit 0/value 1 of log_mtpriority is set)
  • pr - priority (present if bit 0/value 1 of log_priority is set)
  • in - intermediate address (present if bit 0/value 1 of log_intermediate is set and intermediate address information is available)
  • ia - initial (RCPT TO) address (present if bit 1/value 2 of log_intermediate is set and initial address information is available)
  • ui - (new in 8.0) recipient UID (present if bit 0/value 1 of log_uid is set and a recipient UID is available; in particular, will never be present in dequeue "D" records)
  • mu - (new in 7.0.5) IMAP UID and UIDVALIDITY for messages delivered via an ims-ms channel (present if bit 0/value 1 of log_mailbox_uid is set and UID and/or UIDVALIDITY data is available)
  • fr - SMTP extension FUTURERELEASE value (present if bit 0/value 1 of log_futurerelease is set and future release value is present)
  • fl - Sieve filter actions applied (present if bit 0/value 1 of log_filter is set and Sieve filter information is available; in particular, will never be present in dequeue "D" records since there is no Sieve filtering performed (hence no filter information) at that point)
  • re - reason (present if bit 0/value 1 of log_reason is set and a reason string is available)
  • di - diagnostic (present if bit 0/value 1 of log_diagnostics is set (the default) and diagnostic information is available)
  • tr - transport information (present if bit 5/value 32 of log_connection is set and transport information is available)
  • ap - application information (present if bit 6/value 64 of log_connection is set and application information is available)
  • qt - time in queue (present if bit 0/value 1 of log_queue_time is set)
  • tg - (new in 7.0.5) conversion tags (present if bit 0/value 1 of log_conversion_tag is set and any conversion tags are present)
  • if - (new in 7.0.5) IMAP flags (present if bit 0/value 1 of log_imap_flags is set and any IMAP flags are present)
  • df - (new in 7.0.5) delivery flags (present if bit 0/value 1 of log_delivery_flags is set)
  • cd - (new in 8.0) time spent waiting on external service callouts (present if bit 0/value 1 of log_callout_delays is set)
  • tl - (new in 8.0) String produced by "transactionlog" Sieve actions

Here are two sample en entries in XML format showing various different fields being logged (wrapped for printing clarity; the actual log file entries always appear in reality on a single line):


<en ts="2004-12-08T00:40:26.70" pi="0d3730.10.43" sc="tcp_local"
dc="l" ac="E" sz="12" so="info-E8944AE8D033CB92C2241E@whittlesong.com"
od="rfc822;ned+2Bcharsets@mauve.sun.com"
de="ned+charsets@mauve.sun.com" rf="22"
fi="/path/ZZ01LI4XPX0DTM00IKA8.00" ei="01LI4XPQR2EU00IKA8@mauve.sun.com"
mi="<11a3b401c4dd01$7c1c1ee0$1906fad0@elara>" us=""
ss="elara.whittlesong.com ([208.250.6.25])"
in="ned+charsets@mauve.sun.com" ia="ietf-charsets@innosoft.com"
fl="spamfilter1:rvLiXh158xWdQKa9iJ0d7Q==, addheader, keep"/>

<en ts="2016-08-30T06:28:59.93" pi="28e3d.4.233" sc="tcp_local"
dc="process" ac="E" sz="5" so="" od="rfc822;user+2Berrors@example.com"
de="user+errors@example.com" rf="276" ei="01QWOCSBWTPO003L8D@example.com"
mi="<01QWOCSEZD@example.com>,<1658a875@example.net>"
ss="TCP-DAEMON.example.com" se="-1" tr="" ap="" qt="0" df="0"
cd=",,,47,,20:83;24,,,,,,::406,17"/>


And here is a sample field in JSON format (also wrapped for clarity):


{"ty":"en","ts":"2018-10-16T07:14:35.35","pi":"547a.3.3","sc":"tcp_intranet",
"dc":"tcp_local","ac":"E","sz":1,"so":"sender@example.com",
"od":"rfc822;recip@example.net","de":"recip@example.net","rf":20,
"fi":"/opt/sun/comms/messaging64/data/queue/tcp_local/003/ZZk0W5l0MfgU0.00",
"mi":"<0PGP00G053JVOQ00@multke.example.org>","us":"mailsrv",
"ss":"[127.0.0.1] ([127.0.0.1])","pr":3,"in":"recip@example.net","qt":0,
"df":68,"cd":":0,,,0,,,::1567,0"}

Connection entries

Connection transaction (co) entries can have the following attributes/name-value pairs:

  • ts - time stamp (always present; also appears in en entries)
  • no - node name (present if log_node is 1; also appears in en entries)
  • pi - process id (present if log_process is 1; also appears in en entries)
  • sc - source channel (always present; also appears in en entries)
  • dr - direction (always present; specific to connection entries): a "+" for inbound connections, or a "-" for outbound connections
  • ac - entry type or action (always present; also appears in en entries)
  • tr - transport information (always present; may also appear in en entries when bit 5 of log_connection is set);
  • ap - application information (always present; may also appear in en entries when bit 6 of log_connection is set)
  • mi - the name presented on the ETRN command line (present only if bit 0/value 1 of log_message_id is set, and ETRN was used with an argument; note that setting bit 0/value 1 of log_message_id also causes the logging of message ID information, if available, in message transaction (en) log records)
  • us - username (present only if bit 0/value 1 of log_username is set and username information is available; also appears in en entries)
  • ex - (new in 8.0) in "U" records, additional authentication information such as the authentication mechanism (present only if bit 4/value 16 of log_username is set and such extra information is available)
  • di - diagnostic (present only if bit 0/value 1 of log_diagnostic is set and diagnostic information is available; also appears in en entries)
  • ct - time to connect/fail to connect/connection was open (present only if bit 0/value 1 of log_queue_time is set; specific to connection entries)

Here is a sample co entry in XML format (wrapped purely for printing clarity; in reality, such entries always appear on a single line):


<co ts="2004-12-08T00:38:28.41" pi="1074b3.61.281" sc="tcp_local" dr="+"
ac="O" tr="TCP|209.55.107.55|25|209.55.107.104|33469" ap="SMTP"/>

And here is a similar entry in JSON format: (also wrapped for clarity):


{"ty":"co","ts":"2018-10-16T07:14:09.27","pi":"547a.3.0","sc":"tcp_local",
"dr":"+","ac":"O","tr":"TCP|127.0.0.1|25|127.0.0.1|48023","ap":"SMTP"}

Header entries

Header (he) entries have the following attributes/name-value pairs:

  • ts - time stamp (always present; also used in en entries)
  • no - node name (present if log_node is 1; also used in en entries)
  • pi - process id (present if log_process is 1; also used in en entries)
  • va - header line value (always present)

Here is a sample he entry in XML:


<he ts="2004-12-08T00:38:31.41" pi="1074b3.61.281" va="Subject: foo"/>

And one in JSON:


{"ty":"he","ts":"2018-10-16T07:14:35.35","pi":"547a.3.3",
 "va":"Subject: This is a test"}

See also: