Log_format MTA option

From Messaging Server Technical Reference Wiki
Jump to: navigation, search



Transaction logging MTA options: log_format (1-6)

The log_format option controls formatting options for the MTA message transaction log file, mail.log, (as well as the MTA connection transaction log file, connection.log, if separate_connection_log=1 has been set so that connection entries are being recorded separately rather than in mail.log).

A value of 1 (the default) is the standard format. A value of 2 requests non-null formatting: empty address fields are converted to the string "<>". A value of 3 requests counted formatting: all variable length fields are preceded by "N:", where "N" is a count of the number of characters in the field. That is, log_format set to 3 causes length-count tagging of the following fields: The envelope-from address, the original-recipient address, the current-recipient address, the filename, the envelope-message-id, the message-id, the username, the connection information, the intermediate address(es), the Sieve filter action(s), the rejection reason text, the SMTP diagnostic, the transport information, the application information. A value of 4 (new in MS 6.3) requests XML-compatible format.

A value of 5 (new in MS 8.0.2.3) requests JSON-compatiable format. And finally, a value of 6 (new in MS 8.1.0.2) specifies "flat" JSON format.

As of MS 6.3p1, note that setting log_format to a value greater than 3 (XML or JSON format) also changes the defaults for a number of other log_* options. In general, it enables much of the useful (but in other formats disabled by default) optional logging options, including: log_filename, log_filter, log_message_id, log_notary, log_priority, log_process, log_queue_time, log_reason, log_username, and (new options in 7.0.5) log_auth, log_delivery_flags, and log_imap_flags.

In the (new in MS 6.3) XML-compatible format (log_format set to 4), each message log entry appears as a single XML element containing multiple attributes and no subelements. Three elements are currently defined: en for message transaction (e.g., enqueue/dequeue entries), co for connection transaction entries, and he for header entries (the optional additional records in the message transaction log file resulting from setting log_header to 1).

In the (new in MS 8.0.2.3) JSON-compatible format (log_format set to 5), each message log entry appears as a single JSON object containing multiple name-value pairs. All values are strings or integers; at present there are no arrays or nested objects. The first name-value pair always has a name of "ty" and its value specifies the object type. Three object types are currently defined: "en" for message transaction (e.g., enqueue/dequeue entries), "co" for connection transaction entries, and "he" for header entries (the optional additional records in the message transaction log file resulting from setting log_header to 1).

The (new in MS 8.1.0.2) value of 6 also creates a JSON-compatible format, but one which attempts to avoid both JSON structure and structured field content, as well as making the format compatible for consumption by Lumberjack. Specifically, the following changes are made:

  • The ts timestamp attribute is encoded as integer number of milliseconds since the epoch.
  • The tl transactionlog attribute is renamed to msg and is included in every log entry.
  • The tg conversion tag attribute is broken up into as many as 10 separate tag attributes t0, t1, t2, ..., each containing a separate tag. The eleventh and subsequent tags, if they exist, are not logged.
  • The he array of header field values is broken up into as many as 10 separate header attributes h0, h1, h2, ..., each containing a separate header field value. The eleventh and subsequent header fields, if they exist, are not logged.
  • The so attribute contains the envelope from address. The domain from this address is broken out and appears in an additional sd attribute.
  • New in MS 8.1.0.6, the de attribute contains the envelope recipient address. The domain from this address is broken out and appears in an additional rd attribute.
  • The various modifiers to the ac action attribute are moved to a separate sp attribute.
  • When the tr transport information attribute contains TCP address information it is replaced by li and ri attributes containing the local and remote IP addresses, respectively. Port information is not logged.
  • The ap application information attribute is replaced by a po attribute containing the protocol and, if SSL/TLS is being used, a cs attribute containing the TLS/SSL information string. Other parts of the application information string are not logged.

Note that the maximum length of MTA transaction record, both for message transaction records and connection transaction records and regardless of format, is 4096 characters.

Transaction entries

Message transaction (en) elements/objects can have the following attributes (in XML) or name-value pairs (in JSON):

  • ts - time stamp (always present) Note: The format changes from ISO 8601 format to a UNIX epoch value in milliseconds if log_format is set to 6
  • no - node name (present if log_node is 1)
  • pi - process id (present if log_process is 1)
  • sc - source channel (always present)
  • dc - destination channel (field always present, though it will be empty for types of entries other than an enqueue "E" entry)
  • ac - entry type or action (always present)
  • sp - modifier values from action field (Separated out from the action field if log_format is set to 6)
  • sz - message size, reported in units of MTA blocks (field always present, though potentially empty for types of entries such as J or V entries)
  • so - source address (always present)
  • sd - domain from source address (present if log_format is set to 6)
  • od - original destination address (field always present, though potentially empty for types of entries such as J entries)
  • de - destination address (field always present, though potentially empty)
  • rd - domain from destination address (present if log_format is set to 6)
  • rf - recipient flags (present if bit 0/value 1 of log_notary is set)
  • fi - filename (present if bit 0/value 1 of log_filename is set)
  • ei - envelope ID (present if bit 0/value 1 of log_envelope_id is set)
  • mt - (new in 8.0) message tracking id and timeout, in the format tracking-id:timeout (present if bit 0/value 1 of log_tracking is set and the message has a tracking ID and/or timeout)
  • dd - (new in 8.0) deferred delivery time (present if bit 0/value 1 of log_timesis set and the message has a deferred delivery time request)
  • ex - (new in 8.0) expiry time (present if bit 2/value 4 of log_times is set and the message has an expiry time)
  • mi - message ID and optionally the message IDs of related messages, separated by commas (present if bit 0/value 1 of log_message_id is set)
  • us - username (present if bit 0/value 1 of log_username is set and a username is available)
  • as - (new in 8.0) Authenticated user's primary mail address, i.e., normally the value of the user's mail attribute (present if bit 2/value 4 of log_username is set and a primary address is available)
  • au - (new in 7.0.5) SMTP AUTH parameter (present if bit 0/value 1 of log_auth is set and an AUTH parameter was present)
  • ss - source system (present if bit 0/value 1 of log_connection is set and source system information is available)
  • se - sensitivity (present if bit 0/value 1 of log_sensitivity is set)
  • mp - (new in 8.0) MT-PRIORITY (present if bit 0/value 1 of log_mtpriority is set)
  • pr - priority (present if bit 0/value 1 of log_priority is set)
  • in - intermediate address (present if bit 0/value 1 of log_intermediate is set and intermediate address information is available)
  • ia - initial (RCPT TO) address (present if bit 1/value 2 of log_intermediate is set and initial address information is available)
  • ui - (new in 8.0) recipient UID (present if bit 0/value 1 of log_uid is set and a recipient UID is available; in particular, will never be present in dequeue "D" records)
  • mu - (new in 7.0.5) IMAP UID and UIDVALIDITY for messages delivered via an ims-ms channel (present if bit 0/value 1 of log_mailbox_uid is set and UID and/or UIDVALIDITY data is available)
  • fr - SMTP extension FUTURERELEASE value (present if bit 0/value 1 of log_futurerelease is set and future release value is present)
  • fl - Sieve filter actions applied (present if bit 0/value 1 of log_filter is set and Sieve filter information is available; in particular, will never be present in dequeue "D" records since there is no Sieve filtering performed (hence no filter information) at that point)
  • re - reason (present if bit 0/value 1 of log_reason is set and a reason string is available)
  • di - diagnostic (present if bit 0/value 1 of log_diagnostics is set (the default) and diagnostic information is available)
  • tr - transport information (present if bit 5/value 32 of log_connection is set, transport information is available, and log_format is not set to 6)
  • li - local IP address (present if bit 5/value 32 of log_connection is set, transport information is available, and log_format is set to 6)
  • ri - remote IP address (present if bit 5/value 32 of log_connection is set, transport information is available, and log_format is set to 6)
  • ap - application information (present if bit 6/value 64 of log_connection is set, application information is available, and log_format is not set to 6)
  • po - application protocol (present if bit 6/value 64 of log_connection is set, application information is available, and log_format is set to 6)
  • po - SSL/TLS information string (present if bit 6/value 64 of log_connection is set, application information is available, SSL/TLS is being used, and log_format is set to 6)
  • qt - time in queue (present if bit 0/value 1 of log_queue_time is set)
  • tg - (new in 7.0.5) conversion tags (present if bit 0/value 1 of log_conversion_tag is set and any conversion tags are present) Note: This changes from a single tg attribute to per-tag t0, t1, ... attributes if log_format is set to 6
  • if - (new in 7.0.5) IMAP flags (present if bit 0/value 1 of log_imap_flags is set and any IMAP flags are present)
  • df - (new in 7.0.5) delivery flags (present if bit 0/value 1 of log_delivery_flags is set)
  • cd - (new in 8.0) time spent waiting on external service callouts (present if bit 0/value 1 of log_callout_delays is set)
  • sm - smartsend information(present if bit 0/value 1 of log_smartsend is set and smartsend information is available)
  • dk - (new in 8.1.0.6) DKIM signing information (present if bit 0/value 1 of log_dkim is set and DKIM signing was enabled)
  • fm - (new in 8.1.0.2) Address from header From: field (present if bit 0/value 1 of log_from is set and a parsable From: field address is present)
  • tl - (new in 8.0) String produced by "transactionlog" Sieve actions Note: The attribute name changes from tl to msg if log_format is set to 6
  • he - (new in 8.1.0.1) Selected header fields (present if bit 2/value 4 of log_header is set and any of the selected headers are present) Note: This changes from a single he attribute to per-field h0, h1, ... attributes if log_format is set to 6

Here are two sample en entries in XML format showing various different fields being logged (wrapped for printing clarity; the actual log file entries always appear in reality on a single line):


<en ts="2004-12-08T00:40:26.70" pi="0d3730.10.43" sc="tcp_local"
dc="l" ac="E" sz="12" so="info-E8944AE8D033CB92C2241E@whittlesong.com"
od="rfc822;ned+2Bcharsets@mauve.sun.com"
de="ned+charsets@mauve.sun.com" rf="22"
fi="/path/ZZ01LI4XPX0DTM00IKA8.00" ei="01LI4XPQR2EU00IKA8@mauve.sun.com"
mi="<11a3b401c4dd01$7c1c1ee0$1906fad0@elara>" us=""
ss="elara.whittlesong.com ([208.250.6.25])"
in="ned+charsets@mauve.sun.com" ia="ietf-charsets@innosoft.com"
fl="spamfilter1:rvLiXh158xWdQKa9iJ0d7Q==, addheader, keep"/>

<en ts="2016-08-30T06:28:59.93" pi="28e3d.4.233" sc="tcp_local"
dc="process" ac="E" sz="5" so="" od="rfc822;user+2Berrors@example.com"
de="user+errors@example.com" rf="276" ei="01QWOCSBWTPO003L8D@example.com"
mi="<01QWOCSEZD@example.com>,<1658a875@example.net>"
ss="TCP-DAEMON.example.com" se="-1" tr="" ap="" qt="0" df="0"
cd=",,,47,,20:83;24,,,,,,::406,17"/>


And here is a sample field in JSON format (also wrapped for clarity):


{"ty":"en","ts":"2018-10-16T07:14:35.35","pi":"547a.3.3","sc":"tcp_intranet",
"dc":"tcp_local","ac":"E","sz":1,"so":"sender@example.com",
"od":"rfc822;recip@example.net","de":"recip@example.net","rf":20,
"fi":"/opt/sun/comms/messaging64/data/queue/tcp_local/003/ZZk0W5l0MfgU0.00",
"mi":"<0PGP00G053JVOQ00@multke.example.org>","us":"mailsrv",
"ss":"[127.0.0.1] ([127.0.0.1])","pr":3,"in":"recip@example.net","qt":0,
"df":68,"cd":":0,,,0,,,::1567,0"}

Connection entries

Connection transaction (co) entries can have the following attributes/name-value pairs:

  • ts - time stamp (always present; also appears in en entries)
  • no - node name (present if log_node is 1; also appears in en entries)
  • pi - process id (present if log_process is 1; also appears in en entries)
  • sc - source channel (always present; also appears in en entries)
  • dr - direction (always present; specific to connection entries): a "+" for inbound connections, or a "-" for outbound connections
  • ac - entry type or action (always present; also appears in en entries)
  • tr - transport information (always present; may also appear in en entries when bit 5 of log_connection is set);
  • ap - application information (always present; may also appear in en entries when bit 6 of log_connection is set)
  • mi - the name presented on the ETRN command line (present only if bit 0/value 1 of log_message_id is set, and ETRN was used with an argument; note that setting bit 0/value 1 of log_message_id also causes the logging of message ID information, if available, in message transaction (en) log records)
  • us - username (present only if bit 0/value 1 of log_username is set and username information is available; also appears in en entries)
  • ex - (new in 8.0) in "U" records, additional authentication information such as the authentication mechanism (present only if bit 4/value 16 of log_username is set and such extra information is available) (new in MS 8.1.0.3) in connection open/close records, conversion tags associated with the current message being dequeued (present only if bit 4 (value 16) is set and there are conversion tags to log)
  • di - diagnostic (present only if bit 0/value 1 of log_diagnostic is set and diagnostic information is available; also appears in en entries)
  • ct - time to connect/fail to connect/connection was open (present only if bit 0/value 1 of log_queue_time is set; specific to connection entries)

Here is a sample co entry in XML format (wrapped purely for printing clarity; in reality, such entries always appear on a single line):


<co ts="2004-12-08T00:38:28.41" pi="1074b3.61.281" sc="tcp_local" dr="+"
ac="O" tr="TCP|209.55.107.55|25|209.55.107.104|33469" ap="SMTP"/>

And here is a similar entry in JSON format: (also wrapped for clarity):


{"ty":"co","ts":"2018-10-16T07:14:09.27","pi":"547a.3.0","sc":"tcp_local",
"dr":"+","ac":"O","tr":"TCP|127.0.0.1|25|127.0.0.1|48023","ap":"SMTP"}

Header entries

Header (he) entries have the following attributes/name-value pairs:

  • ts - time stamp (always present; also used in en entries)
  • no - node name (present if log_node is 1; also used in en entries)
  • pi - process id (present if log_process is 1; also used in en entries)
  • va - header line value (always present)

Here is a sample he entry in XML:


<he ts="2004-12-08T00:38:31.41" pi="1074b3.61.281" va="Subject: foo"/>

And one in JSON:


{"ty":"he","ts":"2018-10-16T07:14:35.35","pi":"547a.3.3",
 "va":"Subject: This is a test"}

Action modifiers

The following single character modifiers may be associated with the action field:

  • E - Extended SMTP (EHLO) rather than the original SMTP (HELO) was used during connection establishment
  • P - a proxy protocol was used to transfer connection information
  • L - LMTP rather than SMTP was used to transfer the message
  • Q - Pipelining was used during the SMTP/SUBIT transaction
  • A - SASL authentication was done prior to the SMTP/SUBMIT transaction
  • S - SSL/TLS was used to secure the SMTP transaction
  • U - A BURL command was used to create part of the message body (server only)
  • B - Binary SMTP/SUBMIT was used to transfer the message (server only)
  • C - Chunking was used to transfer the message
  • 8 - The SMTPUTF8 (EAI) extension was used
  • X - The message underwent MIME conversion processing (new in MS 8.1.0.6)
  • W - Excessively long lines in the message were wrapped
  • T - Excessively long lines in the message were truncated
  • R - Excessively long lines in the message caused it to be rejected


See also: