Log format MTA option

Transaction logging MTA options:  (1-6)
The  option controls formatting options for the MTA message transaction log file, , (as well as the MTA connection transaction log file,  , if    has been set so that connection entries are being recorded separately rather than in  ).

A value of 1 (the default) is the standard format. A value of 2 requests non-null formatting: empty address fields are converted to the string "&#x3c;&#x3e;". A value of 3 requests counted formatting: all variable length fields are preceded by "N:", where "N" is a count of the number of characters in the field. That is,  set to 3 causes length-count tagging of the following fields: The envelope-from address, the original-recipient address, the current-recipient address, the filename, the envelope-message-id, the message-id, the username, the connection information, the intermediate address(es), the  Sieve filter action(s), the rejection reason text, the SMTP diagnostic, the transport information, the  application information. A value of 4 (new in MS 6.3) requests XML-compatible format.

A value of 5 (new in MS 8.0.2.3) requests JSON-compatiable format. And finally, a value of 6 (new in MS 8.1.0.2) specifies "flat" JSON format.

As of MS 6.3p1, note that setting  to a value greater than 3 (XML or JSON format) also changes the defaults for a number of other   options. In general, it enables much of the useful (but in other formats disabled by default) optional logging options, including: ,  ,   ,  ,   ,  ,   ,  ,   , and (new options in 7.0.5)  ,  ,  and.

In the (new in MS 6.3) XML-compatible format ( set to 4), each message log entry appears as a single XML element containing multiple attributes and no subelements. Three elements are currently defined:  for message transaction (e.g., enqueue/dequeue entries),   for connection transaction entries, and   for header entries (the optional additional records in the message transaction log file resulting from setting   to 1).

In the (new in MS 8.0.2.3) JSON-compatible format ( set to 5), each message log entry appears as a single JSON object containing multiple name-value pairs. All values are strings or integers; at present there are no arrays or nested objects. The first name-value pair always has a name of "ty" and its value specifies the object type. Three object types are currently defined:  for message transaction (e.g., enqueue/dequeue entries),   for connection transaction entries, and   for header entries (the optional additional records in the message transaction log file resulting from setting   to 1).

The (new in MS 8.1.0.2) value of 6 also creates a JSON-compatible format, but one which attempts to avoid both JSON structure and structured field content, as well as making the format compatible for consumption by Lumberjack. Specifically, the following changes are made:



 The  timestamp attribute is encoded as integer number of milliseconds since the epoch. 

 The  transactionlog attribute is renamed to   and is included in every log entry. 

 The  conversion tag attribute is broken up into as many as 10 separate tag attributes ,  ,  , ..., each containing a separate tag. The eleventh and subsequent tags, if they exist, are not logged. 

 The  array of header field values is broken up into as many as 10 separate header attributes  ,  ,  , ..., each containing a separate header field value. The eleventh and subsequent header fields, if they exist, are not logged. 

 The  attribute contains the envelope from address. The domain from this address is broken out and appears in an additional  attribute. 

 The various protocol modifiers to the  action attribute are moved to a separate   attribute. 

 When the  transport information attribute contains TCP address information it is replaced by   and   attributes containing the local and remote IP addresses, respectively. Port information is not logged. 

 The  application information attribute is replaced by a   attribute containing the protocol and, if SSL/TLS is being used, a   attribute containing the TLS/SSL information string. Other parts of the application information string are not logged. </li>

</ul>

Note that the maximum length of MTA transaction record, both for message transaction records and connection transaction records and regardless of format, is 4096 characters.

Transaction entries
Message transaction elements/objects can have the following attributes (in XML) or name-value pairs (in JSON):

<ul>

 - time stamp (always present) Note: The format changes from ISO 8601 format to a UNIX epoch value in milliseconds if       is set to 6 </li>

 - node name (present if  is 1) </li>

 - process id (present if  is 1) </li>

 - source channel (always present) </li>

 - destination channel (field always present, though  it will be empty for types of entries other than an enqueue "E"   entry) </li>

 - entry type or action   (always present) </li>

 - protocol modifier values from action field (Separated out from the action field if  is set to 6) </li>

 - message size, reported in units of   MTA blocks (field always present, though   potentially empty for types of entries such as J or V entries) </li>

 - source address (always present) </li>

 - domain from source address (present if  is set to 6) </li>

 - original destination address (field always  present, though potentially empty for types of entries such as   J entries) </li>

 - destination address (field always present, though  potentially empty) </li>

 - recipient flags (present if bit 0/value 1 of    is set) </li>

 - filename (present if bit 0/value 1 of    is set) </li>

 - envelope ID (present if bit 0/value 1 of    is set) </li>

 - (new in 8.0) message  tracking id and timeout, in the format      (present if bit 0/value 1 of     is set and the message has a tracking ID and/or   timeout) </li>

<li> - (new in 8.0)  deferred delivery time (present if bit 0/value 1 of    is set and the message has a deferred delivery time   request) </li>

<li> - (new in 8.0)  expiry time (present if bit 2/value 4 of   is set   and the message has an expiry time) </li>

<li> - message ID and optionally the message IDs of related  messages, separated by commas (present if bit 0/value 1 of     is set) </li>

<li> - username (present if bit 0/value 1 of    is set and a username is available) </li>

<li> - (new in 8.0) Authenticated user&#x27;s primary mail address, i.e., normally the value of the user&#x27;s     attribute  (present if bit 2/value 4 of   is set and a primary   address is available) </li>

<li> - (new in 7.0.5) SMTP AUTH parameter (present if   bit 0/value 1 of   is set and an AUTH parameter was    present) </li>

<li> - source system (present if bit 0/value 1 of    is set and source system information is    available) </li>

<li> - sensitivity (present if bit 0/value 1 of    is set) </li>

<li> - (new in 8.0) MT-PRIORITY (present if bit 0/value 1 of    is set) </li>

<li> - priority (present if bit 0/value 1 of    is set) </li>

<li> - intermediate address (present if bit 0/value 1 of    is set and intermediate address information is   available) </li>

<li> - initial (RCPT TO) address (present if bit  1/value 2 of   is set and initial address   information is  available) </li>

<li> - (new in 8.0) recipient UID (present if bit 0/value 1 of    is set and a recipient UID is available; in  particular, will never be present in dequeue "D" records) </li>

<li> - (new in 7.0.5) IMAP UID and UIDVALIDITY for messages delivered via an   channel (present if bit 0/value 1 of   is set and UID and/or UIDVALIDITY data is available) </li>

<li> - SMTP extension FUTURERELEASE value (present if bit 0/value 1 of   is set and future release value is present) </li>

<li> - Sieve filter   actions applied (present if bit 0/value 1 of     is set and   Sieve filter information is available; in particular, will never be present in   dequeue "D" records since there is no Sieve filtering performed   (hence no filter information) at that point) </li>

<li> - reason (present if bit 0/value 1 of    is set and a reason string is available) </li>

<li> - diagnostic (present if bit 0/value 1 of    is set (the default) and diagnostic information   is available) </li>

<li> - transport  information (present if bit 5/value 32 of     is set, transport information is   available, and   is not set to 6) </li>

<li> - local IP address (present if bit 5/value 32 of    is set, transport information is   available, and   is set to 6) </li>

<li> - remote IP address (present if bit 5/value 32 of    is set, transport information is   available, and   is set to 6) </li>

<li> - application  information (present if bit 6/value 64 of     is set, application information is   available, and   is not set to 6) </li>

<li> - application protocol (present if bit 6/value 64 of    is set, application information is   available, and   is set to 6) </li>

<li> - SSL/TLS information string (present if bit 6/value 64 of    is set, application information is   available, SSL/tLS is being used, and   is set to 6) </li>

<li> - time in queue (present if bit 0/value 1 of    is set) </li>

<li> - (new in 7.0.5)   conversion tags (present if bit 0/value 1 of     is set and any conversion tags are present)  Note: This changes from a single   attribute to    per-tag ,   , ... attributes if  is set to 6 </li>

<li> - (new in 7.0.5) IMAP flags (present if bit 0/value 1 of     is set and any   IMAP flags are present) </li>

<li> - (new in 7.0.5) delivery flags (present if bit 0/value 1 of    is set) </li>

<li> - (new in 8.0) time spent waiting on external service  callouts (present if bit 0/value 1 of   is set) </li>

<li> - smartsend information(present if bit 0/value 1 of     is set and smartsend information is available) </li>

<li> - (new in 8.1.0.2) Address from header From: field (present if bit 0/value 1 of     is set and a From: field address is present) </li>

<li> - (new in 8.0) String produced by   " "   Sieve actions Note: The attribute name changes from   to   if   is set to 6 </li>

<li> - (new in 8.1.0.1)   Selected header fields (present if bit 2/value 4 of     is set and any of the selected headers are present)  Note: This changes from a single   attribute to    per-field ,  , ... attributes if  is set to 6 </li>

</ul>

Here are two sample  entries in XML format showing various different fields being logged (wrapped for printing clarity; the actual log file entries always appear in reality on a single line): &#x3c;en ts="2004-12-08T00:40:26.70" pi="0d3730.10.43" sc="tcp_local" dc="l" ac="E" sz="12" so="info-E8944AE8D033CB92C2241E@whittlesong.com" od="rfc822;ned+2Bcharsets@mauve.sun.com" de="ned+charsets@mauve.sun.com" rf="22" fi="/path/ZZ01LI4XPX0DTM00IKA8.00" ei="01LI4XPQR2EU00IKA8@mauve.sun.com" mi="&lt;11a3b401c4dd01$7c1c1ee0$1906fad0@elara&gt;" us="" ss="elara.whittlesong.com (&#x5b;208.250.6.25&#x5d;)" in="ned+charsets@mauve.sun.com" ia="ietf-charsets@innosoft.com" fl="spamfilter1:rvLiXh158xWdQKa9iJ0d7Q==, addheader, keep"/&#x3e; &#x3c;en ts="2016-08-30T06:28:59.93" pi="28e3d.4.233" sc="tcp_local" dc="process" ac="E" sz="5" so="" od="rfc822;user+2Berrors@example.com" de="user+errors@example.com" rf="276" ei="01QWOCSBWTPO003L8D@example.com" mi="&lt;01QWOCSEZD@example.com&gt;,&lt;1658a875@example.net&gt;" ss="TCP-DAEMON.example.com" se="-1" tr="" ap="" qt="0" df="0" cd=",,,47,,20:83;24,,,,,,::406,17"/&#x3e; And here is a sample field in JSON format (also wrapped for clarity): {"ty":"en","ts":"2018-10-16T07:14:35.35","pi":"547a.3.3","sc":"tcp_intranet", "dc":"tcp_local","ac":"E","sz":1,"so":"sender@example.com", "od":"rfc822;recip@example.net","de":"recip@example.net","rf":20, "fi":"/opt/sun/comms/messaging64/data/queue/tcp_local/003/ZZk0W5l0MfgU0.00", "mi":"&#x3c;0PGP00G053JVOQ00@multke.example.org&#x3e;","us":"mailsrv", "ss":"&#x5b;127.0.0.1&#x5d; (&#x5b;127.0.0.1&#x5d;)","pr":3,"in":"recip@example.net","qt":0, "df":68,"cd":":0,,,0,,,::1567,0"}

Connection entries
Connection transaction entries can have the following attributes/name-value pairs:

<ul>

<li> - time stamp (always present; also appears in    entries) </li>

<li> - node name (present if  is 1; also   appears in    entries) </li>

<li> - process id (present if  is 1;   also  appears in   entries) </li>

<li> - source channel (always present; also appears in    entries) </li>

<li> - direction (always present; specific to connection  entries): a " " for inbound connections, or a    " " for outbound connections </li>

<li> - entry type or action (always present; also  appears in   entries) </li>

<li> - transport information (always present; may also  appear in   entries when bit 5 of     is set); </li>

<li> - application information (always present; may also  appear in   entries when bit 6 of     is set) </li>

<li> - the name presented on the ETRN command line  (present only if bit 0/value 1 of   is set, and   ETRN was used with   an argument;  note that setting bit 0/value 1 of     also causes   the logging of message ID  information, if available, in message transaction    log  records) </li>

<li> - username (present only if bit 0/value 1 of    is set   and  username information is available; also appears in     entries) </li>

<li> - (new in 8.0) in "U" records, additional authentication information such as the authentication mechanism  (present only if bit 4/value 16 of   is set and such   extra information is available) (new in MS 8.1.0.3) in connection open/close records,   conversion tags associated with the current message being dequeued (present only if bit 4 (value 16) is set   and there are conversion tags to log) </li>

<li> - diagnostic (present only if bit 0/value 1 of    is set and diagnostic information is   available; also appears in   entries) </li>

<li> - time to connect/fail to connect/connection was open (present only if  bit 0/value 1 of   is set; specific to connection   entries) </li>

</ul>

Here is a sample  entry in XML format (wrapped purely for printing clarity; in reality, such entries always appear on a single line): &#x3c;co ts="2004-12-08T00:38:28.41" pi="1074b3.61.281" sc="tcp_local" dr="+" ac="O" tr="TCP&#x7c;209.55.107.55&#x7c;25&#x7c;209.55.107.104&#x7c;33469" ap="SMTP"/&#x3e; And here is a similar entry in JSON format: (also wrapped for clarity): {"ty":"co","ts":"2018-10-16T07:14:09.27","pi":"547a.3.0","sc":"tcp_local", "dr":"+","ac":"O","tr":"TCP&#x7c;127.0.0.1&#x7c;25&#x7c;127.0.0.1&#x7c;48023","ap":"SMTP"}

Header entries
Header entries have the following attributes/name-value pairs:

<ul>

<li> - time stamp (always present; also used in    entries) </li>

<li> - node name (present if  is 1; also   used in    entries) </li>

<li> - process id (present if  is 1;   also used  in   entries) </li>

<li> - header line value (always present) </li>

</ul>

Here is a sample  entry in XML: &#x3c;he ts="2004-12-08T00:38:31.41" pi="1074b3.61.281" va="Subject: foo"/&#x3e; And one in JSON: {"ty":"he","ts":"2018-10-16T07:14:35.35","pi":"547a.3.3", "va":"Subject: This is a test"}

See also:
 * Transaction logging MTA options
 * MTA transaction logging
 * MTA transaction log entry format
 * separate_connection_log MTA Option
 * log_header MTA Option
 * log_header_options MTA Option
 * log_headers_maxchars MTA Option
 * log_node MTA Option
 * log_process MTA Option
 * log_notary MTA Option
 * log_filename MTA Option
 * log_envelope_id MTA Option
 * log_times MTA Option
 * log_tracking MTA Option
 * log_message_id MTA Option
 * log_username MTA Option
 * log_auth MTA Option
 * log_connection MTA Option
 * log_sensitivity MTA Option
 * log_mtpriority MTA Option
 * log_priority MTA Option
 * log_intermediate MTA Option
 * log_uid MTA Option
 * log_mailbox_uid MTA Option
 * log_callout_delays MTA Option
 * log_transactionlog MTA Option
 * log_filter MTA Option
 * log_reason MTA Option
 * log_diagnostics MTA Option
 * log_remote_mta MTA Option
 * log_queue_time MTA Option
 * log_conversion_tag MTA Option
 * log_imap_flags MTA Option
 * log_delivery_flags MTA Option
 * log_futurerelease MTA Option
 * Conversion tags