Difference between revisions of "Message tracking and recall setup and configuration"

From Messaging Server Technical Reference Wiki
Jump to: navigation, search
m (Bulk update)
m (Bulk update)
Line 50: Line 50:
 
   LOG_TRACKING=1
 
   LOG_TRACKING=1
 
  </font>
 
  </font>
=== MTA Channel Setup ===
+
=== MTA channel setup for message tracking and recall ===
  
 
The next step is to tell the tracking subsystem the semantics of the various channels in the deployment using the <code>trackinginternal</code>, <code>trackingrelayed</code>, and <code>trackingdelivered</code>. The general rules are:  
 
The next step is to tell the tracking subsystem the semantics of the various channels in the deployment using the <code>trackinginternal</code>, <code>trackingrelayed</code>, and <code>trackingdelivered</code>. The general rules are:  
Line 57: Line 57:
  
 
<li>
 
<li>
  Internal processing channels such as process, reprocess, conversion, and defragment require no special decoration.  
+
  Internal processing channels such as [[Process channel#Process_channel|<code>process</code>, <code>reprocess</code>]], [[Conversion channel#Conversion_channel|<code>conversion</code>]], and [[Defragmentation channel#Defragmentation_channel|<code>defragment</code>]]  require no special decoration.  
 
</li>
 
</li>
  
Line 65: Line 65:
  
 
<li>
 
<li>
  Any channel that relays messages to systems outside the deployment must be marked with the [[tracking*, notracking* Channel Options#trackingrelayed|<code>trackingrelayed</code>]]  channel option. Of course this includes the <code>tcp_local</code> channel, but would also include, say a special <code>tcp_aol</code> channel set up to handle mail to the aol.com domain.  
+
  Any channel that relays messages to systems outside the deployment must be marked with the [[tracking*, notracking* Channel Options#trackingrelayed|<code>trackingrelayed</code>]]  channel option. Of course this includes the [[Typical TCPIP channels and servers#Typical_TCPIP_channels_and_servers|<code>tcp_local</code> channel]], but would also include, say a special <code>tcp_aol</code> channel set up to handle mail to the aol.com domain.  
 
</li>
 
</li>
  
Line 74: Line 74:
 
</ol>
 
</ol>
  
Note that for tracking and recall to work messages relayed to external systems MUST be handled by a different channel than message relayed to other MTAs inside the deployment. Meeting this requirement may require additional configuration changes.  
+
Note that for tracking and recall to work messages relayed to external systems MUST be handled by a different channel than messages relayed to other MTAs inside the deployment. (At a minimum, this tends to mean using a distinct <code>tcp_local</code>  ''vs.''   <code>tcp_intranet</code> channel.) Meeting this requirement may require additional configuration changes.  
  
 
The MTRK SMTP extension defined in RFC 3885 is used to transfer tracking/recall information from one MTA to another. Use of this extension MUST be enabled on SMTP connections between MTAs inside the deployment. It also must be enabled on SUBMIT channels used by tracking/recall-enabled clients.  
 
The MTRK SMTP extension defined in RFC 3885 is used to transfer tracking/recall information from one MTA to another. Use of this extension MUST be enabled on SMTP connections between MTAs inside the deployment. It also must be enabled on SUBMIT channels used by tracking/recall-enabled clients.  
  
Since MTRK is an SMTP extension, its use is negotiated by the SMTP client and server. So the simplest way to activate this extension is to put the <code>trackingclient</code> and <code>trackingserver</code> on the defaults channel. However, if you wish to avoid use of this extension with systems outside the deployment, a more targeted approach is needed: Place <code>trackingclient</code> on every tcp_&#x2a; channel that has <code>trackinginternal</code> set. Then place <code>trackingserver</code> on every channel that accepts messages from other hosts within the deployment. (Note that once again this may require configuration changes to separate internal and external traffic.)  
+
Since MTRK is an SMTP extension, its use is negotiated by the SMTP client and server. So the simplest way to activate this extension is to put the [[tracking*, notracking* Channel Options#trackingclient|<code>trackingclient</code>]] and [[tracking*, notracking* Channel Options#trackingserver|<code>trackingserver</code>]] on the defaults channel. However, if you wish to avoid use of this extension with systems outside the deployment, a more targeted approach is needed: Place <code>trackingclient</code> on every <code>tcp_&#x2a;</code> channel that has <code>trackinginternal</code> set. Then place <code>trackingserver</code> on every channel that accepts messages from other hosts within the deployment. (Note that once again this may require configuration changes to separate internal and external traffic.)  
  
 
=== MTQP server setup ===
 
=== MTQP server setup ===
Line 84: Line 84:
 
An extended version of the MTRP protocol specified in RFC 3887 is used to perform tracking and recall operations. If only tracking is desired a single MTQP server can be used for the entire deployment and can be run on any host. However, in order to recall messages an MTQP server must be running on every host with an MTA, and if "total recall" is enabled MTQP servers are required on all store hosts as well.  
 
An extended version of the MTRP protocol specified in RFC 3887 is used to perform tracking and recall operations. If only tracking is desired a single MTQP server can be used for the entire deployment and can be run on any host. However, in order to recall messages an MTQP server must be running on every host with an MTA, and if "total recall" is enabled MTQP servers are required on all store hosts as well.  
  
The MTQP server leverages existing MTA facilities which require a channel. Note that the MTQP channel doesn&#x27;t sent or receive messages and has no rewrite rules or associated queue. The specification of such a channel is very simple:  
+
The MTQP server leverages existing MTA facilities which require a channel. Note that the MTQP channel doesn&#x27;t sent or receive messages and has no rewrite rules or associated queue. The specification of such a channel is very simple.  In legacy configuration:  
 
   
 
   
 
  <font color="mediumblue">         
 
  <font color="mediumblue">         
  mtqp smtp
+
mtqp smtp
  mtqp-daemon
+
mtqp-daemon
 
  </font>
 
  </font>
 +
or in Unified Configuration:
 +
 +
<font color="mediumblue">
 +
msconfig&#x3e; '''set channel:mtqp.officical_host_name mtqp-daemon'''
 +
msconfig# '''set channel:mtqp.smtp'''</font>
 
Although somewhat counterintuitive, the "[[smtp, smtp_cr, smtp_crlf, smtp_crorlf, smtp_lf, nosmtp, lmtp, lmtp_cr, lmtp_crlf, lmtp_crorlf, lmtp_lf Channel Options#smtp|<code>smtp</code>]]"  channel option is required to indicate this is a channel with an associated server. Various optional channel options can also be specified, including:  
 
Although somewhat counterintuitive, the "[[smtp, smtp_cr, smtp_crlf, smtp_crorlf, smtp_lf, nosmtp, lmtp, lmtp_cr, lmtp_crlf, lmtp_crorlf, lmtp_lf Channel Options#smtp|<code>smtp</code>]]"  channel option is required to indicate this is a channel with an associated server. Various optional channel options can also be specified, including:  
  
Line 95: Line 100:
  
 
<li>
 
<li>
<code>maytlsserver</code> or <code>musttlsserver</code> are used to control whether SSL/TLS is allowed or required, respectively, before tracking/recall operations may be performed.  
+
[[maytls, maytlsclient, maytlsserver, musttls, musttlsclient, musttlsserver, notls, notlsclient, notlsserver, tlsswitchchannel Options#maytlsserver|<code>maytlsserver</code> or <code>musttlsserver</code>]] are used to control whether SSL/TLS is allowed or required, respectively, before tracking/recall operations may be performed.  
 
</li>
 
</li>
  
 
<li>
 
<li>
<code>slave_debug</code> enables MTQP server debugging.  
+
[[master_debug, nomaster_debug, slave_debug, noslave_debug Channel Options#slave_debug|<code>slave_debug</code>]] enables MTQP server debugging.  
 
</li>
 
</li>
  
 
<li>
 
<li>
<code>maysaslserver</code> may be specified to enable user authentication. User authentication is required in order to use the MTRACK and/or MRECALL commands.  
+
[[maysasl, maysaslclient,  maysaslserver,  mustsasl, mustsaslclient,  mustsaslserver,  nosasl, nosaslclient, nosaslserver,  disconnectbadauthlimit Channel Options#maysaslserver|<code>maysaslserver</code>]]  may be specified to enable user authentication. User authentication is required in order to use the MTRACK and/or MRECALL commands.  
 
</li>
 
</li>
  
 
<li>
 
<li>
<code>disconnectbadauthlimit</code>, <code>disconnectbadcommandlimit</code>, and <code>disconnectcommandlimit</code> have their usual meanings.  
+
[[maysasl, maysaslclient,  maysaslserver,  mustsasl, mustsaslclient,  mustsaslserver,  nosasl, nosaslclient, nosaslserver,  disconnectbadauthlimit Channel Options#disconnectbadauthlimit|<code>disconnectbadauthlimit</code>]], [[recipientlimit, recipientcutoff, deferralrejectlimit, disconnectrecipientlimit, disconnectrejectlimit, disconnectbadcommandlimit, disconnectbadburllimit, disconnectcommandlimit Channel Options#disconnectbadcommandlimit|<code>disconnectbadcommandlimit</code>]], and [[recipientlimit, recipientcutoff, deferralrejectlimit, disconnectrecipientlimit, disconnectrejectlimit, disconnectbadcommandlimit, disconnectbadburllimit, disconnectcommandlimit Channel Options#disconnectcommandlimit|<code>disconnectcommandlimit</code>]]  have their usual meanings.  
 
</li>
 
</li>
  
Line 117: Line 122:
  
 
<dt>
 
<dt>
<code>TOTAL_RECALL</code> (boolean 0 or 1, default 0)  
+
<code>TOTAL_RECALL</code> (boolean 0 or 1, default 0)  
 
</dt>
 
</dt>
  
 
<dd>
 
<dd>
  If set, this option enables recall of messages from the store. Note that if this is used it will result in messages being deleted from recipient&#x27;s folder irrespective of whether or not they have been read. Use of this option should only be undertaken with a full understanding of the possible consequences.  
+
  If set to 1, this option enables recall of messages from the store. Note that if this is used it will result in messages being deleted from recipient&#x27;s folder irrespective of whether or not they have been read. Use of this option should only be undertaken with a full understanding of the possible consequences.  
 
</dd>
 
</dd>
  
Line 160: Line 165:
 
Additionally, the server recognizes and will honor the  [[TCPIP-channel-specific options#TCPIP-channel-specific_options|TCP/IP channel-specific options]]<code> LOG_CONNECTION</code>, <code>TRACE_LEVEL</code>, <code>MAX_THREADS</code>, and <code>IGNORE_BAD_CERT</code>.  
 
Additionally, the server recognizes and will honor the  [[TCPIP-channel-specific options#TCPIP-channel-specific_options|TCP/IP channel-specific options]]<code> LOG_CONNECTION</code>, <code>TRACE_LEVEL</code>, <code>MAX_THREADS</code>, and <code>IGNORE_BAD_CERT</code>.  
  
Finally, a [[Dispatcher#Dispatcher|Dispatcher]]  service definition for the MTQP server is also required:  
+
Finally, a [[Dispatcher#Dispatcher|Dispatcher]]  service definition for the MTQP server is also required. In legacy configuration:  
 
   
 
   
 
  <font color="mediumblue">         
 
  <font color="mediumblue">         
Line 169: Line 174:
 
   STACKSIZE=2048000
 
   STACKSIZE=2048000
 
  </font>
 
  </font>
 +
or in Unified Configuration:
 +
 +
<font color="mediumblue">
 +
msconfig&#x3e; '''set service:MTQP.image IMTA_BIN:mtqp'''
 +
msconfig# '''set service:MTQP.tcp_ports 1038'''
 +
msconfig# '''set service:MTQP.logfilename IMTA_LOG:mtqp_server.log'''
 +
msconfig# '''set service:MTQP.stacksize 2048000'''</font>
  
 
See also:
 
See also:

Revision as of 08:48, 3 October 2015


There are three parts to setting up message tracking and recall capabilities:

  1. Setting up a shared memcached server (or other software that implements the memcache protocol in a compatible fashion) to store tracking/recall information.
  2. Configuring all the MTAs in the deployment to enable tracking and recall.
  3. Setting up Message Tracking and Query Protocol (MTQP) servers on every MTA and possibly additional hosts.

Memcache server setup

Setting up memcached or compatible server is beyond the scope of this document. That said, note that mecmached setup is very simple: Configuration consists of a small number of options on the command line. Indeed, it's often possible to simply say:

        
  memcached -l <ip-address>

Where <ip-address> is the IP address where memcached accepts connections.

Once a server implementing the memcache protocol has been set up, every MTA in the deployment have to be configured to access it. This is done with the memcache_host MTA option; there is no tracking-specific host setting at the present time. The memcache_port MTA option must be explicitly set if the server is listening on a port other than 11211. It may be appropriate to set the memcache_expire MTA option to an appropriate value as well.

Tracking/recall is enabled within the MTA by setting the tracking_mode MTA option to 1. Logging tracking identifiers is good idea; this is controlled by the log_tracking MTA option.

Taken together, and assuming the memcache server is listening on the default port, the MTA option settings should look something like this:


msconfig> show memcache_host
role.mta.memcache_host memcache-server-ip
msconfig> show tracking_mode
role.mta.tracking_mode 1
msconfig> show log_tracking
role.mta.log_tracking 1

or in legacy configuration:

        
  MEMCACHE_HOST=<memcache-server-ip>
  TRACKING_MODE=1
  LOG_TRACKING=1

MTA channel setup for message tracking and recall

The next step is to tell the tracking subsystem the semantics of the various channels in the deployment using the trackinginternal, trackingrelayed, and trackingdelivered. The general rules are:

  1. Internal processing channels such as process, reprocess, conversion, and defragment require no special decoration.
  2. Channels that perform final delivery must be marked with the trackingdelivered channel option. This includes not only ims-ms and tcp_lmtpcs channels, but also the bitbucket and filter_discard channels.
  3. Any channel that relays messages to systems outside the deployment must be marked with the trackingrelayed channel option. Of course this includes the tcp_local channel, but would also include, say a special tcp_aol channel set up to handle mail to the aol.com domain.
  4. Finally, any channel that relays message to other MTAs inside the deployment where tracking is enabled. This would typically be something like a tcp_intranet channel.

Note that for tracking and recall to work messages relayed to external systems MUST be handled by a different channel than messages relayed to other MTAs inside the deployment. (At a minimum, this tends to mean using a distinct tcp_local  vs.  tcp_intranet channel.) Meeting this requirement may require additional configuration changes.

The MTRK SMTP extension defined in RFC 3885 is used to transfer tracking/recall information from one MTA to another. Use of this extension MUST be enabled on SMTP connections between MTAs inside the deployment. It also must be enabled on SUBMIT channels used by tracking/recall-enabled clients.

Since MTRK is an SMTP extension, its use is negotiated by the SMTP client and server. So the simplest way to activate this extension is to put the trackingclient and trackingserver on the defaults channel. However, if you wish to avoid use of this extension with systems outside the deployment, a more targeted approach is needed: Place trackingclient on every tcp_* channel that has trackinginternal set. Then place trackingserver on every channel that accepts messages from other hosts within the deployment. (Note that once again this may require configuration changes to separate internal and external traffic.)

MTQP server setup

An extended version of the MTRP protocol specified in RFC 3887 is used to perform tracking and recall operations. If only tracking is desired a single MTQP server can be used for the entire deployment and can be run on any host. However, in order to recall messages an MTQP server must be running on every host with an MTA, and if "total recall" is enabled MTQP servers are required on all store hosts as well.

The MTQP server leverages existing MTA facilities which require a channel. Note that the MTQP channel doesn't sent or receive messages and has no rewrite rules or associated queue. The specification of such a channel is very simple. In legacy configuration:

        
mtqp smtp
mtqp-daemon

or in Unified Configuration:


msconfig> set channel:mtqp.officical_host_name mtqp-daemon
msconfig# set channel:mtqp.smtp

Although somewhat counterintuitive, the "smtp" channel option is required to indicate this is a channel with an associated server. Various optional channel options can also be specified, including:

Some MTQP-server-specific options are also available:

TOTAL_RECALL (boolean 0 or 1, default 0)
If set to 1, this option enables recall of messages from the store. Note that if this is used it will result in messages being deleted from recipient's folder irrespective of whether or not they have been read. Use of this option should only be undertaken with a full understanding of the possible consequences.
AUTH_DEBUG (debug token list, default "")
This option is used to specify authentication debugging tokens.
COMMAND_TIMEOUT (integer seconds, default 60)
This option specifies the time the server will wait for a command before disconnecting.
STATUS_TIMEOUT (integer seconds, default 60)
This option specifies the time the server will wait for a status write to the client before disconnecting.
CUSTOM_BANNER_STRING (string, default "")
This option specifies the name the server uses to announce itself.

Additionally, the server recognizes and will honor the TCP/IP channel-specific options LOG_CONNECTION, TRACE_LEVEL, MAX_THREADS, and IGNORE_BAD_CERT.

Finally, a Dispatcher service definition for the MTQP server is also required. In legacy configuration:

        
  [SERVICE=MTQP]
  PORT=1038
  IMAGE=IMTA_BIN:mtqp
  LOGFILE=IMTA_LOG:mtqp_server.log
  STACKSIZE=2048000

or in Unified Configuration:


msconfig> set service:MTQP.image IMTA_BIN:mtqp
msconfig# set service:MTQP.tcp_ports 1038
msconfig# set service:MTQP.logfilename IMTA_LOG:mtqp_server.log
msconfig# set service:MTQP.stacksize 2048000

See also: