Difference between revisions of "Smartsend callouts"

From Messaging Server Technical Reference Wiki
Jump to: navigation, search
m (Bulk update)
m (Bulk update)
 
Line 19: Line 19:
 
  </font>
 
  </font>
 
An optional debug level may be specifed; if none is specified a level of 0 (debugging disabled) is assumed. The recipe may be re-run with a different debug level; if this is done the debug level will be updated in the various callouts.  
 
An optional debug level may be specifed; if none is specified a level of 0 (debugging disabled) is assumed. The recipe may be re-run with a different debug level; if this is done the debug level will be updated in the various callouts.  
 +
 +
Although it&#x27;s included as part of the smartsend plugin, the MX rollup capability is an essentially separate facility. Accordingly, a separate recipe is provided to configure it:
 +
 +
<font color="mediumblue">
 +
# msconfig
 +
msconfig&#x3e; run rollup.rcp &#x5b;debug-level &#x5b;rollup-domain-suffix &#x5b;rollup-channel&#x5d;&#x5d;&#x5d;
 +
</font>
 +
An optional debug level, rollup domain suffix, and the channel used to handle rollups may be specified. These default to "0", ".rollup", and "tcp_rollup", respectively.
 +
 +
The script will create the rollup channel if it does not already exist, using the tcp_local channel as a model.
  
 
=== Database entry formats ===
 
=== Database entry formats ===
Line 248: Line 258:
  
 
<span id='ipbackofftimeout_smartsend_parameter'></span>The <code>ipbackofftimeout</code> parameter implements the same functionality as the [[ipbackofftimeout Channel Option#ipbackofftimeout|<code>ipbackofftimeout</code>]] channel option on smartsend domain entries. As with the channel option, the syntax is an integer value in seconds.  
 
<span id='ipbackofftimeout_smartsend_parameter'></span>The <code>ipbackofftimeout</code> parameter implements the same functionality as the [[ipbackofftimeout Channel Option#ipbackofftimeout|<code>ipbackofftimeout</code>]] channel option on smartsend domain entries. As with the channel option, the syntax is an integer value in seconds.  
 +
 +
==== <code>log_headers</code> - Header fields to log in transaction record ====
 +
 +
The <code>log_headers</code> parameter is used to specify one or more additional header fields to log as part of the transaction log entry. The parameter value is a comma-separated list of header field names. Unlike header option file based logging, any field name may be specified, recognized or not. Note that the [[Release Notes#log_header|<code>log_header</code>]] MTA option must also be set appropriately.
  
 
==== <code>maxconnectionsperdomain</code> - Maximum connection to a domain ====
 
==== <code>maxconnectionsperdomain</code> - Maximum connection to a domain ====

Latest revision as of 13:39, 10 October 2019

The smartsend.so library provides a collection of mapping callouts that can be used in to optimize the delivery of opt-in bulk email.

smartsend makes use of a site-provided database that provides information about senders, deployment hosts, recipient domains, and available source IPs. The MTA's general database facility is used for this purpose, although note that this can easily be configured to perform queries using the memcache or Redis protocols.

Many smartsend options also require a cache server. smartsend talks to this server using either the memcache or Redis protocols. A per-host instance of memcached is sufficient for this purpose, although a few functions require deployment of a caching server available across multiple hosts. If the memcache protocol is employed the host and port specified by the memcache_host and memcache_port channel options respectively, with fallback to the corresponding MTA options, will be used.

When used to support multiple independent message sources, it is expected that the first conversion tag associated with each message will be used as a "virtual MTA" identifier. Note that the callouts consume the conversion tag if it is present; they do not provide the means to set it.

The following sections describe the database format as well as the various callouts smartsend provides. It is important to understand that most of the callouts are expected to be configured as a group since they work together to provide a single coherent service. Most of the callouts are designed not to have any effect, i.e., return a failure condition, when they have nothing to do, so they may be combined with other uses of the corresponding mapping.

While the following sections provide details as to how to configure the smartsend callouts, this may be done automatically in a unified configuration by using the provided smartsend recipe:


# msconfig
msconfig> run smartsend.rcp [debug-level]

An optional debug level may be specifed; if none is specified a level of 0 (debugging disabled) is assumed. The recipe may be re-run with a different debug level; if this is done the debug level will be updated in the various callouts.

Although it's included as part of the smartsend plugin, the MX rollup capability is an essentially separate facility. Accordingly, a separate recipe is provided to configure it:


# msconfig
msconfig> run rollup.rcp [debug-level [rollup-domain-suffix [rollup-channel]]]

An optional debug level, rollup domain suffix, and the channel used to handle rollups may be specified. These default to "0", ".rollup", and "tcp_rollup", respectively.

The script will create the rollup channel if it does not already exist, using the tcp_local channel as a model.

Database entry formats

Database entries make use of several common pieces of syntax, the first of which is a parameter list. Parameter lists are represented using a simplified variant of MIME content-type parameter syntax. The ABNF (RFC 2234) syntax for a smartsend parameter list is:


  parameter-list = parameter-name-value [";" parameter-name-value]
  parameter-name-value = *WSP parameter-name *WSP "=" *WSP parameter-value *WSP
  parameter-name = (ALPHA / DIGIT / "-" / "_")1*
  parameter-value = <any CHAR excluding ";">

Note that the allowable characters in a parameter-value is often further constrained by the context in which the parameter-list appears. In particular, vertical bars ("|"), commas (","), and whitespace are all used in some "outer" contexts as delimiters.

Examples of parameter lists include:


  maxmx=5
  maxmessagerateperdomain=30/1200;maxmessagesperconnection=20
  dkimidentity-1=example.net;dkimselector-1=brisbane
  debug=3

A parameterized list of IP addresses (IP list for short) is also used by several types of database entries. The following definition makes use of the IPv4address and IPv6address productions specified in RFC 5954:


  parameterized-ip-list = [global-parameter-list "|"] ip-and-parameters ["," ip-and-parameters]
  global-parameter-list = [parameter-list]
  ip-and-parameters = ["-"] (IPv4address / IPv6address) ["#" (IPv4address / IPv6address)] [";" ip-parameter-list]
  ip-parameter-list = [parameter-list]

Note that IP address parameters allow the two address form supported by the interfaceaddress channel option.

Parameterized IP lists are allowed to be up to 4096 characters in length.

Examples of parameterized IP lists would include:


  1.1.1,1,2.2.2.2,-3.3.3.3,4.4.4.4
  10.59.230.40;banner_host=mauve.example.com,10.59.230.169;banner_host=plum.example.com
  66.218.59.24#10.59.230.40;banner_host=mauve.example.com
  maxmessagerateperdomain=2/300|10.59.230.40,10.59.230.169
  maxmessagesperconnection=30|10.59.230.40;maxmessagsperconnection=10,10.59.230.169,10.59.230.170

A parameterized list of MTA names (MTA list for short) is also used by several types of database entries. The following definition makes use of the Domain productions specified in RFC 5321:


  parameterized-mta-list = [global-parameter-list "|"] mta-and-parameters ["," mta-and-parameters]
  global-parameter-list = [parameter-list]
  mta-and-parameters = ["-"] Domain [";" mta-parameter-list]
  mta-parameter-list = [parameter-list]

Parameterized MTA lists are allowed to be up to 4096 characters in length.

Examples of parameterized MTA lists would include:


  host1.example.com,host2.example.com,host2.example.com
  host1.example.org;received_domain=host1.example.org,host2.example.org;received_domain=host2.example.org,
  id_domain=example.edu|host1.example.edu,host2.example.edu

auth_access callout

The auth_access callout is intended to be called from the AUTH_ACCESS mapping. It provides the ability to:

  • set various limits and operating parameters based on the message's destination domain
  • select a client IP address from a group of available IP addresses for each message in a round-robin fashion
  • impose additional limits and opeational settings on a per-IP basis.

The AUTH_ACCESS mapping should be set up as follows:


AUTH_ACCESS 
 
   *     $C$[IMTA_LIB:smartsend.so,auth_access,<pflags>|$0]$E


Here <pflags> should be replaced with an unsigned integer flag value. The lowest three bits enable increasing levels of debug output, with "0" resulting in no output. Bit 3 (value 8), if set, enables routing-only mode, which is described below.

Note that the callout receives the entire mapping probe. It is designed to process the entire probe and automatically adjusts its behavior to match the variations in probe format caused by setting various MTA options.

Additionally, bit 0, value 1, of the include_retries MTA option should be set, and if conversion tags are to be used to select IP address groups, bit 11, value 2048, of the include_conversiontag MTA option should also be set. Note that the list of conversion tags is interpreted as follows:


<virtual-MTA>,<job-id>,<sender-OCID>,<tenant-OCID>,<compartment-OCID>

The callout normally operates as follows:

  1. Parse the mapping probe into its components. The callout will fail if the input string cannot be parsed.
  2. Construct a key of the form domain_<domain>, where <domain> is the lower cased destination domain for the message, ad look up the key in the general database. The value of any entry found is expected to be a parameter list.
  3. If <tenant-OCID> conversion tag is present construct a key of the form ociddomain_<tenant-OCIDA>_<domain>, where <tenant-OCIDA> and <domain> are the lower cased tenant OCID and destination domain for the message, respectively, and look up the key in the general database. The value of any entry found is expected to be a parameter list.
  4. If <sender-OCID> conversion tag is present construct a key of the form ociddomain_<sender-OCIDA>_<domain>, where <sender-OCIDA> and <domain> are the lower cased sender OCID and destination domain for the message, respectively, and look up the key in the general database. The value of any entry found is expected to be a parameter list.
  5. Construct a key of the form ips_<mtaid>_<virtual-MTA>, where <mtaid> is the MTA id and <virtual-MTA> is the first conversion tag associated with the current message. Both the id and conversion tag are converted to lower case.

    If no MTA id has been specified the official host name on the local channel is used instead. The preceding "_" is omitted if the specified MTA id is blank.

    If there are no conversion tags associated with the message use a key of the form ips_<channel>, where <channel> is the channel where the message is queued.

  6. Look up the key in the general database. The value of any entry found is expected to be an IP list.
  7. If an IP list entry is found select an IP from the list at a random starting point, and proceeding around the list in a strict round-robin fashion on subsequent dequeue attempts.
  8. If an EXTERNAL_TO_INTERNAL mapping exists, apply it to any IP address selected from the IP list that just specifies an IP address (as opposed to an external#internal address pair). The entry will be skipped if the mapping doesn't produce a result and set $Y. The result can be either just the internal address, which will be become the internal IP address for the entry, or an external#internal pair, which will replace the IP address information in the entry.
  9. Impose any limits and set any operational parameters specified by the various parameter lists attached to the destination domain, the IP list as a whole, and the chosen IP.
  10. Limit checks may cause the selected IP addresses to be become ineligible for use. If this happens try the next IP address on the list. If all of the addresses are rejected the message dequeue attempt will fail with a temporary error, causing the MTA to try again later.
  11. If no IP list entry is found and a conversion tag is present construct a third key of the form mta_<conversiontag> and look it up in the general database. Again, the conversion tag is converted to lower case.
  12. If an MTA entry is found it is expected to consist of a comma or whitespace-separated list of MTA names that are able to process messages with the specified conversion tag. The list is checked to see if the current MTA is listed. If it isn't an MTA that is on the list is chosen at random and the message destination is set to that MTA.
  13. If the current MTA is on the list or no MTA list is present the dequeue operation proceeds normally.

If bit 3 (value 8) of <pflags> is set, the plugin operates in routing-only mode:

  1. Parse the mapping probe into its components. The callout will fail if the input string cannot be parsed or if no conversion tag is present
  2. Construct a key of the form mta_<virtual-MTA> and look it up in the general database. The conversion tag is converted to lwoer case.
  3. If an MTA entry is found it is expected to consist of a comma or whitespace-separated list of MTA names that are able to process messages with the specified conversion tag. The list is checked to see if the current MTA is listed. If it isn't an MTA that is on the list is chosen at random and the message destination is set to that MTA.
  4. If the current MTA is on the list or no MTA list is present the dequeue operation proceeds normally.

Note that the addition of an IP address to a IP list entry when a message is being retried unavoidably results in the disruption of the strict round-robin schedule. However, a special mechanism is provided to remove an address from the rotation without causing any disruption: Placing a minus sign "-" in front of an IP address on the list will cause that address to be skipped. A count of the number of IP address skips is maintained in order to preserve the strict round-robin usage. Note that this does NOT work if the IP address is simply removed from the entry.

Also note that the log_smartsend MTA option may be used to include additional information about smartsend callout actions in transaction log records.

The following sections describe the parameters available for use in auth_access database entries. Note that while all parameters are available in all contexts, it may not make sense to, say, specify the banner host on the basis of the destination domain or to control the use of TLS on a per-IP basis.

backoff - Retry frequency for messages

The backoff parameter implements the same functionality as the backoff channel option on smartsend domain entries. The syntax is identical to that of the backoff channel option in unified configuration.

The banner_host parameter specifies the host name to use in the EHLO/HELO command that is issued once a connection is established. This overrides BANNER_HOST TCP/IP-channel-specific option for this connection.

chunking - Control use of SMTP CHUNKING

New in MS 8.1.0.1. The chunking parameter provides the ability to control the use of the SMTP CHUNKING extension by the SMTP client. Possible values are:

  • "disable", which disables the use of chunking, as if nochunkingserver had been specified on the channel, and
  • "optional", which allows the use of chunking if the channel is set to allows it.

debug - Enable smartsend/channel debugging

New in MS 8.1.0.1. The debug parameter provides the ability to enable debugging on a per-IP-list basis. A single integer argument is required. Positive values enable corresponding levels of smartsend debugging; master_debug is also enabled for the duration of processing the current message.

ipbackoff - Retry frequency for messages in IP backoff mode

The ipbackoff parameter implements the same functionality as the ipbackoff channel option on smartsend domain entries. The syntax is identical to that of the ipbackoff channel option in unified configuration.

ipbackofftimeout - Timeout for IP backoff entries

The ipbackofftimeout parameter implements the same functionality as the ipbackofftimeout channel option on smartsend domain entries. As with the channel option, the syntax is an integer value in seconds.

log_headers - Header fields to log in transaction record

The log_headers parameter is used to specify one or more additional header fields to log as part of the transaction log entry. The parameter value is a comma-separated list of header field names. Unlike header option file based logging, any field name may be specified, recognized or not. Note that the log_header MTA option must also be set appropriately.

maxconnectionsperdomain - Maximum connection to a domain

The maxconnectionsperdomain parameter implements limits on the number of simultaneous connections that can be opened from a single source IP to a destination domain. The value of this parameter is one or two space-separated connection limit lists. The first connection limit list value specifies the limits to use when the IP address is operating normally; the second specifies the limits to use when operating in IP backoff mode. If the IP backoff value is omitted it defaults to the first value.

As of MS 8.1, four types of per-domain connection limits are provided:

Per-domain connection limits
Additional Qualification Code Probe Description
Source IP I maxconnectionsperdomain_<IP>_<domain> Per source IP per domain limit
Host H maxconnectionsperdomain_<host>_<domain> Per server host per domain limit
Tag T maxconnectionsperdomain_<tag>_<domain> Per conversion tag per domain limit
  N maxconnectionsperdomain_<domain> Per domain limit

Any or all of these limits may be engaged simultaneously. If multiple limits are used all limits must be satisfied for the connection attempt to proceed.

A connection limit list is specified as one or more limit codes followed by the limit itself, written as an unsigned integer. The "I" code may be omitted from a per-IP limit if it appears first - this is the only supported format in MS 8.0.2.3. Formally, the syntax is:


  connection-limit-list = (ip-limit-value *limits) / 1*limits
  limits = ip-limit / host-limit / tag-limit / limit
  ip-limit = "I" ip-limit-value
  ip-limit-value = 1*DIGIT
  host-limit = "H" host-limit-value
  host-limit-value = 1*DIGIT
  tag-limit = "T" tag-limit-value
  tag-limit-value = 1*DIGIT
  limit = "N" limit-value
  limit-value = 1*DIGIT

For example, a connection limit list value of "I4N10" specifies a 4 connection per-IP-per-domain limit and a 4 connection per-domain limit. A value of "10" specifies a 10 connection per-IP-per-domain limit.

Note that the identically named maxconnectionsperdomain channel option can be used to set connection limits on a per channel basis.

maxmessagerateperdomain - Maximum message rate to a domain

The maxmessagerateperdomain parameter provides the means to limit the rate at which messages are delivered to a destination domain. The value of this parameter is one or two space-separated rate limit lists. The first limit list value specifies the rate limits to use when the IP address is operating normally; the second specifies the limits to use when operating in IP backoff mode. If the IP backoff value is omitted it defaults to the first value.

As of MS 8.1, four types of per-domain rate limits are provided:

Per-domain message rate limits
Additional Qualification Code Probe Description
Source IP I maxmessagerateperdomain_<IP>_<domain> Per source IP per domain limit
Host H maxmessagerateperdomain_<host>_<domain> Per server host per domain limit
Tag T maxmessagerateperdomain_<tag>_<domain> Per conversion tag per domain limit
  N maxmessagerateperdomain_<domain> Per domain limit

Any or all of these limits may be engaged simultaneously. If multiple limits are used all limits must be satisfied before a delivery attempt will be made.

A rate limit list is specified as one or more limit codes followed by the limit itself, written as an unsigned vulgar fraction or integer. The "I" code may be omitted from a per-IP limit if it appears first - this is the only supported format in MS 8.0.2.3. The numerator of the vulgar fraction specifies the number of messages to allow and the denominator specifies the time window in seconds. If a single integer is specified it is treated as a numerator with a default denominator of 3600 (one hour).

Formally, the syntax is:


  rate-limit-list = (ip-limit-value *limits) / 1*limits
  limits = ip-limit / host-limit / tag-limit / limit
  ip-limit = "I" ip-limit-value
  ip-limit-value = 1*DIGIT
  host-limit = "H" host-limit-value
  host-limit-value = 1*DIGIT
  tag-limit = "T" tag-limit-value
  tag-limit-value = 1*DIGIT
  limit = "N" limit-value
  limit-value = 1*DIGIT ["/" 1*DIGIT]

For example, a rate limit list value of "I100N100000/86400" specifies a 100 messages per hour per-IP-per-domain limit and a 100,000 per day per-domain limit. A value of "50" specifies a 50 messages per hour per-IP-per-domain limit.

Note that the identically named maxmessagerateperdomain channel option can be used to set rate limits on a per channel basis.

maxmessagesperconnection - Maximum messages per connection

The maxmessagesperconnection parameter limits the number of messages that can be transferred by a single connection. It overrides the ATTEMPT_TRANSACTIONS_PER_SESSION TCP/IP-channel-specific option for this connection.

max_mx_records - Maximum MX attempts

The max_mx_records parameter limits the number of MX records to consider when attempting to connect. It overrides MAX_MX_RECORDS TCP/IP-channel-specific option for this connection.

override_host - Override destination host

The override_host parameter specifies the host name to use as the destination for connections, overriding any host name that appears in any destination address.

status - Force hold, return of messages

The status parameter provides the means to return or hold messages. Possible values are "active", which causes deliveries to proceed normally, "hold", which suspends deliveries, and "return", which causes messages to be returned with no further delviery attempts.

tls - Control use of TLS

New in MS 8.1.0.1. The tls parameter provides the ability to control the use of the SMTP CHUNKING extension by the SMTP client. Possible values are:

  • "disable"- disables the use of TLS, as if notlsserver had been specified on the channel,
  • "optional", which allows the use of tLS if the channel is set to allows it, and
  • "require", which requires the use of TLS, as if musttlsserver had been specified on the channel,

auth_deaccess callout

The auth_deaccess callout is intended to be called from the AUTH_DEACCESS mapping. It pairs with the auth_access callout described in the previous section to decrement counters associated with some of the limits auth_access implements. At present this is the only functionality provided by this callout.

The AUTH_DEACCESS mapping should be set up as follows:


AUTH_DEACCESS
 
   *     $C$[IMTA_LIB:smartsend.so,auth_deaccess,<pflags>|$0]$E


Here <pflags> should be replaced with an unsigned integer flag value. Currently only the lowest three bits of this bit-encoded value have any meaning -- they enable increasing levels of debug output, with "0" resulting in no output.

Note that the callout receives the entire mapping probe. It is designed to process the entire probe and adjusts its behavior to match the variations in probe format caused by setting various MTA options.

conversions callout

The conversions callout is intended to be called from the CONVERSIONS mapping to provide a database-driven means of configuring multiple DKIM signing keys.

The CONVERSIONS mapping should be set up as follows:


CONVERSIONS
 
   *     $C$[IMTA_LIB:smartsend.so,conversions,pflags=<pflags>;$0]$E


IMPORTANT NOTE: The format of the mapping probe is NOT the same as for the previously AUTH_ACCESS or AUTH_DEACCESS mapping probes. Here <pflags> should be replaced with an unsigned integer flag value. The lowest three bits of this bit-encoded value enable increasing levels of debug output, with "0" resulting in no output.

Note that the callout receives the entire mapping probe. It is designed to process the entire probe and adjusts its behavior to match the variations in probe format caused by setting various MTA options.

Additionally, bit 0, value 1, of the include_domain MTA option should be set, and if conversion tags are to be used to select DKIM keys using "dkim_" lookups. bit 1, value 2, of the include_conversiontag MTA option should also be set. IMPORTANT NOTE: "dkim_" lookups should be disabled if "ocid_" lookups in the AUTH_REWRITE mapping are enabled for DKIM use. In this case bit 4 (value 16) of pflags should be set to disable such lookups.

The callout operates as follows:

  1. Parse the mapping probe into its components. The callout will fail if the input string cannot be parsed.
  2. Construct a key of the form domain_<domain>, where <domain> is the lower cased destination domain (DOMAIN value in the CONVERSIONS mapping probe) for the message.
  3. Look up the key in the general database. The value of any entry found is expected to be a parameter list.
  4. If bit 4 (value 16) of the pflags value is clear, construct a second key of the form dkim_<conversiontag>, where <conversiontag> is the first conversion tag (TAG value in the CONVERSIONS mapping probe) associated with the current message and converted to lower case. If there are no conversion tags associated with the message use a key of the form ips_<channel>, where <channel> is the destination channel (OUT-CHAN value in the CONVERSIONS mapping probe) for the message.
  5. Look up the key in the general database. The value of any entry found is expected to be another parameter list.
  6. Use the combined parameter lists to select appropriate DKIM keys to use to sign the message, and generate a mapping result that will cause this to happen.
  7. If bit 3 (value 8) of the pflags value is set and a conversion tag is present, construct a key of the form mta_<conversiontag> and look up the corresponding entry in the general database. If an entry is found it is expected to consist of a list of the MTAs capable of handling this conversion tag value. If an entry is found and the current MTA is not on the list the is forced into the tcp_intranet channel queue so it can be routed to the correct MTA.

dkimidentity-N, dkimselector-N - DKIM parameters

The actual DKIM parameters correspond exactly to the DKIM template keywords (dkimidentity-N and dkimselector-N) used in the CONVERSIONS mapping table and have the same semantics as the destinationdkimidentityN channel option. Note, however, that the parameter separator is a semicolon while the CONVERSIONS template keyword separator is a comma.

ip_backoff callout

New in MS 8.1. The ip_backoff callout is used to activate or deactivate IP backoff mode for a specified IP address and destination domain. Unlike other smartsend callouts, the ip_backoff callout is not associated with any specific mapping. Rather, it is designed to be used as part of site-specific checks that determine when IP backoff mode needs to be in effect.

A call to the ip_backoff callout should appear as follows:



    $[IMTA_LIB:smartsend.so,ip_backoff,pflags=<pflags>|<channel>|<ip>|<domain>|<value>|<timeout>|]

  The arguments are:  
<pflags>
Unsigned integer flag value. Currently only the lowest three bits of this bit-encoded value have any meaning -- they enable increasing levels of debug output, with "0" resulting in no output.
<channel>
Channel associated with the IP address.
<ip>
IP address for which IP backoff is to be enabled or disabled.
<domain>
Destination domain for which backoff is to be enabled or disabled.
<valuet>
The backoff entry's value. At the present time the entry value is ignored and this field should be left blank - but see the special value and their semantics below.
<timeout>
Timeout, in seconds, for the bckoff entry. A value of 0 causes the timeout specified by the ipbackofftimeout channel option to be used. If no timeout value has been specified a default value of 3600 (one hour) is used. The timeout argument can be omitted, in which case it's value defaults to 0.

The callout succeeds even if the entry cannot be set.

The special value "?" causes a lookup to be performed on the backoff entry instead of a set. The callout fails if the entry is not present. If the entry is present the callout succeeds and returns the entry's value.

The specil value "-" causes the specified backoff entry to be deleted. The callout always succeeds.

The ip_backoff callout operates by creating and deleting entries in memcache or Redis. The entry's name is of the form "ip_backoff_<ip>" and at present the value is the string "mode=1".

auth_rewrite callout

The auth_rewrite callout is intended to be called from the AUTH_REWRITE mapping. It provides the ability to confirm that a conversion tag appearing in a header field has a corresponding IP list or MTA entry, the ability to activate DKIM signing, and can be used to override the received_domain and id_domain settings for the current message.

The confirmation functionality is useful in preventing messages from being routed to the wrong MTA during configuration updates. The ability to override domain settings can be used to obfuscate local host information.

If DKIM support and obfuscation support is not enabled the callout succeeds if the confirmation attempt fails, so that the mapping template itself determines the appropriate action to be taken. If, on the other hand, DKIM or obfuscation support is enabled, the callout succeeds when it needs to return a result and fails otherwise, in which case the plugin necessarily provides all results.

Note that this callout is only needed in specific circumstances and therefore is not configured by the smartsend.rcp recipe.

The AUTH_REWRITE mapping should be set up approximately as follows if bit 4 (value 16) of pflags is clear:


AUTH_REWRITE 
 
   *     $C$[IMTA_LIB:smartsend.so,auth_rewrite,<pflags>|$0]$E$N$X4.7.0|Unknown$ tag


Note that the template in this example causes a temporary failure to be returned should confirmation fail.

The callout should be configured as follows if either bit 4 (value 16, DKIM) or bit 5 (value 32, domain obfuscation) are set:


AUTH_REWRITE 
 
   *     $C$[IMTA_LIB:smartsend.so,auth_rewrite,<pflags>|$0]$E


Note that the template in this example simply returns the callout result if the callout succeeds.

Here <pflags> should be replaced with an unsigned integer flag value. The lowest three bits enable increasing levels of debug output, with "0" resulting in no output. Bit 3 (value 8), controls the type of lookup that is performed for confirmation and domain obfuscation information. A lookup for "ips_" entries is performed if the bit is clear; a lookup for "mta_" entries is performed if the bit is set. If bit 5 (value 32) is set the result may contain override values for received_domain and/or id_domain.

Bit 4 (value 16) or bit 5 (value 32), if set, cause a lookup to be performed on a key of the form ocid_<sender-OCID>, where <sender-OCID> is the sender OCID (see below) If this lookup fails an additional query is done for ocid_<tenant-OCID>, where <tenantr-OCID> is the tenant OCID. The result is interpreted as a parameter list and can contain DKIM parameters (bit 4 set) and override values for received_domain and/or id_domain (bit 5 set).

Additionally, the authrewrite channel option should be set to 16 on all appropriate source channels (or on the defaults channel if the mapping is to be applied to all message flows). The header where the virtual MTA (first conversion tag) is located may be specified as the first value of the authrewrite_extra_headers MTA option. The header field containing the tenant and sender OCIDs may be specified as the second value if DKIM processing is enabled. At present the only supported syntax for this header field is a JSON string of the form:


{"tenantId":"<tenant-OCID>","senderId":"<sender-OCID>","compartmentId":"<compartment-OCID>"}

The tenantId and senderId OCIDs must be present. The compartmentId is optional; if it is not supplied its value defaults to that of the tenantId. Note that the parameters may appear in any order. Any additional parameters will be silently ignored.

Alternately, bit 13, value 8192 of the include_conversiontag MTA option, can be set to cause conversion tag information to be included in the probe if it is available. The conversion tag list is interpreted as follows:


<virtual-MTA>,<job-id>,<sender-OCID>,<tenant-OCID>,<compartment-OCID>

The compartment-OCID will be omitted if it has the same value as the tenant-OCID, which is the case when it specifies the root compartment for the tenancy.

send_access callout

The send_access callout is intended to be called from either the SEND_ACCESS or ORIG_SEND_ACCESS mapping. It provides the ability to confirm that a conversion tag transferred using the XCONVTAG extension has a corresponding IP list or MTA entry. This functionality is useful in preventing messages from being routed to the wrong MTA during configuration updates. The callout succeeds if the confirmation attempt fails, so that the mapping template itself determines the appropriate action to be taken.

Note that this callout is only needed in specific circumstances and therefore is not configured by the smartsend.rcp recipe.

The SEND_ACCESS mapping should be set up approximately as follows:


SEND_ACCESS
 
   *     $C$[IMTA_LIB:smartsend.so,send_access,<pflags>|$0]$E$N$X4.7.0|Unknown$ tag


Note that the template in this example causes a temporary failure to be returned when confirmation fails.

Here <pflags> should be replaced with an unsigned integer flag value. The lowest three bits enable increasing levels of debug output, with "0" resulting in no output. Bit 3 (value 8), controls the type of lookup that is performed. A lookup for "ips_" entries is performed if the bit is clear; a lookup for "mta_" entries is performed if the bit is set. (The latter is appropriate for an intermediate routing MTA.) Finally, bit 4 (value 16) should be set if this callout is placed in the ORIG_SEND_ACCESS mapping (as opposed to its normal location in the SEND_ACCESS mapping).

Additionally, bit 4, value 16, of the include_conversiontag MTA option should also be set so that the conversion tag is included in the SEND_ACCESS mapping probe. Bit 3, value 8 should be set if the callout is done from the ORIG_SEND_ACCESS mapping.

forward callout

The forward callout is used in conjunction with the mx_access callout (described below) to implement a general purpose MX rollup facility that's superior to what was provided by the dns_get_first_mx mapping callout. With this approach rollups are specified in the general database, which in turn can be linked to either memcache or Redis.

The first step in configuring this functionality is to select a domain suffice to identify all of the rollups that will be defined. ".rollup" is used in other products for this purpose, but any domain can be used as long as it isn't a valid top-level domain (TLD) or internal domain.

Once the domain has been selected a rewrite rule needs to be added to route rollups to the proper tcp_ that will process rolled up messages. Assuming that ".rollup" is the rollup domain suffix, the tcp_local channel is to be used, and the channel has the usual official host name of "tcp_local-daemon", the rule would be:


.rollup                   $U%$H$D@tcp_local-daemon

The forward callout should be now added to the FORWARD mapping table as follows:


FORWARD 
 
   *|*.rollup $E
   *          $C$[IMTA_LIB:smartsend.so,forward,<pflags>|$0]$E

Of course ".rollup" should be replaced with whatever domain is to be used to identify rollups.

The mx_access callout must also be set up, as described below, and the entries need to be added to the general database to create specific MX rollups.

The forward callout performs the following actions:

  • Perform an MX record lookup on the routing domain part of the envelope recipient address. No action is taken if the lookup fails or no records are found.
  • Construct a list of MX records with the lowest precedence (highest priority).
  • For each MX host on the list, construct a probe of the form rolluphost_<mxhost> and look it up in the general database. The MX host name is forced to lower case as part of constructing the probe. The values of these entries are expected to either be of the form:

    
    mxhost1,mxhost2,mxhost3,...,mxhostN;rollupname
    
    

    or:

    
    [mxhost-ip-1],[mxhost-ip-2],[mxhost-ip-3],[mxhost-ip-N];rollupname
    
    

    Here "mxhost1...mxhostN" are the names of all the MX hosts included in the rollup, mxhost-ip-1...mxhost-ip-N are the IP addresses of all the MX hosts included in the rollup, and "rollupname" is the domain name associated with the rollup. In the case of a host name list, the list should be exactly the same as the list specified in the corresponding "rollupmx_" entry used by the MX_ACCESS mapping, and he domain name must end with the rollup domain suffix.

  • If the initial lookup for a given host fails it is repeated with the first element of the host name replaced by a "*". If this fails a lookup is attempted with the first two elements replaced with a "*".
  • If a match is found the corresponding entry value is checked to make sure all of the MXes it specified are also on the MX list for the destination domain. If they are a result is returned that adds the rollup name to the address as a source route. If not the process continues with the next host.

Setting up an MX rollup

A rollup consists of two sets of general database entries, one set to match the various MX hosts to the rollup and another to specify the MX hosts for the rollup. This sounds more complex than it actually is because the MX hosts in the first set have to match the set in the second.

For example, let's suppose that a large service provider, example.net, is known to provide service for many different domain, but all of the domains share the MX hosts mx1.example.net, mx2.example.net, and mx3.example.net. An appropriate set of database entries for this service provider would be:


rolluphost_mx1.example.net   mx1.example.net,mx2.example.net,mx3.example.net;example.net.rollup
rolluphost_mx2.example.net   mx1.example.net,mx2.example.net,mx3.example.net;example.net.rollup
rolluphost_mx3.example.net   mx1.example.net,mx2.example.net,mx3.example.net;example.net.rollup

rollupmx_example.net.rollup  mx1.example.net,mx2.example.net,mx3.example.net

If this seems highly duplicative, that's because it is: Denormalization is required to turn a bidirectional mapping into a set of entries in a name/value store.

In this case a wildcard could be used to lower the number of entries:


rolluphost_*.example.net     mx1.example.net,mx2.example.net,mx3.example.net;example.net.rollup

rollupmx_example.net.rollup  mx1.example.net,mx2.example.net,mx3.example.net

When wildcards are used it may be necessary to specify the matching list as a serious of IP addresses known to be associated with the service instead of a series of host names:


rolluphost_*.example.net     [ip1],[ip2],[ip3];example.net.rollup

rollupmx_example.net.rollup  mx1.example.net,mx2.example.net,mx3.example.net

mx_access callout

The mx_access callout is used in conjunction with the forward callout (described above) to implement a general purpose MX rollup facility.

Once the steps for setting up the forward callout have been done, the mx_access callout should be added to the MX_ACCESS mapping table as follows:


MX_ACCESS
 
   *|*.rollup     $C$[IMTA_LIB:smartsend.so,mx_access,<pflags>|$0.rollup]$E

The ".rollup" that appears on both sides of the mapping should be replaced with whatever domain is to be used to identify rollups.

The mx_access callout constructs a probe of the form rollupmx_<domaint>, where "domain" is the destination domain, and looks it up in the general database. If a match is found the value of the entry is expected to be of the form:


mxhost1,mxhost2,mxhost3,mxhostN

This value is then returned as the result, which then becomes the MX list for the domain.

log_action callout

The log_action callout is intended to be called from either the LOG_ACTION mapping. It provides the ability to store per-destinion-domain counter information using Redis sorted sets. This provides the ability to determine things like the most active domains, the domain generating the most temporary failures, and so on. Currently the callout always fails, so it can be safely inserted before any other actions in the mapping.

Note that this callout is only needed in specific circumstances and therefore is not configured by the smartsend.rcp recipe.

Counters are stored under keys with names of the form counter_<mtaid>_<operation> and counters_<conversiontag>_<operation>, where <mtaid> is the MTA id and <conversiontag> is the first conversion tag associated with the current message, both converted to lower case. Operation is determined from the logging action as shown in the following table:

Redis counter operation names
Operation Action
delivered D
enqueued E
rejected J
attempted Q
failed R

The LOG_ACTION mapping should be set up as follows


LOG_ACTION
 
   *     $C$[IMTA_LIB:smartsend.so,log_action,<pflags>|$0]


Here <pflags> should be replaced with an unsigned integer flag value. The lowest three bits enable increasing levels of debug output, with "0" resulting in no output. Bit 3 (value 8), if set, disables per-MTA counter entries. Bit 4 (value 16), if set, disables per-conversion-tag entries.

Additionally, bit 1, value 2, of the ilog_conversion_tag MTA option should also be set so that the conversion tag is included in the LOG_ACTION mapping probe. Bit 2, value 4 of the ilog_intermediate MTA option should also be set to make intermediate address information available to the mapping; this is necessary to get accurate domain information when delivering to the message store.


See also: