Sieve external lists

RFC 6134 (Sieve Extension: Externally Stored Lists)defines a Sieve extension for external lists, EXTLISTS,  which is intended for cases where a Sieve script would like to consult an externally-stored list: this might be data in an LDAP directory, a flat file  containing a list of something, a database, a personal addressbook, etc. In order to use a Sieve external list, a  Sieve script must, per Sieve syntax, declare that it will use this  extension via require "extlists"; The MTA&#x27;s implementation of EXTLISTS operates as follows. Sieve scripts may use the " " argument in a  " " action or in tests such as  " ", " ",  " ", " ",  " ", or  " ", etc., or in the test  component of the " " or  " " actions, to indicate a wish to  access an "external list". The MTA then makes use of the  mapping table to determine  the meaning and contents of the referenced external list.

When an external list is referenced in a Sieve script, via a " " argument, e.g., if address :list" "from" "pab" { keep; } or redirect :list "friends"; the MTA uses the list name (as well as other data) to construct a probe  into its   mapping table. In the case of tests, the probe  syntax is:  test-name&#x7c;sieve-owner&#x7c;spare_4-value&#x7c;spare_5-value&#x7c;spare_6-value&#x7c;list-name&#x7c;argument-value Here   can be any of  ,  ,  ,   ,  ,  ,   ,  ,   ,  ,   ,  .   The    is normally the (canonical address of  the) "owner" of the Sieve---so the user&#x27;s address for  user-level Sieves, or normally the  postmaster address for system-level  Sieves; or the field can be blank if the Sieve has no owner address.  The  ,   , and    fields contain the values of those  LDAP attributes named by the MTA&#x27;s   ,   , and    MTA options associated with the  current envelope recipient; these   options have no  defaults, so by default no values appear here. Furthermore, even if   MTA options are defined, the    fields will be blank if the current recipient address was not obtained  from an LDAP entry. (And prior to Messaging Server 7.2-0.01, the    fields  would be blank for system Sieve probes; as of   Messaging Server 7.2-0.01, the current  recipient address&#x27;s LDAP attribute values are used even for system  Sieve probes.) The intention is that such   MTA  options (and  corresponding attributes) may be defined and used to store, or a  per-user basis, information about how to construct appropriate access  URLs for different "types" of external lists: for instance,  one of these attributes might store a value which is a sort of template  for constructing lookups of lists in that user&#x27;s personal addressbook,  while another of these attributes might store a value which is a sort  of template for constructing lookups of lists in that user&#x27;s calendar  (so a CardDAV lookup template). The   is the argument to the " " parameter; for  instance, in the examples above " " and  " ", respectively. (In fact, consulting multiple external lists is supported, in which case the multiple list  names are present in the probe, separated with the vertical bar  character.) Finally, the   is  whatever string from the message corresponds to the Sieve test.

For " " actions, the syntax is: redirect&#x7c;sieve-owner&#x7c;spare_4-value&#x7c;spare_5-value&#x7c;spare_6-value&#x7c;list-name where note that there is no final  . (The Sieve external lists draft has another, alternate construct for " ", where the argument assumed  above to be a list name is instead a URL to which to do the  redirection. While the MTA is capable of supporting such usage,  enabling it by matching and returning the argument blindly would be  extremely dangerous and is not recommended. Recommended  configuration and usage is instead to have the   mapping  table: (1) expect and match only when a list name is the argument, and  (2) construct and return a sensible URL based upon that list name as  well as the other fields of the probe.)

The  flag will be set if the Sieve is a system Sieve, and  may be tested in the template using the usual   (flag  set) or   (flag clear) flag test syntax.

The mapping template must set the  flag if the test succeeds and $N if the test fails. $F may be set to report an error; in this case the mapping result will be used as the error message.

Failure to set $Y, $N, or $F will cause an "unknown list" error to be signaled.

Note that $Y takes precedence over $N, so a single mapping template can set $N before performing a lookup, $Y after, and both the success and failure case will be handled.

In the case of a successful  action, the template  should return a URL pointing to the desired list of addresses.

In addition, if a Sieve test or " " making use of a Sieve external list employs any of the    fields, then the mapping template must  also set the   flag. tells the Sieve machinery that the test is recipient-specific and the script must be  reevaluated for each recipient. Failure to set  can lead  to botched test results for multiple recipient messages.

An MTA-specific extension to the Sieve external lists draft is that the MTA will also potentially return properties associated with list  entries. If Sieve variables are enabled, then the use of " " with a test sets variables in a fashion similar to  " ": that is,   is set to  whatever was found on the list,   is set to the first  property value associated with the list entry,   is the  second property value, etc. When variables are enabled, the  result of the mapping consists of properties separated by vertical bars.

So for instance, suppose a site has the    MTA option defined naming an LDAP  attribute in which the users store the leading portion of the LDAP URL  for where each user&#x27;s own, personal, PAB information is stored; for  instance, suppose that in the user attribute   is  each user&#x27;s PAB DN location information in a form of: ldap:///user&#x27;s-pab-dn Then suppose further that the site wants users to be able to access their personal PAB information for   use in their  personal Sieve filters. In particular, the site wants the list named " ", if used in an ,  , or   test, to mean "any address found in the  user&#x27;s own PAB", and for any other, more specifically named list   , to mean the contact entries in a  user&#x27;s own PAB that are in the   PAB  group; that is, the contact entries marked with  "  " where the    was defined in a group entry in the  user&#x27;s PAB (an entry with   and with     and    ). Also suppose that the site uses the   attribute to store the address most  suitable for using to send to users. Then the site might want an   setting of where in legacy configuration mode, the setting would be made in the MTA option file as  LDAP_SPARE_4=psroot And the might would also want a   mapping table including entries such as: SIEVE_EXTLISTS ! Check for the special case of testing whether an &#x27;address&#x27; is merely ! present in the user&#x27;s PAB somewhere -- the test of external list "pab": ! ! test-name&#x7c;sieve-owner&#x7c;spare_4&#x7c;spare_5&#x7c;spare_6&#x7c;pab&#x7c;address-from-message !   address&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;pab&#x7c;&#x2a;   \ $N$&#x5d;pab$1?piEmail1?sub?(&#x7c;(piEmail1=$=$4$_)(piEmail2=$=$4$_)(piEmail3=$=$4$_))&#x5b;$Y envelope&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;pab&#x7c;&#x2a; \ $N$&#x5d;pab$1?piEmail1?sub?(&#x7c;(piEmail1=$=$4$_)(piEmail2=$=$4$_)(piEmail3=$=$4$_))&#x5b;$Y header&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;pab&#x7c;&#x2a;   \ $N$&#x5d;pab$1?piEmail1?sub?(&#x7c;(piEmail1=$4)(piEmail2=$4)(piEmail3=$4))&#x5b;$Y ! ! Note that no entry to allow redirect to the pseudo-list "pab" is included ! above: this is intentional as making it "too easy" for users to resend to ! all their contacts seems unwise. Instead, redirect is enabled below merely ! for specifically named and defined PAB groups. ! ! Now check for named groups (named external lists); that is, for a named ! group, find the piEntryID for that group. ! ! test-name&#x7c;sieve-owner&#x7c;spare_4&#x7c;spare_5&#x7c;spare_6&#x7c;group&#x7c;address-from-message !   address&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;   \ $CGROUP&#x7c;address&#x7c;$2&#x7c;$5&#x7c;$&#x5d;pab$1?piEntryID?sub?(&(objectClass=piTypeGroup)(displayName=$4))&#x5b; envelope&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;  \ $CGROUP&#x7c;envelope&#x7c;$2&#x7c;$5&#x7c;$&#x5d;pab$1?piEntryID?sub?(&(objectClass=piTypeGroup)(displayName=$4))&#x5b; header&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;  \ $CGROUP&#x7c;header&#x7c;$2&#x7c;$5&#x7c;$&#x5d;pab$1?piEntryID?sub?(&(objectClass=piTypeGroup)(displayName=$4))&#x5b; redirect&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;  \ $CGROUP&#x7c;redirect&#x7c;$2&#x7c;$&#x5d;pab$1?piEntryID?sub?(&(objectClass=piTypeGroup)(displayName=$4))&#x5b; ! ! Now find contact entries that have the correct piEntryID, probing with ! ! GROUP&#x7c;test-name&#x7c;spare_4&#x7c;address-from-message&#x7c;piEntryID !   GROUP&#x7c;address&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;%&#x2a;    \ $N$&#x5d;pab$0?piEmail1?sub?(&(memberOfPiGroup=$2$3)(&#x7c;(piEmail1=$=$1$_)(piEmail2=$=$1$_)(piEmail3=$=$1$_)))&#x5b;$Y GROUP&#x7c;envelope&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;%&#x2a;   \ $N$&#x5d;pab$0?piEmail1?sub?(&(memberOfPiGroup=$2$3)(&#x7c;(piEmail1=$=$1$_)(piEmail2=$=$1$_)(piEmail3=$=$1$_)))&#x5b;$Y GROUP&#x7c;header&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;%&#x2a;   \ $N$&#x5d;pab$0?piEmail1?sub?(&(memberOfPiGroup=$2$3)(&#x7c;(piEmail1=$=$1$_)(piEmail2=$=$1$_)(piEmail3=$=$1$_)))&#x5b;$Y ! ! Note redirect case needs to return PAB URL itself, rather than lookup result. ! It is assumed that for all lists, the most appropriate user addresses to use ! for redirection are stored in the users&#x27; piEmail1 attribute. !   GROUP&#x7c;redirect&#x7c;&#x2a;&#x7c;%&#x2a;   pab$0?piEmail1?sub?(memberOfPiGroup=$1$2)$Y Note here how the user&#x27;s initial portion of an LDAP URL value, stored in the attribute named by , is converted by the    mapping table into the MTA&#x27;s    URL format  (the format that tells the MTA that this is an LDAP URL for  querying the PAB directory---the LDAP directory that the MTA locates  via its    options). For the special case of an " " or " "  test against the so-called list named merely  " ", the user&#x27;s own entire PAB is searched,  looking for entries where the test argument matches the value of any of  the attributes ,  , or    in a PAB entry. For other named lists/groups, the list/group name is used to attempt to locate a corresponding entry  (with   of   and a    matching the specified group/list name), and  when found, retrieve the   value that indicates  that group/list. Then a second lookup is performed, to find those contact entries that have a   value for the  group/list in question. In the case of an " " or " "  test against named lists, the user&#x27;s PAB is searched looking for  entries where both the list ID is present in the entry in a    attribute, and the entry&#x27;s    value matches the test argument. Just in case a user configures a " " test for address  matching purposes (though note that this is poor user practice, as  " " or " "  tests are more appropriate for such purposes),  " " tests are set up similarly, though  including matching against alias values (  and    in addressbook attributes, analogous to the MTa&#x27;s  usual   and    attributes) as well as against the  canonical   (analogous to MTA&#x27;s    attribute) value. And in the case of a " " action, the mapping returns not the  actual addresses, but rather a " " URL  specifying where to find the appropriate addresses to which to  redirect; this URL will be the new address (the address to which to  redirect) enqueued to the reprocess channel, which in turn will perform  the actual list expansion (expand that URL into the addresses it  specifies). Note that the    MTA option  limits how many addresses will actually be  used from such a list; additional addresses will be ignored.
 * 1) msconfig set ldap_spare_4 psroot

With Sieve access to user personal PAB&#x27;s set up as above, then users can make use of the addresses in their PAB in various ways in their  Sieve scripts.

As a simple example, suppose that a system-level Sieve would do a discard; on a message for whatever reason: perhaps because a spam/virus filter package callout returned a verdict suggesting that a message is spam;  perhaps due to the source-IP being suspect; whatever. But if a user wishes to keep any message purporting to come from one of their known  correspondents, then that user might use a personal Sieve filter with  explicit  " and " "  actions to override the system-level " ",  e.g.,  require &#x5b;"envelope","extlists"&#x5d;;  ...other-actions-such-as-list-fileintos...  if envelope :list "from" "pab" { keep; } Or for another example, suppose that a system-level Sieve filter  (perhaps configured as part of a spam/virus filter package callout so  configured via a        MTA option value) has a Sieve effect of:  require &#x5b;"spamtest","relational"&#x5d;;  if spamtest :value "ge" "200" { discard; } Then a user might use a Sieve test and action such as:  require &#x5b;"envelope","extlists"&#x5d;;  if envelope :list "from" "pab" { spamadjust "-10000"; } In this example above, the user adjusts the spamtest value downward  (drastically) for any sender address found in the user&#x27;s own PAB. With such a drastically lowered spamtest value, the message will likely be  safe from any system-level Sieve spamtest that might otherwise choose  to discard or reject the message. (And the user&#x27;s Sieve can continue on to do any further  " or other operations that  may seem desirable to the user. Leaving open the potential for the  user&#x27;s own Sieve to perform further actions, such as  " " or " "  or " ", is a reason why a user might  prefer to do a " " rather than an  explicit " " for known senders.)

Or the user&#x27;s Sieve might do more subtle adjusting and testing: require &#x5b;"envelope","extlists","fileinto","spamtest","relational"&#x5d;; /&#x2a; First, lower the spam score for senders in my PAB  &#x2a;/ if envelope :list "from" "pab" { spamadjust "-100"; } /&#x2a; But better check open-list postings -- that list gets postings with forged From addresses. If the spam score if over 200 even after any adjustment for being sent by a recognized contact, discard the message. &#x2a;/ if allof (header :contains &#x5b;"To","Cc","Bcc"&#x5d; "open-list@domain.com",           spamtest :value "ge" "200") { discard; stop; }    /&#x2a; File messages to or from my buddies in my "softball" PAB list or with softball in the subject, to my softball folder &#x2a;/ if anyof (envelope :list &#x5b;"from","to"&#x5d; "softball",           header :contains "Subject" "softball") { fileinto "softball"; } /&#x2a; Discard mildly spammy messages (mild after adjustment for known       senders, above) that don&#x27;t show me on a recipient header line &#x2a;/ if allof (not header :contains             &#x5b;"To","Cc","Bcc","Resent-to","Resent-Cc","Resent-Bcc"&#x5d;              "my-own-address",            spamtest :value "ge" "50") { discard; stop; }     /&#x2a; Keep messages from known (in my PAB) correspondents &#x2a;/ if envelope :list "from" "pab" { keep; }  Comparing date with list of holidays

For another example, suppose that a site keeps a list of site-wide holiday days stored in the MTA&#x27;s general database, in general database  entries of the form: HOLIDAY&#x7c;yyyy-mm-dd      Yes (where for purposes of this example, it is the mere existence of an entry that matters, not the details of what its right hand side  translation value---here Yes---may be), and that the site wishes users  to be able to perform  " and  " " tests against that list using the special  list named " ". Then the site might use:  SIEVE_EXTLISTS    currentdate&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;holiday&#x7c;&#x2a;   $N$C${HOLIDAY&#x7c;$4}$Y$E    date&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;holiday&#x7c;&#x2a;          $N$C${HOLIDAY&#x7c;$4}$Y$E This would then allow users to use tests such as  require "&#x5b;"date","extlists"&#x5d;; if currentdate :list "date" "holiday" { redirect "mobile-address"; } A more sophisticated use would be to store a label specifying the type of holiday (e.g., " ",  " ", " ",  " ", etc.) as the right hand  side of each general database entry. Then Sieve scripts could check that returned value as a  property (using the  Sieve variables extension  and checking the returned property value via  ) and  perform additional decision making based on the type of holiday.

Sieve external list tests don&#x27;t have to involve checking an actual list. In such cases a string test is typically used since no value actually comes from the message. For example, the mechanism can be used to implement a user attribute which, if present, is inserted into all messages delivered to the user as a header. The first step is to make the attribute, which we&#x27;ll call userAddHeader, available to the external list machinery as a spare attribute: The SIEVE_EXTLISTS mapping would then be: SIEVE_EXTLISTS string&#x7c;&#x2a;&#x7c;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;addheader&#x7c;&#x2a;   $N$&#x2a; string&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;&#x2a;&#x7c;addheader&#x7c;&#x2a;  $Y$1$&#x2a; Note the use of $&#x2a; to make this sieve recipient-specific.
 * 1) msconfig set ldap_spare_4 userAddHeader

And the sieve to perform the header addition would be: require &#x5b;"extlists", "editheader", "variables"&#x5d;; if string :list "addheader" "addheader" { addheader "X-Added-Header" "${1}";} This would probably best be implemented as a destination channel sieve, although any type of sieve would work.

See also:
 * ldap_spare_4 MTA Option
 * ldap_spare_5 MTA Option
 * ldap_spare_6 MTA Option
 * MTA URL types
 * Postmaster addresses
 * LDAP PAB MTA options
 * max_redirect_addresses MTA Option
 * Example Sieve external lists with properties
 * Sieve variables extension
 * Testing Sieve external lists
 * Sieve filters
 * mapping_paranoia MTA Option
 * LDAP external directory lookup MTA options