Management of Mail Servers against incoming Spam and avoid outgoing emails are regarded as SPAM.

Managing your own email server is not an easy task.  It involves continue monitoring of all inbound and  outbound email to ensure outbound email can be delivered to recipient's mailboxes, while there will be no excess SPAM email (or filtered out legitmate email) to the mailbox of internal users.

Here's some tips that I have used to improve the reliability and accuracy of our inbound and outbound email gateway (MTA):

For outbound email, the objective is: avoid outbound email being tagged as SPAM.  Therefore, we need to provide more information for MTA to distinguish genuine email and fake email (SPAM).

1. Enable sending of Non-delivery report (NDR) with cautions.  
   If your SMTP server is being used to send SPAM by NDR, your mail server reputation will be decreased, or even blacklisted.
2. Realtime Blacklist Checking
   Check your server's reputation regularly. Make sure your mail server is not listed as on these blacklist.  Online blacklist checking here.
3. Open Relay
   Ensure your SMTP gateway is not an open relay.  Open relay means your mail server accept outside sender to send email to email address other than your own domain.  Online open relay checking tools - mxtoolbox.
4. Implement Domain Key Identified Mail (DKIM).  
   It involves generate a public and private key pair to "sign" the email.  The public key will be published to Internet via DNS TXT record.  The private key will be stored at email gateway server.  All outbound email will be "signed".  If the recipient server supports DKIM, it will check the the "digital signature" to verify the Integrity and Authenticity.
   
   DKIM DNS record generation tool here
   DKIM testing tool 1
   DKIM testing tool 2
   Automatic DKIM and SPF testing: send an email to  checkmyauth@auth.returnpath.net
5. Implement Security Policy Framework (SPF)
   SPF involves the domain owner published "authorized" SMTP server IP address on the Internet by DNS TXT record.  The recipient SMTP server (if support SPF) will lookup the SPF record on DNS server and check if the source IP matches those listed.  You can check the SPF record of any domain using mxtoolbox.
6. RDNS
   Contact your ISP to add the Reverse Lookup (RDNS) address for the IP address of the mail server.  Good MTA will check the RDNS result of the source IP address.  If there is a valid RDNS record, this will reduce the "likelihood" of being tagged as SPAM.

For inbound email, the objective is:

• Filter out SPAM email (false negative)
• Prevent legitimate email tagged/filtered out as SPAM (false positive)

1. Avoid direct exposure of email server to internet.  Implement SMTP gateways with SPAM filtering features.  Commercial option includes GFI, McAfee, etc.
2. For SMTP Gateways, turn on security checking features.
1. Directory Harvesting - LDAP lookup to verify the validity of the recipient.  If the recipient is not valid, disconnect the SMTP connection and avoid sending Non Delivery Report (NDR).  You may setup your email gateway to LDAP lookup the user name from Domain Controllers.
2. Phising - check the email content for phising URL.  (e.g. fake Citibank or UPS website inviitation).
3. Anti-virus Scan
4. Email blacklist - should be able to support blacklist on a domain based on individual email base.  Should be precaution between SMTP to/from or MIME to/from information.
5. DNS blacklist - There are various online DNS black list, such as spamhaus.org, backscatterer.org, sorbs.net.  These provider provides free realtime Blacklist, based on real time report from the Internet.  Some of these list will also list Dial-Up IP addresss which should not be used to host email servers.  The good side is the blacklist is updated rather frequent.  The down side is these list often mis-blacklist useful IP address, particular IP address of some Hotmail and Yahoo server.  Here's my favourite server list:
• zen.spamhaus.org
• dul.dnsbl.sorbs.net
• smtp.dnsbl.sorbs.net
• http.dnsbl.sorbs.net
• socks.dnsbl.sorbs.net
• new.spam.dnsbl.sorbs.net
• nomail.rhsbl.sorbs.net
6. Verify sender address domain - some SPAM email forge fake domain which does not exist at all.  Standard software package allows checking on the validity of the sender email domain.
7. SPF and DKIM checking - enable these option to perform SPF and DKIM checking on inbound email.
8. Languages - If you think your business will not receive any email in Russian or Chinese, you may select to filter email in those languages.
9. Maximum number of email recipients - a normal email will not have a large number of email recipient.  You can put a maximum number on it.  Email with too many recipients will be regarded as SPAM.  If your email server serves complaints, sometimes customer may send the complaint to multiple recipient (e.g. Press, Councillor, Government Officials) but the content is still valid.  Use it with caution.
10. Subject and Body keyword checking
11. Malicious attachment - Files of types which are vunlerable to infrected by virus and trojan, but not commonly used by end-users, such asexecutable  batch file or screen saver, shall be banned from being received via Email.
3. Should implement IP whitelist of important sender (e.g. SMTP server from headquarter or important customer).
4. Turn on/off NDR feature with cautions.  NDR features can be used by Spammers to send spam email via your server.  NDR should be disabled for high-confidence filtering rules (e.g. email address blacklist, SPF check) only.
5. Get an agreement on email filtering policy with users.  Get users concensus on handling SPAM tagged email, whether those SPAM tagged emails shall be sent to end-users or dispose my mail server.
6. For all filtered email, it should be reviewed regularly to avoid mis-filtered.  You may regularly review the Quarantine vault.  You may also setup SPAM folder at user side so that the responsibilities of double check remains at end-user side.
7. MX records - make sure you have correct MX record.

(updated on 2013-05-08 for additional online tools to implement DKIM.)

Other resource:
What is Email Header