[Updated May 14th, 2014]
Today, maturing ESPs and MACs are collaborating closely with their peers to improve and further automate the processes of “onboarding” digital marketers, filtering outbound mail for spam, detecting potentially breached data points, and others related to mitigating abuse. With new technologies being introduced to our industry daily, one important process is vetting new customers during the onboarding phase. Unfortunately, mitigation abuse tools are generally considered an afterthought or even a distraction. I’m looking forward to more discussion on this issue at the next M3AAWG events in Boston late this year and San Francisco in early 2015.
Three years ago, MailChimp introduced their Freemium plan because they felt their abuse mitigation and onboarding processes were technologically sufficient. Many, many other ESPs followed suit creating a windfall of VC funding. Now the strategy of building a client user base and giving away your product or service has become commonplace among companies that intend to scale rapidly. [Enter the age of #GrowthHackers] At that time, their genetic algorithm for Omnivore was built to detect abuse and protect customers while also preserving the overall health of MailChimp’s delivery platform. While there is no silver bullet when it comes to mitigating abuse, the engineers at MailChimp quickly found that additional data and technologies significantly strengthened Omnivore.
There is no shortage of data services, tools and organizations specializing in abuse mitigation. The combination of data/services from SURBL, Spamhaus, Cloudmark and other vendors are used by Email Service Providers to combine various admin and anti-abuse systems. Generally, ESPs use personal information provided during signup to vet the validity of marketers and/or customers’ sends; some ESPs have stringent requirements and many ESPs are not automated at all. Data services like SURBL and Spamhaus can be utilized to tell if a user’s domain or URLs in the content are from blacklisted sources.
These services currently serve as a stellar litmus test for the quality and/or validity of a potential client or send. However, they do not tell the entire story, especially if the domains are being used for the first time. It is possible the sender is referencing URLs in their content that might not belong to the customer. Layered protection that includes multiple datapoints is safer than blocking on a single piece of fragmented data.
As the MailChimp user base grew and as more tools were introduced, the MailChimp engineers saw the need for additional data to make more intelligent decisions. External data services were added into Omnivore to increase accuracy. When it comes to onboarding and sending, domains and URLs are a great way to determine the user’s primary intentions. MailChimp quickly found a partner in Dissect Cyber. Dissect Cyber’s innovative toolset and data services can be used to inspect further the age, registrar, nameservers and other detailed information of a domain or URL. (Dissect Cyber offers multiple methods of pre-emptive detection that can identify certain types of malicious domains the “first time” they are used in outbound mail.)
All of this data is fed back into Omnivore, which builds a detailed fingerprint of the potential user and/or their content. According to MailChimp engineers, an abuse mitigation system like Omnivore provides many ways to detect good users and improve the quality of legitimate users’ experience. One thing that quickly became apparent is that users make frequent mistakes in adding invalid URLs to their content. Dissect Cyber’s “IsNu” data service can not only be used to determine the age of a domain, but can also be used to detect invalid domains accidentally used by customers in campaign target URLs.
The IsNu query service warns you when absolutely nothing is known about a given domain. This condition is very rare for a legitimate domain and it should be considered highly suspicious. Data such as registrar, nameserver, and other various pieces of DNS information provided by Dissect Cyber “D8s” and “RPZone” services allowed the MailChimp engineers to stack tools for internal products, scan all outbound mail, and improve the onboarding and human review processes.
MailChimp ultimately makes heavy use of data provided by Dissect Cyber to decide when to automate the onboarding of customers, when to interrupt an account from sending, and when to get humans involved for further vetting.
Dissect Cyber also offers tools for humans to research further on domains and user information. A tool that is currently under development by Dissect Cyber will give researchers the ability to quickly stitch together all the domains owned by a single individual/organization. As all of this data is fed into the various systems, MailChimp is working with Dissect Cyber and other vendors to extend the toolset and data services, so that detection of abuse happens faster and overall quality improves.
As ESPs continue to mature, restructuring of the customer onboarding process is vital. Applications like Dissect Cyber tools offer a wealth of information to raise the barriers of entry and become a legitimate application housed in your ESP model.
This blog post was inspired by industry colleagues April Lorenzen and Brandon Fouts.