@radekmieBy Radosław Miernik · Published on · Updated on · Comment on Hacker News, Reddit
How is your week going? I just finished wrapping up an internal postmortem on how we lost 29 hours’ worth of emails (tens of thousands). It wasn’t the first (but hopefully the last) time a 3rd party wreaked such havoc on our product, but this time I decided to share it. It’s anonymized, but I hope you understand.
Below you can find our story, including (almost all) the emails we exchanged with the Mailchimp teams, as well as some comments on what we did wrong and what we could have done better. To some extent, I wanted to share it so the next company facing it would know what to do.
If you are in such a position, jump straight to What should I do!?
Update 14 March 2026: Case closed, it were fake emails. Some text below got redacted, some more got added. In any case, it still stings.
Our email provider stopped delivering our emails and locked us out of the account. Then, they didn’t reply for over 16 hours (or more, depending on what you consider a reply), and when they did, they suggested that our API key was leaked. When asked about why they think so, they mentioned that we sent two “fake shipping messages”… And yet, they cannot tell us which ones are these.
Overall, we exchanged roughly 30 emails, called them 6 times, and tried to get help from their automated chatbots… It still took us over a day to resolve the underlying issue. Well, technically it’s still unresolved, as we’re yet to hear back from them why some messages were considered “fake”. See the update below.
We’re now in a process of investigating a backup email provider1, organizing ourselves better to communicate such outages faster, and a kill switch that’d allow us to reduce the number of lost emails.
13:10 We received an email from the Mailchimp Compliance Team2, that our account “has been temporarily suspended due to mailchimp3 account review”. Based on the looks of it, a fully automated message.
13:50 The above email was finally opened (lunch time, right?) and shared internally for further investigation. Most importantly, our API keys were still working (API happily accepted our emails), but nothing was delivered.
Funnily enough, no one from our team could log in, so we couldn’t follow the instructions mentioned in the email. However, the error message said maintenance, not suspension. At this point, we assumed it was some sort of an error on their end, so we didn’t panic.
14:06 We reached out to both the Mailchimp Compliance Team and the Mailchimp Account Recovery. Both email addresses we found online2, as the initial message was unclear about how we should contact them.
21:59 After contacting the support via another account’s in-app chat, on-site chat, and phone, we decided to send another email to the Mailchimp Compliance Team to the same email that mailed us first.
The weirdest part was that the support told us that they cannot help us, as the Compliance Team is somehow special, and they cannot contact them about our case. Now, when I look at it, even their email address suggests that they’re a different part of the organization.
02:14 (+1) We got our first “not necessarily automated” reply… Well, sort of. On the one hand, it seems automated, but on the other, all the previous emails were replied to with something much sooner, not 4 hours later.
14:31 (+1) We got another reply. While it still looks like a standard reply, there’s finally some name (censored) we can address. Most importantly for us, they said that “it appears that an unauthorized individual has gained access to our account” and implied that our API key has leaked.
At this point, emails were not delivered for over 24 hours, so we were actively working on plan B. To be precise, we were working on a fallback to SMS communication for all of our customers (for free, of course).
14:44 (+1) We replied that we cannot do any of the suggested steps, as we still couldn’t log in. In the meantime, we double-checked all places where the API key has been used to make sure it wasn’t mishandled at any point.
14:59 (+1) Another reply, we could finally log in.
15:23 (+1) While we were still working on rolling the API keys, we decided to ask them about the implied “unauthorized activity”. As a Swiss company handling thousands of restaurant reservations each day, we must take such words seriously. Like, we already contacted our lawyers seriously.
15:31 (+1) Non-production API keys got rolled. It took longer than I’d like to – partly because Mandrill’s4 UI is not the fastest, but also because it’s not trivial on our end. That’s already noted, and we will work on that.
15:53 (+1) Production API key got rolled.
16:40 (+1) Over an hour later (!), we got the reply we all waited for. They confirmed that our “compromised API key” has been used to send some “fake shipping messages” with the subject “Reservierungsbestätigung”…
17:37 (+1) …which is literally “reservation confirmation” in German. Oh, and by the way, we send out a few tens of thousands of such emails every day, as we operate mostly in the DACH region.
18:20 (+1) After a small back and forth to confirm whether we absolutely must drop all queued emails. As it turned out, there was no way around it, so we went ahead and did that.
Yes, it took another half an hour, but we really didn’t want to lose these emails. Some of them may have already been outdated, but we couldn’t easily resend the still-relevant ones. And as the first email mentioned the Rule Engine, we hoped that at least some emails could have been released.
18:24 (+1) First emails got delivered, yay! Minutes later, we informed our customers what messages were lost, and how they can resend them (we have such an option in the app).
20:30 (+1) We once again asked for more details about the suggested unauthorized access. Yes, it is a copy-paste of the previous email with additional questions… But what else should we do if they didn’t reply?
08:46 (+5) This blog post was published and posted on r/MailChimp.
13:19 (+7) One of the moderators left a comment that I should reach out to them with our case numbers. We exchanged a few messages.
06:45 (+11) We finally got an answer on what exactly happened! Yes, it turns out that one of our users’ was hacked (most likely a password leak), and their account was used to send out exactly seven spam messages. The access got revoked ~15 minutes after we got that email.
Listen, I have a theory… Some automated spam filter triggered on one of our outbound emails. I don’t know why exactly, but let’s say one of our customers configured something weird in their email templates – it doesn’t matter.
The first thing that went wrong was that it somehow blocked us from logging in. Note that there was no notification, no email, no warning – just an instant ban.
Then, when we asked for details on which email it was, the second thing went wrong… Someone checked it manually and noticed that it was a mistake, so… They stopped replying.
I know that at our scale, we’re basically a rounding error in Mailchimp’s books, but it left a really bad impression. To some extent, I’d still recommend their services – we’ve been using Mandrill for over 11 years now, and except for occasional outages (it happens to everyone), it has been good.
First of all, I’m sorry for you. If you’d like to share your story, please do send me an email. Now, full focus on resolving the issue. Good luck!
Such things happen. In today’s era of more and more autonomous content classification virtually everywhere, such false positives will happen. Of course, there’s still a chance that some of our emails were really “fake”, but given our track record, and the fact that so far the app is working, it’s rather unlikely. Yes, it turned out that it was a fake message, but the way MailChimp handled it was simply terrible. Since we reached out via Reddit the communication got much better, though. I guess the takeaway is… Don’t hesitate to write posts like this?
So… Any email provider recommendations?
A problem with having an alternative provider is that our customers would need to set up DKIM and DMARC for both. Of course, it only affects the ones who’d like to use a custom domain, but still.
Below, I gathered all emails to the relevant people, in case you need them. Keep in mind that while we were told to contact all of them (depending on who we asked), only the first one really helped us to solve the issue.
As the name of the service was misspelled (lowercased), it felt like a scam at first. If anyone from Mailchimp or Intuit is reading it: please, fix this template.
Yes, we’re using Mandrill, not Mailchimp. Technically speaking, it’s Mailchimp Transactional Email now, but I like the pre-merge name more.