Trouble in paradise: Our issue with Google mail for domains

Brian Ambrozy 28 Oct, 2009 at 8:20pm ET Article published in Tech

A couple of years ago, we switched mail for our domains to Google instead of hosting it ourselves. Using Google Mail for Domains, we were able to easily manage email for over 25 users scattered all over the world, without having to deal with POP3 servers, and explaining to people how to configure their mail clients. Everyone is familiar with Gmail, and the web interface is second to none.

We switched several of our domains over. The configuration was a bit tricky, as Google provides several alternate servers, and offers different MX priorities for each server.

Back in 2007, when we migrated, the official Google server configuration looked like this:

icrontic.com. MX (10) aspmx.l.google.com.
icrontic.com. MX (20) alt1.aspmx.l.google.com.
icrontic.com. MX (20) alt2.aspmx.l.google.com.
icrontic.com. MX (30) aspmx2.googlemail.com.
icrontic.com. MX (30) aspmx3.googlemail.com.
icrontic.com. MX (30) aspmx4.googlemail.com.
icrontic.com. MX (30) aspmx5.googlemail.com.

And that was that. Then, a few months back, there was a problem…

Several months ago a client said to us: “I’ve been sending emails to you…” and we never got them. We insisted that the problem was on his end, something he was doing wrong. After all, this was Gmail. It just worked. We got emails from everybody, everywhere, every time. No problem, right?

We checked our spam folders, didn’t find anything from him, and foisted the problem off as “his problem.”

A month or so later, it happened again, with a totally different person. It was the same story, “I sent you an email, didn’t you get it?” and the answer was the same: No, we didn’t get it, and no it wasn’t in any spam folder. Again we wrote it off as a configuration issue on his end.

The other day, an internal email sent from one Icrontic staffer to another never made it. After fervently searching in spam folders, we confirmed that the message was indeed sent, but never received.

And then the shit hit the fan: today an email was in the Icrontic inbox: “Not sure if you saw my first email, reaching out to you … Let me know when you are available and we can set up a conversation.” This was from a very important contact that now gets the impression that we have dropped the ball on returning his initial contact.

After a frantic search of the spam folder, we had come to the irrefutable conclusion: some emails were simply never showing up.

That’s when we started digging. A visit to our DNS setup and recommended Google Apps MX settings revealed the problem: Whereas our configuration looked like the one recommended by Google two years ago, the current recommended configuration did not match. Google Apps recommended MX settings look like this:

icrontic.com. MX (0) aspmx.l.google.com.
icrontic.com. MX (5) alt1.aspmx.l.google.com.
icrontic.com. MX (5) alt2.aspmx.l.google.com.
icrontic.com. MX (10) aspmx2.googlemail.com.
icrontic.com. MX (10) aspmx3.googlemail.com.

Notice two key things: The MX priorities are different, and aspmx4 and aspmx5 are not on the list.

Now, various searches reveal that Google has not decommissioned 4 and 5. They are no longer recommending them, but they are still up. However, we are at a loss to explain what happened to our mail, other than to assume the MX priority changes had something to do with it.

Google has not notified us in any way (for any of our domains) that a change was made. Even if the changes were minor or superficial, we should have been notified so we can make sure we’re running the latest recommended MX configuration for our domains. Not only does our mail rely on it, but several of our associates and colleagues have switched to Google Apps on our recommendation. Situations like this undermine our authority and our associates’ sense that we know what we’re doing.

Dear Google: If you change things, please tell us. Love, Icrontic.

Comments

28 Oct 2009 ~ 8:54pm Frank That's definitely weird. The change in priority shouldn't have an effect on mail delivery at all as long as they are still in the same order, which it looks like they are. I've definitely noticed this discrepancy in their documentation... I think the old priority and server list is still up on some pages. I know that I do still get requests for MX records using both ways. Anyway, yeah I definitely think the Google documentation is inconsistant for apps for domains...
28 Oct 2009 ~ 9:34pm Garg Anybody want to translate this from email science to common man?
28 Oct 2009 ~ 9:44pm UPSLynx thats..... not good.
28 Oct 2009 ~ 10:42pm MachineDog It's mildly distressing if Google is capable of dropping the ball like this on notifying it's customers of changes on a cloud product.
29 Oct 2009 ~ 12:59am RyanMM This is the kind of thing that deserves a #google #fail hash usage and multiple pings to people like Matt Cutts and other high-profile googlers. It's crap, it's amateurish, and shouldn't happen with a supposedly enterprise product.

I say this as someone who deploys Google Apps for clients and expects it to work as promised.

Blow this up, guys.
29 Oct 2009 ~ 12:05pm Dave What @RyanMeray said. Werd.
29 Oct 2009 ~ 12:11pm ardichoke All Google would have had to do is make a cname for aspmx4 and aspmx5 which pointed them to aspmx2 and aspmx3 and this never would have been a problem. Way to fail Google. Way. To. Fail.
29 Oct 2009 ~ 12:57pm Leonardo

Anybody want to translate this from email science to common man?

Hmm, I guess not.
29 Oct 2009 ~ 1:03pm Thrax Mail gets routed through servers. Icrontic uses Google's servers to host its mail. Google publishes a list of the servers it uses to route/deliver mail. These servers are used on our end to configure how mail is sent/received. Google dropped two of the mail servers from the recommended configuration without telling us. This means the config on our end still contained entries for these servers. This means that some of our mail appears to have been sent to servers that are no longer in service, or are not functioning correctly. This means Icrontic has definitely lost important emails.
29 Oct 2009 ~ 3:10pm photodude Odd, I'll have to check my MX settings now. I hope I'm not losing mail.....
30 Oct 2009 ~ 7:25am Leonardo Thanks, Thrax, it's not quite clear.
31 Oct 2009 ~ 12:25pm chrisWhite This is really lame.
3 Nov 2009 ~ 12:05pm Matt I encountered similar issues this weekend. I just posted to the gmail help forums here: http://www.google.com/support/forum/p/gmail/thread?tid=41a707289b6affb3&hl=en

I have the same seven MX records (different priorities) as you have, and ran thorough tests on each which showed everything working fine. I checked on Google's Apps MX setup instructions and aspmx4.googlemail.com. and aspmx5.googlemail.com. are still listed (and the priorities don't matter as long as they're in the right order, as @Frank points out). So, they are still recommending them. Furthermore, I know exactly what MX record my missing emails went through, and it was the primary one - aspmx.l.google.com.

Interested to hear if anyone else is experiencing issues, and if anyone has any more insight into the problem.
3 Nov 2009 ~ 1:24pm primesuspect Thanks for letting me know. I'm gonna post in your thread.
16 Nov 2009 ~ 4:15am photodude Curious if your also experiencing Spoofing issues with your google for domains mail?

My domain has been hit by a spoofer, and nothing seems to help, added SPF records still no change. Sometimes I think that my incoming mail is effected due to the spoofing on my domian.
16 Nov 2009 ~ 11:04am ardichoke Spoofing of that type is nearly impossible to do. The only way to pull it off would be if someone who hosts at the same place that their DNS is hosting was ARP cache poisoning and redirecting all DNS traffic to their box which was pointing Icrontic's mail elsewhere. Any decent hosting company will catch something like that nearly immediately and shut it down. That said it's not technically impossible...