Another day, another mystery. This time, our mystery involves a popular email security product, an even more popular productivity services provider, and a rather obscure bug. The bug was never triggered before our attempt to configure a certain feature, but hey, there is a first time for everything.
The Problem
We had multiple domain names (as people and companies do) and we used one of them as the main domain for creating employees email addresses. The remaining domains were configured as domain aliases for the main domain. When a user is created on the main domain, they get aliases on all the other domains, pretty neat, right??
It's neat and saves a ton of money. However, this configuration presented a grave problem when we started configuring our email security platform. None of the email aliases were showing up in the platform, which meant we couldn't apply policies on these addresses nor could we forward emails correctly, a huge bummer and definitely a dealbreaker. The road towards figuring out the root cause and resolving this issue was a treacherous but really educative one. Let's dig into it, shall we?
First Thoughts
Our first thought was that we messed up the configuration. After all, we aren't experts using the security platform and mistakes might have happened. However, we reviewed our configuration multiple times and even got one of the experts on the vendor side to review it. Nothing was out of order, yet things weren't working as expected.
Broken Permissions Maybe?
Integrating the email provider, and the security one requires giving the email security platform access to the email provider –sometimes pretty generous access– to allow it to handle user management and email operations (as expected).
So, it was logical that our next suspect were the permissions we granted for the service user used by the email security platform. Once again, we reviewed the permissions multiple times and went through them with our assigned engineer on the vendor side. Everything seemed to be in order when we tested directly through interacting with the Email vendor APIs, but no aliases showed up on the platform.
Could It Be DNS?
Surprisingly, it wasn't DNS this time :). In fact, DNS was a long shot, as there is very little relationship between the records we configure and why aliases wouldn't show up. After all, we are getting the normal user addresses –those created under the main domain– so DNS had nothing to do with this problem.
Hitting The Wrong Jackpot
For a moment, we thought we identified the issue. A confusion around how the API for our mail provider works made us think this was nothing more than misconfigured permissions and access scopes. After some testing, this theory was proven to be wrong. At this point, we are getting ready to sacrifice a goat in exchange for some pointers towards the root cause of this frustrating issue (for the record, I am joking, I don't condone sacrificing animals to make up for lacking knowledge 😅).
We Found Waldo
After months of investigation, debugging, and trials, we found out the root cause of all of our pain. It was a bug in the product we are onboarding from the email security platform. The email security platform didn't account for the unique setup we had at our company, and their system wasn't designed to accommodate this particular configuration, so it failed to fetch the correct information from the email provider.
For those of you who are into technical details, here is what happened. The email provider has an API endpoint that returns all users within the organization of the customer. The huge JSON response includes all data about the users, including their aliases, the editable and non-editable types. The way we configured our organization with that email provider, all aliases were assigned automatically and thus were non-editable, do you see where this is going? 🤣
The email security platform used the right API endpoint but only looked for aliases in the field for Editable aliases, and ignored the other field where all non-editable aliases were included, and that led to the issue of all of our aliases not showing up when we connected the platform to our email providers.
In the defence of the email security provider, Google named the field for Editable aliases, aliases while naming the other field nonEditableAliases which can be a bit confusing when trying to figure which one has the full list of aliases (surprise, neither one does) but still not really an excuse for missing the difference when developing a complex platform focused on security.
The Solution
Once the root cause was identified, the vendor's RnD team worked out a fix and deployed it into their environments. Soon after that, we confirmed the issue has been indeed resolved and everything is now working as expected.
Epilogue
This was a challenging issue and while it was entertaining at times, it was very frustrating as it caused serious delays to our implementation timeline. I am thrilled that it's now resolved and that all of our vendor future clients won't have to go through the same ordeal if they ever end up having a similar use case or configuration.