Migrating to Exchange Online (Part 1)

Yep, you got that right.

I volunteered and got involved  in a migration to Exchange Online these days and I thought it would be quick, nice and easy job to help the IT team in a mid-sized organization. Not even close!

The scope is the following:

3 IMAP e-mail solutions (5 domains, 3 geo-locations of the servers) -> Exchange Online

No migration tools and no external consultants will be involved in the migration.

The first thing you'd need to know when you plan and estimate your migration is speed. Depending on how important this is for you, you can choose the different options ahead.

Option 1: PST Import service 

Caveat: If your e-mail servers can't export a PST, then you need a 3rd party tool to do that.

Limitation: You can't import PST files larger than 10 GB or containing more than 1 000 000 items. So you might as well need a tool to split your largest mailboxes into multiple PST files.

1.1. Copy your PSTs to a secure Azure BLOB storage and then import them through the Admin Panel (you need to be a Global Admin)

We were considering to use the new Office 365 Import service launched last year (and still free, MS are planning to charge for it if you ship your drives) and we were very frustrated to find out the capping of 0.5 GB / hour when you import your PSTs. There's an additional prerequisite step to that, first you'd need to copy your PSTs in an Azure Blob, which for us worked as a charm - over 300 Mbps - and that's when the speed traffic police comes in and your happy face is gone :)

By speed traffic police I meant the Office 365 User Throttling - I still dunno how the PST import of 500 mailboxes, for instance could be considered user import. It's purely the migration team's job to export all PSTs, prepare PST mapping files and all that. I totally don't see it the same as if you'd import your data through Outlook - this is where there should the throttling apply, and not if these actions are associated from the Office 365 interface, but anyway... you do get the perks like calendars, tasks, contacts. It's never perfect.

Tip: If you run multiple PST import jobs at the same time - you get 0.5 GB / h for each of them.

1,2. Ship your hard drives to the nearest Microsoft data center and then import through the Admin Panel (still, you need to be a Global Admin)

This is usually the preferred method if you have slow bandwidth to Azure. On our tests, 300 Mbps were more than satisfactory and we don't use an ExpressRoute to Azure, This means that we can transfer 1 TB for ~13 hours. So in our particular scenario, shipping the hard drives would definitely take more than the copying to Azure, but if you have 10 TB of e-mail and slower connection to Azure, your choices are different.

If you still decide to go that dirt road (PST import in general), the official MS Import guide is here.

Option 2: IMAP migration

Obviously the only other option (I'm excluding user-handled migration through Outlook and all that, of course) is the IMAP migration. It comes in two flavors.

2.1. DIY e.g. get one of the many free 3rd party scripts out there and run it on your own environment - why would you do that?

This will get the source and target mailboxes in sync, but a big trouble with that is you need your source e-mail account passwords, and the destination (Office 365) passwords for the users as well. Not a big deal, unless your users are already in Office 365 and using stuff like SharePoint, OneDrive, Yammer, etc. A no-brainer to reset or ask them for their passwords.

2.2. Initiate the IMAP migration from the Exchange Admin Center

Before you start, a good survival guide about it is found here. The most obvious limitations:

• Only items in a user's inbox or other mail folders can be migrated. You can’t migrate contacts, calendar items, or tasks.
• A maximum of 500,000 items can be migrated from a user’s mailbox.
• The maximum message size that can be migrated is 35 MB.

Provided this works for you, let's imagine you've already tried the PST import and you want to clear all stuff that you've imported and start anew. The most intuitive thing will be to delete the users - but they will be retained for about 30 days. A workaround is to rename and delete them :) 

Or... as a best practice empty those mailboxes through PowerShell.

Use the Search-Mailbox cmdlet and pipe it with DeleteContent.

Example: Search-Mailbox -Identity "John Doe" -DeleteContent

It's a bit slow, but maybe faster than opening the user's mailbox through Outlook (and for that you'd need to grant yourself Full Access to that mailbox first - lame and unprofessional approach).

So... the trouble with IMAP migration. If you run a single mailbox - the speed is still not much better - between 0.5 and 1 GB per hour. A disaster I'd say.

The key is concurrency - you'd need to run at least 20 mailboxes per batch so you can get your migration done in some realistic schedules that you can present to the business.

With the concurrency, we've easily reached speed of 10 GB per hour with 15 concurrent mailboxes. This totally corresponds with the Microsoft provided data, so you can bet on it. Of course, this excludes a lot of other factors which might block you from reaching such speeds - your ISP, your firewall, your current e-mail server bandwidth and so on and so forth.

In the process of setting this up we've found a nice little calc that helps you do the math.

So once you have your estimations and you prepare your CSV file for the IMAP migration, you start your sync. Everything goes well over the weekend, the Spring flowers are blooming and etc. NO!

There are many reasons why the IMAP sync can fail - wrong CSV format, wrong passwords, your e-mail server goes down, Microsoft Exchange Online goes down...but all of those you can either fix or just hope for Microsoft to fix as quickly as possible. But what to do when the sync fails and you want to start afresh?

Well, first the sync will not always stop if some of the above problems occur. It will continue running even if it runs fine for 1 mailbox and all the rest fail. So you need to stop it manually. From the GUI - I've noticed that stopping it manually leaves it in "Removing" state for more than 12 hours - dunno if it's a bug on our tenant or not, but then your only way is to kill it with PowerShell. The Remove-MigrationBatch command comes handy in this case. If you miss this part, your new sync with all the correct data will NOT run as the same mailboxes are already part of another sync which is not totally gone. Use the "-force" parameter, otherwise there is no result (at least in our tests).

Once you clear the migration job, you go to the GUI in EAC, you see it gone and you import your same CSV file and hope to get this running. No! 2 reasons:

If you import the same CSV file, EAC tries to be smart and doesn't detect the mailboxes which were part of the previous synchronizations. So rename your CSV file.

Your next attempt would also fail. Even if your mailboxes get detected and they go into the batch, they will fail. Don't ignore the error report, but download it and read the following:

The user "dimitar.miriyski@abc.xyz" already exists, but the migration batch that includes it couldn't be found.

Does that make sense? Yes, of course. Your Office 365 user is not deleted. But the migration batch is. So what's the problem with that? There's a history of all those batches... which you need to sweep off manually with the Remove-MigrationUser command. Hopefully you haven't messed with too many users in your failed batch :) Ideally it should never have failed, but things like that happen.

Another trick when running the migration batches is not to use CAPS in the batch title.

Option 3: A mix of 1 and 2.

It seems as the perfect cocktail - you do the bulk stuff with IMAP sync to gain the speed advantage and then you do the calendars, tasks and contacts through PST export and import. Not tested yet.

Conclusion

So... from the short time of research and testing this, I'd say there are a lot of things you'd need to consider and Exchange Online is not a toy. No wonder the Exchange Administrators are still out there, but in the next post from this series I'll see if I can prove that a SharePoint guy with some Office 365 knowledge and experience can do the trick for a small to medium sized company.