Character Import Status Error Retrieving Character List Try Again Ssl Connect Error

This weblog mail service is a part of a 5 post series consisting of:

• Substitution Hybrid Migrations: More Than Merely a Pretty Confront (Part ane)
• Digging Into Hybrid Migration Move Report Data (Part 2)
• Troubleshooting Failed Migrations (Function 3 - you are here)
• Troubleshooting Ho-hum Migrations (Office four)
• What to do if a migration is Completed With Warnings (Part v)

Continuing the blog post series, we arrived at troubleshooting failed migrations.

A 'failed migration' is when the condition of the move asking shows equally 'failed', and we accept one or more failures logged in the movement report. The motion is stopped and needs the administrator's attending to investigate the reason of failure. Sometimes, resuming of the movement can help, especially if there were some temporary issues on the Exchange Online side that were addressed.

Before getting into troubleshooting, I recommend you check the following 'Minimum Requirements'; those are the things we know will break migrations (and nosotros see them do so):

• MRSProxy needs to exist enabled and running on Exchange on-premises.
• Substitution Online requires Negotiate (NTLM) authentication for MRSProxy.
• Brand certain your migration users are synchronized with AADconnect tool and corresponding post users are provisioned correctly on the Exchange Online side for respective on-premises mailboxes (ExchangeGuid present, alias, recipient blazon correct, accepted domains for the e-mail addresses and secondary smtp accost user@tenant.mail.onmicrosoft.com).
• Maximum storage quota in Commutation Online for both primary and archive mailbox is 100GB each. Even if you take auto-expanding enabled in Exchange Online, as of this writing the maximum mailbox size for migration of chief archive is yet 100GB – reference here.
• Maximum number of items per regular mailbox folder is 1 million and 3 million for the dumpster, reference here
• On-bounds migration admin needs to have the minimum required permissions and valid credentials.
• You cannot offboard to Exchange 2010 an Exchange Online Mailbox that has whatsoever hold other than Litigation Concord. This is considering Exchange 2010 doesn't know virtually in-identify hold which was introduced in Exchange 2013 or almost system-wide holds in Office 365. More info on the types of holds can be found hither.
• You cannot offboard an Exchange Online annal mailbox which has auto-expanding enabled in Office 365
• You cannot offboard a mailbox and a primary archive to Substitution 2007
• Yous cannot offboard a remote mailbox without ExchangeGuid assault it
• Any network load-balancing for Substitution 2010 MRSProxy servers requires IP persistence (affinity).
• SSL offloading is non supported for MRSProxy.
• For classic hybrid – where nosotros crave inbound connectivity from Substitution Online to on-premises Commutation, allow all Exchange Online IP addresses to connect to on-premises EWS / Autodiscover.
• For archetype hybrid, pre-authentication for EWS / Autodiscover virtual directories is non supported.(*)
• For classic hybrid, a valid three^rd party certificate is required for EWS / IIS. Also see this.
• TLS1.2 should be enabled in the on-bounds infrastructure.
• If you have an Substitution organization with Exchange 2013/2016 in coexistence with legacy Exchange servers (Exchange 2010), and so you need to betoken the MRSproxy namespace to the newer Commutation version in your environment. This is required considering, for example, Exchange 2010 cannot proxy to Exchange 2016 in order to motility an Exchange 2016 mailbox to or from Exchange Online through an Commutation 2010 MRSProxy endpoint.

(*) Strictly speaking, for hybrid migrations and EWS/mrsproxy.svc endpoint, it would piece of work if the device supported NTLM pre-authentication. It would not work for other hybrid scenarios similar Free/Busy, which cannot piece of work with pre-hallmark on EWS and Autodiscover.

We as well recommend that yous bypass the network devices such every bit firewalls and contrary proxies during migrations in order to reduce source network latency and avoid frequent communication transient errors that would event in mailbox locks and slow migrations.

Often when troubleshooting Office 365 migrations, the Exchange Admin Center GUI is helpful and quite verbose regarding the reason of failure, and it many times includes a link to the respective documentation page for more than data on specific outcome.

Let me briefly show you lot some useful info that we tin can see in the (Classic) Exchange Admin Center. Equally a note, at the time of writing this commodity, the New Exchange Admin Center doesn't currently show all of this information.

The following can be seen in the above screenshot:

• We accept 1 migration batch chosen "Test Hybrid Migration" of type Substitution Remote Move
• Management of the move is Onboarding (from on-premises to the cloud)
• The current status Syncing (things are going well and then far)
• At that place is only ane migration user in the batch (looking at the Full column)
• The user is not Synced (hasn't reached the Incremental Sync at 95%), non Finalized (hasn't reached the 100% completion) and not Failed (didn't meet a fatal failure)

Afterwards clicking on View details, we also see:

• Who created the batch (crystal@mytenant.onmicrosoft.com),
• When it was created and started (New-MigrationBatch -AutoStart),
• When it should complete (after the initial sync will be done)
• At that place is no terminal synced time because the status is syncing and no initial sync has been washed
• Also, the associated endpoint is the name of my migration endpoint (Go-MigrationEndpoint) through which I am running the batch.

After a little while, the user failed because of the ExchangeGuid missing on the mail user object in Substitution Online:

In such situation, the migration service failed to inject the move request because the user failed validation. This means that nosotros don't take a motion asking for this user and therefore will have no motion report.
If yous were to click on 'Download the report for this user', you would get an empty .txt file.
Let me testify you how this failure looks like in PowerShell and what objects are created and available for us to check there.
With Get-MigrationBatch command, we tin meet the proper name of the batch, the condition, the type and how many users are contained in the batch:

To see all properties, run Become-MigrationBatch |FL.

Another attributes values that yous saw in the Exchange Admin Center GUI, for this batch were:

CreationDateTime           : 6/one/2020 eight:23:35 AM
StartDateTime              : vi/ane/2020 viii:23:34 AM
LastSyncedDateTime         :
SubmittedByUser            : crystal@<mytenant>.onmicrosoft.com
BatchDirection             : Onboarding
SourceEndpoint             : Hybrid Commutation Miry

If I had multiple batches and I was interested in seeing this detail ane, I would run: Get-MigrationBatch "Test Hybrid Migration" or if I wanted to see all batches that are failed, I would run: Get-MigrationBatch -Status SyncedWithErrors

Going further with PowerShell, if I want to see the migration user contained in that batch, I would do information technology like this: Get-MigrationUser -BatchId "Test Hybrid Migration". To see all the details on the migration user, I would again append |FL

This error is self-explanatory, ExchangeGuid is missing on the user and I tin can likewise encounter information technology with Get-MailUser command for this migration user:

From the Get-MigrationUser output, I can likewise come across the RequestGuid is empty, so this also tells me that at that place is no move request / movement report for this migration user. I tin can run Get-MoveRequest <user> or Go-MigrationUser -BatchId "Test Hybrid Migration" | Get-MoveRequest to confirm this.

In cases where the fault message on the migration user is not and then obvious and you still don't take a motility request created for information technology, y'all can check Get-MigrationUserStatistics with DiagnosticInfo verbose switch: Become-MigrationUserStatistics <user identity> -DiagnosticInfo verbose |FL and meet if any more than details found.
I will now become through some more control examples if you lot want to play around and bank check uncomplicated or more complicated stuff in PowerShell. Also, some things can be simply checked from PowerShell and if you take a motility request created and this is failed or is progressing slow, you can come across more on analyzing move reports with PowerShell in later function of this blog series.

To get an overview of migration statistics:

Go-MigrationStatistics

To become all migration users, their status and corresponding batches:

Get-MigrationUser

To go a specific migration user:

Get-MigrationUser <email accost>

To check the mistake on a specific migration user:

Get-MigrationUser <email address> |FL errorsummary
Get-MigrationUser <email address> |FL

To go all failed migration users:

Get-MigrationUser -Status Failed

To get all failed migration users and their errors:

Become-MigrationUser -Status Failed | FT identity , errorsummary
Go-MigrationUser -Status Failed | FL identity , errorsummary

To become migration users from a detail batch:

Get-MigrationUser -Batch "Batch Name"

To get all migration batches:

Get-MigrationBatch

To get a particular batch:

Get-MigrationBatch "Batch Name"

Checking move requests (specific for hybrid remote moves)

To get all existing move requests:

Go-MoveRequest

To get movement request statistics for a specific move asking:

Go-MoveRequestStatistics "User"

Know that in that location are 2 primary types of failures:

• Transient Exceptions, example DataExportTransientException
• Permanent Exceptions, example StoragePermanentException

Note: For a move request to be in a Failed state, we would need to accept a permanent failure. Too many transient failures (commonly more than 60) will eventually crusade a permanent failure. Too many transient failures tin can likewise slow down your migration considerably.

To meet the failures (transient or permanent), you would run commands similar to these or export the statistics to an XML file (discussed in the afterward role of this weblog series)

To store the movement report in a variable:

$stats = Get-MoveRequestStatistics "Affected User" -IncludeReport

To bank check all failures and their count:

$stats.report.Failures | group failuretype | Format-Tabular array -AutoSize

To check total details of the terminal failure:

$stats.report.Failures[-one]

To check the last 2 failures:

$stats.Report.Failures | select -concluding ii

To bank check the starting time failure:

$stats.report.Failures[0]

To cheque the first 3 failures:

$stats.Study.Failures | select -beginning 3

If there are a lot of failures, you can create a list of the failures with the PowerShell Index number associated with each failure by running the following:

$i=0;$stats.written report.Failures | % { $_ | Select-Object @{name="index";expression={$i}},timestamp,failurecode,failuretype,failureside;$i++} | ft

Using this output, yous can then easily identify the index number you want to focus on by enclosing the failure index number in [brackets], example:

$stats.report.Failures[four]

To get failed motility requests:

Go-MoveRequest -MoveStatus Failed

Most frequent failures

Hither is a list of virtually frequently seen failures in hybrid migrations (and when I say 'almost frequent' I mean 'most frequent' issues that we encounter in back up, not that you will see those errors in every migration). Annotation that not all are permanent failures, meaning not all these will cause your migrations to fail.

• "User is already being moved" – reference here
• "You lot tin't use the domain because it's not an accepted domain for your organization" – reference here
• "Target mailbox doesn't accept an smtp proxy matching '.mail.onmicrosoft.com'" – reference here
• "MigrationPermanentException: Cannot detect a recipient that has mailbox GUID" – reference here. Note that another possible scenario for this mistake is when we cannot find a ComponentShared Mailbox by its GUID on the Exchange Online side. A ComponentShared mailbox is used to host data from other Office 365 workloads like Teams, OneDrive for Business and SharePoint. You would check (in Substitution Online PowerShell) these mailbox GUIDs with the command: Become-MailboxLocation -User <SMTP>. If the mailbox GUID in the fault belongs to a component shared mailbox, please log a case with Microsoft Back up.
• "Yous must specify the PrimaryOnly parameter" – reference here
• "The remote server returned an Fault 404" or "HTTP request has exceeded the allotted timeout" – reference here
• "The remote server returned an fault: (403) Forbidden" – reference hither
• "Admission is denied" – reference hither
• "Couldn't switch the mailbox into Sync Source mode" – reference here
• "CommunicationErrorTransientException - The remote endpoint no longer recognizes this sequence. This is most likely due to an abort on the remote endpoint. The value of wsrm:Identifier is not a known Sequence identifier. The reliable session was faulted." – reference here
• "The server was unable to procedure the request due to an internal error.  For more information about the error, either plough on IncludeExceptionDetailInFaults ..." – references here and here
• "TooManyBadItemsPermanentException" - Failed to find a principal from the source forest or target woods – references here and here
• "The information consistency score (Investigate) for this asking is besides low" – reference here. Note that we will have more on Data Consistency Score later in the weblog mail series.
• "Exception has been thrown by the target of an invocation." – reference here
• "Transient error CommunicationErrorTransientException has occurred. The system will retry" – reference here
• "The Mailbox '<username>@contoso.com' isn't enabled for unified messaging." – reference hither
• "Failed to catechumen the source mailbox 'Chief (00000000-0000-0000-0000-000000000000)' to postal service-enabled user after the motility." or "Unable to update Active Directory information for the source mailbox at the finish of the move." – reference hither
• "Target user <User> already has a primary mailbox". Annotation: pay special attention to the scenario, it matters if you get this mistake in onboarding (motion to Exchange Online) or offboarding (move from Exchange Online). For onboarding moves, please see this, and for offboarding see this. Note on scenario 1 step 7 in that article: it is not supported to remote restore a disconnected mailbox from Substitution 2010 on-bounds source server version, it needs to exist minimum Exchange 2013 version.
• "StalledDueTo_Target*" when you lot move mailboxes to Commutation Online – reference hither. More on this when we will exist discussing boring migrations in next function of this web log post serial.
• "MapiExceptionTooComplex: Unable to query table rows. (hr=0x80040117, ec=-2147221225)" – reference hither.
• "Mailbox Replication Proxy Service tin't process this request because information technology has reached the maximum number of active MRS connections allowed" – reference here.

Migration failures due to Exchange Online mailbox folder limits

I want to discuss 2 common migration errors related to mailbox binder quota limits in Exchange Online when onboarding a mailbox to Office 365:

• MapiExceptionFolderHierarchyChildrenCountQuotaExceeded
• MapiExceptionFolderHierarchyDepthQuotaExceeded

First one, FolderHierarchyChildrenCount refers to Maximum number of subfolders per mailbox binder (ten,000 subfolders per folder). MapiExceptionFolderHierarchyChildrenCountQuotaExceeded is thrown when a folder has more than 10,000 same level subfolders in it on the source mailbox.
Second one, FolderHierarchyDepth refers to maximum folder hierarchy depth (300 folders 'deep').
MapiExceptionFolderHierarchyDepthQuotaExceeded is thrown when the source mailbox has a binder hierarchy with more than than 300 folder levels depth.
Here is a visual of these two:

The mailbox binder limits are documented in the above and you can read them with PowerShell (Get-MailboxStatistics) ran against an Exchange Online Mailbox. These item two folder limits are:

• FolderHierarchyChildrenCountReceiveQuota
• FolderHierarchyDepthReceiveQuota

You can export the mailbox folders to CSV, on the source side with the following commands:

Go-MailboxFolderStatistics <user> -FolderScope NonIpmroot | Export-Csv -NoTypeInformation PrimaryFolderStats.csv
Get-MailboxFolderStatistics <user> -Annal -FolderScope NonIpmroot | Export-Csv -NoTypeInformation ArchiveFolderStats.csv

Note that NonIpmRoot Binder scope is not available on Exchange 2010 Servers, you lot will need to use MFCMAPI to see system folders.

The solution is to merge affected folders or move them across the folder bureaucracy. You volition then exist able to drift the mailbox.
However, to avoid the warnings that the end-user will receive after migration, you take to make certain you are beneath warning quotas (that is 9,000 for kid folders and 250 for folder depth).

Here is an example of such warning for folder bureaucracy children count:

There is another folder limit (maximum number of messages in a mailbox folder) that is quite often seen in hybrid migrations: 1 one thousand thousand items per folder limit and more rarely the 3 1000000 items per Dumpster (Recoverable Items folder).
These limits are too documented here and you can see read them using Become-MailboxStatistics:

The failure is: MapiExceptionMessagePerFolderCountQuotaExceeded. Y'all tin can export the folders items to CSV using the following commands:

Become-MailboxFolderStatistics <user> -FolderScope NonIpmroot | sort itemsinfolder -Descending | select folderpath, itemsinfolder |Export-Csv -NoTypeInformation PrimaryFolderItems.csv
Become-MailboxFolderStatistics <user> -Archive -FolderScope NonIpmroot | sort itemsinfolder -Descending | select folderpath, itemsinfolder |Consign-Csv -NoTypeInformation ArchiveFolderItems.csv
Get-MailboxFolderStatistics <user> -FolderScope RecoverableItems | sort itemsinfolder -Descending | select folderpath, itemsinfolder | Export-Csv -NoTypeInformation PrimaryDumpsterFolderItems.csv
Become-MailboxFolderStatistics <user> -Archive -FolderScope RecoverableItems | sort itemsinfolder -Descending | select folderpath, itemsinfolder |Export-Csv -NoTypeInformation ArchiveDumpsterFolderItems.csv

Solution is to reduce the number of items in the folder (by deleting or moving them to a different folder when possible).
You volition need to recreate migration (remove batch or afflicted migration user from migration batch and create new batch with the user) in gild to successfully prepare this failure.

A few more than troubleshooting tips

MoveOptions Parameter

Frequently mailbox moves fail because of corrupt items or elements in a mailbox. These mailbox move failures tin be avoided past excluding those (often corrupt) elements from existence migrated.

The MoveOptions parameter (previously known as the SkipMoving parameter which is being deprecated) tin be added to the onboard or offboard request from PowerShell with the values of:

'SkipFolderRules, SkipFolderACLs, SkipFolderPromotedProperties, SkipFolderViews, SkipFolderRestrictions, SkipContentVerification, SkipPerObjectIndex'.

This will tell the migration to skip these elements when performing the move. We recommend you perform these skips under the guidance of Microsoft Support.

You tin can review a move report from a previously failed movement attempt and get some clues on what exclusions yous should consider making.

For case, this failure beneath means that nosotros have a search binder on the source mailbox where the query (restriction) is too complex and cannot be created on the target.

Sometimes the failure identifies the actual problematic source binder so yous tin can look more than at the DataContext content. You tin can and then either delete the query on the source mailbox or just skip the migration of the queries (search folders) so that you can complete the migration:

Set-MoveRequest user@contoso.com -MoveOptions @{add="SkipFolderRestrictions"}

Mailbox Integrity checks

If you lot migrate a mailbox (primary mailbox or annal) to Commutation Online and the size is bigger than 10GB, this is considered a big mailbox and the MRS will perform an ISinteg task to ensure integrity of the mailbox that is being moved.

If you suspect that your move is stuck on ISinteg job, you can cheque the move study in EXO PowerShell and search for all strings containing isinteg keyword:

$stats = Become-MoveRequestStatistics <user> -IncludeReport
$stats.report.Entries | where { [string] $_ -similar "*IsInteg*" } | % {[string] $_}

If that shows completed, this means there are no problems. Otherwise, you can try running the same command MRS is using on your Exchange on-bounds environs, in Ems:

New-MailboxRepairRequest <migration user identity> -CorruptionType MessageId

For more info on the New-MailboxRepairRequest cmdlet, you can check here.

Depending on the Exchange Server Version you can check then the status of the repair request.

For Exchange 2013 and later, utilise this cmdlet:

Get-MailboxRepairRequest -Mailbox <user identity>

For Exchange 2010 version, y'all would demand to look in Consequence Viewer for the following events:

• Event 10047 when the repair request is started
• Event 10062 when a corruption is detected and repaired
• Effect 10048 when the repair completes successfully

You tin as well endeavour to motility a mailbox locally from one server to another, remove the local move request and then retry migration of the mailbox to Exchange Online.

Testing MRS service

Ane utility that can be used for troubleshooting the mailbox move operation is the Examination-MRSHealth cmdlet.  I thing to realize is that it cannot be tested from Role 365 side since the cmdlet is not available to a tenant administrators. However, at to the lowest degree from my experience, I take never encountered a state of affairs where MRS service would be stopped on the Office 365 side (and was not automatically recovered within seconds). We can employ this utility to test the mailbox replication service health on-premises. Also on-premises, y'all can cheque if the MRSProxy is enabled on the EWS virtual directories and if EWS awarding pool is started in IIS managing director.

Result Viewer Diagnostic logging

When performing a mailbox motion, you tin turn up diagnostic logging on the mailbox replication service or other component like asp.net to get better, more granular events in the effect log on-premises.

In most situations, yous don't actually become useful events in the on-bounds outcome viewer when troubleshooting an Substitution Online remote motility due to the fact that those events would be written in the datacenter. The default event logging can provide y'all with enough information on what the event would be, accept for case event 1309 from ASP.NET where the clarification is cocky-explanatory: MRSproxy service beingness disabled.

If you practice find a relevant event log for the afflicted Exchange Online remote motility in the effect viewer and this is related to MRS, you can turn upward diagnostic logging for the MRS service with the post-obit cmdlet:

Become-EventLogLevel 'MSExchange Mailbox Replication*' | Ready-EventLogLevel -Level Expert

Then reproduce the issue or wait for it to be reproduced again and then bank check in the Effect Viewer logs for whatsoever relevant events.

Tracking incoming failed requests from EXO

Peculiarly useful in communication or timeout failures, in that location are 3 chief logs to runway the MRS requests on the Exchange on-bounds servers in order for you to understand if an MRS Commutation Online request reached your Exchange server, or non. These frequently help u.s. narrow down the issue to a most likely network device (in front end of Exchange Server) that could terminate the connection and not pass information technology to Exchange Servers. Or if the asking reaches the Exchange servers, nosotros tin see where this is stuck and get a better understanding on what's the problem on the Substitution server on-bounds.

Exchange on-bounds server logs to track an EXO Incoming MRS request:

• HTTPerr logs: %SystemRoot%\System32\LogFiles\HTTPERR
• IIS logs for Default Web Site (DWS): %SystemDrive%\inetpub\logs\LogFiles\W3SVC1 – UTC Timezone

The name of the IIS logs contains the date of the log, for example u_ex190930.log is from Sept 30, 2019.

• HTTPProxy logs for EWS (bachelor in Commutation 2013 or afterwards): %ExchangeInstallPath%Logging\HttpProxy\Ews

The name of the HTTPProxy logs contains the date and 60 minutes starting to log, for instance HttpProxy_2019093014-10.LOG (10^th log from Sept xxx, 2019, starting hr 14:00 UTC)

Few things to mention here:

• Ever correlate the timestamp of a failure HH:MM:SS in move study with these logs (IIS and HTTPProxy are in UTC timezone)
• A failed asking will never accept 200 Status code (if you meet information technology with 200 in logs, information technology means y'all are not looking at the failed one). Note that for a asking that times out, you might notwithstanding be able to encounter it here with 200 status code and possibly a higher time-taken
• If you lot run across the failed asking  in HTTPerr logs, this won't probably be nowadays in IIS logs or HTTPProxy logs – it is stuck in forepart of IIS, cheque the particular reason in HTTPerr logs and cheque for IIS misconfiguration
• If you see the failed requests in IIS logs , so you tin can do IIS failed request tracing on that status code and cheque further the detailed mistake in HttpProxy logs

This concludes Part iii of these weblog series. We will be talking well-nigh troubleshooting dull migrations next!

I would like to thank the following persons for contributing to this blog and for their time and patience to read this: Angus Leeming, William Rall, Brad Hughes, Chris Boonham, Ben Winzenz,  Cristian Dimofte, Nicu Simion, Nino Bilic,  Timothy Heeney

balltinglainfire.blogspot.com

Source: https://techcommunity.microsoft.com/t5/exchange-team-blog/troubleshooting-failed-migrations/ba-p/1746234

Share this post