Cloud Management Gateway Troubleshooting Deepdive HTTP/1.1 500 CMGConnector_InternalServerError

Before starting into the troubleshooting part, let me just give you an overview of my lab environment:

  • 1 Primary site TP2005 upgraded to TP2006
  • Management Point in HTTPS mode
  • Public SSL Cert on the CMG
  • Client are Hybrid/and AAD joined

Scenario:

Co-Management over CMG was working fine until I upgraded to TP2006…

After the upgrade clients on the internal network were successfully communicating with the MP in HTTPS mode but clients connecting via the CMG couldn’t connect anymore and where hitting the following error in the ccm_messaging log:

So here is where the fun part begins…Troubleshooting:

So let’s start by running the Cloud management gateway connection analyzer

As we could see there is an issue from the Cloud Management Gateway Connection Point forwarding the client requests to the MP as we are receiving an HTTP Status code 500.

So where is HTTP error code is coming from?

Let’s look into the SMS_CLOUD_PROXYCONNECTOR.log if we can find some additional hints..

We can see the same HTTP 500 status code thrown but we also could see that is coming from the CCM_STS service, so probably some authentication issue.

Next Step is to enable Failed Request Tracing (Kudos to my PFE colleague Herbert Fuchs who helped me getting some more info beside a stupid HTTP error telling me something is not working along with the CCM_STS service)

And create a rule for the Status code 500

Now we are getting an additional Log Folder in IIS for those failed requests

Let’s open the first xml and look in the Request Details

When scrolling down what caused the issue we could see that it could not load the System.IdentityModel.Token.Jwt with the version highlighted below.

Now let’s look into the properties of the Program Files\SMS_CCM\CCM_STS\binSystem.IdentityModel.Tokens.Jwt.dll and check the version:

So, we found the reason for our problem (and btw our product group already confirmed the issue and working on a fix)!

After copying over an older version of the dll (version 4.0.20622.1351) from a different environment all started working again but let’s wait for the fix 😊

Resources on Servicing the Modern Workplace and WFH

We recently published a bunch of great articles with regards to Servicing Windows and Microsoft Apps for Enterprise (formally known as Office 365 Pro Plus) to help you overcome challenges you might face when it comes to Work from home scenarios.

So as we want our devices stay healthy and secure as well in those remote times I see a lot of customers re-think their current design and also moving to Windows Update for Business.

For this reason I put together a list of resources to help you moving forward and optimize your Servicing approach:

New on Microsoft Learn: Stay current with Windows 10 and Microsoft 365 Appshttps://techcommunity.microsoft.com/t5/windows-it-pro-blog/new-on-microsoft-learn-stay-current-with-windows-10-and/ba-p/1483656
A calendar approach for keeping the modern workplace up to datehttps://techcommunity.microsoft.com/t5/windows-it-pro-blog/a-calendar-approach-for-keeping-the-modern-workplace-up-to-date/ba-p/1475666
Deploying a new version of Windows 10 in a remote worldhttps://techcommunity.microsoft.com/t5/windows-it-pro-blog/deploying-a-new-version-of-windows-10-in-a-remote-world/ba-p/1419846
Optimize Windows monthly update deployment for remote deviceshttps://techcommunity.microsoft.com/t5/windows-it-pro-blog/optimize-windows-monthly-update-deployment-for-remote-devices/ba-p/1309917
Transform Windows feature updates with a servicing calendarhttps://techcommunity.microsoft.com/t5/windows-it-pro-blog/transform-windows-feature-updates-with-a-servicing-calendar/ba-p/1475672
Roll out updates faster with the Update Baseline for Windows 10https://techcommunity.microsoft.com/t5/windows-it-pro-blog/roll-out-updates-faster-with-the-update-baseline-for-windows-10/ba-p/1468950
Real-world practices to optimize Windows 10 update deploymentshttps://techcommunity.microsoft.com/t5/windows-it-pro-blog/real-world-practices-to-optimize-windows-10-update-deployments/ba-p/1227825
Optimize on-premises monthly update delivery using the cloudhttps://techcommunity.microsoft.com/t5/windows-it-pro-blog/optimize-on-premises-monthly-update-delivery-using-the-cloud/ba-p/1483519
Office hours: servicing Windows 10 in a remote worldhttps://techcommunity.microsoft.com/t5/windows-it-pro-blog/office-hours-servicing-windows-10-in-a-remote-world/ba-p/1316322
WUfB and Servicing Enhancements with Windows 10 2004https://techcommunity.microsoft.com/t5/windows-it-pro-blog/what-s-new-for-it-pros-in-windows-10-version-2004/ba-p/1419764
Co-Management of Windows Updates Workloadshttps://techcommunity.microsoft.com/t5/core-infrastructure-and-security/co-management-of-windows-updates-workloads/ba-p/922378
Troubleshooting Windows 10 Update Ring Policieshttps://techcommunity.microsoft.com/t5/intune-customer-success/support-tip-troubleshooting-windows-10-update-ring-policies/ba-p/714046
Managing Patch Tuesday with Configuration Manager in a remote work worldhttps://techcommunity.microsoft.com/t5/configuration-manager-blog/managing-patch-tuesday-with-configuration-manager-in-a-remote/ba-p/1269444
Mastering Configuration Manager Bandwidth limitations for VPN connected Clientshttps://techcommunity.microsoft.com/t5/premier-field-engineering/mastering-configuration-manager-bandwidth-limitations-for-vpn/ba-p/1280002
Modern Content Distribution: Microsoft Endpoint Manager and Connected Cachehttps://techcommunity.microsoft.com/t5/premier-field-engineering/modern-content-distribution-microsoft-endpoint-manager-and/ba-p/1148669
Keeping Windows 10 devices up to date with Microsoft Intune and Windows Update for Businesshttps://www.microsoft.com/en-us/itshowcase/keeping-windows-10-devices-up-to-date-with-microsoft-intune-and-windows-update-for-business
Configuring Office 365 ProPlus updates for remote workers using VPNhttps://techcommunity.microsoft.com/t5/office-365-blog/configuring-office-365-proplus-updates-for-remote-workers-using/ba-p/1253491
Optimize Office 365 connectivity for remote users using VPN split tunnelinghttps://docs.microsoft.com/en-us/office365/enterprise/office-365-vpn-split-tunnel