Cloud Management Gateway Troubleshooting Deepdive HTTP/1.1 500 CMGConnector_InternalServerError

Before starting into the troubleshooting part, let me just give you an overview of my lab environment:

  • 1 Primary site TP2005 upgraded to TP2006
  • Management Point in HTTPS mode
  • Public SSL Cert on the CMG
  • Client are Hybrid/and AAD joined


Co-Management over CMG was working fine until I upgraded to TP2006…

After the upgrade clients on the internal network were successfully communicating with the MP in HTTPS mode but clients connecting via the CMG couldn’t connect anymore and where hitting the following error in the ccm_messaging log:

So here is where the fun part begins…Troubleshooting:

So let’s start by running the Cloud management gateway connection analyzer

As we could see there is an issue from the Cloud Management Gateway Connection Point forwarding the client requests to the MP as we are receiving an HTTP Status code 500.

So where is HTTP error code is coming from?

Let’s look into the SMS_CLOUD_PROXYCONNECTOR.log if we can find some additional hints..

We can see the same HTTP 500 status code thrown but we also could see that is coming from the CCM_STS service, so probably some authentication issue.

Next Step is to enable Failed Request Tracing (Kudos to my PFE colleague Herbert Fuchs who helped me getting some more info beside a stupid HTTP error telling me something is not working along with the CCM_STS service)

And create a rule for the Status code 500

Now we are getting an additional Log Folder in IIS for those failed requests

Let’s open the first xml and look in the Request Details

When scrolling down what caused the issue we could see that it could not load the System.IdentityModel.Token.Jwt with the version highlighted below.

Now let’s look into the properties of the Program Files\SMS_CCM\CCM_STS\binSystem.IdentityModel.Tokens.Jwt.dll and check the version:

So, we found the reason for our problem (and btw our product group already confirmed the issue and working on a fix)!

After copying over an older version of the dll (version 4.0.20622.1351) from a different environment all started working again but let’s wait for the fix 😊

Resources on Servicing the Modern Workplace and WFH

We recently published a bunch of great articles with regards to Servicing Windows and Microsoft Apps for Enterprise (formally known as Office 365 Pro Plus) to help you overcome challenges you might face when it comes to Work from home scenarios.

So as we want our devices stay healthy and secure as well in those remote times I see a lot of customers re-think their current design and also moving to Windows Update for Business.

For this reason I put together a list of resources to help you moving forward and optimize your Servicing approach:

New on Microsoft Learn: Stay current with Windows 10 and Microsoft 365 Apps
A calendar approach for keeping the modern workplace up to date
Deploying a new version of Windows 10 in a remote world
Optimize Windows monthly update deployment for remote devices
Transform Windows feature updates with a servicing calendar
Roll out updates faster with the Update Baseline for Windows 10
Real-world practices to optimize Windows 10 update deployments
Optimize on-premises monthly update delivery using the cloud
Office hours: servicing Windows 10 in a remote world
WUfB and Servicing Enhancements with Windows 10 2004
Co-Management of Windows Updates Workloads
Troubleshooting Windows 10 Update Ring Policies
Managing Patch Tuesday with Configuration Manager in a remote work world
Mastering Configuration Manager Bandwidth limitations for VPN connected Clients
Modern Content Distribution: Microsoft Endpoint Manager and Connected Cache
Keeping Windows 10 devices up to date with Microsoft Intune and Windows Update for Business
Configuring Office 365 ProPlus updates for remote workers using VPN
Optimize Office 365 connectivity for remote users using VPN split tunneling