Cloud Management Gateway Troubleshooting Deepdive HTTP/1.1 500 CMGConnector_InternalServerError

Intune, System Center 1 Minute

Before starting into the troubleshooting part, let me just give you an overview of my lab environment:

  • 1 Primary site TP2005 upgraded to TP2006
  • Management Point in HTTPS mode
  • Public SSL Cert on the CMG
  • Client are Hybrid/and AAD joined

Scenario:

Co-Management over CMG was working fine until I upgraded to TP2006…

After the upgrade clients on the internal network were successfully communicating with the MP in HTTPS mode but clients connecting via the CMG couldn’t connect anymore and where hitting the following error in the ccm_messaging log:

So here is where the fun part begins…Troubleshooting:

So let’s start by running the Cloud management gateway connection analyzer

As we could see there is an issue from the Cloud Management Gateway Connection Point forwarding the client requests to the MP as we are receiving an HTTP Status code 500.

So where is HTTP error code is coming from?

Let’s look into the SMS_CLOUD_PROXYCONNECTOR.log if we can find some additional hints..

We can see the same HTTP 500 status code thrown but we also could see that is coming from the CCM_STS service, so probably some authentication issue.

Next Step is to enable Failed Request Tracing (Kudos to my PFE colleague Herbert Fuchs who helped me getting some more info beside a stupid HTTP error telling me something is not working along with the CCM_STS service)

And create a rule for the Status code 500

Now we are getting an additional Log Folder in IIS for those failed requests

Let’s open the first xml and look in the Request Details

When scrolling down what caused the issue we could see that it could not load the System.IdentityModel.Token.Jwt with the version highlighted below.

Now let’s look into the properties of the Program Files\SMS_CCM\CCM_STS\binSystem.IdentityModel.Tokens.Jwt.dll and check the version:

So, we found the reason for our problem (and btw our product group already confirmed the issue and working on a fix)!

After copying over an older version of the dll (version 4.0.20622.1351) from a different environment all started working again but let’s wait for the fix 😊

Author

Published