Field Notes: The case of the stopped Azure AD Connect synchronization – stale Internet proxy server

This is a continuation of a series on Azure AD Connect. In this blog post, I cover a specific case where an export to Azure AD fails due to stale Internet proxy settings configured on the server running Azure AD Connect. I go through various tools, some of which we have covered in our previous blog posts, to provide different perspectives.

Background

Azure AD Connect should be made aware when it is running on a server that is sitting behind a proxy server. This is achieved by updating the machine.config file to include proxy and port settings. This file is located in the C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config folder. The Troubleshoot Azure AD connectivity document details this process.

The case of the stale proxy server

So, we have successfully managed to install and configure Azure AD Connect following one of the methods I covered in the previous blog posts (I provide links in the summary below). Synchronization that has been working fine for a few months suddenly stops. Troubleshooting begins – let’s look at some tools and methods.

Azure AD Connect Troubleshooting Tool

We covered an introduction to the troubleshooting tool here. This tool has the ability to help troubleshoot and diagnose object synchronization issues.

We go ahead and select necessary menu options and specify a distinguished name of an object that we are using to troubleshoot.

Interesting! We get a confirmation that there is a problem, but let’s focus on one message that stands out: An error occurred while sending the request – OperationStopped [Get-MsolDomain], HttpRequestException

This is a nice clue, but let’s move on.

Synchronization Service Manager

We move on to the Synchronization Service Manager and discover that export to Azure AD profile has a status of stopped-extension-dll-exception.

If you do a search on the Internet on this status, you will find some blogs and documents pointing to a stale or expired credential. Could this be a problem in our case? Let’s have a look somewhere else to gather more clues.

Synchronization Scheduler

Let’s also check if the synchronization scheduler has not been suspended and everything is health from that perspective. Oh no – we run Get-ADSyncScheduler and get and error! Start-ADSyncSyncCycle throws a similar error.

System.Net.Http.HttpRequestException: An error occurred while sending the request. System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: A connection
attempt failed because the connected party did not properly respond after a period of time, or established connection
failed because connected host has failed to respond 10.0.0.2:8080

Someone passing by makes a remark — “that’s a lot of red right there”, but we already have a good idea on we are dealing with at this stage. Before we get to the fix though, let’s look at two more tools we could leverage.

Windows Event Viewer

Everyone’s favourite! A quick look at the application event log gives us an array of event ID 906 errors.

One of these events confirms our challenge with connectivity “... connection failed because connected host has failed to respond…” You know what this is – proxy:port! We probably should have just started at the Event Viewer right? Did someone change or decommission the proxy server without our knowledge? Hmmm…

Azure AD Connect installer

The fix is coming up next, but let’s check what the Azure AD Connect installer would show us when an attempt to connect to Azure AD is made.

Unable to connect to the remote server

I highlight this because the error is different in a case where the proxy is still there but we cannot get to it due to name resolution for instance.

The remote name could not be resolved.

The fix is easy

In our case, the proxy server is no longer around and the Azure AD Connect server was still attempting to go through it. The server now has a more direct route to the necessary Azure AD endpoints. We need to remove the proxy settings. So, we navigate to C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config and remove the stale proxy settings from the machine.config file.

<system.net>
    <defaultProxy>
        <proxy
        usesystemdefault="true"
        proxyaddress="http://proxy.<server>:<port>"
        bypassonlocal="true"
        />
    </defaultProxy>
</system.net>

You may need to restart the Microsoft Azure AD Sync service

Start-ADSyncSyncCycle now completes without errors and synchronization is working again.

Recap

This is kind of a reverse of what we would normally do in most production deployments as servers running Azure AD Connect rely on a proxy server to get to the required endpoints. In this case, we needed to remove proxy settings as the server running the proxy service is no longer around. I covered a few tools to demonstrate different approaches and perspectives.

Here’s a little extra before I go: Aaron Guilmette‘s Azure AD Connect Network and Name Resolution Prerequistes Test script that you can also explore. — If you are uncertain about your server’s ability to connect to Office 365 for the purposes of deploying Azure AD Connect or to local network resources for configuring a multi-forest deployment, you can attempt to use this tool to report on connectivity and name resolution success. —

Till next time…

Beystor Makoala