Troubleshooting Invisible / Hidden Characters in your PowerShell Code…

Recently I was helping a colleague with a PowerShell script, things were going great until we ran the code and nothing was outputting. Now this wouldn’t normally be a problem but we knew there should be output. So we started troubleshooting our code…

For this blog I’m going to use some really basic examples but you will get the idea…

The following two blocks of Code look identical visually, but under the covers lurks an invisible foe.

Code Block #1: (Bad Character Code)

$ServiceState = "Running"
Write-Host "Getting Services with State: $ServiceState"
$Services = Get-Service | Where-Object {$_.Status -eq "Running"}

ForEach ($Service in $Services){
    Write-Host "$($Service.Name)"
}

Code Block #2: (Good Code)

$ServiceState = "Running"
Write-Host "Getting Services with State: $ServiceState"
$Services = Get-Service | Where-Object {$_.Status -eq "Running"}

ForEach ($Service in $Services){
    Write-Host "$($Service.Name)"
}

Now, when we run this code we should receive a list of all the Services on the local machine which are in a Running State like this…

However, this is what we recieved.

Now, first reaction was that we had a logic error in the code, but as you can see this is not the case.

Next thought was that we had a “Bad Character” in the script, as I have seen “Bad Characters” interfere with scripts in the past a quick solution to this was to drop the code into Notepad, and copy it back. Still no luck…

So I manually retyped line 3 of the Script as this is where we suspected the problem to reside since $Services was returning a $Null.

$Services = Get-Service | Where-Object {$_.Status -eq "Running"}

Ran the script and success, now I was confident that we had a Bad Character in the Script, but this left me scratching my head as to why the “Notepad” trick didn’t work. In the past this has typically removed hidden characters, but it would appear that in Windows 10 (1909) where I am testing this that this does not work anymore, speculation that since Notepad can do more formatting we can no longer rely on it to strip these characters out anymore.

So I started looking for a solution to help me in the future so I don’t waste cycles troubleshooting a syntax error I can’t see. Fortunately Visual Studio Code (if you aren’t using this to code PowerShell you really should take a look.) has extensions that you can use to assist with all sort of items, including finding and Highlighting Bad Characters. The extension I’m going to demonstrate is Highlight Bad Chars by Kevin Wenger.

Once we find the extension in Visual Code Extension Market, we can install it.

Now, with this extension installed we can now visually see the difference in the code.

Now you might be asking, what character was this? Well I was curious as well, so I did the following:

I isolated the highlighted character, and dumped the ASCII Code for it. Putting “ABC” in front and behind the “Hidden” allowed me to identify the hidden Character.

A = 65
B = 66
C = 67

Anything other than those codes, would be the offending item.

Code Block #3: (Isolate Bad Character)

"ABC ABC".ToCharArray() | %{[int][char]$_}

Looks like I have two offending characters, 194 and 160.

So, now we know exactly why this happened, and what we can do to assist in identifying these types of issues sooner so we don’t waste as much time looking for a “Hidden” foe.

The Hidden Characters are still there in Code Block #1: (Bad Character Code) and Code Block #3: (Isolate Bad Character) if you want to cut and paste them into your Editor and try this out for your self. This is why it is incredibly important to know what you are cutting and pasting as these hidden characters come along for the ride.

I would like to thank my colleague Courtney Marr from Long View Systems for letting me help him with this head scratcher…

Chris.