You know the problem, one you've probably seen before, where you open a document and for some reason some of the symbols have morphed into something much less intelligble. For example, you're reading a document and suddenly you see "on‑premise" and you have to decipher what was missing. Normally you can pick it up from context, but if it's incredibly important that you transcribe the correct character, I can walk you through correcting those mistakes.

## Simple Solution

The easiest way to solve this is to try and force the document to open in [UTF-8 encoding](https://en.wikipedia.org/wiki/UTF-8) which 9 times out of 10 will solve your problem. It depends on the programs you're using and what you can install, but when you do this you should see everything returned to it's proper character.

## Manual Process

If you can't change the encoding in the program you're using (maybe it's not available or IT has locked out that feature) you can usually solve it with some creative online researching. This process involves a few different steps, so let's get started.

### Determine Problematic Encoding

First we need to figure out how the document is currently being read. Given this issue crops up a lot on Windows machines, it's safe to assume that it's usually a problem with [Windows-1252](https://en.wikipedia.org/wiki/Windows-1252) encoding which often gets referred to as ANSI encoding. In the cases where it's not, it's probably [ISO/IEC 8859-1](https://en.wikipedia.org/wiki/ISO/IEC_8859-1) or [ISO/IEC-8859-15](https://en.wikipedia.org/wiki/ISO/IEC_8859-15) which are slightly different but can be solved using a similar method.

The way to determine exactly what encoding is to look at the characters you can see. Usually there will be exactly three of them (to match the three bytes in UTF-8 encoding) and you'll need to look very closely. Using the earlier example we have `‑` which if you look closely is an `a` with a circumflex above it, a Euro symbol, and an open single-quotation mark (not a standard apostrophe!). That last character is found in the ANSI encoding set but not the two others we have, so by the process of elimination we know we're dealing with ANSI/Windows-1252.

### Determine Hex Values

Now we need to determine the hex values of those characters. Since each character is one byte long, the hex value will be exactly 2 digits, each from 0-F (if you're unfamiliar, hex goes 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F because it is hexadecimal, a base-16 counting system). You can use the tables on the wiki pages to determin, by finding the character in the [Character Set](https://en.wikipedia.org/wiki/Windows-1252#Character_set) section and then looking at the row heading for the first digit and the column heading for the second digit.

In this case we start with `â` which is in row `E_` and column `_2` and thus it's hex value is `E2`.
Next we have `€` which is in row `8_` and column `_0` and thus is `80`.
Lastly `‘` is in `9_` and `_1` or `91`.

### Determine Unicode Character.

There's a wonderful page that lists all the unicode characters and their descriptive names that you can find [here](https://www.utf8-chartable.de/unicode-utf8-table.pl). It uses a table system and we just need to work out how our bytes translate. On this page make sure you set the "display format for UTF-8 encoding" setting to "hex". Now we need to find the right initial character, you'll notice in the UTF-8 column on the first page that it shows a single UTF-8 hex value, so we need to go much higher. In the "go to other block" select box, scroll down to "U+08A0... U+08FF: Arabic Extended-A", this should reload the page and we'll see that we're into UTF-8 characters with three hex bytes and they start with `e0`. We're looking for `e2`, so we probably still need to go a bit further. In the "go to other block" section change to "U+2000... U+206F: General Punctuation". You'll notice we now start with `e2 80` in the hex section which is really close. Scroll down and you'll see `e2 80 91` is `‑`, also known as `NON-BREAKING HYPHEN`. We've found the character that had issues being encoded!

If you're concerned that there's so many Unicode pages and how do you find the right one so quickly aside from enumerating them in order until you hit the right bytes, well you can also use some assumptions. I knew it was probably some form of punctuation given the type of sentence, and probably a type of hyphen or dash, thus I thought it would be in something to do with punctuation or diacritical marks, which narrows down where I needed to look.

Good luck and happy decoding!