Unicode converter

Unicode gives every character a number, its code point, written U+ and at least four hex digits. There are 1,114,112 of them, from U+0000 to U+10FFFF. Encodings decide how those numbers become bytes. Here are the same four characters in each:

UTF-8 uses 1 to 4 bytes per character and is the same as ASCII for the first 128, which is why it is the default for the web, JSON and most files. UTF-16 uses one 16-bit unit for characters up to U+FFFF and two (a surrogate pair from the range D800-DFFF) for the rest. JavaScript, Java, C# and Windows store strings as UTF-16, which is where escapes like \uD83D\uDE00 for 😀 come from.

The first byte says how many bytes follow; each following byte starts with the bits 10 and carries 6 bits of the code point.

Example: é is U+00E9, binary 00011 101001. Into the 2-byte pattern that gives 11000011 10101001, or C3 A9. When that pair is read as Latin-1 or Windows-1252 instead of UTF-8, you see é, the classic mojibake. The bytes C0, C1 and F5-FF never appear in valid UTF-8, and surrogates (U+D800-DFFF) may not be encoded at all. The decoder above reports each of these with its byte position.

A CSS escape takes up to six hex digits and ends at the first character that is not hex. If the next character is a hex digit or a space, add a space after the escape: \E9 b. The converter does this for you. The Unicode → text mode reads all of these, mixed into ordinary text, and joins surrogate pairs.

// JavaScript
'😀'.codePointAt(0).toString(16)        // '1f600'
String.fromCodePoint(0x1F600)            // '😀'
[...new TextEncoder().encode('é')]       // [195, 169]

# Python
hex(ord('😀'))                           # '0x1f600'
'é'.encode('utf-8')                       # b'\xc3\xa9'
b'\xf0\x9f\x98\x80'.decode('utf-8')         # '😀'

# Terminal
printf '😀' | xxd -p                      # f09f9880
printf '\xf0\x9f\x98\x80\n'              # 😀

Some characters take no space on screen but still change the string, so ===, searches and database lookups fail for no visible reason. The usual suspects:

Bidirectional overrides can make source code display in a different order from how the compiler reads it (CVE-2021-42574, "Trojan Source"). Lookalikes are the other trap: pаypal.com with a Cyrillic а is a different domain. Unicode's UTS #39 lists thousands of confusable pairs; the tool flags the common Cyrillic and Greek ones plus styled letters (fullwidth, bold or script math letters) that NFKC normalization turns back into ASCII.

"How long is this string" has four answers, and each language picks a different default:

Database column limits, API size limits and token counts each use one of these, so check which before you truncate text. The same letter can also be stored two ways: é as one code point (U+00E9, NFC) or as e plus a combining accent (U+0065 U+0301, NFD). They look identical but are not equal until you normalize with s.normalize('NFC'); the converter tells you when text is not NFC.

Related tools: escape strings for JSON, HTML, SQL and shell, decode URLs, and Base64. Working on app security? See security tools on DevHunt.