Unicode does not fit into 8 bits, not even into 16. Although only 110,116 code points are in use, it has the capability to define up to 1,114,112 of them, which would require 21 bits.

However, computers have advanced since the 1970s. An 8 bit microprocessor is a bit out of date. New computers now have 64 bit processors, so why can’t we move beyond an 8 bit character and into a 32 bit or 64 bit character?

The first answer is: we can!

A lot of software is written in C or C++, which supports a “wide character”. This is a 32 bit character called wchar_t. It is an extension of C’s 8 bit char type. Internally, modern Web browsers use these wide characters (or something similar) and can theoretically quite happily deal with over 4 billion distinct characters. This is plenty for Unicode. So –internally, modern Web browers use Unicode.

Unicode in chrome can be fickle, especially when working outside of the BMP.

So if browsers can deal with Unicode in 32 bit characters, where is the problem? The problem is in the sending and receiving, and reading and writing of characters.

The problem remains because:

1. A lot of existing software and protocols send/receive and read/write 8 bit characters
2. Using 32 bits to send/store English text would quadruple the amount of bandwidth/space required

Although browsers can deal with Unicode internally, you still have to get the data from the Web server to the Web browser and back again, and you need to save it in a file or database somewhere. So you still need a way to make 110,000 Unicode code points fit into just 8 bits.

There have been several attempts to solve this problem such as UCS2 and UTF-16. But the winner in recent years is UTF-8, which stands for Universal Character Set Transformation Format 8 bit.

UTF-8 is a clever. It works a bit like the Shift key on your keyboard. Normally when you press the H on your keyboard a lower case “h” appears on the screen. But if you press Shift first, a capital H will appear.

UTF-8 treats numbers 0-127 as ASCII, 192-247 as Shift keys, and 128-192 as the key to be shifted. For instance, characters 208 and 209 shift you into the Cyrillic range. 208 followed by 175 is character 1071, the Cyrillic Я. The exact calculation is (208%32)*64 + (175%64) = 1071. Characters 224-239 are like a double shift. 226 followed by 190 and then 128 is character 12160: ⾀. 240 and over is a triple shift.

UTF-8 is therefore a multi-byte variable-width encoding. Multi-byte because a single character like Я takes more than one byte to specify it. Variable-width because some characters like H take only 1 byte and some up to 4.

Best of all it is backward compatible with ASCII. Unlike some of the other proposed solutions, any document written only in ASCII, using only characters 0-127, is perfectly valid UTF-8 as well – which saves bandwidth and hassle.

Unicode chart

Three ways to enter Unicode characters in Windows

In Microsoft Word you can insert Unicode characters by typing the hex value of the character then typing Alt-x. You can also see the Unicode value of a character by placing the cursor immediately after the character and pressing Alt-x. This also works in applications that use the Windows rich edit control such as WordPad and Outlook.

First create a registry key under HKEY_CURRENT_USER of type REG_SZ called EnableHexNumpad, set its value to 1, and reboot. Then you can enter Unicode symbols by holding down the Alt key and typing the plus sign on the numeric keypad  followed by the character value. When you release the Alt key, the symbol will appear.

Another option is to install the UnicodeInput utility.

Inserting characters by using hexidecimal Unicode values

If you know the *Unicode (hexadecimal) value of any character, you can use the “ALT X” keyboard shortcut to enter the character directly in your document in some programs such as Microsoft Word. This is particularly useful for all special characters that are not included in the list above (such as the R-hacek  Ř (0158 ALT X )). (You must press and HOLD DOWN the “ALT” key while pressing the X)

1. Type the 4-digit Unicode (hexadecimal) value of the character, including the leading zero if applicable. (Optionally, the value string can also begin with U+)
2. Press “ALT X” – this is the ALT key and the X key at the same time.
Microsoft Word, Wordpad and some other applications will replace the string to the left of the insertion point with the character you specified.
*(Unicode: A character encoding standard developed by the Unicode Consortium. By using more than one byte to represent each character, Unicode enables almost all of the written languages in the world to be represented by using a single character set.)

http://www.alt-codes.net/

End of the alt key codes list.

If you enjoyed this excerpt, buy a copy of Unicode Explained.

http://www.oreillynet.com/pub/a/xml/excerpts/unicode-explained/tables-writing-characters.html