The official identify and spelling of this encoding is UTF-8, where UTF stands for UCS Transformation Format. Please do not write UTF-eight in any documentation textual content in other ways (akin to utf8 or UTF_8), until of course you seek advice from a variable identify and never the encoding itself.
An essential note for developers of UTF-8 decoding routines: For safety reasons, a UTF-8 decoder should not accept UTF-8 sequences which are longer than essential to encode a personality. For instance, the character U+000A (line feed) have to be accepted from a UTF-eight stream solely within the type 0x0A, however not in any of the next 5 potential overlong kinds: 0xC0 0x8A 0xE0 0x80 0x8A 0xF0 0x80 0x80 0x8A 0xF8 0x80 0x80 0x80 0x8A 0xFC 0x80 0x80 0x80 0x80 0x8A Any overlong UTF-8 sequence could be abused to bypass UTF-8 substring tests that look just for the shortest possible encoding. All overlong UTF-eight sequences begin with one in all the next byte patterns: 1100000x (10xxxxxx) 11100000 100xxxxx (10xxxxxx) 11110000 1000xxxx (10xxxxxx 10xxxxxx) 11111000 10000xxx (10xxxxxx 10xxxxxx 10xxxxxx) 11111100 100000xx (10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx) Also note that the code positions U+D800 to U+DFFF (UTF-sixteen surrogates) in addition to U+FFFE and U+FFFF should not happen in regular UTF-8 or UCS-four information. UTF-eight decoders should deal with them like malformed or overlong sequences for security reasons. Markus Kuhn’s UTF-eight decoder stress test file incorporates a scientific assortment of malformed and overlong UTF-eight sequences and can assist you to to verify the robustness of your decoder. Who invented UTF-8?
The encoding recognized as we speak as UTF-eight was invented by Ken Thompson. It was born in the course of the evening hours of 1992-09-02 in a new Jersey diner, the place he designed it within the presence of Rob Pike on a placemat (see Rob Pike’s UTF-eight history). It replaced an earlier try and design a FSS/UTF (file system secure UCS transformation format) that was circulated in an X/Open working doc in August 1992 by Gary Miller (IBM), Greger Leijonhufvud and John Entenmann (SMI) as a replacement for the division-heavy UTF-1 encoding from the primary version of ISO 10646-1. By the tip of the first week of September 1992, Pike and Thompson had turned AT&T Bell Lab’s Plan 9 into the world’s first working system to make use of UTF-8. They reported about their experience at the USENIX Winter 1993 Technical Conference, San Diego, January 25-29, 1993, Proceedings, pp. 43-50. FSS/UTF was briefly also known as UTF-2 and later renamed into UTF-8, and pushed through the requirements process by the X/Open Joint Internationalization Group XOJIG. Where do I discover good UTF-8 instance information?
A number of interesting UTF-8 instance information for assessments and demonstrations are: UTF-eight Sampler internet page by the Kermit challenge Markus Kuhn’s instance plain-text recordsdata, including among others the traditional demo, decoder check, TeX repertoire, WGL4 repertoire, euro take a look at pages, and Robert Brady’s IPA lyrics. Unicode Transcriptions Generator for Indic Unicode test informationWhat totally different encodings are there?
Both the UCS and Unicode standards are to begin with massive tables that assign to each character an integer quantity. If you utilize the term “UCS”, “ISO 10646”, or “Unicode”, this simply refers to a mapping between characters and integers. This doesn't yet specify easy methods to store these integers as a sequence of bytes in memory. ISO 10646-1 defines the UCS-2 and UCS-4 encodings. These are sequences of two bytes and four bytes per character, respectively. ISO 10646 was from the beginning designed as a 31-bit character set (with attainable code positions ranging from U-00000000 to U-7FFFFFFF), nevertheless it took till 2001 for the primary characters to be assigned past the essential Multilingual Plane (BMP), that is beyond the primary 216 character positions (see ISO 10646-2 and Unicode 3.1). UCS-four can characterize all UCS and Unicode characters, UCS-2 can characterize solely those from the BMP (U+0000 to U+FFFF). “Unicode” initially implied that the encoding was UCS-2 and it initially didn’t make any provisions for characters outdoors the BMP (U+0000 to U+FFFF). When it became clear that more than 64k characters can be needed for certain particular functions (historic alphabets and ideographs, mathematical and musical typesetting, and so forth.), Unicode was turned into a sort of 21-bit character set with doable code factors within the range U-00000000 to U-0010FFFF. The 2×1024 surrogate characters (U+D800 to U+DFFF) were introduced into the BMP to allow 1024×1024 non-BMP characters to be represented as a sequence of two 16-bit surrogate characters. This fashion UTF-sixteen was born, which represents the prolonged “21-bit” Unicode in a method backwards appropriate with UCS-2. The time period UTF-32 was introduced in Unicode to describe a 4-byte encoding of the extended “21-bit” Unicode. UTF-32 is the very same factor as UCS-4, except that by definition UTF-32 is rarely used to signify characters above U-0010FFFF, while UCS-four can cover all 231 code positions as much as U-7FFFFFFF. The ISO 10646 working group has agreed to change their standard to exclude code positions past U-0010FFFF, in order to show the new UCS-4 and UTF-32 into virtually the identical thing. In addition to all that, UTF-8 was launched to provide an ASCII backwards compatible multi-byte encoding. The definitions of UTF-8 in UCS and Unicode differed originally slightly, as a result of in UCS, as much as 6-byte lengthy UTF-eight sequences had been potential to signify characters as much as U-7FFFFFFF, whereas in Unicode only as much as 4-byte lengthy UTF-8 sequences are defined to characterize characters up to U-0010FFFF. (The distinction was in essence the same as between UCS-four and UTF-32.) No endianess is implied by the encoding names UCS-2, UCS-4, UTF-16, and UTF-32, though ISO 10646-1 says that Bigendian should be most popular except in any other case agreed. It has turn out to be customary to append the letters “BE” (Bigendian, high-byte first) and “LE” (Littleendian, low-byte first) to the encoding names so as to explicitly specify a byte order. In order to allow the automated detection of the byte order, it has change into customary on some platforms (notably Win32) to begin every Unicode file with the character U+FEFF (ZERO WIDTH NO-BREAK Space), additionally recognized because the Byte-Order Mark (BOM). Its byte-swapped equal U+FFFE shouldn't be a legitimate Unicode character, therefore it helps to unambiguously distinguish the Bigendian and Littleendian variants of UTF-sixteen and UTF-32. A full featured character encoding converter will have to offer the following 13 encoding variants of Unicode and UCS: UCS-2, UCS-2BE, UCS-2LE, UCS-4, UCS-4LE, UCS-4BE, UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, UTF-32LE Where no byte order is explicitly specified, use the byte order of the CPU on which the conversion takes place and in an input stream swap the byte order each time U+FFFE is encountered. The distinction between outputting UCS-4 versus UTF-32 and UTF-16 versus UCS-2 lies in handling out-of-range characters. The fallback mechanism for non-representable characters must be activated in UTF-32 (for characters > U-0010FFFF) or UCS-2 (for characters > U+FFFF) even the place UCS-4 or UTF-16 respectively would offer a representation. Really simply of historic interest are UTF-1, UTF-7, SCSU and a dozen other less broadly publicised UCS encoding proposals with various properties, none of which ever loved any significant use. Their use should be averted. A good encoding converter may even provide options for adding or removing the BOM: - Unconditionally prefix the output text with U+FEFF. - Prefix the output textual content with U+FEFF until it's already there. - Remove the primary character whether it is U+FEFF.It has also been prompt to use the UTF-eight encoded BOM (0xEF 0xBB 0xBF) as a signature to mark the beginning of a UTF-8 file. This practice ought to positively not be used on POSIX programs for several causes: - On POSIX techniques, the locale (and not a magic file-sort code) defines the encoding of plain text recordsdata. Mixing the 2 concepts would add a number of complexity and break present functionality. - Adding a UTF-eight signature at first of a file would interfere with many established conventions such because the kernel looking for “#!” in the beginning of a plaintext executable to locate the appropriate interpreter. - Handling BOMs properly would add undesirable complexity even to simple applications like cat or grep that combine contents of several files into one.Along with the encoding alternatives, Unicode also specifies varied Normalization Forms, which give reasonable subsets of Unicode, particularly to take away encoding ambiguities attributable to the presence of precomposed and compatibility characters: Normalization Form D (NFD): Split up (decompose) precomposed characters into combining sequences where doable, e.g. use U+0041 U+0308 (LATIN CAPITAL LETTER A, COMBINING DIAERESIS) instead of U+00C4 (LATIN CAPITAL LETTER A WITH DIAERESIS). Also avoid deprecated characters, e.g. use U+0041 U+030A (LATIN CAPITAL LETTER A, COMBINING RING ABOVE) as a substitute of U+212B (ANGSTROM Sign). Normalization Form C (NFC): Use precomposed characters instead of combining sequences where doable, e.g. use U+00C4 (“Latin capital letter A with diaeresis”) as a substitute of U+0041 U+0308 (“Latin capital letter A”, “combining diaeresis”). Also keep away from deprecated characters, e.g. https://x-x-x.tube/ use U+00C5 (LATIN CAPITAL LETTER A WITH RING ABOVE) instead of U+212B (ANGSTROM Sign).NFC is the popular type for Linux and WWW. Normalization Form KD (NFKD): Like NFD, but avoid as well as using compatibility characters, e.g. use “fi” instead of U+FB01 (LATIN SMALL LIGATURE FI). Normalization Form KC (NFKC): Like NFC, but avoid in addition the usage of compatibility characters, e.g. use “fi” as an alternative of U+FB01 (LATIN SMALL LIGATURE FI).A full-featured character encoding converter should also supply conversion between normalization forms. Care must be used with mapping to NFKD or NFKC, as semantic info is likely to be lost (as an example U+00B2 (SUPERSCRIPT TWO) maps to 2) and extra mark-up information would possibly must be added to preserve it (e.g., 2 in HTML). What programming languages help Unicode?
More moderen programming languages that had been developed after around 1993 already have special knowledge types for Unicode/ISO 10646-1 characters. This is the case with Ada95, Java, TCL, Perl, Python, C# and others. ISO C ninety specifies mechanisms to handle multi-byte encoding and broad characters. These amenities had been improved with Amendment 1 to ISO C ninety in 1994 and even additional improvements had been made within the ISO C 99 commonplace. These facilities had been designed originally with varied East-Asian encodings in thoughts. They're on one aspect slightly more refined than what could be necessary to handle UCS (dealing with of “shift sequences”), but in addition lack assist for more advanced features of UCS (combining characters, etc.). UTF-8 is an example of what the ISO C commonplace calls multi-byte encoding. The kind wchar_t, which in modern environments is normally a signed 32-bit integer, can be utilized to carry Unicode characters. (Since wchar_t has ended up being a 16-bit type on some platforms and a 32-bit kind on others, additional varieties char16_t and char32_t have been proposed in ISO TR 19769 for future revisions of the C language, to present application programmers extra control over the representation of such broad strings.) Unfortunately, wchar_t was already extensively used for numerous Asian 16-bit encodings throughout the nineteen nineties. Therefore, the ISO C ninety nine normal was sure by backwards compatibility. It couldn't be modified to require wchar_t for use with UCS, like Java and Ada95 managed to do. However, the C compiler can no less than signal to an software that wchar_t is assured to carry UCS values in all locales. To do so, it defines the macro __STDC_ISO_10646__ to be an integer constant of the form yyyymmL. The yr and month seek advice from the model of ISO/IEC 10646 and its amendments that have been implemented. For instance, __STDC_ISO_10646__ == 200009L if the implementation covers ISO/IEC 10646-1:2000. How ought to Unicode be used beneath Linux?
Before UTF-8 emerged, Linux customers all around the world had to use various completely different language-specific extensions of ASCII. Most popular had been ISO 8859-1 and ISO 8859-2 in Europe, ISO 8859-7 in Greece, KOI-8 / ISO 8859-5 / CP1251 in Russia, EUC and Shift-JIS in Japan, BIG5 in Taiwan, and so forth. This made the trade of information difficult and utility software had to fret about various small differences between these encodings. Support for these encodings was often incomplete, untested, and unsatisfactory, as a result of the appliance developers rarely used all these encodings themselves. Because of these difficulties, main Linux distributors and software developers are actually phasing out these older legacy encodings in favour of UTF-8. UTF-eight support has improved dramatically over the previous couple of years and many individuals now use UTF-8 every day in - textual content information (source code, HTML information, email messages, etc.) - file names - customary input and commonplace output, pipes - surroundings variables - cut and paste choice buffers - telnet, modem, and serial port connections to terminal emulatorsand in every other places the place byte sequences used to be interpreted in ASCII. In UTF-8 mode, terminal emulators resembling xterm or the Linux console driver rework each keystroke into the corresponding UTF-eight sequence and ship it to the stdin of the foreground process. Similarly, any output of a process on stdout is distributed to the terminal emulator, where it's processed with a UTF-eight decoder after which displayed using a 16-bit font. Full Unicode functionality with all bells and whistles (e.g. high-quality typesetting of the Arabic and Indic scripts) can only be anticipated from subtle multi-lingual word-processing packages. What Linux helps right this moment on a broad base is way easier and mainly aimed toward replacing the previous 8- and 16-bit character units. Linux terminal emulators and command line instruments usually solely assist a Level 1 implementation of ISO 10646-1 (no combining characters), and solely scripts reminiscent of Latin, Greek, Cyrillic, Armenian, Georgian, CJK, and many scientific symbols are supported that need no further processing support. At this level, UCS support may be very comparable to ISO 8859 support and the only vital difference is that we have now 1000's of different characters accessible, that characters can be represented by multibyte sequences, and that ideographic Chinese/Japanese/Korean characters require two terminal character positions (double-width). Level 2 support within the type of combining characters for chosen scripts (in particular Thai) and Hangul Jamo is in components also available (i.e., some fonts, terminal emulators and editors help it by way of easy overstringing), but precomposed characters needs to be most well-liked over combining character sequences where accessible. More formally, the preferred way of encoding text in Unicode under Linux needs to be Normalization Form C as outlined in Unicode Technical Report #15. One influential non-POSIX Pc operating system vendor (whom we shall go away unnamed right here) instructed that each one Unicode files ought to begin with the character ZERO WIDTH NOBREAK Space (U+FEFF), which is on this function additionally referred to because the “signature” or “byte-order mark (BOM)”, in an effort to determine the encoding and byte-order utilized in a file. Linux/Unix doesn't use any BOMs and signatures. They would break far too many present ASCII syntax conventions (resembling scripts starting with #!). On POSIX methods, the chosen locale identifies already the encoding expected in all enter and output recordsdata of a process. (It has additionally been steered to call UTF-eight files with out a signature “UTF-8N” information, but this non-customary time period is usually not used within the POSIX world.) Before you swap to UTF-8 beneath Linux, update your set up to a recent distribution with up-to-date UTF-eight help. That is specific the case if you utilize an set up older than SuSE 9.1 or Red Hat 8.0. Before these, UTF-eight support was not yet mature sufficient to be recommendable for day by day use. Red Hat Linux 8.0 (September 2002) was the primary distribution to take the leap of switching to UTF-8 because the default encoding for most locales. The only exceptions have been Chinese/Japanese/Korean locales, for which there were on the time still too many specialised tools available that didn't but help UTF-8. This first mass deployment of UTF-8 underneath Linux brought on most remaining points to be ironed out somewhat quickly throughout 2003. SuSE Linux then switched its default locales to UTF-eight as properly, as of model 9.1 (May 2004). It was followed by Ubuntu Linux, the primary Debian-derivative that switched to UTF-eight as the system-vast default encoding. With the migration of the three hottest Linux distributions, UTF-8 associated bugs have now been fixed in virtually all properly-maintained Linux instruments. Other distributions may be expected to observe quickly. How do I've to modify my software?
If you are a developer, there are several approaches so as to add UTF-eight help. We are able to break up them into two categories, which I'll call comfortable and hard conversion. In delicate conversion, knowledge is stored in its UTF-eight type in every single place and solely very few software adjustments are mandatory. In laborious conversion, any UTF-8 knowledge that this system reads might be converted into large-character arrays and will be dealt with as such in all places inside the applying. Strings will solely be converted again to UTF-8 at output time. Internally, a personality stays a hard and fast-dimension memory object. We also can distinguish onerous-wired and locale-dependent approaches for supporting UTF-8, relying on how a lot the string processing relies on the usual library. C affords various string processing functions designed to handle arbitrary locale-particular multibyte encodings. An utility programmer who relies totally on these can stay unaware of the particular particulars of the UTF-eight encoding. Likelihood is then that by merely altering the locale setting, momxxxxxx 2021 several other multi-byte encodings (equivalent to EUC) will mechanically be supported as nicely. The other means a programmer can go is to hardcode knowledge about UTF-eight into the application. This may occasionally lead in some conditions to important performance improvements. It could also be the most effective method for functions that may solely be used with ASCII and UTF-8. Even the place support for each multi-byte encoding supported by libc is desired, it might nicely be value adding additional code optimized for UTF-8. Because of UTF-8’s self-synchronizing options, it may be processed very effectively. The locale-dependent libc