UTF-EBCDIC

UTF-EBCDIC is a character encoding used to represent Unicode characters. It is meant to be EBCDIC-friendly, so that legacy EBCDIC applications on mainframes may process the characters without much difficulty. Its advantages for existing EBCDIC-based systems are similar to UTF-8's advantages for existing ASCII-based systems. Details on UTF-EBCDIC are defined in Unicode Technical Report #16.

To produce the UTF-EBCDIC encoded version of a series of Unicode code points, an encoding based on UTF-8 (known in the specification as UTF-8-Mod) is applied first (creating what the specification calls an I8 sequence). The main difference between this encoding and UTF-8 is that it allows Unicode code points U+0080 through U+009F (the C1 control codes) to be represented as a single byte and therefore later mapped to corresponding EBCDIC control codes. In order to achieve this, UTF-8-Mod uses 101XXXXX instead of 10XXXXXX as the format for trailing bytes in a multi-byte sequence. As this can only hold 5 bits rather than 6, the UTF-8-Mod encoding of codepoints above U+009F is generally larger than the UTF-8 encoding.

The UTF-8-Mod transformation leaves the data in an ASCII-based format (for example, U+0041 "A" is still encoded as 01000001), so each byte is fed through a reversible (one-to-one) lookup table to produce the final UTF-EBCDIC encoding. For example, 01000001 in this table maps to 11000001; thus the UTF-EBCDIC encoding of U+0041 (Unicode's "A") is 0xC1 (EBCDIC's "A").

This encoding form is rarely used, even on the EBCDIC-based mainframes for which it was designed. IBM EBCDIC-based mainframe operating systems, such as z/OS, usually use UTF-16 for complete Unicode support. For example, DB2 UDB, COBOL, PL/I, Java and the IBM XML toolkit support UTF-16 on IBM mainframes.

Codepage layout

There are 160 characters with single-byte encodings in UTF-EBCDIC (compared to 128 in UTF-8). As can be seen, the single-byte portion is similar to IBM-1047 instead of IBM-37 due to the location of the square brackets. CCSID 37 has [] at hex BA and BB instead of at hex AD and BD respectively.

UTF-EBCDIC

	_0	_1	_2	_3	_4	_5	_6	_7	_8	_9	_A	_B	_C	_D	_E	_F
0_	NUL 0000	SOH 0001	STX 0002	ETX 0003	ST 009C	HT 0009	SSA 0086	DEL 007F	EPA 0097	RI 008D	SS2 008E	VT 000B	FF 000C	CR 000D	SO 000E	SI 000F
1_	DLE 0010	DC1 0011	DC2 0012	DC3 0013	OSC 009D	LF 000A	BS 0008	ESA 0087	CAN 0018	EM 0019	PU2 0092	SS3 008F	FS 001C	GS 001D	RS 001E	US 001F
2_	PAD 0080	HOP 0081	BPH 0082	NBH 0083	IND 0084	NEL 0085	ETB 0017	ESC 001B	HTS 0088	HTJ 0089	VTS 008A	PLD 008B	PLU 008C	ENQ 0005	ACK 0006	BEL 0007
3_	DCS 0090	PU1 0091	SYN 0016	STS 0093	CCH 0094	MW 0095	SPA 0096	EOT 0004	SOS 0098	SGCI 0099	SCI 009A	CSI 009B	DC4 0014	NAK 0015	PM 009E	SUB 001A
4_	SP 0020	• +00	• +01	• +02	• +03	• +04	• +05	• +06	• +07	• +08	• +09	. 002E	< 003C	( 0028	+ 002B	| 007C
5_	& 0026	• +0A	• +0B	• +0C	• +0D	• +0E	• +0F	• +10	• +11	• +12	! 0021	$ 0024	* 002A	) 0029	; 003B	^ 005E
6_	- 002D	/ 002F	• +13	• +14	• +15	• +16	• +17	• +18	• +19	• +1A	• +1B	, 002C	% 0025	_ 005F	> 003E	? 003F
7_	• +1C	• +1D	• +1E	• +1F	2 0000	2 0020	2 0040	2 0060	2 0080	` 0060	: 003A	# 0023	@ 0040	' 0027	= 003D	" 0022
8_	2 00A0	a 0061	b 0062	c 0063	d 0064	e 0065	f 0066	g 0067	h 0068	i 0069	2 00C0	2 00E0	2 0100	2 0120	2 0140	2 0160
9_	2 0180	j 006A	k 006B	l 006C	m 006D	n 006E	o 006F	p 0070	q 0071	r 0072	2 01A0	2 01C0	2 01E0	2 0200	2 0220	2 0240
A_	2 0260	~ 007E	s 0073	t 0074	u 0075	v 0076	w 0077	x 0078	y 0079	z 007A	2 0280	2 02A0	2 02C0	[ 005B	2 02E0	2 0300
B_	2 0320	2 0340	2 0360	2 0380	2 03A0	2 03C0	2 03E0	3 0000	3 0400	3 0800	3 0C00	3 1000	3 1400	] 005D	3 1800	3 1C00
C_	{ 007B	A 0041	B 0042	C 0043	D 0044	E 0045	F 0046	G 0047	H 0048	I 0049	3 2000	3 2400	3 2800	3 2C00	3 3000	3 3400
D_	} 007D	J 004A	K 004B	L 004C	M 004D	N 004E	O 004F	P 0050	Q 0051	R 0052	3 3800	3 3C00	4 4000	4 8000	4 10000	4 18000
E_	\ 005C	4 20000	S 0053	T 0054	U 0055	V 0056	W 0057	X 0058	Y 0059	Z 005A	4 28000	4 30000	4 38000	5 40000	5 100000
F_	0 0030	1 0031	2 0032	3 0033	4 0034	5 0035	6 0036	7 0037	8 0038	9 0039						APC 009F

  Letter   Number   Punctuation   Symbol   Other   Undefined

   Blue cells containing a large single-digit number are the start bytes for a sequence of that many bytes. The unbolded hexadecimal code point number shown in the cell is the lowest character value encoded using that start byte. This value can be greater than the value which would be obtained by following the start byte with continuation bytes which are all 65 (hex 0x41), if this would result in an invalid overlong form.

   Orange cells with one dot are continuation bytes. The hexadecimal number shown after a "+" plus sign is the value of the 5 bits they add.

   Red cells indicate start bytes (for a sequence of that many bytes) which can never appear in properly encoded UTF-EBCDIC text, because any possible continuation would result in an invalid overlong form. For example, 0x76 is marked in red because even 0x76 0x73 (which maps to the UTF-8-Mod sequence 0xC2 0xBF) would merely be an overlong encoding of U+005F (properly encoded as UTF-8-Mod 0x5F, UTF-EBCDIC 0x6D).

Oracle UTFE

Oracle UTFE is a Unicode 3.0 UTF-8 Oracle database variation, similar to the CESU-8 variant of UTF-8, where supplementary characters are encoded as two 4-byte characters rather than a single 4- or 5-byte character. It is used only on EBCDIC platforms.[1]

Advantages:

• Only Unicode character set for EBCDIC.
• Length of SQL CHAR types can be specified in number of characters.
• Binary order of the SQL CHAR columns is same as binary order of the SQL NCHAR columns if the data consists of same supplementary characters. Consequently, these columns sort the same for identical strings.[1]

Disadvantages:

• Supplementary characters occupy six bytes instead of four bytes only. Consequently, supplementary characters need to be converted.
• UTFE is not a Unicode standard encoding. Clients requiring UTF-8 encoding must convert data on retrieval and storage.[1]

See also

• UTF-1
• UTF-8
• BOCU-1

References

1. Baird, Cathy; Chiba, Dan; Chu, Winson; Fan, Jessica; Ho, Claire; Law, Simon; Lee, Geoff; Linsley, Peter; Matsuda, Keni; Oscroft, Tamzin; Takeda, Shige; Tanaka, Linus; Tozawa, Makoto; Trute, Barry; Tsujimoto, Mayumi; Wu, Ying; Yau, Michael; Yu, Tim; Wang, Chao; Wong, Simon; Zhang, Weiran; Zheng, Lei; Zhu, Yan; Moore, Valarie (2002) [1996]. "Appendix A: Locale Data". Oracle9i Database Globalization Support Guide (PDF) (Release 2 (9.2) ed.). Oracle Corporation. Oracle A96529-01. Archived (PDF) from the original on 2017-02-14. Retrieved 2017-02-14.

External links

• V.S. Umamaheswaran, Unicode Technical Report #16: the definition of UTF-EBCDIC (2002-04-16)