National Language Support Guide and Reference

Using the iconv Command

Any converter installed in the system can be used through the iconv command, which uses the iconv library. The iconv command acts as a filter for converting from one code set to another. For example, the following command filters data from PC Code (IBM-850) to ISO8859-1:

cat File | iconv -f IBM-850 -t ISO8859-1 | tftp -p - host /tmp/fo

The iconv command converts the encoding of characters read from either standard input or the specified file and then writes the results to standard output.

Understanding libiconv

The iconv application programming interface (API) consists of the following subroutines that accomplish conversion:

iconv_open

Performs the initialization required to convert characters from the code set specified by the FromCode parameter to the code set specified by the ToCode parameter. The strings specified are dependent on the converters installed in the system. If initialization is successful, the converter descriptor, iconv_t, is returned in its initial state.

iconv

Invokes the converter function using the descriptor obtained from the iconv_open subroutine. The inbuf parameter points to the first character in the input buffer, and the inbytesleft parameter indicates the number of bytes to the end of the buffer being converted. The outbuf parameter points to the first available byte in the output buffer, and the outbytesleft parameter indicates the number of available bytes to the end of the buffer.

For state-dependent encoding, the subroutine is placed in its initial state by a call for which the inbuf value is a null pointer. Subsequent calls with the inbuf parameter as something other than a null pointer cause the internal state of the function to be altered as necessary.

iconv_close

Closes the conversion descriptor specified by the cd variable and makes it usable again.

In a network environment, the following factors determine how data should be converted:

• Code sets of the sender and the receiver
• Communication protocol (8-bit or 7-bit data)

The following table outlines the conversion methods and recommends how to convert data in different situations. See the Interchange Converters--7-bit and the Interchange Converters--8-bit for more information.

If the sender uses the same code set as the receiver, the following possibilities exist:

• When protocol allows 8-bit data, the data can be sent without conversions.
• When protocol allows only 7-bit data, the 8-bit code points must be mapped to 7-bit values. Use the iconv interface and one of the following methods:

  uucode	Provides the same mapping as the uuencode and uudecode commands. This is the recommended method. For more information, see Interchange Converters--uucode.
  7-bit	Converts internal code sets using 7-bit data. This method passes ASCII without any change. For more information, see Interchange Converters--7-bit.

If the sender uses a code set different from the receiver, there are two possibilities:

• When protocol allows only 7-bit data, use the fold7 method.
• When protocol allows 8-bit data and you know the receiver's code set, use the iconv interface to convert the data. If you do not know the receiver's code set, use the following method:

  8-bit

  Converts internal code sets to standard interchange formats. The 8-bit data is transmitted and the information is preserved so that the receiver can reconstruct the data in its code set. For more information, see Interchange Converters--8-bit.

Using the iconv_open Subroutine

The following examples illustrate how to use the iconv_open subroutine in different situations:

• When the sender and receiver use the same code sets, and if the protocol allows 8-bit data, you can send data without converting it. If the protocol allows only 7-bit data, do the following:

  Sender:
   cd = iconv_open("uucode", nl_langinfo(CODESET));
  
  
  Receiver:
   cd = iconv_open(nl_langinfo(CODESET), "uucode"); 

• Whne the sender and receiver use different code sets, and if the protocol allows 8-bit data and the receiver's code set is unknown, do the following:

  Sender:
   cd = iconv_open("fold8", nl_langinfo(CODESET));
  
  
  Receiver:
   cd = iconv_open(nl_langinfo(CODESET),"fold8" );

  If the protocol allows only 7-bit data, do the following:

  Sender:
   cd = iconv_open("fold7", nl_langinfo(CODESET));
  
  
  Receiver:
   cd = iconv_open(nl_langinfo(CODESET), "fold7" );

The iconv_open subroutine uses the LOCPATH environment variable to search for a converter whose name is in the following form:

iconv/FromCodeSet_ToCodeSet

The FromCodeSet string represents the sender's code set, and the ToCodeSet string represents the receiver's code set. The underscore character separates the two strings.

Because the iconv converter is a loadable object module, a different object is required when running in the 64-bit environment. In the 64-bit environment, the iconv_open routine uses the LOCPATH environment variable to search for a converter whose name is in the following form:

iconv/FromCodeSet_ToCodeSet__64.

The iconv library automatically chooses whether to load the standard converter object or the 64-bit converter object. If the iconv_open subroutine does not find the converter, it uses the from,to pair to search for a file that defines a table-driven conversion. The file contains a conversion table created by the genxlt command.

The iconvTable converter uses the LOCPATH environment variable to search for a file whose name is in the following form:

iconvTable/FromCodeSet_ToCodeSet

If the converter is found, it performs a load operation and is initialized. The converter descriptor, iconv_t, is returned in its initial state.

Converter Programs versus Tables

Converter programs are executable functions that convert data according to a set of rules. Converter tables are single-byte conversion tables that perform stateless conversions. Programs and tables are in separate directories, as follows:

After a converter program is compiled and linked with the libiconv.a library, the program is placed in the /usr/lib/nls/loc/iconv directory.

To build a table converter, build a source converter table file. Use the genxlt command to compile translation tables into a format understood by the table converter. The output file is then placed in the /usr/lib/nls/loc/iconvTable directory.

Unicode and Universal Converters

Unicode (or UCS-2) conversion tables are found in:

$LOCPATH/uconvTable/*CodeSet*

The $LOCPATH/uconv/UCSTBL converter program is used to perform the conversion to and from UCS-2 using the iconv utilities.

A Universal converter program is provided that can be used to convert between any two code sets whose conversions to and from UCS-2 is defined. Given the following uconv tables:

X     -> UCS-2
UCS-2 -> Y

a universal conversion can be defined that maps the following:

X -> UCS-2 -> Y

by use of the $LOCPATH/iconv/Universal_UCS_Conv.

Universal UCS Converter

UCS-2 is a universal 16-bit encoding that can be used as an interchange medium to provide conversion capability between virtually any code sets. The conversion can be accomplished using the Universal UCS Converter, which converts between any two code sets XXX and YYY as follows:

XXX <-> UTF-32 <-> YYY

The XXX and YYY conversions must be included in the supported List of UCS-2 Interchange Converters, and must be installed on the system.

The universal converter is installed as the file /usr/lib/nls/loc/iconv/Universal_UCS_Conv.

The conversion between multibyte and wide character code depends on the current locale setting. Do not exchange wide character codes between two processes, unless you have knowledge that each locale that might be used handles wide character codes in a consistent fashion. Most locales for this operating system use the Unicode character value as a wide character code, except locales based on IBM-eucTW codesets.

Using Converters

The iconv interface is a set of the following subroutines used to open, perform, and close conversions:

• iconv_open
• iconv
• iconv_close

Code Set Conversion Filter Example

The following example shows how you can use these subroutines to create a code set conversion filter that accepts the ToCode and FromCode parameters as input arguments:

#include <stdio.h>
#include <nl_types.h>
#include <iconv.h>
#include <string.h>
#include <errno.h>
#include <locale.h>

#define ICONV_DONE() (r>=0)
#define ICONV_INVAL() (r<0) && (errno==EILSEQ))
#define ICONV_OVER() (r<0) && (errno==E2BIG))
#define ICONV_TRUNC() (r<0) && (errno==EINVAL))

#define USAGE 1
#define ERROR 2
#define INCOMP 3

char ibuf[BUFSIZ], obuf[BUFSIZ];

extern int errno;

main (argc,argv)
int argc;
char **argv;
{

 size_t  ileft,oleft;
 nl_catd catd;
 iconv_t cd;
 int r;
 char *ip,*op;

 setlocale(LC_ALL,"");
 catd = catopen (argv[0],0);

 if(argc!=3){
  fprintf(stderr,
   catgets (catd,NL_SETD,USAGE,"usage;conv fromcode tocode\n"));
  exit(1);
 }

 cd=iconv_open(argv[2],argv[1]);

ileft=0;

while(!feof(stdin)) {

 /*
 * After the next operation,ibuf will
 * contain new data plus any truncated
 * data left from the previous read.
 */
 ileft+=fread(ibuf+ileft,1,BUFSIZ-ileft,stdin);
 do {
  ip=ibuf;
  op=obuf;
  oleft=BUFSIZ;

  r=iconv(cd,&ip,&ileft,&op,&oleft);

  if(ICONV_INVAL()){
   fprintf(stderr,
      catgets(catd,NL_SETD,ERROR,"invalid input\n"));
   exit(2);
 }

 fwrite(obuf,1,BUFSIZ-oleft,stdout);

 if(ICONV_TRUNC() || ICONV_OVER())
  /*
  *Data remaining in buffer-copy
  *it to the beginning
  */

  memcpy(ibuf,ip,ileft);

  /*
  *loop until all characters in the input
  *buffer have been converted.
  */
 } while(ICONV_OVER());
}

 if(ileft!=0){
  /*
  *This can only happen if the last call
  *to iconv() returned ICONV_TRUNC, meaning
  *the last data in the input stream was
  *incomplete.
  */
 fprintf(stderr,catgets(catd,NL_SETD,INCOMP,"input incomplete\n"));
 exit(3);
 }

 iconv_close(cd);
 exit(0);
}

Naming Converters

Code set names are in the form CodesetRegistry-CodesetEncoding where:

The from,to variable used by the iconv command and iconv_open subroutine identifies a file whose name should be in the form /usr/lib/nls/loc/iconv/%f_%t or /usr/lib/nls/loc/iconvTable/%f_%t, where:

List of Converters

Converters change data from one code set to another. The sets of converters supported with the iconv library are listed in the following sections. All converters shipped with the BOS Runtime Environment are located in the /usr/lib/nls/loc/iconv/* or /usr/lib/nls/loc/iconvTable/* directory.

These directories also contain private converters; that is, they are used by other converters. However, users and programs should only depend on the converters in the following lists.

Any converter shipped with the BOS Runtime Environment and not listed here should be considered private and subject to change or deletion. Converters supplied by other products can be placed in the /usr/lib/nls/loc/iconv/* or /usr/lib/nls/loc/iconvTable/* directory.

Programmers are encouraged to use registered code set names or code set names associated with an application. The X Consortium maintains a registry of code set names for reference. See Code Sets for National Language Support for more information about code sets.

PC, ISO, and EBCDIC Code Set Converters

These converters provide conversion between PC, ISO, and EBCDIC single-byte stateless code sets. The following types of conversions are supported: PC to/from ISO, PC to/from EBCDIC, and ISO to/from EBCDIC.

Conversion is provided between compatible code sets such as Latin-1 to Latin-1 and Greek to Greek. However, conversion between different EBCDIC national code sets is not supported. For information about converting between incompatible character sets, refer to the Interchange Converters--7-bit and the Interchange Converters--8-bit.

Conversion tables in the iconvTable directory are created by the genxlt command.

Compatible Code Set Names

The following table lists code set names that are compatible. Each line defines to/from strings that may be used when requesting a converter.

Files

The following table describes the inconvTable converters found in the /usr/lib/nls/loc/iconvTable directory:

Multibyte Code Set Converters

Multibyte code-set converters convert characters among the following code sets:

• PC multibyte code sets
• EUC multibyte code sets (ISO-based)
• EBCDIC multibyte code sets

The following table lists code set names that are compatible. Each line defines to/from strings that may be used when requesting a converter.

1. Conversions between Simplified and Traditional Chinese are provided (IBM-eucTW <--> IBM-eucCN and big5 <--> IBM-eucCN).
2. UTF-8 is an additional code set. See UTF-8 Interchange Converters for more information.

Files

The following list describes the Multibyte Code Set converters that are found in the /usr/lib/nls/loc/iconv directory.

Interchange Converters--7-bit

This converter provides conversion between internal code and 7-bit standard interchange formats (fold7). The fold7 name identifies encodings that can be used to pass text data through 7-bit mail protocols. The encodings are based on ISO2022. For more information about fold7, see Understanding libiconv.

The fold7 converters convert characters from a code set to a canonical 7-bit encoding that identifies each character. This type of conversion is useful in networks where clients communicate with different code sets but use the same character sets. For example:

The following escape sequences designate standard code sets:

When converting from a code set to fold7, the escape sequence used to designate the code set is chosen according to the order listed. For example, the JISX0208.1983-0 characters use 01/11 01/04 04/02 as the designation.

Files

The following list describes the fold7 converters that are found in the /usr/lib/nls/loc/iconv directory:

Interchange Converters--8-bit

This converter provides conversions between internal code and 8-bit standard interchange formats (fold8). The fold8 name identifies encodings that can be used to pass text data through 8-bit mail protocols. The encodings are based on ISO2022. For more information about fold8, see Understanding libiconv.

The fold8 converters convert characters from a specific code set encoding to a canonical 8-bit encoding that identifies each character. This type of conversion is useful in networks where clients communicate with different code sets but use the same character sets. For example:

The following escape sequences designate standard code sets.

When converting from a code set to fold8, the escape sequence used to designate the code set is chosen according to the order listed. For example, the JISX0208.1983-0 characters use 01/11 02/04 02/08 04/02 as the designation.

Files

The following list describes the fold8 converters found in the /usr/lib/nls/loc/iconv directory:

Interchange Converters--Compound Text

Compound text interchange converters convert between compound text and internal code sets.

Compound text is an interchange encoding defined by the X Consortium. It is used to communicate text between X clients. Compound text is based on ISO2022 and can encode most character sets using standard escape sequences. It also provides extensions for encoding private character sets. The supported code sets provide a converter to and from compound text. The name used to identify the compound text encoding is ct.

The following escape sequences are used to designate standard code sets in the order listed below.

01/11 02/05 02/15 03/01 M L 04/09 04/02 04/13 02/13 03/08 03/05 03/00 00/02

GR right half of IBM-850 unique characters. Characters common to ISO8859-1 should not use this escape sequence.

01/11 02/05 02/15 03/02 M L 04/09 04/02 04/13 02/13 07/05 06/04 06/03 04/10 05/00 00/02

GR right half of Japanese user-definable characters.

01/11 02/05 02/15 03/01 M L 06/09 06/02 06/13 02/13 03/08 03/05 03/00 00/02

GL right half of IBM-850 unique characters. Characters common to ISO8859-1 do not use this escape sequence.

01/11 02/05 02/15 03/02 M L 06/09 06/02 06/13 02/13 07/05 06/04 06/03 04/10 05/00 00/02

GL Japanese (IBM-udcJP) user-definable characters.

Files

The following list describes the compound text converters that are found in the /usr/lib/nls/loc/iconv directory:

Interchange Converters--uucode

This converter provides the same mapping as the uuencode and uudecode commands.

During conversion from uucode, 62 bytes at a time (including a new-line character trailing the record) are converted, and generating 45 bytes in outbuf.

Files

The following list describes the uucode converters found in the /usr/lib/nls/loc/iconv directory:

UCS-2 Interchange Converters

UCS-2 uses a universal 16-bit encoding. Conversions for each code set are provided in both directions, between the code set and UCS-2. For more information, see Code Sets for National Language Support.

UCS-2 converters are found in /usr/lib/nls/loc/uconvTable and /usr/lib/nls/loc/uconv directories. The uconvdef command is used to generate new converters or to customize existing UCS-2 converters.

UTF-8 Interchange Converters

UTF-8 is a universal, multibyte encoding described in the UCS-2 and UTF-8. Conversions for each code set are provided in both directions, between the code set and UTF-8.

UTF-8 conversions are usually done by using the Universal_UCS_Conv and /usr/lib/nls/loc/uconv/UTF-8 converter. For more information, see UCS-2 Interchange Converters.

Miscellaneous Converters

A set of low-level converters used by the code set and interchange converters is provided. These converters are called miscellaneous converters. These low-level converters may be used by some of the interchange converters. However, the use of these converters is discouraged because they are intended for support of other converters.

Files

The following list describes the miscellaneous converters found in the /usr/lib/nls/loc/iconv and /usr/lib/nls/loc/iconvTable directories: