ICU%20Build%20/%20Installation%20Issues%20and%20Beyond

1
ICU Build / Installation Issues and Beyond

• Yves Arrouye
• ltyves_at_realnames.comgt
• Steven R. Loomis
• ltsrl_at_jtcsv.comgt

2
Topics

• Configuring and building ICU
• Integrating ICU into your product
• Porting ICU to a new platform
• Migrating an existing product to ICU
• Adding new data to ICU
• Packaging application data

3
Build Process Directory Structure

• icu
• data mostly text source files for data
• source
• common - core unicode functionality
• config build configuration files
• data data files used during the build
• extra - ustdio (file handling, used by tools)
• i18n higher level resource-based localization
• samples
• test
• cintltst (C), intltest (C)
• tools for processing and packaging data files

4
File Name Conventions

• Library
• Win32 icuuc.dll, icui18n.dll, icudata.dll
• Unix libicu-uc.so, libicu-i18n.so, libicudata.so
• (Could be .a, .sl, .x depending on platform)
• Include file
• Public APIs under unicode/ directories
• Example search path icu/source/common
• include unicode/ucnv.h

5
Dependencies

• Required
• ANSI C C compiler
• NO templates, exceptions, RTTI, STL
• Working math library
• stdio (or wrapper, especially for tools)
• Nice to have
• Dynamic loading
• Memory mapping
• Threads
• IEEE 754 numerics

6
Unix configure script

• GNU Autoconf system
• Detects platform capabilities
• Generates Makefiles
• Install prefix --prefix/usr/local
• /usr/local/lib/libicu-uc.so
• /usr/local/share/icu (data)
• /usr/local/include/unicode
• /usr/local/sbin (tools)
• /usr/local/bin (samples)

7
Unix configure options

• Things to turn off or on (use --help)
• Threads, hpuxcma, Mapped, Tests, Extras, Samples
• Out of source build (--srcdir)
• Share one read-only source base with multiple
  targets
• mkdir /build/hpux11-hppa-gcc/
• cd /build/hpux11-hppa-gcc/
• /icu/source/configure srcdir/icu/source
• Example call
• sh ./configure --enable-threadsno --prefix/opt

8
Unix Building

• gmake all all code, data, tests, tools
• gmake check run all tests
• gmake install optional install to (prefix)
• (Use just make if your make is GNU..)

9
Unix configure variables

• Shell variables
• CC/usr/opt/bin/cc
• CXX/usr/opt/bin/CC (C)
• CC and CXX can need full paths
• CPPFLAGS -Dmydefine3 -I/path
• (both C and C compiler)
• LDFLAGS-L/usr/local/lib lotherlib
• FORCE_LIBS-lm lpthread lstdc
• PATH
• sh CC/usr/opt/bin/cc ./configure

10
Unix runConfigureICU

• For a specific set of platforms
• Pre-selects compiler and options for reference
  platforms and then some.
• Use instead of configure
• sh runConfigureICU AIX4.3xlC
• Supported platforms
• AIX4.1.5xlC SOLARISCC SOLARISGCC LinuxRedHat5.2
  LinuxRedHat6.x HP/UX10.2CC HP/UX10.2ACC HP/UX11CC
  HP/UX11ACC AIX4.3xlC SOL2.7CC PTX SOLARISX86
  AIX4.3VA

11
Win32

• Use allinone project
• ICU_DATA sets target location for icudata.dll
  (use a trailing backslash \)
• (Default same as other DLL locations)
• Target Debug vs. Release
• Makedata project builds and updates ICU data
• Hit F7!

12
Running the Tests

• Set ICU_DATA (trailing slash or backslash) and
  set TZPST8PDT .. (or gmake check)
• Intltest shows errors after each level
• Intltest LIST list possible tests
• Intltest format
• Intltest rbbi/LIST
• Intltest v format
• Cintltest
• Cintltest l - list possible tests
• Cintltest /tsvconv
• Cintltest v /tsformat

13
Integrating ICU Compiling

• UNIX can include /share/icu/Makefile.inc or,
  -I(prefix)/include
• Win32 add icu\include\ to search path

14
Integrating ICU Linking

• May not need to link in i18n
• Unix use (ICULIBS) or
• licu-uc licu-i18n
• Win32 icu\lib\Debug\icuuc.lib,
  icu\lib\Debug\icui18n.lib (or Release)

15
Integrating ICU Data

• u_setDataDirectory() or ICU_DATA
• Set these before any ICU calls!
• Modes
• DLL - copy libicudata.so or icudata.dll with
  other libs
• Common set a path to dir containing icudata.dat
• Linked Link app with data dll, call
  u_setCommonData() before any ICU calls are made.
  (see udata.h )

16
Port ICU to new platform

• Keep us in the loop!
• (You will be glad, the next time you dont have
  to back-port)
• Bring up incrementally, prioritize (if needed)
• If you fix a general bug, dont ifdef it
• Test for features, not platforms (except putil.c)
• if U_HAVE_XXX instead of if MYPLATFORM
• Read and respect the ICU coding guidelines

17
Port Sane order

• Common
• umachine.h ? platform include.h
• Not needed under autoconf
• Base types (int32_t,..)
• cmemory.h, cstring.h, putil.c, umutex.c
• Platform utilities strcpy() through default time
  zone.
• putil.c has POSIX routines for many of these
  tasks
• udata.c dynamic loading, memory map..
• I18n
• samples/date test linking an executable!
• Toolutils

18
Port Sane order, contd

• Tools
• Can skip pkgdata/genccode/gencmn if need be
• Makeconv gencnval converter files
• Ustdio UFILE library, used in..
• Genrb Resource Bundle compiler
• Ctestfw used in cintltest
• Test
• Intltest C tsmthred.cpp requires threads
• Cintltest C

19
Port Sane order, 3

• Other tools (gentz, gencnval, ..)
• Data tools
• Determine data loading options (dll, memory map,
  ..)
• Go back and update udata.c if needed
• Pkgdata/genccode/gencmn
• Gotchas
• Sometimes C linkage is needed for icudata.
• Exporting symbols (see U_EXPORT macros)
• Tests sensitive to limits of floating point
  capability

20
Port keeping autoconf happy

• autoconf 2.13 depends on GNU m4
• Only edit .in files if possible!
• Add compiler choice in runConfigureICU
• Find a close mh- file in source/config, copy to
  mh-yourplatform
• Platform detection
• Aclocal.m4, config.guess, config.sub
• Custom tweaks in configure.in
• Search for pthreads, headers, math lib
• rpath separator
• Test other platforms!

21
Migrating to ICU - Context

• What does the international contexts of your
  application consist of?
• Locale
• System/Default Locale for OS interface
• Locale of an end user
• Locale of external services
• Codepage
• System/Default
• File format or communications protocol
• Internal processing
• Timezone

22
Migrating Binding

• To what are these contexts bound?
• User (may be the same as system settings)
• Connection, thread, transaction (server)
• What type of international data does my
  application need?
• Localized messages
• Collation sequences
• New converters
• See ICU Design Principles Services Guide
• Converters, Formatters,etc. are per-thread

23
Migrating Data Path

• Either widen the data path
• Unicode everywhere
• Direct manipulation easier
• Or build wrappers
• Knowledge of Unicode only at certain levels
• Pass data as opaque blobs of text elsewhere

24
Migrating Input/Output

• Application data input and output
• Conversion APIs to obtain data in Unicode, output
  in legacy codepages
• ICU locale suport to manipulate dates, times,
  currencies
• ICU collation support for all comparison, break
  iterators instead of scanning for whitespace to
  separate words and sentences.

25
Migrating - Messages

• Use Resource Bundles to externalize
  locale-depentant data such as messages.
• Use MessageFormat, ustdio
• Translate the messages.

26
Adding Data to ICU

• Locale language _ country _ variant
• Determine where data belongs in hierarchy
• Language
• Dont set specific currency (format with OK)
• Base translations country/language names, days
  of weeks, months
• Country
• Currency codes
• Formatting variations
• Variant (e.g. Euro)
• Add to data/index.txt, tools/genrb/genrbfiles.mk
  
• Reslocal.mk for temporary additions
• Use Locale Explorer for examples

27
Add Converter Alias

• Alias convrtrs.txt
• ibm-367 us-ascii MIME ANSI_X3.4-1968 IANA
  ascii Table file ibm-367.ucm
• Standard name follows alias (above us-ascii is
  the MIME name)
• Algorithmic types listed in ucnv_bld.c
• Locale argument ISO_2022,localejp
• Rebuild alias table with gencnval
• (gmake all in tools, or rebuild
  makedataproject)

28
Add Converter Table

• Format Modified IBM CDRA
• Choose a type
• SBCS single byte
• ltU0127gt \xB1 comment 0
• ltUFF01gt \x20 fallback 1
• DBCS double byte
• ltU2537gt \x43\xEC 0
• MBCS / EBCDIC_STATEFUL
• Mixed single and double byte. See documentation
• The LAST entry overrides if there is a conflict

29
Add Converter Table to Build

• Add .ucm file to tools/makeconv/ucmfiles.mk
• ucmlocal.mk for temporary additions

30
Packaging Data

• What is a package?
• 1.6 160 converters, 170 bundles 7.6Mb
• udata unified API to create, locate, and manage
  data
• Raw form of data (usable in this way)
• Single directory (no subdirs)
• Individual files (.res, .dat, .cnv, ..)
• Each file has versioning, endianness, etc..
• Created with unewdata.h (toolutils library)

31
Pkgdata Modes

• Common (icudata.dat)
• Single file, with internal table of contents
• Memory mapped
• Portable among platforms which share endianness
  and base charset (ASCII vs EBCDIC)
• Shared Library (icudata.dll)
• Data converted into static byte arrays
• One symbol per library table of contents
• Can be used with u_setCommonData() for compile
  time linkage
• DLL search path finds your data!

32
Pkgdata List File

• Manifest for package creation
• Root.resja_jp.resc\otherfiles\otherdata.dat
• Relative to s source directory

33
Pkgdata Options

• -m mode (DLL or Common)
• -O buildoptions (Unix path to Makefile.inc,
  Win32 path to ICU dir)
• -p shortname (e.g. icudata or myappdata)
• -I installdir
• Examples
• pkgdata m dll p xyz O /opt/icu/Makefile.inc
  xyz.lst
• pkgdata m dll p xyz O DE\Work\icu xyz.lst
• (Win32 Debug)

34
QA