From 9b14a0bf760b0cc08b1a134c21ced70e27dbf440 Mon Sep 17 00:00:00 2001 From: Henri Sivonen Date: Tue, 21 May 2019 12:06:27 +0300 Subject: [PATCH] Document how to minimize unmappables when using the encoders. --- README.md | 12 ++++++++++++ src/lib.rs | 12 ++++++++++++ 2 files changed, 24 insertions(+) diff --git a/README.md b/README.md index 095a921..d40f1b3 100644 --- a/README.md +++ b/README.md @@ -86,6 +86,18 @@ one directly. (It wraps this crate and adds UTF-7 decoding.) For mappings to and from Windows code page identifiers, use the [`codepage`](https://crates.io/crates/codepage) crate. +## Preparing Text for the Encoders + +Normalizing text into Unicode Normalization Form C prior to encoding text into +a legacy encoding minimizes unmappable characters. Text can be normalized to +Unicode Normalization Form C using the +[`unic-normal`](https://crates.io/crates/unic-normal) crate. + +The exception is windows-1258, which after normalizing to Unicode Normalization +Form C requires tone marks to be decomposed in order to minimize unmappable +characters. Vietnamese tone marks can be decomposed using the +[`detone`](https://crates.io/crates/detone) crate. + ## Licensing Please see the file named diff --git a/src/lib.rs b/src/lib.rs index 56fbd2e..a978a8b 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -248,6 +248,18 @@ //! [charset](https://crates.io/crates/charset) wraps encoding_rs and adds //! UTF-7 decoding for email purposes. //! +//! # Preparing Text for the Encoders +//! +//! Normalizing text into Unicode Normalization Form C prior to encoding text +//! into a legacy encoding minimizes unmappable characters. Text can be +//! normalized to Unicode Normalization Form C using the +//! [`unic-normal`](https://crates.io/crates/unic-normal) crate. +//! +//! The exception is windows-1258, which after normalizing to Unicode +//! Normalization Form C requires tone marks to be decomposed in order to +//! minimize unmappable characters. Vietnamese tone marks can be decomposed +//! using the [`detone`](https://crates.io/crates/detone) crate. +//! //! # Streaming & Non-Streaming; Rust & C/C++ //! //! The API in Rust has two modes of operation: streaming and non-streaming.