unichar: Use surrogate macros in ucs4 validity check

unichar: Add surrogate handling

unichar: Add uni_is_valid_ucs4

lib: Added UTF8_IS_START_SEQ() helper macro

lib: API change - have uni_utf8_get_char*() return _char_bytes Often the two functions are called in close proximity (both ways round). As _get_char*() calls _char_bytes() early on the success path, we may as well return that value to the caller for immediate use. The callers which call _char_bytes() first are simply rejecting the truncated case quickly - all other invalid cases still call both functions, and all other valid cases (which should be the fast path) likewise call both. Signed-off-by: Phil Carmody <phil@dovecot.fi>

lib: Added UNICODE_REPLACEMENT_CHAR_UTF8

lib: Fixed NUL-handling in uni_utf8_*strlen*() uni_utf8_strlen() could have skipped over the ending NUL byte and caused read buffer overflows with invalid input. uni_utf8_strlen_n() and uni_utf8_partial_strlen_n() now allow NUL bytes in the input and they're treated as regular control characters. Previously the size was actually treated as max_size with early NUL byte termination. Technically this is an API change, but I'm not aware of anything using these functions in an incompatible way.

lib: Added uni_utf8_partial_strlen_n()

Reversed recent "short utf8" changes. Solr code needs to parse the UTF8 input explicitly anyway to encode the XML characters. And all the character checks were already done in it.

liblib: Added uni_utf8_short_*() for handling UTF8 data where [56]-byte sequences are invalid.

Backported parts of normalizer_func_t changes from v2.2 tree.

Replaced "decomposed titlecase" conversions with more generic normalizer function. Plugins can now change mail_user.default_normalizer. Specific searches can also use different normalizers by changing mail_search_context.normalizer.

uni_utf8_to_decomposed_titlecase(): Require input length to be exact now. Most of the callers did that already anyway

Added uni_utf8_strlen().

liblib: Added uni_utf8_to_ucs4_n().

Added uni_utf8_data_is_valid().

Added a global utf8_replacement_char variable.

UTF-8 string validity was still checked incorrectly.

Added uni_utf8_str_is_valid(). --HG-- branch : HEAD

Added some UTF16_ macros for helping UTF-16 conversions. --HG-- branch : HEAD

Added more consts, ATTR_CONSTs and ATTR_PUREs. --HG-- branch : HEAD

Replace invalid UTF8 input with a replacement character. --HG-- branch : HEAD

Define unichars array type and use it for uni_utf8_to_ucs4() output. --HG-- branch : HEAD

uni_utf8_get_valid_data() API changed. --HG-- branch : HEAD

Moved uni_utf8_get_valid_data() to lib/ --HG-- branch : HEAD

Changed .h ifdef/defines to use <NAME>_H format. --HG-- branch : HEAD

Added uni_ucs4_to_titlecase() and uni_utf8_to_decomposed_titlecase(). They use a unicharmap.c file generated from UnicodeData.txt. --HG-- branch : HEAD

Rewrote some code and cleaned up the API --HG-- branch : HEAD

Added unichar_t UCS-4 type and some ucs4/utf8 functions. --HG-- branch : HEAD