Updated copyright notices to include the year 2018.

Updated copyright notices to include the year 2017.

lib-fts: Change fts_icu_*() to use ARRAY_TYPE(icu_utf16) for UTF16 input. This makes it clearer how the API is intended to be used.

lib-fts: Change normalizer filter to use new truncate.

lib-fts: Added maxlen parameter to icu-normalizer. This is needed because the normalization can increase the token's length.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

lib-fts: Move ICU transliterator creation to fts-icu.h This allows others to use transliterators without so much code duplication. The transliterator still has to be destroyed by the user, with utrans_close().

lib-fts: fts-filter API changed to have a non-pointer vfuncs variable. The main benefit being that the fts-filter implementations can save a few lines of code.

lib-fts: Include fts-filter.h in fts-filter-private.h This avoids having to always include them both.

lib-fts: Removed unnecessary array.h include from fts-filter.h

lib-fts: ICU normalization changes some characters to spaces - remove them. We don't really want to add spaces to our index. It would be nice if the words between spaces were actually split to different tokens, but that's more of the fts-tokenizer's job and at filter stage that's probably not wanted anymore.

lib-fts: Partially reverted d097a9779c37 - don't use lib_atexit() Because fts is loaded as plugin lib_atexit() is called after the plugin is already unloaded, so it crashes.

lib-fts: Call u_clean() at deinit to free up all of libicu's memory.

lib-fts: Rewrite ICU handling functions. Some of the changes: - Use buffers instead of allocating everything from data stack. - Optimistically attempt to write the data directly to the buffers without first calculating their size. Grow the buffer if it doesn't fit first. - Use u_strFromUTF8Lenient() instead of u_strFromUTF8(). Our input is already supposed to be valid UTF-8, although we don't check if all code points are valid, while u_strFromUTF8() does check them and return failures. We don't really care about if code points are valid or not and u_strFromUTF8Lenient() passes through everything. Added unit tests to make sure all the functions work as intended and all the UTF-8 input passes through them successfully.

lib-fts: ICU normalizer code cleanup. Fold some long lines. Rename the internal struct to be more aligned with the other filters.

lib-fts: normalizer-icu no longer returns empty tokens.

lib-fts: Fixed compiling without libicu

lib-fts: Minor code cleanups

lib-fts: Removed supports() function from filter API I think the original idea was that this could be used to automatically create filters for many languages, but this probably won't be needed or wanted.

lib-fts: Removed filter name macros from fts-filter.h The filters can be directly accessed via their class structs.

lib-fts: Compiler warning fix when building without libicu

lib-fts: Changed filter's internal APIs to return error_r directly. It's not very safe to store strings allocated from data stack to any permanent structs.

fts: Improve filter error handling. In lib-fts: Move error storage to the generic filter struct level. Change make_utf8 helper to void also. In fts: Add the error argument to _filter() calls.

lib-fts: Removed make_uchar() return value. It was already de facto meaningless, so best we remove it.

lib-fts: normalizer-icu now delays initialization of libicu. This way the normalizer can be quickly created even if it's never even used.

fts: Change filter API to be able to return errors Modify fts_filter_filter() to return integer status codes. It returns 1 if a token was returned, 0 if it was filtered away and -1 on error.

lib-fts: Renamed normalizer to icu-normalizer, including the source code.