Updated copyright notices to include the year 2018.

lib-fts: Fix test-fts-filter to use valid unicode

Updated copyright notices to include the year 2017.

global: Use const for all test_functions[] arrays

global: Use "static const" for all struct tests[] arrays

lib-fts: Change normalizer filter to use new truncate.

lib-fts: Cut overlong strings in lowercase filter. Added new common truncate function for filters. It also removes any partial characters, that would remain from plain truncation.

lib-fts: Added maxlen parameter to icu-normalizer. This is needed because the normalization can increase the token's length.

lib-fts: Improved stopword file reading. The reading tries to be a little bit stricter now. Only stopwords at the start of a new line are accepted now. Changed fi stopwords accordingly. Also removed superfluous stack allocation in parsing.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

lib-fts: Fix compilation for systems without libicu. The earlier patch, 194e3622d5e6, did not consider both library dependencies.

lib-fts: Fix compilation for systems without libstemmer.

lib-fts: Silence clang warnings.

lib-fts: Add Norwegian. Norwegian has two main dialects, Bokmal(nb) and Nynorsk(nn). They are detected separately by libexttextcat, but the stemmer only knows Norwegian. Thus they are treated as a single language, Norwegian (no). This might also make more sense in everyday use of mixed writing style Norwegian. Caveat: The default normalizer filter does not modify U+00F8 (Latin Small Letter O with Stroke). In some configurations it might be desirable to rewrite it to e.g. o. Same goes for the upper case version. This can be done by passing a modified "id" setting to the normalizer filter.

lib-fts: Add Swedish (sv) to supported languages.

lib-fts: Add l' contraction to French unit test.

various - remove 8-bit characters from literal strings in test cases C has a portable way of expressing characters not in the basic character set, namely \xNN escaping. Otherwise, the interpretation of the raw utf-8 is implentation dependent. This has the benefit of making some tests' expected output more obvious, such as "=c3=a4" matching "\xC3\xA4", even if it hinders the readability of some natural-language-based tests. Signed-off-by: Phil Carmody <phil@dovecot.fi>

lib-fts: Add missing "j'" to French contractions.

lib-fts: Add UTF-8 unit test for lowercase filter.

lib-fts: Add prefixing contraction filter. Filters away prefixing contracted words, e.g. "l'homme" -> "homme". Tokens to be filtered must be lower case. Only supports French in this initial version.

lib-fts: Removed unnecessary data stack frames from test-fts-filter. They are already added by lib-test/test-common.c

lib-fts: Added "english-possessive" filter.

lib-fts: ICU normalization changes some characters to spaces - remove them. We don't really want to add spaces to our index. It would be nice if the words between spaces were actually split to different tokens, but that's more of the fts-tokenizer's job and at filter stage that's probably not wanted anymore.

lib-fts: Fixed memory leaks in test-fts-filter unit test

lib-fts: Rewrite ICU handling functions. Some of the changes: - Use buffers instead of allocating everything from data stack. - Optimistically attempt to write the data directly to the buffers without first calculating their size. Grow the buffer if it doesn't fit first. - Use u_strFromUTF8Lenient() instead of u_strFromUTF8(). Our input is already supposed to be valid UTF-8, although we don't check if all code points are valid, while u_strFromUTF8() does check them and return failures. We don't really care about if code points are valid or not and u_strFromUTF8Lenient() passes through everything. Added unit tests to make sure all the functions work as intended and all the UTF-8 input passes through them successfully.

lib-fts: Fixed test-fts-filter when building without libstemmer

lib-fts: Fixed/improved test-fts-filter unit tests for previous changes.

lib-fts: Removed filter name macros from fts-filter.h The filters can be directly accessed via their class structs.

lib-fts: Added "lowercase" filter. For now it handles only ASCII characters, but that's enough for our use.

lib-fts: Minor unit test cleanups

lib-fts: Minor unit test fix - use case sensitive checks for word comparison.

lib-fts: Assume that fts_language.name is always lowercased and has two letters. Anything else just unnecessarily complicates the code.

lib-fts: Fixed test-fts-filter unit test to handle errors a bit better.

lib-fts: Delay stopwords filter full initialization until it's needed.

fts: Improve filter error handling. In lib-fts: Move error storage to the generic filter struct level. Change make_utf8 helper to void also. In fts: Add the error argument to _filter() calls.

lib-fts: normalizer-icu now delays initialization of libicu. This way the normalizer can be quickly created even if it's never even used.

fts: Change filter API to be able to return errors Modify fts_filter_filter() to return integer status codes. It returns 1 if a token was returned, 0 if it was filtered away and -1 on error.

lib-fts: Fixed test-fts-filter to work again after previous normalizer changes.

lib-fts: Don't crash in test-fts-filter if libstemmer support isn't built in

lib-fts: Fixed some wrong macro name checks. Forgot to change these ones while renaming them..

Initial import for lib-fts. Parts of what this code does was already implemented internally by fts-lucene. lib-fts is intended to be usable for all the FTS backends. The APIs are still going to change a bit, but hopefully not after v2.2.17 release. Mostly written by Teemu Huovila.