fts: Fix searching headers with TEXT/BODY TEXT is searching headers and BODY is searching MIME headers. Those headers were indexed with data language, so search must also include data language when looking up words. We'll just include the data language for all searches now, so it should always work correctly.

fts: Contexts are now required or checked Satisfied static analyzer

Updated copyright notices to include the year 2018.

Updated copyright notices to include the year 2017.

global: Fixed mismatched bool vs. int/pointer handling I don't think these fix any actual bugs.

fts: When searching data headers, don't expand search keys to all languages. For example "SEARCH HEADER Message-Id foo@example.com" works better the less expansion there is for "foo@example.com".

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

fts: Added support for per-language tokenizer settings. fts_tokenizer_<lang> now overrides fts_tokenizers setting. fts_tokenizer_<name>_<lang> now overrides fts_tokenizer_<name> setting.

fts: Remove default filters and tokenizers. There aren't any specially good defaults that work for all the languages, so it's better to just enforce explicit settings.

fts: Fixed memory leak at deinit when using multiple fts plugins at the same time using libfts.

lib-fts: fts_language_list_init() API changed to return errors.

fts: Renamed filter and tokenizer specific setting keys. Having the plural in the name didn't make a lot of sense, so all the settings are now: fts_filters = name1 name2 fts_filters_<lang> = name3 name 4 fts_filter_<name> = key1=value1 fts_filter_<lang>_<name> = key2=value2 fts_tokenizers = name1 name2 fts_tokenizer_<name> = key1=value1 So this also removes the confusion in settings if there existed a filter with one of the language names.

fts: Simplify error messages a little bit to be more consistent with other such errors. Only the setting name is really important.

fts: Use key=value instead of "key value" settings for tokenizers and filters.

fts: If fts_filters setting doesn't exist, use built-in defaults.

fts: Prefer language-specific filter settings over global fts_filters setting.

fts: Lowecase non-human language input while indexing.

fts: When tokenizing a search word, give "search" parameter to all the tokenizers.

fts: Replace '-' with '_' in tokenizer-specific fts_tokenizers_* settings

fts: Create tokenizers differently Create tokenizers earlier. Create separate tokenizers for search and indexing. Enable configuration of tokenizers. Add some helpers in fts-tokenizer.h api. Change tokenizer unit tests to match those changes. lib-fts: Refactor lib-fts settings a bit Turned address tokenizer settings into "boolean" values. Changed have_parent to "no_parent" and added "search" setting. Added documentation in fts-tokenizer.h. Change unit tests accordingly.

fts: Create filters earlier This builds on the assumption that early initialization will be made less costly, in a coming change.

fts: Do not deinit uncreated fts context Fixes crash with fts-lucene and fts-solr when lib-fts wasn't used. Patch by Teemu Huovila

fts: Fixed infinite looping at deinit. fts_mail_user_deinit() was calling itself. Now the fts backend is responsible for calling it at deinit.

fts: Don't always try to initialize lib-fts. Require backend to explictly do it now.

fts: Replace '-' with '_' in filter-specific fts_filters_* settings. So for example using fts_filters_normalizer_icu instead of fts_filters_normalizer-icu.

fts: Added FTS_BACKEND_FLAG_TOKENIZED_INPUT, which is implemented via lib-fts.