fts: Fix searching SEARCH_HEADER_ADDRESS/COMPRESS_LWSP These are "non-language" headers that are being searched, so they need to be searched using data language.

Updated copyright notices to include the year 2018.

global: Add/change switch case "fall through" comments These fix warnings with gcc 7's -Wimplicit-fallthrough

fts: Fix search query generation when tokens are filtered out. For example with English stopwords when searching "foo and bar" it should be equilevant to "foo bar", since "and" is filtered out.

Updated copyright notices to include the year 2017.

global: Change string position/length from unsigned int to size_t Mainly to avoid truncating >4GB strings, which might potentially cause some security holes. Normally there are other limits, which prevent such excessive strings from being created in the first place. I'm sure this didn't find everything. Maybe everything could be found with compiler warnings. -Wconversion kind of does it, but it gives way too many unnecessary warnings. These were mainly found with: grep " = strlen" egrep "unsigned int.*(size|len)"

fts: When searching data headers, don't expand search keys to all languages. For example "SEARCH HEADER Message-Id foo@example.com" works better the less expansion there is for "foo@example.com".

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

fts: Fixed searching for HEADER "" with lib-fts

fts: Fixed FTS search query generation after earlier per-language tokenizer changes.

fts: Error handling fix for previous per-language tokenizer change.

fts: Added support for per-language tokenizer settings. fts_tokenizer_<lang> now overrides fts_tokenizers setting. fts_tokenizer_<name>_<lang> now overrides fts_tokenizer_<name> setting.

fts: Avoid expanding mail_search_args multiple times (for optimization & crash-avoidance) Primarily this fixes the crash: index-search-result.c: line 132 (index_search_result_update_flags): assertion failed: (result->search_args->args == &search_arg) It could be triggered by: a search return (update) body body seen b store 1 +flags \seen c store 1 -flags \seen

fts + lib-fts: Fixed crash with some search queries that contained uninitialized search args.

lib-fts: Changed fts_tokenizer_next/final() to return error string. The current tokenizers can't fail, but if we're doing tokenization via external services they could fail.

lib-fts: Added fts_tokenizer_final() as a convenience wrapper.

fts: Reset tokenizers before using them This is mainly needed if the previous tokenization had failed.

fts: Don't crash if search arg string expands to empty token list.

fts: If filtering fails during search arg expansion, fail the fts search.

fts: Improve filter error handling. In lib-fts: Move error storage to the generic filter struct level. Change make_utf8 helper to void also. In fts: Add the error argument to _filter() calls.

fts: Change filter API to be able to return errors Modify fts_filter_filter() to return integer status codes. It returns 1 if a token was returned, 0 if it was filtered away and -1 on error.

fts: Change tokenizer API to be able to return errors Modify fts_tokenizer_next() to return integer status codes. It returns 1 if a token was returned in *token_r, 0 if more input is needed and -1 on error.

fts: Create tokenizers differently Create tokenizers earlier. Create separate tokenizers for search and indexing. Enable configuration of tokenizers. Add some helpers in fts-tokenizer.h api. Change tokenizer unit tests to match those changes. lib-fts: Refactor lib-fts settings a bit Turned address tokenizer settings into "boolean" values. Changed have_parent to "no_parent" and added "search" setting. Added documentation in fts-tokenizer.h. Change unit tests accordingly.

fts: Create filters earlier This builds on the assumption that early initialization will be made less costly, in a coming change.

fts: Fixed handling NOT when using lib-fts

fts: Added FTS_BACKEND_FLAG_TOKENIZED_INPUT, which is implemented via lib-fts.