Updated copyright notices to include the year 2018.

spelling fixes

Updated copyright notices to include the year 2017.

lib-fts: Lift helper function out of generic tokenizer.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

lib-fts: Removed TODO comment. The call to uni_utf8_get_char_n() already does easy cases first. Adding more special code paths is not necessary.

lib-fts: Minor code cleanup - Rename some internal functions. Maybe the names are more logical this way.

lib-fts: Add Unicode TR29 rule WB5a setting to tokenizer. Splits prefixing contracted words from base word. E.g. "l'homme" -> "l" "homme". Together with a language specific stopword list unnecessary contractions can thus be filtered away. This is disabled by default and only works with the TR29 algorithm. Enable by "fts_tokenizer_generic = algorithm=tr29 wb5a=yes"

lib-fts: Update comment on tr29 rules.

lib-fts: Use UTF8_IS_START_SEQ()

lib-fts: Moved IS_APOSTROPHE() to fts-common.h

lib-fts: Optimized truncation of partial trailing UTF-8 characters in tokenizers.

lib-fts: Fixed tr29 tokenizer to delete last character correctly when it's preceded by non-ASCII

lib-fts: Use new uni_utf8_get_char*() interface

lib-fts: tokenizers - Fixed removal of trailing character in truncated tokens. If the token is truncated, we don't want to remove the trailing character since it's not actually there. Also we don't want to remove trailing apostrophes from a truncated word, because they're not actually at the end of the (untruncated) token there. This doesn't make a big difference, but it's slightly more correct.

lib-fts: Optimize tokenizers - Rewrite of apostrophe handling. We parse through the UTF8 characters only once now, not twice.

lib-fts: tr29 tokenizer - rename variable in preparation for the next patch

lib-fts: tokenizers - don't include removed apostrophes as part of the token size

lib-fts: simple tokenizer minor cleanup - removed unnecessary token length > 0 check fts_tokenizer_generic_simple_current_token() will check it in any case.

lib-fts: tr29 tokenizer cleanup - Avoid unnecessary goto.

lib-fts: simple tokenizer optimization - don't check unicode word breaks for ASCII chars.

lib-fts: simple tokenizer cleanup - make prev_letter updating more explicit. It was now hidden inside one of the functions, which didn't make the prev_letter very consistent when a word break was found. It didn't actually matter what the prev_letter was at that point, but now the behavior is more consistent.

lib-fts: simple tokenizer cleanup - removed unnecessary variables

lib-fts: tr29 cleanup - consistently call valid chars "token" and "non-token" chars. Instead of word/token/text.

lib-fts: tr29 cleanup - Avoid i++ in the for loop to avoid extra calculations

lib-fts: tr29 cleanup - token can never be empty by the time it's being returned.

lib-fts: Optimization for tr29 - we don't need to track last_size explicitly

lib-fts: Correct internal helper function for tr29.

lib-fts: Change TR29 tokenizer to break at full stop (and others). Diverge from the TR29 rules and always break at MidNumLet letters. This fixes tokenizing first.last@domain.tld email addresses.

lib-fts: Fixed handling tokens that contain only apostrophes

lib-fts: Fix simple tokenizer apostrophe handling. Apostrophes and quotation marks are now treated as word breaks, except U+0027 between non-wordbrek characters. The characters U+2019 and U+FF07 are transformed to U+0027 before processing.

lib-fts: Fix tr29 tokenizer apostrophe handling. U+0027, which is called Single Quote in tr29, was not properly handled as a word boundary.

lib-fts: Fixed assert-crash in fts-tokenizer-generic

lib-fts: Changed fts_tokenizer_next/final() to return error string. The current tokenizers can't fail, but if we're doing tokenization via external services they could fail.

lib-fts: Minor code cleanup - avoid functions always returning same value Makes it clearer to see in the code that the function can't return any failures.

lib-fts: Fixed token truncation.

lib-fts: fts-tokenizer-generic-private.h had content that didn't really belog there.

lib-fts: Removed tokenizer name macros from fts-tokenizer.h The tokenizers can be directly accessed via their class structs.

lib-fts: Added fts_tokenizer_reset()

fts: When tokenizing a search word, give "search" parameter to all the tokenizers.

lib-fts: Use case-sensitive settings comparisons in fts-tokenizer Dovecot in general doesn't allow case-insensitive settings.

lib-fts: Improve using max_length in tr29 tokenizer

lib-fts: Fixed using max_length setting in simple tokenizer

lib-fts: Default to simple tokenizer algorithm

fts: Change tokenizer API to be able to return errors Modify fts_tokenizer_next() to return integer status codes. It returns 1 if a token was returned in *token_r, 0 if more input is needed and -1 on error.

Initial import for lib-fts. Parts of what this code does was already implemented internally by fts-lucene. lib-fts is intended to be usable for all the FTS backends. The APIs are still going to change a bit, but hopefully not after v2.2.17 release. Mostly written by Teemu Huovila.