Updated copyright notices to include the year 2018.

global: use i_rand_limit() and i_rand_minmax() when possible

global: Replace rand with i_rand

test: lib-fts: Ignore uni_utf8_get_valid_data return value Makes static analyzers happier

Updated copyright notices to include the year 2017.

global: Use const for all test_functions[] arrays

global: Use "static const" for all struct tests[] arrays

global: Change string position/length from unsigned int to size_t Mainly to avoid truncating >4GB strings, which might potentially cause some security holes. Normally there are other limits, which prevent such excessive strings from being created in the first place. I'm sure this didn't find everything. Maybe everything could be found with compiler warnings. -Wconversion kind of does it, but it gives way too many unnecessary warnings. These were mainly found with: grep " = strlen" egrep "unsigned int.*(size|len)"

lib-fts: Add randomness test to test-fts-tokenizer

lib-fts: Make sure address tokenizer can't return empty tokens. This happened when address was a token that first looked like it could be a valid address, but then got truncated due to reaching maxlen, followed by truncating the UTF8-sequence and finally all the rest of the '-' or '.' chars that were valid at the beginning of the address are stripped away by fts_tokenizer_delete_trailing_invalid_char(), leaving nothing left. Fixes: Panic: file fts-tokenizer.c: line 206 (fts_tokenizer_next): assertion failed: (ret <= 0 || (*token_r)[0] != '\0')

lib-fts: Fix fts_tokenizer_delete_trailing_partial_char() unit test

lib-fts: add unit test to fts_tokenizer_delete_trailing_partial_char()

lib-fts: allow hyphen character in domain part Allow hyphen character and remove possible trailing hyhpen character when email tokenization is done.

lib-fts: remove trailing period character from email-address any trailing period character '.' should be removed when email tokenization is done.

lib-fts: Fix unit tests for lib-fts.

lib-fts: Limit maximum length of addresses found. The address tokenizer now takes a "maxlen" parameter, which defaults to 254 bytes. Previously addresses, or something looking like it, could be of any length. This could cause trouble in fts backends.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

various - remove 8-bit characters from literal strings in test cases C has a portable way of expressing characters not in the basic character set, namely \xNN escaping. Otherwise, the interpretation of the raw utf-8 is implentation dependent. This has the benefit of making some tests' expected output more obvious, such as "=c3=a4" matching "\xC3\xA4", even if it hinders the readability of some natural-language-based tests. Signed-off-by: Phil Carmody <phil@dovecot.fi>

Remove now-unnecessary direct stdlib.h #includes.

lib-fts: Add Unicode TR29 rule WB5a setting to tokenizer. Splits prefixing contracted words from base word. E.g. "l'homme" -> "l" "homme". Together with a language specific stopword list unnecessary contractions can thus be filtered away. This is disabled by default and only works with the TR29 algorithm. Enable by "fts_tokenizer_generic = algorithm=tr29 wb5a=yes"

lib-fts: Fixed tr29 tokenizer to delete last character correctly when it's preceded by non-ASCII

lib-fts: Added unit testing for unicode apostrophe handling.

lib-fts: tokenizers - Fixed removal of trailing character in truncated tokens. If the token is truncated, we don't want to remove the trailing character since it's not actually there. Also we don't want to remove trailing apostrophes from a truncated word, because they're not actually at the end of the (untruncated) token there. This doesn't make a big difference, but it's slightly more correct.

lib-fts: tokenizers - don't include removed apostrophes as part of the token size

lib-fts: Fix unit tests for TR29 full stop change.

lib-fts: Fixed handling tokens that contain only apostrophes

lib-fts: Fix simple tokenizer apostrophe handling. Apostrophes and quotation marks are now treated as word breaks, except U+0027 between non-wordbrek characters. The characters U+2019 and U+FF07 are transformed to U+0027 before processing.

lib-fts: Test trailing "number." for TR29 in test-fts-tokenizer

lib-fts: Improved test-fts-tokenizer to run multiple text inputs

lib-fts: Changed fts_tokenizer_next/final() to return error string. The current tokenizers can't fail, but if we're doing tokenization via external services they could fail.

lib-fts: Various improvements to test-fts-tokenizer

lib-fts: Removed tokenizer name macros from fts-tokenizer.h The tokenizers can be directly accessed via their class structs.

lib-fts: test-fts-tokenizer cleanup - moved tokenizers_init/deinit() to be done globally

lib-fts: Added fts_tokenizer_reset()

lib-fts: fts-tokenizer-address didn't reset state properly when input ended.

lib-fts: Implemented "search" parameter to fts-tokenizer-address.

lib-fts: Removed explicit no_parent setting from fts-address-tokenizer It was used only by unit tests and we can check it without the setting as well.

lib-fts: Don't treat empty domains as valid addresses. This actually pretty much reverts the previous patch and just fixes the original code's chars_after_at() to work correctly. Also renamed the function to make it a bit clearer what was intended.

lib-fts: Removed unnecessary code from fts-address-tokenizer. chars_after_at() was only used to check if local-part was empty, but it was checked at a state where it never could have been empty.

lib-fts: Improve using max_length in tr29 tokenizer

lib-fts: Fixed using max_length setting in simple tokenizer

fts: Change tokenizer API to be able to return errors Modify fts_tokenizer_next() to return integer status codes. It returns 1 if a token was returned in *token_r, 0 if more input is needed and -1 on error.

fts: Create tokenizers differently Create tokenizers earlier. Create separate tokenizers for search and indexing. Enable configuration of tokenizers. Add some helpers in fts-tokenizer.h api. Change tokenizer unit tests to match those changes. lib-fts: Refactor lib-fts settings a bit Turned address tokenizer settings into "boolean" values. Changed have_parent to "no_parent" and added "search" setting. Added documentation in fts-tokenizer.h. Change unit tests accordingly.

Initial import for lib-fts. Parts of what this code does was already implemented internally by fts-lucene. lib-fts is intended to be usable for all the FTS backends. The APIs are still going to change a bit, but hopefully not after v2.2.17 release. Mostly written by Teemu Huovila.