Updated copyright notices to include the year 2018.

Updated copyright notices to include the year 2017.

global: Change string position/length from unsigned int to size_t Mainly to avoid truncating >4GB strings, which might potentially cause some security holes. Normally there are other limits, which prevent such excessive strings from being created in the first place. I'm sure this didn't find everything. Maybe everything could be found with compiler warnings. -Wconversion kind of does it, but it gives way too many unnecessary warnings. These were mainly found with: grep " = strlen" egrep "unsigned int.*(size|len)"

lib-fts: Make sure address tokenizer can't return empty tokens. This happened when address was a token that first looked like it could be a valid address, but then got truncated due to reaching maxlen, followed by truncating the UTF8-sequence and finally all the rest of the '-' or '.' chars that were valid at the beginning of the address are stripped away by fts_tokenizer_delete_trailing_invalid_char(), leaving nothing left. Fixes: Panic: file fts-tokenizer.c: line 206 (fts_tokenizer_next): assertion failed: (ret <= 0 || (*token_r)[0] != '\0')

lib-fts: allow hyphen character in domain part Allow hyphen character and remove possible trailing hyhpen character when email tokenization is done.

lib-fts: remove trailing period character from email-address any trailing period character '.' should be removed when email tokenization is done.

lib-fts: Limit maximum length of addresses found. The address tokenizer now takes a "maxlen" parameter, which defaults to 254 bytes. Previously addresses, or something looking like it, could be of any length. This could cause trouble in fts backends.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

lib-fts: Changed fts_tokenizer_next/final() to return error string. The current tokenizers can't fail, but if we're doing tokenization via external services they could fail.

lib-fts: Minor code cleanup - avoid functions always returning same value Makes it clearer to see in the code that the function can't return any failures.

lib-fts: Minor code cleanups

lib-fts: Removed tokenizer name macros from fts-tokenizer.h The tokenizers can be directly accessed via their class structs.

lib-fts: Added fts_tokenizer_reset()

lib-fts: fts-tokenizer-address didn't reset state properly when input ended.

lib-fts: Implemented "search" parameter to fts-tokenizer-address.

lib-fts: Minor fts-tokenizer-address cleanups

lib-fts: Removed explicit no_parent setting from fts-address-tokenizer It was used only by unit tests and we can check it without the setting as well.

lib-fts: Use rfc822-parser in fts-tokenizer-address instead of duplicating its code.

lib-fts: Don't treat empty domains as valid addresses. This actually pretty much reverts the previous patch and just fixes the original code's chars_after_at() to work correctly. Also renamed the function to make it a bit clearer what was intended.

lib-fts: Removed unnecessary code from fts-address-tokenizer. chars_after_at() was only used to check if local-part was empty, but it was checked at a state where it never could have been empty.

fts: Change tokenizer API to be able to return errors Modify fts_tokenizer_next() to return integer status codes. It returns 1 if a token was returned in *token_r, 0 if more input is needed and -1 on error.

fts: Create tokenizers differently Create tokenizers earlier. Create separate tokenizers for search and indexing. Enable configuration of tokenizers. Add some helpers in fts-tokenizer.h api. Change tokenizer unit tests to match those changes. lib-fts: Refactor lib-fts settings a bit Turned address tokenizer settings into "boolean" values. Changed have_parent to "no_parent" and added "search" setting. Added documentation in fts-tokenizer.h. Change unit tests accordingly.

Initial import for lib-fts. Parts of what this code does was already implemented internally by fts-lucene. lib-fts is intended to be usable for all the FTS backends. The APIs are still going to change a bit, but hopefully not after v2.2.17 release. Mostly written by Teemu Huovila.