lib-fts: Add Unicode TR29 rule WB5a setting to tokenizer. Splits prefixing contracted words from base word. E.g. "l'homme" -> "l" "homme". Together with a language specific stopword list unnecessary contractions can thus be filtered away. This is disabled by default and only works with the TR29 algorithm. Enable by "fts_tokenizer_generic = algorithm=tr29 wb5a=yes"

lib-fts: tokenizers - Fixed removal of trailing character in truncated tokens. If the token is truncated, we don't want to remove the trailing character since it's not actually there. Also we don't want to remove trailing apostrophes from a truncated word, because they're not actually at the end of the (untruncated) token there. This doesn't make a big difference, but it's slightly more correct.

lib-fts: Optimization for tr29 - we don't need to track last_size explicitly

lib-fts: Change TR29 tokenizer to break at full stop (and others). Diverge from the TR29 rules and always break at MidNumLet letters. This fixes tokenizing first.last@domain.tld email addresses.

lib-fts: Fix simple tokenizer apostrophe handling. Apostrophes and quotation marks are now treated as word breaks, except U+0027 between non-wordbrek characters. The characters U+2019 and U+FF07 are transformed to U+0027 before processing.

lib-fts: Minor code cleanups

lib-fts: fts-tokenizer-generic-private.h had content that didn't really belog there.

Initial import for lib-fts. Parts of what this code does was already implemented internally by fts-lucene. lib-fts is intended to be usable for all the FTS backends. The APIs are still going to change a bit, but hopefully not after v2.2.17 release. Mostly written by Teemu Huovila.