global: Use check-local in Makefile.am instead of overriding check directly This helps with dependency problems, like running "make check" in lib-storage without "make" first would try to compile the test programs too early and fail.

{lib,lib-fts}: fix builds with BSD make Without this change, BSD make doesn't know how to make a couple of the generated files because the BUILT_SOURCES file names don't match exactly the left hand sides of the rules. (GNU make somehow manages to match the rule even though it is not an exact match.)

lib-fts: use full path to word-properties script This is a step toward fixing builds where object dir != source dir.

lib-fts: download data files into srcdir This is a step toward fixing builds where object dir != source dir.

lib: Download unicode.org files from dovecot.org

lib-fts: Cut overlong strings in lowercase filter. Added new common truncate function for filters. It also removes any partial characters, that would remain from plain truncation.

lib-fts: Improved stopword file reading. The reading tries to be a little bit stricter now. Only stopwords at the start of a new line are accepted now. Changed fi stopwords accordingly. Also removed superfluous stack allocation in parsing.

lib-fts: Add stopword files for more languages.

lib-fts: Move stopwords to subdirectory. All files incluided in dist are explicitly mentioned. The whole subdirectory 'stopwords' could also be distributed, but that is more error prone.

lib-fts: Lift helper function out of generic tokenizer.

lib-fts: Add lib-fts to CPPFLAGS as include dir Without this, VPATH builds fail because the includes cannot be found as they are not on same directory.

lib-fts: Create library for development packages.

fts: Added fts_library_init() and _deinit() Replaces calling three different functions on init and deinit.

lib-fts: Add Norwegian. Norwegian has two main dialects, Bokmal(nb) and Nynorsk(nn). They are detected separately by libexttextcat, but the stemmer only knows Norwegian. Thus they are treated as a single language, Norwegian (no). This might also make more sense in everyday use of mixed writing style Norwegian. Caveat: The default normalizer filter does not modify U+00F8 (Latin Small Letter O with Stroke). In some configurations it might be desirable to rewrite it to e.g. o. Same goes for the upper case version. This can be done by passing a modified "id" setting to the normalizer filter.

lib-fts: Add Swedish (sv) to supported languages.

lib-fts: Add prefixing contraction filter. Filters away prefixing contracted words, e.g. "l'homme" -> "homme". Tokens to be filtered must be lower case. Only supports French in this initial version.

lib-fts: Install headers on make install.

lib-fts: Added "english-possessive" filter.

lib-fts: Moved IS_APOSTROPHE() to fts-common.h

lib-fts: Rewrite ICU handling functions. Some of the changes: - Use buffers instead of allocating everything from data stack. - Optimistically attempt to write the data directly to the buffers without first calculating their size. Grow the buffer if it doesn't fit first. - Use u_strFromUTF8Lenient() instead of u_strFromUTF8(). Our input is already supposed to be valid UTF-8, although we don't check if all code points are valid, while u_strFromUTF8() does check them and return failures. We don't really care about if code points are valid or not and u_strFromUTF8Lenient() passes through everything. Added unit tests to make sure all the functions work as intended and all the UTF-8 input passes through them successfully.

Reverted d592417ec815 which added unnecessary code to Makefiles. The original problem it tried to solve was properly fixed by 46969c4cc57e. make will actually wait for processes to finish creating files before it continues to the next program that wants to access the file. As long as the dependencies are correct.

lib-fts: Makefile compiling dependency fix

lib-fts: autogenerate C arrays using perl The sh script had bashisms, the awk script crashed mawk, so let's try perl... Signed-off-by: Phil Carmody <phil@dovecot.fi>

lib-fts: .sh scripts weren't executable - changed them to be run via bash directly. Better to avoid relying on the executable bit.

lib-fts: Reverted e80969ea8684 which replaced .sh scripts with awk Bugs in older awk versions (used at least by Debian squeeze & wheezy) caused awk to crash while processing the script.

Makefile: Fixed build concurrency issues with lib-fts

lib-fts: Replaced word-boundary/break-data.sh with more portable awk scripts Patch by Michael Grimm.

lib-fts: Added "lowercase" filter. For now it handles only ASCII characters, but that's enough for our use.

lib-fts: Removed "simple" normalizer. It translated input to titlecase, which wasn't suitable for snowball stemming that wanted lowercase input. Since that doesn't work, there's probably no good for the existence of this (perhaps in future it's replaced by unicode-aware lowercaser).

lib-fts: Use rfc822-parser in fts-tokenizer-address instead of duplicating its code.

lib-fts: Added normalizer-simple for doing normalization without libicu.

lib-fts: Renamed normalizer to icu-normalizer, including the source code.

configure: s/normalizer/libicu/ since we it could be used for something else as well.

lib-fts: Added udhr_fra.txt to EXTRA_DIST

lib-fts: Added PropList.txt to EXTRA_DIST

lib-fts: Fixed using FTS_NORMALIZER_CFLAGS/LIBS.

lib-fts: Added missing stopwords_fi.txt

lib-fts: Fixed test-fts-language to use TEXTCAT_DATADIR This may still make too many assumptions about what data exists where.. So we may need to remove this test from "make check". But for now leave it there.

Initial import for lib-fts. Parts of what this code does was already implemented internally by fts-lucene. lib-fts is intended to be usable for all the FTS backends. The APIs are still going to change a bit, but hopefully not after v2.2.17 release. Mostly written by Teemu Huovila.