fts-tika: Fixed HTTPS support by properly propagating SSL client settings from mail_user.

Add PIE and RELRO to binaries

fts plugin: Install fts-storage.h

fts: Added "doveadm fts lookup" command. This is mainly useful for debugging lib-fts. It doesn't perform any of the lib-fts tokenization / filtering so you can do raw lookups.

fts: Install fts-user.h header

fts: Added FTS_BACKEND_FLAG_TOKENIZED_INPUT, which is implemented via lib-fts.

fts: Parse HTML MIME parts using lib-mail's mail-html2text.

fts: Install some of the header files that other external plugins might want to use.

fts: Added fts-parser support for Tika It can be enabled by setting: plugin { fts_tika = http://localhost:9998/tika/ }

Makefile: Again changes to handling test program dependencies. .la libraries need to link .lo files, .a libraries need to link .o files. Changed dsync to be .la library for consistency.

Makefile: Fixed test program dependencies Explicit dependencies on .o or .lo files didn't seem to work properly and with high enough make -j value the build failed. Depending on the entire library guarantees that all the individual .o files have been built.

fts: Added fts_index_timeout setting to abort search if indexing hasn't finished by then. This timeout shows up to client as: tag NO [INUSE] Timeout while waiting for indexing to finish

Redesigned mail precaching APIs. There's no longer a one monolithic mailbox_sync(MAILBOX_SYNC_FLAG_PRECACHE) call, but rather one mail_precache() call for each mail to be precached. This allows the callers to show the progress and in general is cleaner.

fts: Added generic "fts expunge log". This is intended to be used by fts backends that can't directly expunge from the index, but rather need to log the expunges and do the actual expunging later. Added also "doveadm dump" support for the log.

fts: Added "doveadm fts optimize" command.

Makefile fixes to include everything in dist

fts: Define pkglibexecdir explicitly in Makefile.am, since older automakes don't.

fts: Install decode2text to libexecdir.

fts: Added xml2text binary

fts: Added attachment decoding via an external script.

fts: Strip text/html mails to plaintext before sending them to FTS backend.

fts: Redesigned/enhanced FTS API and how virtual plugin works with it. The changes include: - Only indexer process and "doveadm index" do FTS indexing now. Other processes connect to the indexer process via UNIX socket and request indexing a specific mailbox. - FTS backends can now index/search any search key. Current backends implement indexing for some specific header fields (Subject, From, To, Cc, Bcc), but it would also be possible to add indexing for e.g. message size or date. - CLucene support is fixed and fully functional. - Solr is split to "solr" and "solr_old" backends. The new "solr" backend supports the new header fields. It also uses mailbox GUIDs as mailbox identifiers, so that renaming a mailbox doesn't require reindexing. The "solr_old" uses the old Solr schema and doesn't support any new features.

Use -Wl,--as-needed -Wl,--no-undefined GNU ld flags for non-plugins.

Changed header/body searching to do search key charset translation earlier. This cleans up and simplifies error handling.

Moved fts/virtual plugin communication functions away from lib-storage.

Get rid of imap/, pop3/, lda/ and sql/ module subdirs. It's all in lib/dovecot/ now. --HG-- branch : HEAD

Symlink FTS plugins to pop3 directory too. --HG-- branch : HEAD

Previous changes broke plugin symlink creation in make install. --HG-- branch : HEAD

Use dynamic module suffix provided by libtool rather than hardcoded ".so". --HG-- branch : HEAD

Fixed non-ASCII searches. --HG-- branch : HEAD

FTS API changes and Squat rewrite. Squat is still missing expunge handling. --HG-- branch : HEAD

Better type safety to module_contexts arrays. Already fixed some bugs. --HG-- branch : HEAD

Changed the plugin filename numbering (ie. the ordering in which the plugins are loaded). This doesn't change anything for current plugins, but future plugins will need this change so they can get themselves loaded between specific plugins. --HG-- branch : HEAD

Added full text search plugin framework. Still missing support for handling expunges. Also it could use better logic with more complex SEARCH queries to figure out when the FTS index should be used. --HG-- branch : HEAD