Updated copyright notices to include the year 2018.

global: timeout_remove(NULL) is a no-op

fts-lucene: Fix potential memory leak

global: remove unnecessary ENOENT checks from unlink_directory() calls

global: Log internal storage error on failure

Updated copyright notices to include the year 2017.

lib: Changed unlink_directory() API to return error string. There are many different reasons why it could fail, which a simple errno can't very clearly indicate.

fts-lucene: Fixed crash on error or auto-rebuild conditions.

lib-storage: add no_fts in struct mail_search_arg When flag is set, FTS will not be performed. fts plugin: handle no_fts flag No fts will be performed once flag is set. Original patch from Timo Sirainen <timo.sirainen@dovecot.fi>

fts-lucene: Fixed crash on rescan Broken by af2564c7f9e05ad245a032efdfbc5abbb9b70f1e - rescan_ctx can be NULL.

fts-lucene: Rescan skipped namespace roots.

fts-lucene: Cleanup - move code to a separate function

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

Compiler warning fixes

fts-lucene: Fix to earlier commit: Header names must be indexed without libfts, not with.

fts-lucene: Optionally use lib-fts instead of CLucene's own analyzers. fts_lucene = use_libfts enables this.

configure: Fixed/improved finding and using libexttextcat. libexttextcat wasn't actually being used at all by fts-lucene. Now we'll first prefer finding it via pkg-config, next finding exttextcat and only last look up textcat.

configure: Stemmer and textcat checks are now done even if CLucene isn't used. They'll be useful for other FTS backends as well. This patch doesn't change any actual functionality.

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

fts-lucene: No .. the rebuilding warning was still in the wrong place.

fts-lucene: Compiling fix for earlier change.

fts-lucene: When deleting corrupted lucene-indexes/ directory, don't rmdir it. This at least avoids errors with NFS when some of the files are still open inside it by other processes.

fts-lucene: Automatically close index after 2 minutes of idling.

fts-lucene: Make sure Lucene indexes are always closed at deinit to avoid memory leaks. At least optimize leaked memory.

fts-lucene: If whitespace_chars was set, we may have ended up indexing some garbage words. The final \0 was replaced with space, so everything beyond that was also being indexed.

fts: Added fts_no_autofuzzy setting to require exact matches for found results. This is done by using the FTS search results as only filters on which the regular non-FTS search is done.

fts-lucene: Don't mix definite + maybe queries for now, since it returns broken results.

fts-lucene: Added mime_parts parameter to index MIME parts as separate documents. This is in preparation for being able to actually read such information and show it to clients via some new IMAP extension. Also it might be better to index some MIME parts together to avoid wasting disk space, but for now this is all or nothing. For example anything that is "message body" could probably be indexed without the part number at all.

fts-lucene: Fixed "doveadm dump" crash when built with stemmer

fts-lucene: Fix to previous change. We allocated way too little memory.

fts-lucene: Don't allocate large amounts of memory from stack. This probably only happened for huge header values.

fts-lucene: Skip returned duplicate results. This caused assert-crash at least when searching from virtual mailbox because the score array was larger than the actual number of results.

fts-lucene: Fixed crash in doveadm dumping lucene index.

fts-lucene: Fixed building without libstemmer

fts-lucene: Support normalize setting also without snowball. Added no_snowball setting. Snowball seems to be converting / breaking words down rather annoyingly.

Avoid strict aliasing warnings.

fts-lucene: doveadm fts rescan crashed with mailbox_list_index=yes

fts-lucene: Compile fix for previous unlink_directory() change

fts-lucene: Compiling fix for previous change

fts-lucene: Added "normalize" option to put data through user's normalizer function.

fts-lucene: Added "normalize" option to put data through normalize().

Renamed buffer_create_*data() to buffer_create_from_*data() for consistency. Especially i_stream_create_from_data() was a very similar function, which made it difficult to remember which one should have "from" and which one shouldn't.

Reverted "support for non-pointers" part of the hash table API changes. Originally I wrote it using clang, which didn't give as many warnings as gcc did. I guess this way is safer anyway..

Array API changed: ARRAY_DEFINE(name, type) -> ARRAY(type) name Easy way to update your existing code: perl -i -pe 's:ARRAY_DEFINE\(([^,]+), *([^)]+)\);:ARRAY($2) $1;:' **/*.[ch]

Hash table API is now (mostly) type safe.

hash_table_create(): Removed table_pool parameter. Every single caller was using default_pool there, so there's no point in having it.

Renamed struct mailbox_info.name to vname.

fts-lucene: Fixed handling non-lowercase SEARCH HEADER FROM/TO/SUBJECT/CC/BCC Based on patch by Matthew Powell.

Debian names libtextcat as libexttextcat. Patch by Stephan Bosch.

seq_range_array_add() API changed. Added other functions to provide the less common use cases.

fts-lucene: Fixed search when all terms were MUST_NOT.

fts-lucene: Optimize searching for existence of an indexed header.

fts-lucene: Index the header name tokenized, or we can't search it.

fts-lucene: Fixed handling "maybe" queries (unindexed headers)

fts-lucene: Added whitespace_chars subsetting to fts_lucene. A value of "@." could be useful so that user@domain.tld allows searching user, domain and tld separately instead of requiring the whole string to match.

fts-lucene, fts-solr: Try to optimize searching for existence of header name.

lib-storage: Replaced MAILBOX_FLAG_KEEP_RECENT flag with reverse MAILBOX_FLAG_DROP_RECENT. Very few places actually want to drop recent flags, so this way is easier.

fts-lucene: If default_language changes, reindex mailbox.

fts-lucene: Error logging fix.

Moved GUID code to liblib. Use guid_128_t type consistently everywhere.

fts-lucene: Added default_language setting and separated stemmer/textcat support.

fts-lucene: Add '/' suffix to textcat_dir if it's not already there.

fts-lucene: Free all Lucene's memory on plugin deinit.

fts-lucene: Use TermQuery instead of PrefixQuery for searches. Although PrefixQuery gives a more IMAP-like result, it's slower and can fail if the prefix string is small.

fts-lucene: Automatically delete corrupted indexes and mark all mailboxes unindexed.

fts-lucene: doveadm fts rescan now resets "last indexed uid" for all mailboxes.

fts-lucene: Avoid unnecessarily creating the lucene-indexes directory.

fts-lucene: Yet another (final!) attempt at avoiding "Too Many Clauses" error.

fts-lucene: Another attempt to avoid "Too Many Clauses" error by not using ? wildcard.

fts-lucene: Fix to previous change: actually use WildcardQuery, not TermQuery..

fts-lucene: Use WildcardQuery rather than RangeQuery to limit what UIDs to fetch.

fts-lucene: Don't crash if expunging didn't actually do anything.

fts-lucene: Actually delete messages from index on expunge/rescan.

fts-lucene: Avoid crashing on doveadm dump.

fts-lucene: Use fts expunge log instead of scanning the entire index for expunged mails.

fts-lucene: Added "doveadm dump" support for lucene index.

fts-lucene: Several fixes and improvements.

fts-lucene: Store indexed header names lowercased so they can actually be searched.

fts-lucene: Crashfix for handling header searches with stop words (etc).

fts-lucene: Fixed compiling without textcat/snowball

fts-lucene: Added initial support for language detection and stemming. This requires textcat and snowball (libstemmer) libraries. This can be enabled by setting: fts_lucene = textcat_dir=/usr/share/libtextcat \ textcat_conf=/etc/dovecot/textcat.conf I'm not yet sure how good of an idea it is to actually use this, so it needs some testing.. For example how bad is it if language is detected incorrectly?

fts-lucene: Don't crash when trying to search an unsearchable word.

fts-lucene: Fixed checking if we need to track expunges or optimize.

fts-lucene: Don't crash if optimize can't open the index.

fts-lucene: If optimize doesn't find an index, set all UIDs as missing.

fts-lucene: Delete stale write locks.

fts-lucene: Added fuzzy search support.

fts: Redesigned/enhanced FTS API and how virtual plugin works with it. The changes include: - Only indexer process and "doveadm index" do FTS indexing now. Other processes connect to the indexer process via UNIX socket and request indexing a specific mailbox. - FTS backends can now index/search any search key. Current backends implement indexing for some specific header fields (Subject, From, To, Cc, Bcc), but it would also be possible to add indexing for e.g. message size or date. - CLucene support is fixed and fully functional. - Solr is split to "solr" and "solr_old" backends. The new "solr" backend supports the new header fields. It also uses mailbox GUIDs as mailbox identifiers, so that renaming a mailbox doesn't require reindexing. The "solr_old" uses the old Solr schema and doesn't support any new features.

fts-lucene: Fixes to work with CLucene v2.3.3.4 and new FTS API. It's still not recommended to actually use this.

Updated copyright notices to include year 2010. --HG-- branch : HEAD

When dropping last_uid fields, finish the search before removing them. Otherwise lucene could crash. --HG-- branch : HEAD

FTS API changes and Squat rewrite. Squat is still missing expunge handling. --HG-- branch : HEAD

Compiler warning fixes --HG-- branch : HEAD

Fixes --HG-- branch : HEAD

Create a locks directory under the lucene-indexed directory. Detect and delete stale locks. Fixed expunges to work. --HG-- branch : HEAD

Separate headers and body in building and searching. Added support for Lucene to index/search them separately. --HG-- branch : HEAD

Don't call addDocument() after each block. --HG-- branch : HEAD

Fixed filtering to work with Lucene. Generic FTS code now implements the filtering if the backend doesn't. --HG-- branch : HEAD

Removed utf8_strlen_n() here and used the one from unichar.h --HG-- branch : HEAD

Add support for expunges. --HG-- branch : HEAD

Fixes and cleanups --HG-- branch : HEAD

Some fixes. --HG-- branch : HEAD

Create only a single index into INBOX's index dir. Did several fixes. However still a bit buggy. --HG-- branch : HEAD

Our field length was set to way too small value. For query keys use key* by default, or "key key2" if it contains spaces. If the query parser anyway gives an error, don't bother logging it unless DEBUG is enabled. --HG-- branch : HEAD

--with-lucene now enables lucene full text search indexing. Note that using it breaks IMAP RFC. It also seems to have problems finding texts that have special characters in them, such as email addresses. You can anyway enable it by loading fts and fts_lucene plugins and setting fts=lucene in plugin section. --HG-- branch : HEAD