Updated copyright notices to include the year 2018.

fts: Code cleanup

fts: Don't try to retry parsing if indexing already fails

fts: Use consistent naming for retriable_err_msg It's only for retriable errors, not errors in general

fts: Make sure may_need_retry is reset between attempts. It wasn't if the second attempt failed early.

fts: Retry indexing for tika backend in case of internal server error

fts: Extend parser's deinit function interface with retriable_err_msg_r parameter the new parameter will be set to a error string in case indexing may need retrying, so it will be logged later when decision about retry will be made.

fts: Parser move try_init arguments to a single structure parser_context.

global: start relying on buffer_free(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - buffer_free(&E); - } + buffer_free(&E);

global: Log internal storage error on failure

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

global: Handle broken Content-Type headers consistently. 1) Only the first Content-Type header is used. (Simpler than using the last.) 2) Invalid Content-Types are parsed as far as we can. This is mainly to make sure they aren't treated as missing Content-Types, which could cause them to become text/plain.

Use mail_get_*stream_because() wherever possible.

lib-storage: When cached mime.parts is detected to be inconsistent, log it as hex-encoded.

Use mail_set_cache_corrupted_reason() wherever possible.

fts: Code cleanup - moved fts_header_has_language() to a global function

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

fts: When indexing header names with lib-fts, add them using "data" language.

fts: Added support for per-language tokenizer settings. fts_tokenizer_<lang> now overrides fts_tokenizers setting. fts_tokenizer_<name>_<lang> now overrides fts_tokenizer_<name> setting.

fts: If reading mail fails, log an error.

fts: If we fail while indexing headers, return failure immediately.

fts: Error logging fix. 1) We were logging the error after it was already freed from data stack. 2) We were logging uninitialized error string when fts indexing was the one that failed.

fts: Avoid excessive data stack usage with lib-fts

fts: Fixed crash when not using lib-fts

lib-fts: Changed fts_tokenizer_next/final() to return error string. The current tokenizers can't fail, but if we're doing tokenization via external services they could fail.

fts: Lowecase non-human language input while indexing.

fts: Reset tokenizers before using them This is mainly needed if the previous tokenization had failed.

fts: Improve filter error handling. In lib-fts: Move error storage to the generic filter struct level. Change make_utf8 helper to void also. In fts: Add the error argument to _filter() calls.

fts: Change filter API to be able to return errors Modify fts_filter_filter() to return integer status codes. It returns 1 if a token was returned, 0 if it was filtered away and -1 on error.

fts: Change tokenizer API to be able to return errors Modify fts_tokenizer_next() to return integer status codes. It returns 1 if a token was returned in *token_r, 0 if more input is needed and -1 on error.

fts: Create tokenizers differently Create tokenizers earlier. Create separate tokenizers for search and indexing. Enable configuration of tokenizers. Add some helpers in fts-tokenizer.h api. Change tokenizer unit tests to match those changes. lib-fts: Refactor lib-fts settings a bit Turned address tokenizer settings into "boolean" values. Changed have_parent to "no_parent" and added "search" setting. Added documentation in fts-tokenizer.h. Change unit tests accordingly.

fts: Create filters earlier This builds on the assumption that early initialization will be made less costly, in a coming change.

fts: Minor error messagage prefix change.

fts: If backend has FTS_BACKEND_FLAG_TOKENIZED_INPUT set, index also the header name. We were sending the hdr_name to the backend without tokenizing it or filtering it, so the backend couldn't have done anything useful about it.

fts: Added FTS_BACKEND_FLAG_TOKENIZED_INPUT, which is implemented via lib-fts.

fts: Code cleanup to do all index updates via fts_build_data() This doesn't actually change the behavior.

fts: Fixed assert-crash if fts-parser was created for a body part that backend didn't want to index.

fts: If fts-parser fails, stop indexing instead of ignoring the error and continuing. This is especially important in case there's just some temporary error.

fts: Added missing error logging/setting.

fts: When indexing address headers, don't add MISSING_MAILBOX or MISSING_DOMAIN.

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

fts: Added message_part to fts_backend_build_key. This allows FTS backends to index MIME parts separately from each others.

fts: Free fts-parser on failures.

Updated copyright notices to include year 2014.

Replaced all -1U and (unsigned int)-1 with UINT_MAX. It's somewhat clearer this way. Also clang's -fsanitize=integer gives runtime errors about -1U (but not about explicit casts, so no need to change (type)-1 casts).

fts: Don't send binary MIME parts to backend through UTF8 text conversion. Based on patch by Mike Abbott / Apple.

Oops :) Update copyrights to 2013 without breaking all .c files.

Updated copyright notices to include year 2013.