lib-storage: Make index_mail_want_cache() global

lib-storage: convert index to use container_of

lib-storage: Add index_mail_get_cached_body[structure]() This just moves the code to these functions without changing any of the logic.

lib-mail: message-part-data: Renamed struct message_part_envelope_data to message_part_envelope.

lib-storage: Add index_mail_get_vsize_extension()

lib-storage: Move index_mail.vsize_ext_id to mailbox.mail_vsize_ext_id It's mailbox-specific, so it doesn't have to be looked up every time. Also this is needed for the following fix.

lib-storage: Add vsize extension to index Keep virtual size in index, instead of cache when it's less than 2^32-1. This helps when cache becomes corrupted, and goes away, we still have virtual sizes for quota calculations.

lib-storage: rename get_real_mail to get_backend_mail Also change the function signature so that it returns an int to indicate success or failure.

lib-storage: remove set_cache_corrupted_reason from mail_vfuncs Change set_cache_corrupted signature to what set_cache_corrupted_reason used to be.

Use mail_get_*stream_because() wherever possible.

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

lib-storage: When cached mime.parts is detected to be inconsistent, log it as hex-encoded.

lib-storage: Added mail_set_cache_corrupted_reason()

lib-storage: Added MAIL_FETCH_BODY_SNIPPET.

lib-storage: Mail prefetching was still not working right. I wonder if it ever actually worked (except for SEARCH TEXT).

lib-storage: Don't fetch wanted_fields for messages that don't match the search query. For example "doveadm fetch text subject foo" was opening all the mails, even though only a few matched the subject (that was hopefully already in cache file). The behavior still isn't perfect though. The wanted_fields should probably be split into search_wanted_fields and fetch_wanted_fields, but the current behavior is likely good enough for now.

lib-storage: Headers were sometimes added to dovecot.index.cache with wrong line number value. This happened if the header parsing was done multiple times for the same mail. The end result being that when fetching multiple headers as a stream, the headers could have been returned in the wrong order. This messed up at least dsync's header hash comparisons.

lib-storage: istream-mail updates mail->expunged if it notices ENOENT from parent stream.

lib-storage: Log mail istream read failures in one place. Also handle ENOENT errors by checking if the mail has already been expunged, and if so don't log an error, just return "mail is already expunged" error to client.

lib-storage: "Message has no NUL characters" flag was being set wrong to cache file. Only the first MIME part was checked for its existence. When this flag was wrong, IMAP FETCH may have returned NUL characters instead of converting them to 0x80 character. This apparently caused Outlook to hang.

lib-storage: Partially reverted the mail.get_real_mail() API change for backwards compatibility. I hadn't realized Pigeonhole was also using it. (I thought it was using only the mail_get_real_mail() public API.) This fixes Pigeonhole v0.4.2 to work again.

lib-storage: Replaced mail_get_real_mail() with mail_get_backend_mail() that can fail. For now the mail_get_real_mail() can still be used for backwards compatibility.

dbox: Added support for POP3 message order.

lib-storage: Catch input stream errors when parsing mail.

lib-storage: Added support for "private modseqs" for INDEXPVT index. Used by dsync (next commit).

lib-storage: Check and log stream errors when parsing/searching messages.

imap: URLFETCH BINARY BODYPARTSTRUCTURE returns binary-decoded line counts.

lib-storage: struct mail.close() now clears all of its data.

lib-storage: Moved index_mail.data_pool to mail_private.data_pool

Array API changed: ARRAY_DEFINE(name, type) -> ARRAY(type) name Easy way to update your existing code: perl -i -pe 's:ARRAY_DEFINE\(([^,]+), *([^)]+)\);:ARRAY($2) $1;:' **/*.[ch]

lib-storage: Added caching for mail_get_binary_size()

IMAP BINARY extension supports now FETCH BINARY command.

Marked functions parameters that are allowed to be NULL. Some APIs were also changed. The non-obvious APIs where NULL parameter was changed to "" are master_service_init() and auth_master_user_list_init(). These checks can currently be enabled only on a patched clang: http://llvm.org/bugs/show_bug.cgi?id=6786

maildir: Remember the mail's GUID/filename in memory once it's fetched once. This allows retrieving it later even if the message gets expunged.

lib-storage: Preserve from_envelope in save context's dest_mail even if it's temporary. This allows looking it up while the mail is being saved.

lib-storage: Added mail_log_update_wanted_fields()

imapc: Fixed closing mail properly.

lib-storage: Verify that cached message size matches actually read size.

Redesigned mail precaching APIs. There's no longer a one monolithic mailbox_sync(MAILBOX_SYNC_FLAG_PRECACHE) call, but rather one mail_precache() call for each mail to be precached. This allows the callers to show the progress and in general is cleaner.

lib-storage: mail->saving was set too late, which could have caused crashes.

lib-storage: Added mail_parse().

lib-storage: Added mail prefetching support for searching.

Search supports now prefetching data for returned mails. Dropped imapc's own prefetching. mail_prefetch_count specifies how many mails can be kept open and issue a prefetch. This works using posix_fadvise(POSIX_FADV_WILLNEED) for maildir, sdbox and cydir backends. Apparently only Linux supports this. imapc backend also implements this internally by sending wanted IMAP commands to remote server. The command pipelining helps with latency. This change also makes it actually possible for imapc backend to first check if wanted data is already cached in local index and avoid sending unnecessary IMAP commands to remote server.

lib-storage: Changed mailbox_search_*() API to return pointer struct mail. This reduces the flexibility, because now different mailbox_search_next() calls can't have different mail parameters with possibly different wanted_fields settings. This flexibility was never used though, and removing it gives a couple of benefits: * it's now easier for backends to implement prefetch and parallel search * usage is easier since mail_alloc()/mail_free() is no longer needed

lib-storage: Small optimizations to setting mail's access_parts.

lib-storage: Removed unused index_mail.uid_validity.

imapc: Send UID FETCH commands with larger uidset parameter if possible. This code assumes that server doesn't reorganize FETCH replies when it's given an increasing UID range, which in theory IMAP server would be allowed to do, but I doubt there are any.

lib-storage: Unvirtualized mailbox_header_lookup_*()

lib-storage: Removed struct index_transaction_context. All of its contents are now in struct mailbox_transaction_context.

lib-storage: Added istream_opened() method to struct mail. This can be hooked into by plugins that want to do something with the message's stream in the message file, rather than the virtual resulting stream visible to client. For example when using dbox attachments, this stream contains only the data in the dbox files without any external attachments.

Maildir: If mail's virtual size can be found from filename/uidlist, do it instead of using cache. This is especially useful with POP3 to avoid opening cache file.

Removed mail_update_uid() / mail_index_update_uid(). It was working properly only with mdbox and there wasn't really a need for it. --HG-- branch : HEAD

lib-storage: Added mail_get_real_mail() that returns physical mail in a virtual mailbox. --HG-- branch : HEAD

lib-storage: mail_get_parts() no longer returns const pointer. The struct message_part contained context-pointer and other APIs were using it, so the pointer really couldn't be const without casting it away sometimes. --HG-- branch : HEAD

lib-storage: If plugin changes mail's input stream, don't assert-crash on close. --HG-- branch : HEAD

lib-storage: *_mailboxes don't descend from index_mailbox anymore, it's now a context. --HG-- branch : HEAD

Use separate read block sizes when reading only message header vs. full message. --HG-- branch : HEAD

index-mail: Expose tee_stream so plugins can read the stream too while mail is being read. --HG-- branch : HEAD

lib-storage: Use less memory from data stack. --HG-- branch : HEAD

Added ability to specify message's minimum modseq value. --HG-- branch : HEAD

Added ability to change existing messages' UIDs with mail_[index_]update_uid(). --HG-- branch : HEAD

struct mail_private now contains all kinds of statistics about file accesses. --HG-- branch : HEAD

Don't cache Date: header if we don't really want it. --HG-- branch : HEAD

Don't cache Date: header if we don't really want it. --HG-- branch : HEAD

Mail copying: Avoid updating cache file because of copying. The mail is being opened in any case, so it's unlikely to help anything. --HG-- branch : HEAD

mail_get_flags/keywords() now returns updated values if they've been changed within transaction. --HG-- branch : HEAD

Added the concept of Global UIDs that are preserved across copies. They can be fetched, searched and specified for the save API. dbox format supports them correctly, but with maildir the base filename is used so maildir_copy_preserve_filename=yes is required to preserve GUIDs when copying. --HG-- branch : HEAD

dbox: Forgot to add pop3.uidl cache variables in previous commit. --HG-- branch : HEAD

struct mail can't necessarily be casted to struct index_mail. Added a new get_index_mail() method to return it. --HG-- branch : HEAD

struct mail can't necessarily be casted to struct index_mail. Added a new get_index_mail() method to return it. --HG-- branch : HEAD

Initial CONDSTORE support. --HG-- branch : HEAD

Added mail_set_cache_corrupted() to rebuild the cache file and force recalculating the given field. --HG-- branch : HEAD

Added mail_set_cache_corrupted() to rebuild the cache file and force recalculating the given field. --HG-- branch : HEAD

If mail saving fails, don't try to update cache file (only to have the changes rollbacked). --HG-- branch : HEAD

Assert-crash if mail's input stream is closed too early or too late. --HG-- branch : HEAD

Added mail_get_keyword_indexes() --HG-- branch : HEAD

Cleanups/rewrites to how/when date/size caching is done. --HG-- branch : HEAD

Don't cache all headers just because they're listed in cache file. First check if we actually want to cache them. --HG-- branch : HEAD

Changed .h ifdef/defines to use <NAME>_H format. --HG-- branch : HEAD

mail.close() virtual function finishing. --HG-- branch : HEAD

Added mail.close() virtual function. --HG-- branch : HEAD

mail_*() APIs changed to return int and return the actual data as pointer. Changed some code to do error handling a bit better. --HG-- branch : HEAD

mail_index_lookup*() can't fail anymore. Changed several APIs not to return failure anymore. --HG-- branch : HEAD

Don't store data->rec. It may not stay usable. --HG-- branch : HEAD

Cache received date, sent date, save date and physical size when saving mails if they're wanted. --HG-- branch : HEAD

Added mail_get_first_header_utf8() and mail_get_headers_utf8(). --HG-- branch : HEAD

Moved tee stream handling to index-mail code. --HG-- branch : HEAD

We didn't cache correctly headers that existed in cache file for other messages, but we weren't currently interested of. --HG-- branch : HEAD

Store virtual sizes to dovecot-uidlist if we want to use them all the time. If nothing else than virtual size is wanted to be known (ie. POP3 or POP3-like IMAP client), don't bother updating cache file at all. --HG-- branch : HEAD

Don't feed index_mail_parser headers while searching message bodies, unless we actually need something from there. --HG-- branch : HEAD

Compiling fix --HG-- branch : HEAD

Don't use time_t anymore in cache file. uint32_t is large enough and makes cache files compatible between 32bit and 64bit systems. --HG-- branch : HEAD

index_mail_cache_add() was used with lib-storage's internal indexes for most of the received date, physical size and virtual size cache additions. This may have caused all kinds of trouble, but since it went unnoticed this long perhaps not.. --HG-- branch : HEAD

Added mail_get_save_date() and some cleanups. --HG-- branch : HEAD

Array API redesigned to work using unions. It now provides type safety without having to enable DEBUG, as long as the compiler supports typeof(). Its API changed a bit. It now allows directly accessing the array contents, although that's not necessarily recommended. Changed existing array usage to be type safe in a bit more places. Removed array_t completely. Also did s/modifyable/modifiable/. --HG-- branch : HEAD

Added mail_set_uid() to select a mail by UID. --HG-- branch : HEAD

Changed mailbox_save_*() API a bit: Moved the struct mail *dest_mail to save_init() instead of being in save_finish(). This way you can request wanted fields from the mail while it's being saved. With maildir the message is being parsed at the same time as it's being saved, and the results are stored into cache file. --HG-- branch : HEAD

Cleanups and minor logic changes as to what should be cached. --HG-- branch : HEAD

Added mail_cache_min_mail_count setting. --HG-- branch : HEAD

Header caching was more or less broken. --HG-- branch : HEAD

Mark *all* non-found headers that are registered in cache file as non-found at the end of header parsing. --HG-- branch : HEAD

Added MAIL_CACHE_FLAG_TEXT_PLAIN_7BIT_ASCII to compress simple BODY and BODYSTRUCTURE replies into a single flag + message parts. Also did other cleanups. --HG-- branch : HEAD

Try to be a bit smarter about when to parse header/body. --HG-- branch : HEAD

Keywords are now stored in X-Keywords headers in mbox. Did several related API changes to get better performance. --HG-- branch : HEAD

Renamed mail_get_header() to mail_get_first_header() and mail_gets_headers() to mail_get_header_stream(). Added new mail_get_headers() which returns NULL-terminated string list of all found headers. --HG-- branch : HEAD

Changed some buffers to arrays. --HG-- branch : HEAD

Major mail-storage API changes. It's now a bit cleaner and much more plugin friendly. Removed proxy_mailbox* stuff, they were difficult to use and there's now much easier way to replace them. --HG-- branch : HEAD

Keyword fixes. --HG-- branch : HEAD

mail->expunged field wasn't actually set. Now it's always set if wanted_fields contains message header or body. --HG-- branch : HEAD

Removed "mail.uid". It's not used and will not be used. --HG-- branch : HEAD

Initial support for keywords. Syncing to mbox/maildir doesn't work yet. --HG-- branch : HEAD

Cache field indexes are cache file specific, not globals. --HG-- branch : HEAD

Fixed cache flags to work again. Flags are cached by default now. --HG-- branch : HEAD

Added mail.get_physical_size(). --HG-- branch : HEAD

Fixed a crash with fetch (rfc822.size body). Also fixed it so it doesn't anymore parse the message body twice. --HG-- branch : HEAD

get_index() wasn't working with caching disabled --HG-- branch : HEAD

header caching fixes --HG-- branch : HEAD

Header caching redesigned. New design allows caching decisions per field, so they can be divided to temporary/permanent. Cached headers are now always returned in original order, old code didn't guarantee it. Some other caching changes. (still missing code to store changes in caching decisions) --HG-- branch : HEAD

Always create cache transaction. Easier that way .. and fixes a crash. --HG-- branch : HEAD

Cache file fixes, API changes, etc. It's still in somewhat ugly state, but getting better.. --HG-- branch : HEAD

Cache API updates. Don't return direct pointers to mmaped cache file anymore. --HG-- branch : HEAD

Cache updating is done now by first reserving space where to write, and then writing to it whenever buffer gets full. There is no persistent cache file locks anymore because of this, but it also means that the same cached field may be written multiple times to the file by different processes. Also since we reserve more space than we actually need at first, it some space can be wasted if multiple processes are updating the cache. --HG-- branch : HEAD

Added a framework for getting POP3 UIDL strings from mail-storage. The default still is uidvalidity.uid. Added beginnings of overriding it from X-UIDL header in mbox. --HG-- branch : HEAD

mbox code compiles again, but syncing is only partially implemented so accessing mboxes fails.. Also some cleanups to index-storage and maildir code. --HG-- branch : HEAD

importing new index code. mbox still broken. --HG-- branch : HEAD

Stop parsing cached headers when we've got everything. --HG-- branch : HEAD

Message parsing can now be done in two parts - header and body. We're now more eager at parsing MIME structure for message. It's done whenever body has to be fully read (eg. rfc822.size). --HG-- branch : HEAD

If BODY/BODYSTRUCTURE is requested with some other headers, parse the headers only once. If body contains multiple MIME parts, cache the internal MIME structure so BODY[part] fetching doesn't need to parse it again. --HG-- branch : HEAD

Changed how cached headers are stored. Previously when we added new headers, all the old headers were duplicated as well which wasted space. Also fixed several bugs in caching. --HG-- branch : HEAD

Index cache file rewrite. It's not finished yet and mbox support is completely broken. But it's getting difficult to maintain outside cvs :) --HG-- branch : HEAD

API change for expunging messages. Not exactly what I wanted, but good enough. --HG-- branch : HEAD

API change for updating message flags. --HG-- branch : HEAD

Better handling for multiline headers. Before we skipped headers larger than input buffer size (8k with read (default), 256k with mmap). The skipping was also a bit buggy. Now we parse the lines one at a time. There's also a way to read the header fully into memory before parsing it, if really needed. --HG-- branch : HEAD

FETCH ENVELOPE failed if it wasn't already cached. --HG-- branch : HEAD

Forgot to add --HG-- branch : HEAD