imapc: Fix sending initial FETCH after reconnection SELECTs mailbox Move sending the FETCH when the SELECT returns tagged OK reply instead of delaying it until mailbox is next synced. Most importantly this allows sending the FETCH before any retried commands that are also sent after SELECT receives tagged reply.

imapc: Move imapc_mailbox_fetch_state*() to imapc-mailbox.c Mainly to make the next commit smaller.

lib-storage: convert imapc to use container_of

imapc: Cleanup - Rename auth_error to auth_failed_reason for consistency

imapc: Change auth_failed boolean to more specific auth_failed_state Since we now rely on auth_failed_state being initialized to zero, explicitly set IMAPC_COMMAND_STATE_OK to be defined as 0.

imapc: include imapc-client.h in imapc-storage.h It was already required to be able to successfully include imapc-storage.h.

imapc: Add imapc_mailbox.capabilities Use it instead of imapc_client_get_capabilities(). Simplifies the following patch.

imapc: Rename imapc_storage_has_modseqs() to imapc_mailbox_has_modseqs() Simplifies the following patch.

imapc: Fail user creation if login to imapc_host fails. This causes imapc to actually wait for the login to succeed or fail. Such a wait was already done by the imap code, which will be removed by the next patch.

imapc: Cleanup - change delayed_expunged_uids to seq_range. This simplifies the next commit.

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

imapc: Added support for imapc_features=modseq If the remote server supports CONDSTORE or QRESYNC extensions we'll use the remote's MODSEQ and HIGHESTMODSEQ counts. There are some situations where the HIGHESTMODSEQ isn't updated exactly correctly on an open mailbox, so this feature shouldn't be fully relied on. It was primarily implemented for dsync+imapc support - both for preserving modseqs and also for HIGHESTMODSEQ lookups.

imapc: Fixed escaping/unescaping mailbox names. This fixes accessing mailbox names with '%' (the escape_char). It also fixes local namespace separator being different from the remote separator.

imapc: If login is aborted at deinit, don't log an error.

imapc: If authentication fails, preserve the error message as storage/list error. This isn't very helpful yet, because the mailbox list creation itself fails.

imapc: If imapc_features has gmail-migration, fetch X-GM-MSGID at startup and cache it. This works only if index files haven't been disabled.

imapc: Fixed STATUS_FIRST_RECENT_UID to return the (mostly) correct UID.

imapc: Added imapc_features=search support for sending SEARCH commands. Currently requires the remote server to support ESEARCH (but this would be easy to avoid). This is only minimally tested for now, so bugs may exist (especially related to sub-queries).

imapc: Implemented imapc_features=gmail-pop3 that also migrates pop3_deleted_flag. This also caused gmail-pop3-uidl feature to be renamed to simply gmail-pop3, which includes both the features.

imapc: Avoid hanging when running into server problems. imapc_mail_fetch_flush() caused imail->fetch_count to drop to 0, but imapc_mailbox_run() still started imapc_client_run() which was doing nothing.

imapc: Merge identical FETCH commands together (only updating UID range)

imapc: Reorganize code so that imapc_list works without imapc_storage. Most importantly fixes crashes when imapc_list is trying to lookup hierarchy separator before storage is created.

imapc: If imapc isn't the inbox=yes namespace, do the login and initial LIST in background.

imapc: If APPEND to selected mailbox doesn't send EXISTS, try if NOOP sends it. This makes Dovecot behave better with Courier.

Reverted 269104a0821b (Build imapc and pop3c always as plugins.) This is no longer needed now that lib-ssl-iostream can be linked without linking openssl.

Replaced *_INDEX_PREFIX macros with a common MAIL_INDEX_PREFIX.

lib-storage: Added support for "broken_char" setting for escaping invalid mailbox names. This is currently enabled only for imapc backend, where '~' character is used for it. Most importantly this allows migrating mailboxes from other IMAP servers that contain mailbox names that Dovecot's imapc backend couldn't otherwise access.

Build imapc and pop3c always as plugins. This avoids having to link openssl library to all binaries that use lib-storage. It appears that simply by linking with openssl causes each such process to use 100-200 kB of more memory. With 10k imap processes this is 1-2 GB of wasted memory. The imapc and pop3c are still registered as stubs, and their plugins are automatically loaded when needed.

lib-storage: Added MAILBOX_METADATA_BACKEND_NAMESPACE

Array API changed: ARRAY_DEFINE(name, type) -> ARRAY(type) name Easy way to update your existing code: perl -i -pe 's:ARRAY_DEFINE\(([^,]+), *([^)]+)\);:ARRAY($2) $1;:' **/*.[ch]

imapc: Added imapc_features=rfc822.size setting to use RFC822.size for physical sizes.

Compiler warning fixes.

imapc: Added support for fetching GUID from remote server, if supported. Currently this is only done for GMail.

imapc: Don't use separate indexes/ directory anymore. Fixes/cleanups to make it possible.

imapc: Support retrying some IMAP commands if we get disconnected.

imapc: Fixed detecting when messages are missing from index.

imapc: Initial support for automatically reconnecting to remote server.

imapc: Fixed handling keywords. The status->keywords must point to mail_index's keywords.

imapc: Code cleanup: avoid unnecessarily accessing imapc-client-private.h

imapc: Fixed fetching data for a saved, but uncommitted, mail.

imapc: Keep the last fetched message body cached until mailbox is closed. This primarily helps partial IMAP fetches so each partial fetch doesn't redownload the message body.

iampc: Small code cleanup.

imapc: Added more checks to catch buggy IMAP server responses.

imapc: Another try at fixing mUTF-7 mailbox names.

imapc: Use mUTF-7 for mailbox names.

imapc: Don't crash if a newly seen uncommitted message is expunged.

imapc: mailbox_status now returns permanent flags/keywords as they are on remote server.

imapc: Redesigned remote sequence <-> index file record mapping is done. The previous code didn't work when multiple connections modified the same index files.

imapc: Delay handling new messages and setting uidvalidity/uidnext until sync.

imapc: Fixed handling immediate changes when starting IDLE.

imapc: Error logging cleanups. Don't bother to log disconnection errors for NOOP.

imapc: Fixed error handling while opening mailbox.

imapc: Fixed syncing external changes to mailbox when opening it.

imapc: Removed support for connection locking and multiple connections to mailbox. If needed they can be added back by reversing this change, but for now they're no longer needed.

Search supports now prefetching data for returned mails. Dropped imapc's own prefetching. mail_prefetch_count specifies how many mails can be kept open and issue a prefetch. This works using posix_fadvise(POSIX_FADV_WILLNEED) for maildir, sdbox and cydir backends. Apparently only Linux supports this. imapc backend also implements this internally by sending wanted IMAP commands to remote server. The command pipelining helps with latency. This change also makes it actually possible for imapc backend to first check if wanted data is already cached in local index and avoid sending unnecessary IMAP commands to remote server.

lib-storage: Changed mailbox_search_*() API to return pointer struct mail. This reduces the flexibility, because now different mailbox_search_next() calls can't have different mail parameters with possibly different wanted_fields settings. This flexibility was never used though, and removing it gives a couple of benefits: * it's now easier for backends to implement prefetch and parallel search * usage is easier since mail_alloc()/mail_free() is no longer needed

imapc: Initial support for local index files.

imapc: Use multiple client connections when necessary. Some other fixes.

imapc: Many fixes related to syncing and error handling.

imapc: Support IDLE even if remote IMAP server doesn't support it.

imapc: Moved settings from plugin {} section to proper imapc_* settings.

imapc: Write large message bodies to temp files rather than keeping in memory.

imapc: Send UID FETCH commands with larger uidset parameter if possible. This code assumes that server doesn't reorganize FETCH replies when it's given an increasing UID range, which in theory IMAP server would be allowed to do, but I doubt there are any.

imapc: Added support for saving and copying messages.

imapc: Fixed mail_set_seq/uid() to work.

imapc: Implemented more methods.

imapc: Handle properly mailbox changes while it's selected. Added support for IDLE.

imapc: Code cleanups and fixes.

imapc: Added support for LIST/LSUB.

Added initial implementation of "imapc" storage. It can be used to create a "smart IMAP proxy" where Dovecot uses remote IMAP server as a mail storage. This is a very rough early implementation. Performance isn't good, many required features are missing, error handling is lacking and code needs de-uglification. Still, it should be enough for selecting INBOX and accessing mails in it.