bcb4e51a409d94ae670de96afb8483a4f7855294Stephan Bosch/* Copyright (c) 2004-2018 Dovecot authors, see the included COPYING file */
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen Users can be divided to three groups:
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen 1. Most users will use only a single IMAP client which caches everything
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen locally. For these users it's quite pointless to do any kind of caching
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen as it only wastes disk space. That might also mean more disk I/O.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen 2. Some users use multiple IMAP clients which cache everything locally.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen These could benefit from caching until all clients have fetched the
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen data. After that it's useless.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen 3. Some clients don't do permanent local caching at all. For example
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen Pine and webmails. These clients would benefit from caching everything.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen Some locally caching clients might also access some data from server
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen again, such as when searching messages. They could benefit from caching
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen only these fields.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen After thinking about these a while, I figured out that people who care
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen about performance most will be using Dovecot optimized LDA anyway
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen which updates the indexes/cache immediately. In that case even the first
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen user group would benefit from caching the same way as second group. LDA
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen reads the mail anyway, so it might as well extract some information
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen about it and store them into cache.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen So, group 1. and 2. could be optimally implemented by keeping things
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen cached only for a while. I thought a week would be good. When cache file
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen is compressed, everything older than week will be dropped.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen But how to figure out if user is in group 3? One quite easy rule would
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen be to see if client is accessing messages older than a week. But with
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen only that rule we might have already dropped useful cached data. It's
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen not very nice if we have to read and cache it twice.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen Most locally caching clients always fetch new messages (all but body)
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen when they see them. They fetch them in ascending order. Noncaching
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen clients might fetch messages in pretty much any order, as they usually
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen don't fetch everything they can, only what's visible in screen. Some
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen will use server side sorting/threading which also makes messages to be
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen fetched in random order. Second rule would then be that if a session
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen doesn't fetch messages in ascending order, the fetched field type will
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen be permanently cached.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen So, we have three caching decisions:
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen 1. Don't cache: Clients have never wanted the field
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen 2. Cache temporarily: Clients want this only once
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen 3. Cache permanently: Clients want this more than once
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen Different mailboxes have different decisions. Different fields have
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen different decisions.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen There are some problems, such as if a client accesses message older than
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen a week, we can't know if user just started using a new client which is
325d4ad220bd13f6d176391d962a0e33c856a7f6Timo Sirainen just filling its local cache for the first time. Or it might be a
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen client user hasn't just used for over a week. In these cases we
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen shouldn't have marked the field to be permanently cached. User might
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen also switch clients from non-caching to caching.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen So we should re-evaluate our caching decisions from time to time. This
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen is done by checking the above rules constantly and marking when was the
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen last time the decision was right. If decision hasn't matched for two
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen months, it's changed. I picked two months because people go to at least
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen one month vacations where they might still be reading mails, but with
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen different clients.
1b97a59edb073e9a89ac43a21a9abe5d590d4a56Timo Sirainenvoid mail_cache_decision_state_update(struct mail_cache_view *view,
a3738999f5a661fe9336f7418a23afc43140d385Timo Sirainen if (dec == (MAIL_CACHE_DECISION_NO | MAIL_CACHE_DECISION_FORCED)) {
a3738999f5a661fe9336f7418a23afc43140d385Timo Sirainen /* don't update last_used */
008a83e9f680f04f69789fb702232416eab2a86cTimo Sirainen if (ioloop_time - cache->fields[field].field.last_used > 3600*24) {
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen /* update last_used about once a day */
008a83e9f680f04f69789fb702232416eab2a86cTimo Sirainen cache->fields[field].field.last_used = (uint32_t)ioloop_time;
0beb5d6c661ee68130a954ed0f31a34c19195fb7Timo Sirainen if (cache->field_file_map[field] != (uint32_t)-1)
450b902af091adca7b3fe8165a3c64c64e41e5c5Timo Sirainen /* a) forced decision
450b902af091adca7b3fe8165a3c64c64e41e5c5Timo Sirainen b) not cached, mail_cache_decision_add() will handle this
450b902af091adca7b3fe8165a3c64c64e41e5c5Timo Sirainen c) permanently cached already, okay. */
450b902af091adca7b3fe8165a3c64c64e41e5c5Timo Sirainen /* see if we want to change decision from TEMP to YES */
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen if (uid < cache->fields[field].uid_highwater ||
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen /* a) nonordered access within this session. if client doesn't
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen request messages in growing order, we assume it doesn't
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen have a permanent local cache.
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen b) accessing message older than one week. assume it's a
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen client with no local cache. if it was just a new client
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen generating the local cache for the first time, we'll
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen drop back to TEMP within few months. */
4bbee99b3aef449a9a2a11a5b5cf1ca486915c49Timo Sirainen cache->fields[field].field.decision = MAIL_CACHE_DECISION_YES;
0beb5d6c661ee68130a954ed0f31a34c19195fb7Timo Sirainen if (cache->field_file_map[field] != (uint32_t)-1)
992a13add4eea0810e4db0f042a595dddf85536aTimo Sirainenvoid mail_cache_decision_add(struct mail_cache_view *view, uint32_t seq,
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen unsigned int field)
8fa86f7ef06aa6cf0239c7ca2eb98889691d40d4Timo Sirainen if (MAIL_CACHE_IS_UNUSABLE(cache) || view->no_decision_updates)
4bbee99b3aef449a9a2a11a5b5cf1ca486915c49Timo Sirainen if (cache->fields[field].field.decision != MAIL_CACHE_DECISION_NO) {
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen /* a) forced decision
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen b) we're already caching it, so it just wasn't in cache */
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen /* field used the first time */
4bbee99b3aef449a9a2a11a5b5cf1ca486915c49Timo Sirainen cache->fields[field].field.decision = MAIL_CACHE_DECISION_TEMP;
215cdc8dbdeee2e0e91e6f67d9c6a575a90e118aAki Tuomivoid mail_cache_decisions_copy(struct mail_index_transaction *itrans,