mail-cache-decisions.c revision e015e2f7e7f48874495f9df8b0dd192b7ffcb5cc
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen/* Copyright (C) 2004 Timo Sirainen */
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen Users can be divided to three groups:
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen 1. Most users will use only a single IMAP client which caches everything
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen locally. For these users it's quite pointless to do any kind of caching
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen as it only wastes disk space. That might also mean more disk I/O.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen 2. Some users use multiple IMAP clients which cache everything locally.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen These could benefit from caching until all clients have fetched the
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen data. After that it's useless.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen 3. Some clients don't do permanent local caching at all. For example
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen Pine and webmails. These clients would benefit from caching everything.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen Some locally caching clients might also access some data from server
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen again, such as when searching messages. They could benefit from caching
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen only these fields.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen After thinking about these a while, I figured out that people who care
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen about performance most will be using Dovecot optimized LDA anyway
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen which updates the indexes/cache immediately. In that case even the first
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen user group would benefit from caching the same way as second group. LDA
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen reads the mail anyway, so it might as well extract some information
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen about it and store them into cache.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen So, group 1. and 2. could be optimally implemented by keeping things
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen cached only for a while. I thought a week would be good. When cache file
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen is compressed, everything older than week will be dropped.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen But how to figure out if user is in group 3? One quite easy rule would
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen be to see if client is accessing messages older than a week. But with
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen only that rule we might have already dropped useful cached data. It's
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen not very nice if we have to read and cache it twice.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen Most locally caching clients always fetch new messages (all but body)
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen when they see them. They fetch them in ascending order. Noncaching
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen clients might fetch messages in pretty much any order, as they usually
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen don't fetch everything they can, only what's visible in screen. Some
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen will use server side sorting/threading which also makes messages to be
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen fetched in random order. Second rule would then be that if a session
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen doesn't fetch messages in ascending order, the fetched field type will
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen be permanently cached.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen So, we have three caching decisions:
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen 1. Don't cache: Clients have never wanted the field
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen 2. Cache temporarily: Clients want this only once
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen 3. Cache permanently: Clients want this more than once
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen Different mailboxes have different decisions. Different fields have
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen different decisions.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen There are some problems, such as if a client accesses message older than
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen a week, we can't know if user just started using a new client which is
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen just filling it's local cache for the first time. Or it might be a
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen client user hasn't just used for over a week. In these cases we
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen shouldn't have marked the field to be permanently cached. User might
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen also switch clients from non-caching to caching.
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen So we should re-evaluate our caching decisions from time to time. This
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen is done by checking the above rules constantly and marking when was the
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen last time the decision was right. If decision hasn't matched for two
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen months, it's changed. I picked two months because people go to at least
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen one month vacations where they might still be reading mails, but with
3c57664b9dce82cd3e43347394b92ef3591b8901Timo Sirainen different clients.
992a13add4eea0810e4db0f042a595dddf85536aTimo Sirainenvoid mail_cache_decision_lookup(struct mail_cache_view *view, uint32_t seq,
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen unsigned int field)
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen if (cache->fields[field].decision != MAIL_CACHE_DECISION_TEMP) {
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen /* a) forced decision
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen b) not cached, mail_cache_decision_add() will handle this
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen c) permanently cached already, okay. */
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen /* see if we want to change decision from TEMP to YES */
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen if (mail_index_lookup_uid(view->view, seq, &uid) < 0 ||
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen if (ioloop_time - cache->fields[field].last_used > 3600*24) {
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen /* update last_used about once a day */
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen if (uid < cache->fields[field].uid_highwater ||
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen /* a) nonordered access within this session. if client doesn't
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen request messages in growing order, we assume it doesn't
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen have a permanent local cache.
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen b) accessing message older than one week. assume it's a
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen client with no local cache. if it was just a new client
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen generating the local cache for the first time, we'll
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen drop back to TEMP within few months. */
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen cache->fields[field].decision = MAIL_CACHE_DECISION_YES;
992a13add4eea0810e4db0f042a595dddf85536aTimo Sirainenvoid mail_cache_decision_add(struct mail_cache_view *view, uint32_t seq,
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen unsigned int field)
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen if (cache->fields[field].decision != MAIL_CACHE_DECISION_NO) {
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen /* a) forced decision
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen b) we're already caching it, so it just wasn't in cache */
8aacc9e7c84f8376822823ec98c2f551d4919b2eTimo Sirainen /* field used the first time */
e015e2f7e7f48874495f9df8b0dd192b7ffcb5ccTimo Sirainen cache->fields[field].decision = MAIL_CACHE_DECISION_TEMP;