Updated copyright notices to include the year 2018.

global: Rename timing API to stats-dist API The API wasn't really about timings specifically, but about statistics in general. The new stats service was already using it for tracking statistics for non-timing related numbers.

lib-fs: Add fs_file/iter_init_with_event() Use the event for logging critical errors

lib-fs: Add fs_file/iter_init_parent()

lib-fs: Set fs_iter.flags immediately in fs_iter_init() This way backends don't need to set it.

lib-fs: Split allocation from fs.iter_init() to a separate fs.iter_alloc()

lib-fs: Split allocation from fs.file_init() to a separate fs.file_alloc()

global: Replace o_stream_ignore_last_errors() with o_stream_abort() wherever possible

lib-fs: Fix spelling mistakes Original work by @jsoref

global: Replace o_stream_nfinish() with o_stream_finish() whenever possible

lib-fs: Add init function that can use fs string directly This avoids doing splitting the fs driver and args everywhere and ensures that separators are consistent.

lib-fs: Add FS_METADATA_ORIG_PATH This allows fs backends to keep track of the original filename and use it for logging purposes and such.

lib-fs: Allow fs_set_metadata() to update already added metadata. Previously we were just appending the metadata multiple times with different values, which could have caused problems.

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

lib-fs: Fixed fs_metadata_find() to actually work

lib-fs: Added internal fs_metadata_find() helper function

lib-fs: Optimize changing FS_METADATA_INTERNAL_PREFIX metadata. file->metadata_changed doesn't need to change in that situation. It's used by fs-metawrap to find out if it needs to rewrite metadata during a write, but internal metadata isn't written to its header.

lib-fs: Lazily initialize timings to save memory.

fs-api: Retrieve errno before dest->copy_output is NULL'ed CID 10144: When aborting output, errno retrieval is attempted after dest->copy_output has already been unreffed. Found by coverity.

lib-fs: abort parent in fs_file_write_stream_abort_parent

lib-fs: Fixes to previous fs_write_stream_abort*() changes.

lib-fs: Add fs_write_stream_abort_parent and use it Will do all the things abort_error does, but leaves error untouched. You are expected to set one yourself.

lib-fs: Remove fs_write_stream_abort Do not use fs_write_stream_abort anymore, instead use fs_write_stream_abort_error to provide upstream some reason when it's used.

fs-api: Add and use fs_write_stream_abort_error This lets caller to specify error instead of setting it with fs_error. Doing it like this lets us percolate the error upwards.

lib-fs: Added fs_get_nlinks() Although fs_stat() could return this, its caller can't indicate whether it actually wants the link count. Usually fs_stat() is used only to get the file's size. In some backends it's not cheap to get the link count, so adding this function allows the caller to explicitly ask for it.

lib-fs: Added fs_metadata_init_or_clear() helper

lib-fs: Added write_bytes to statistics

lib-fs: Added fs_switch_ioloop()

lib-fs: fs_wait_async() can't fail, so make it return void. The individual requests can timeout, but not the waiting itself.

lib-fs: do NOT feed i_strconcat() with NULL fill the string with valid content beforehand.

lib-fs: Added more asserts

lib-fs: Clarified fs_write_stream_abort() API and dropped its _async(). We can't handle an abort after an async fs_write_stream_finish() is already going.

lib-fs: Added fs_write_stream_abort_async()

o_stream_send_istream() API changed again The new API enforces the caller to correctly handle all the possible situations. It also makes in unambiguous whether to wait for input or output stream.

global: Simplify code with the new o_stream_send_istream() API

global: Don't assume o_stream_send_istream() return bytes sent. This won't be true soon after its API changes.

lib: istream - more migration to i_stream_read_bytes() This time a coccinelle semantic patch, again hand checked: git grep 'i_stream_read_data' | sed s/:.*// | \ while read f; do spatch --sp-file istream.cocci --in-place "$f" ; done -- 8< --- istream.cocci --- @@ expression e1, e2, e3, e4; @@ - i_stream_read_data(e1, e2, e3, e4 - 1) + i_stream_read_bytes(e1, e2, e3, e4) -- 8< --- end Signed-off-by: Phil Carmody <phil@dovecot.fi>

lib: istream - migrate to i_stream_read_more() Scripted to find all the low-hanging-fruit (single line calls), but hand-checked: git grep i_stream_read_data.*,\ 0\) | sed s/:.*// | \ xargs sed -i -e 's/i_stream_read_data(\(.*\), \(.*\), \(.*\), 0)/i_stream_read_more(\1, \2, \3)/' Signed-off-by: Phil Carmody <phil@dovecot.fi>

lib-fs: Minor code cleanup

lib-fs: Set fs error on fs_write_stream_abort() Probably would be nice for fs_write_stream_abort() to have an error string parameter, which could be used instead of this generic error.

lib-fs: Added asserts to make sure async writes are finished before close

lib-fs: Fixes to stats count tracking fs_exists(), fs_copy(), fs_rename() and fs_delete() could have increased the count multiple times on async operations.

lib-fs: Allow multiple fs_read_stream() calls to seekable istreams. This fixes for example calling fs_default_copy() for an istream which was already opened: Panic: file fs-api.c: line 528 (fs_read_stream): assertion failed: (!file->istream_open)

lib-fs: Fixed multiple concurrent fs_read_stream() calls Return a new limit-istream, so each istream can have its own independent offset.

lib-fs: Assert-crash if file's istream isn't closed at fs_file_close() Some backends can't guarantee that the istream stays usable after the file is closed.

lib-fs: Added fs-test backend for helping with creation of fs-wrapper unit tests.

lib-fs: Added default implementations for fs_iter_*(), fs_copy() and fs_stat() The backends can now leave them as NULL, and the callers will get an error that they're not supported.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

lib-fs: Added fs_stats_get_read/write_usecs() These can be easily used to sum up all the timings for read and write categories.

lib-fs: Added asserts to fs_set_metadata() to make sure key and value are non-NULL Otherwise the bug would be noticed only much later when trying to dereference NULL pointer.

lib-fs: debug and enable_timing settings weren't being permanently copied to fs->set

lib-fs: timing_deinit()s was called too late - fs was already freed.

lib-fs: Track operation timing statistics if enable_timing setting is set.

lib-fs: Added "dict" backend, which is a wrapper to using lib-dict.

lib-fs: Fix to earlier write_stream_finish() commit 2f1378beeef6 accidentally reversed == and != check.

lib-fs: fs_copy*() didn't update metadata_changed correctly. It was supposed to be (only) set when the copying actually finished.

lib-fs: Small code cleanup for handling write_stream_finish()

lib-fs: fs_write() didn't update stats.write_count correctly. If backend implemented asynchronous write(), the write_count was updated multiple times. If backend didn't implement write(), then it was counted twice.

lib-fs: Call o_stream_nfinish() automatically for all backends in fs_write_stream_finish() Otherwise each backend needs to do it internally. fs-metawrap for example was missing this and causing assert-crashes.

lib-fs: Added "randomfail" driver. Using this in front of fs drivers allows randomly injecting failures. For example: mail_attachment_fs = randomfail:all=10,read=30,read-range=2000-3000:sis posix This means that all FS operations have a 10% chance of failing, except reads have a 30% chance of failing. If the read fails, it'll fail somewhere between offsets 2000-3000 (the default is 0, so it'll fail at the start of file). The supported operations are: wait metadata prefetch read write lock exists stat copy rename delete iter. "all" applies to all of them. The supported ranges are: read-range, write-range, iter-range.

lib-fs: fs_write_stream_abort() now ignores missing o_stream_nfinish() call

lib-fs: fs_write_stream() now automatically corks the ostream and uncorks at _finish().

lib-fs: Fixed fs-compress code to actually build and run. It's a plugin (because it depends on extra compression libraries). We want to register it only when it's explicitly attempted to be used, not before.

lib-fs: Added fs-compress wrapper. Future TODO could include automatically detecting the format of the input file, but this should be optional.

lib-fs: Forgot to initialize refcount in previous commit

lib-fs: Added fs_ref() and fs_unref() for reference counting.

lib-fs: Use the new t_str_replace() instead of doing it ourself.

lib-fs: fs_class_register() can now be called externally.

lib-fs: Added support for module_contexts in fs. They could also be added to fs_file if needed, but for now I didn't bother adding it.

lib-fs: When autoloading fs plugins with '-' in the name, translate them to '_' It's not possible for the function names to contain '-' in any case. Also Dovecot in general uses '-' instead of '_' in the configuration names. This change actually allows using either of them.

lib-fs: Minor code cleanup: Merge read_counted & prefetch_counted into read_or_prefetch_counted

lib-fs: Track how many different operations have been done on the fs.

lib-fs: Added fs_get_parent() and fs_get_driver() These can be used to iterate through fs wrappers and see what they are.

lib-fs: Added flag for iteration returning object IDs. The flag is only allowed to be used if FS_PROPERTY_OBJECTIDS is set (to avoid writing extra code for backends that don't support this).

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

lib-fs: Added fs_lookup_metadata() wrapper to fs_get_metadata()

lib-fs: Improved error logging

lib-fs: Keep track of all fs_iters in fs.

lib-fs: Added wait_ioloop/prev_ioloop helpers to struct fs.

lib-fs: Keep a linked list of all files within fs to help debugging.

lib-fs: Added FS_PROPERTY_COPY_METADATA and fs_set_metadata() tracking.

lib-fs: Added username and session_id to struct fs directly.

lib-fs: Make sure we don't leak data stack.

Updated copyright notices to include year 2014.

lib-fs: Added FS_OPEN_FLAG_ASYNC_NOQUEUE

lib-fs: If backend doesn't implement exists(), emulate it with stat().

lib-fs: Added support for asynchronous fs iteration.

lib-fs: Added support for giving a hash of the written data to be verified by storage.

Memory leak fixes.

lib-fs: If fs_default_copy() fails, close the copy stream immediately. The stream's origin may already be freed by the time the fs_file_close() is trying to close the stream.

lib-fs: Added fs_get_root_driver()

lib-fs: Added fs_file_close() to explicitly close all streams that the file has open.

lib-fs: Add data stack frames to most API calls, so the backends/callers don't need to.

lib-fs: Always keep the error in the parentmost fs. This fixes problems with getting the correct error message when wrapper filesystems are used.

lib-fs: Fixed fs_default_copy() to work with async writes.

lib-fs: If fs_read_stream() creates a seekable stream, keep it open for future reads.

fs-api: Fixed fs_copy() generic implementation's error messages

lib-fs: Avoid memory leaks if file was closed before fs_read()/fs_copy() was finished.

lib-fs: Added FS_OPEN_FLAG_SEEKABLE. Globally handle fs_read_stream() seekability and waits.

lib-fs: fs_write_stream_finish(file, NULL) is now alias to fs_write_stream_finish_async(file) This makes wrapper implementations easier.

Oops :) Update copyrights to 2013 without breaking all .c files.