tree-wide: remove Emacs lines from all files This should be handled fine now by .dir-locals.el, so need to carry that stuff in every file.

journal: move mmap() ENOMEM loop to function Introduces mmap_try_harder()

journal: move window initialization to window_add

journal: fix boolean handling in MMapCache Let's use bitfields for our booleans, and don't try to apply binary OR or addition on them, because that's weird and we should instead use logical OR only.

journal: make mmap_cache_unref() a NOP when NULL is passed, like all other destructors

tree-wide: make macros for converting fds to pointers and back generic and use them everywhere

tree-wide: sort includes Sort the includes accoding to the new coding style.

util-lib: split out allocation calls into alloc-util.[ch]

remove unused includes This patch removes includes that are not used. The removals were found with include-what-you-use which checks if any of the symbols from a header is in use.

journald: process SIGBUS for the memory maps we set up Even though we use fallocate() it appears that file systems like btrfs will trigger SIGBUS on certain low-disk-space situation. We should handle that, hence catch the signal, add it to a list of invalidated pages, and replace the page with an empty memory area. After each write check if SIGBUS was triggered, and consider the write invalid if it was. This should make journald a lot more robust with file systems where fallocate() is not reliable, for example all CoW file systems (btrfs...), where changing written data can fail with disk full errors. https://bugzilla.redhat.com/show_bug.cgi?id=1045810

journal: replace contexts hashmap with a plain array try_context() is such a hot path that the hashmap lookup is expensive. The number of contexts is small - it is the number of object types. Using a hashmap is overkill. A plain array will do. Before: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m9.445s user 0m9.228s sys 0m0.213s After: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m5.438s user 0m5.266s sys 0m0.170s

journal: delete unused function mmap_cache_close_context This never had any callers. Contexts are freed when the MMapCache is freed.

journal: remove journal_file_object_keep/release functions The only user is sd_journal_enumerate_unique() and, as explained in the previous commit (fed67c38e3 "journal: map objects to context set by caller, not by actual object type"), the use of them there is now superfluous. Let's remove them. This reverts major parts of commits: ae97089d49 journal: fix access to munmapped memory in sd_journal_enumerate_unique 06cc69d44c sd-journal: fix sd_journal_enumerate_unique skipping values Tested with an "--enable-debug" build and "journalctl --list-boots". It gives the expected number of results. Additionally, if I then revert the previous commit ("journal: map objects to context set by caller, not to actual object type"), it crashes with SIGSEGV, as expected.

journal: add debug mode for mmap-cache (--enable-debug=mmap-cache) This is useful for exposing unsafe access to mmapped objects after the context that they were mapped in was already moved. For example: journal_file_move_to_object(f1, OBJECT_DATA, p1, &o1); journal_file_move_to_object(f2, OBJECT_DATA, p2, &o2); t = o1->object.type; /* this usually works, but is unsafe */

sd-journal: fix sd_journal_enumerate_unique skipping values sd_journal_enumerate_unique will lock its mmap window to prevent it from being released by calling mmap_cache_get with keep_always=true. This call may return windows that are wider, but compatible with the parameters provided to it. This can result in a mismatch where the window to be released cannot properly be selected, because we have more than one window matching the parameters of mmap_cache_release. Therefore, introduce a release_cookie to be used when releasing the window. https://bugs.freedesktop.org/show_bug.cgi?id=79380

hashmap: introduce hash_ops to make struct Hashmap smaller It is redundant to store 'hash' and 'compare' function pointers in struct Hashmap separately. The functions always comprise a pair. Store a single pointer to struct hash_ops instead. systemd keeps hundreds of hashmaps, so this saves a little bit of memory.

journal: do not leak mmaps on OOM After a section of memory is succesfully allocated, some of the following actions can still fail due to lack of memory. In this case -ENOMEM is returned without actually freeing the already mapped memory. Found with coverity. Fixes: CID#1237762

journal: fix access to munmapped memory in sd_journal_enumerate_unique sd_j_e_u needs to keep a reference to an object while comparing it with possibly duplicate objects in other files. Because the size of mmap cache is limited, with enough files and object to compare to, at some point the object being compared would be munmapped, resulting in a segmentation fault. Fix this issue by turning keep_always into a reference count that can be increased and decreased. Other callers which set keep_always=true are unmodified: their references are never released but are ignored when the whole file is closed, which happens at some point. keep_always is increased in sd_j_e_u and later on released.

journald: keep statistics on how of we hit/miss the mmap cache

list: make our list macros a bit easier to use by not requring type spec on each invocation We can determine the list entry type via the typeof() gcc construct, and so we should to make the macros much shorter to use.

journal: fix hashmap leak in mmap-cache hashmap_free() wasn't being called on m->contexts and m->fds resulting in a leak. To reproduce do: while(1) { sd_journal_open(&j, SD_JOURNAL_LOCAL_ONLY); sd_journal_close(j); } Memory usage will increase until OOM.

Rearrange a few fields to reduce holes

Add __attribute__((const, pure, format)) in various places I'm assuming that it's fine if a _const_ or _pure_ function calls assert. It is assumed that the assert won't trigger, and even if it does, it can only trigger on the first call with a given set of parameters, and we don't care if the compiler moves the order of calls.

journal: Set the last_unused pointer correctly when attaching an unused window It seems the previous code was copy/pasted from context_detach_window() but not updated.

journal: Properly track the number of allocated windows. Checks were already in place to make sure that the number of windows was limited to 64, but the count was never incremented or decremented.

journal: always keep marked mmap windows around

journal: completely rework the mmap cache as I too dumb to actually understand it Instead of doing hand optimized fd bisect arrays just use plain old hashmaps. Now I can understand my own code again. Yay! As a side effect this should fix some bad memory accesses caused by accesses after mmap(), introduced in 189.

man: fix a bunch of typos in docs https://bugs.freedesktop.org/show_bug.cgi?id=54501

journal: be more careful when keeping around mmaps we still need

mmap: resize arrays dynamically

journal: add superficial structure verifier

journal: implement basic journal file verification logic

journal: implement generic sharable mmap caching logic instead of having one simple per-file cache implement an more comprehensive one that works for multiple files and can actually maintain multiple maps per file and per object type.