tree-wide: remove Emacs lines from all files This should be handled fine now by .dir-locals.el, so need to carry that stuff in every file.

journal: Drop monotonicity check when appending to journal file Remove the check that triggers rotation of the journal file when the arriving log entry had a monotonic timestamp smaller that the previous log entry. This check causes unnecessary rotations when journal-remote was receiving from multiple senders, therefore monotonicity can not be guaranteed. Also, it does not offer any useful functionality for systemd-journald.

journal: add missing space to switch statement

journal: add asserts on f->(data|field)_hash_table Functions dereferencing these members should assert their non-NULL state.

journal: add asserts for f->header Just some additional asserts in functions dereferencing f->header.

journald: minor fixes This primarily contains some minor coding style fixups for 7a24f3bf2fb181243a1957a0cdd54cd919396793 and earlier changes. Specifically: * Don't log at log levels above LOG_DEBUG from "library" code like journal-file.c * Don't negate errno values before passing them to log_debug_errno(), as the call can handle this fine anyway * Cast some calls we knowingly ignore the return values of to (void) * Don't clobber function call-by-ref return values on failure * Don't mix function calls and variable declarations in one line There's also one more relevant change: when failing to enqueue a journal change fs event, we'll run it immediately.

journal: coalesce ftruncate()s in 250ms windows Prior to this change every journal append causes an ftruncate() for the sake of inotify propagation of the mmap-based writes. With this change the notification is deferred up to ~250ms, coalescing any repeated journal writes during the deferred period into a single ftruncate(). The ftruncate() call isn't free and doing it on every append adds unnecessary overhead and latency in the journald event loop. Introduces journal_file_enable_post_change_timer() which manages a timer on the provided sd-event instance for scheduling coalesced ftruncates. The ftruncate() behavior is unchanged unless journal_file_enable_post_change_timer() is called on the JournalFile. While not a tremendous improvement, profiling systemd-journald event loop latencies using instrumentation as introduced by 34b8751 it was observed that coalescing the ftruncates was low-hanging fruit worth pursuing. Note orders 12 and 13 shifting left into order 11 and order 6 dipping into order 5: Unmodified: log2(us) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ----------------------------------------------------------- [10685.414572] 0 0 0 0 38 602 61 2 290 60 1643 2554 13 1 4 1 0 0 1 [10690.415114] 0 0 0 0 0 646 54 7 309 44 2073 2148 17 1 3 0 0 0 1 [10695.415509] 0 0 0 0 1 650 73 3 324 37 2071 2270 9 0 0 1 0 1 0 [10700.416297] 0 0 0 0 0 659 50 4 318 38 2111 2152 6 0 1 0 0 1 1 [10705.417136] 0 0 0 0 2 660 48 4 320 38 2129 2146 12 1 1 0 0 1 1 [10710.489114] 0 0 0 0 0 673 38 3 321 37 1925 2339 7 0 0 0 0 1 1 [10715.489613] 0 0 0 0 3 656 64 8 317 48 2365 2007 7 0 0 0 0 0 1 Coalesced: log2(us) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ----------------------------------------------------------- [ 6169.161360] 0 0 0 1 24 786 54 11 389 24 4192 771 6 4 0 0 1 0 1 [ 6174.161705] 0 0 0 1 18 800 35 6 380 27 3977 893 3 1 0 0 1 0 1 [ 6179.162741] 0 0 0 1 28 768 51 4 391 16 3998 831 5 3 0 0 0 0 2 [ 6184.162856] 0 0 0 0 19 770 60 2 376 26 3795 1004 9 5 1 0 1 0 1 [ 6189.163279] 0 0 0 0 28 761 49 7 372 27 3729 1056 3 2 0 0 1 0 1 [ 6194.164255] 0 0 0 0 25 785 49 7 394 19 3996 908 6 3 2 0 0 0 1 [ 6199.164658] 0 0 0 0 29 797 35 5 389 18 3995 898 3 4 1 1 1 0 1 The remaining high-order delays are a result of the synchronous fsyncs in systemd-journald, beyond the scope of this commit.

journal: add dst_allocated_size parameter for compress_blob compress_blob took src, src_size, dst and *dst_size, but dst_size wasn't used as an input parameter with the size of dst, but only as an output parameter. dst was implicitly assumed to be at least src_size-1. This code wasn't *wrong*, because the only real caller in journal-file.c got it right. But it was misleading, and the tests in test-compress.c got it wrong, and worked only because the output buffer happened to be the same size as input buffer. So add a seperate dst_allocated_size parameter to make it explicit what the size of the buffer is, and to allow test to proceed with different output buffer sizes.

journal: make mmap_cache_unref() a NOP when NULL is passed, like all other destructors

journal: reduce minimum journal file size to 512 KiB For low end embedded systems 4 MiB for each journal file is a lot of memory. Journald will use at least 512 KiB even if JOURNAL_FILE_SIZE_MIN is set to less than that so just use 512 KiB.

journal: return better error for empty files When reading stuff, we should only return EIO when an actual read error occured, not when we don't like the data for whatever reason. We already return ENODATA for all other kinds of file truncation, hence do the same for the most obvious kind, so that callers know what ENODATA means.

util-lib: split out allocation calls into alloc-util.[ch]

util-lib: split out file attribute calls to chattr-util.[ch]

util-lib: split xattr-related calls into xattr-util.[ch]

util-lib: split string parsing related calls from util.[ch] into parse-util.[ch]

util-lib: split out fd-related operations into fd-util.[ch] There are more than enough to deserve their own .c file, hence move them over.

util-lib: split our string related calls from util.[ch] into its own file string-util.[ch] There are more than enough calls doing string manipulations to deserve its own files, hence do something about it. This patch also sorts the #include blocks of all files that needed to be updated, according to the sorting suggestions from CODING_STYLE. Since pretty much every file needs our string manipulation functions this effectively means that most files have sorted #include blocks now. Also touches a few unrelated include files.

journal: irrelevant coding style fixes

journal: fix error handling when compressing journal objects Let's make sure we handle compression errors properly, and don't misunderstand an error for success. Also, let's actually compress things if lz4 is enabled. Fixes #1662.

journal: rework vacuuming logic Implement a maximum limit on number of journal files to keep around. Enforcing a limit is useful on this since our performance when viewing pays a heavy penalty for each journal file to interleve. This setting is turned on now by default, and set to 100. Also, actully implement what 348ced909724a1331b85d57aede80a102a00e428 promised: use whatever we find on disk at startup as lower bound on how much disk space we can use. That commit introduced some provisions to implement this, but actually never did. This also adds "journalctl --vacuum-files=" to vacuum files on disk by their number explicitly.

journal: improve some messages Indicate that we are ignoring errors, when we ignore them, and log that at LOG_WARNING level. Use the right error code for the log message.

journal: simplify things by using the LESS_BY() macro

journal: make journal_file_close() return NULL The way it is customary everywhere else in our sources.

tree-wide: never use the off_t unless glibc makes us use it off_t is a really weird type as it is usually 64bit these days (at least in sane programs), but could theoretically be 32bit. We don't support off_t as 32bit builds though, but still constantly deal with safely converting from off_t to other types and back for no point. Hence, never use the type anymore. Always use uint64_t instead. This has various benefits, including that we can expose these values directly as D-Bus properties, and also that the values parse the same in all cases.

Bug #944: Deletion of unnecessary checks before calls of the function "free" The function "free" is documented in the way that no action shall occur for a passed null pointer. It is therefore not needed that a function caller repeats a corresponding check. http://stackoverflow.com/questions/18775608/free-a-null-pointer-anyway-or-check-first This issue was fixed by using the software Coccinelle 1.0.1.

journal: avoid mapping empty data and field hash tables When a new journal file is created we write the header first, then sync and only then create the data and field hash tables in them. That means to other processes it might appear that the files have a valid header but not data and field hash tables. Our reader code should be able to deal with this. With this change we'll not map the two hash tables right-away after opening a file for reading anymore (because that will of course fail if the objects are missing), but delay this until the first time we access them. On top of that, when we want to look something up in the hash tables and we notice they aren't initialized yet, we consider them empty. This improves handling of some journal files reported in #487.

journal: don't force FS_NOCOW_FL on new journal files, but warn if it is missing This way users have the freedom to set or unset the FS_NOCOW_FL flag on their journal files by setting it on the journal directory. Since our default tmpfiles configuration now sets this flag on the directory the flag is set by default on new files, however people can opt-out of this by masking the tmpfiles file for it.

shared: add random-util.[ch]

util: merge change_attr_fd() and chattr_fd()

fix gcc warnings about uninitialized variables like: src/shared/install.c: In function ‘unit_file_lookup_state’: src/shared/install.c:1861:16: warning: ‘r’ may be used uninitialized in this function [-Wmaybe-uninitialized] return r < 0 ? r : state; ^ src/shared/install.c:1796:13: note: ‘r’ was declared here int r; ^

journal: fix return code Introduced in fa6ac76083b8ff. Might be related to CID #1261724, but I don't know if coverity can recurse this deep.

journal-file: update format string to remove cast

journal: align comments to make them more legible

journal: fix Inappropriate ioctl for device on ext4 Logs constantly show systemd-journald[395]: Failed to set file attributes: Inappropriate ioctl for device This is because ext4 does not support FS_NOCOW_FL. [zj: fold into one conditional as suggested on the ML and fix (preexisting) r/errno confusion in error message.]

journal: make skipping of exhausted journal files effective again Commit 668c965af "journal: skipping of exhausted journal files is bad if direction changed" fixed a correctness issue, but it also significantly limited the cases where the optimization that skips exhausted journal files could apply. As a result, some journalctl queries are much slower in v219 than in v218. (e.g. queries where a "--since" cutoff should have quickly eliminated older journal files from consideration, but didn't.) If already in the initial iteration find_location_with_matches() finds no entry, the journal file's location is not updated. This is fine, except that: - We must update at least f->last_direction. The optimization relies on it. Let's separate that from journal_file_save_location() and update it immediately after the direction checks. - The optimization was conditional on "f->current_offset > 0", but it would always be 0 in this scenario. This check is unnecessary for the optimization.

remove unused includes This patch removes includes that are not used. The removals were found with include-what-you-use which checks if any of the symbols from a header is in use.

journald: don't specify inline in local functions Leave it to the compiler to figure out whether it shall inline stuff or not. Only place where using static inline is OK to use is in in header files, really.

Fix some format strings for enums, they are signed

util: make it easy to initialize the crtime from the current time in fd_setcrtime()

journald: turn off COW for journal files on btrfs btrfs' COW logic results in heavily fragment journal files, which is detrimental for perfomance. Hence, turn off COW for journal files as we create them. Turning off COW comes at the cost of data integrity guarantees, but this should be acceptable, given that we do our own checksumming, and generally have a pretty conservative write pattern. Also see discussion on linux-btrfs: http://www.spinics.net/lists/linux-btrfs/msg41001.html

journal: consider file deletion errors a reason for rotation

journald: whenever we rotate a file, btrfs defrag it Our write pattern is quite awful for CoW file systems (btrfs...), as we keep updating file parts in the beginning of the file. This results in fragmented journal files. Hence: when rotating files, defragment them, since at that point we know that no further write accesses will be made.

journald: when we detect the journal file we are about to write to has been deleted, rotate https://bugzilla.redhat.com/show_bug.cgi?id=1171719

journald: add some additional checks before we divide by values read from journal file headers Since the file headers might be replaced by zeroed pages now due to sigbus we should make sure we don't end up dividing by zero because we don't check values read from journal file headers for changes.

journald: process SIGBUS for the memory maps we set up Even though we use fallocate() it appears that file systems like btrfs will trigger SIGBUS on certain low-disk-space situation. We should handle that, hence catch the signal, add it to a list of invalidated pages, and replace the page with an empty memory area. After each write check if SIGBUS was triggered, and consider the write invalid if it was. This should make journald a lot more robust with file systems where fallocate() is not reliable, for example all CoW file systems (btrfs...), where changing written data can fail with disk full errors. https://bugzilla.redhat.com/show_bug.cgi?id=1045810

util: make creation time xattr logic more generic

journal: journal_file_next_entry() does not need pointer to current Object The current offset is sufficient information.

journal: compare candidate entries using JournalFiles' locations When comparing the locations of candidate entries, we can rely on the location information stored in struct JournalFile.

journal: keep per-JournalFile location info during iteration In next_beyond_location() when we find a candidate entry in a journal file, save its location information in struct JournalFile. The purpose of remembering the locations of candidate entries is to be able to save work in the next iteration. This patch does only the remembering part. LOCATION_SEEK means the location identifies a candidate entry. When a winner is picked from among candidates, it becomes LOCATION_DISCRETE. LOCATION_TAIL here signifies we've iterated the file to the end (or the beginning in the case of reversed direction).

journal: abstract the resetting of JournalFile's location

journal: delete unused function journal_file_skip_entry() Its only caller is a test.

journal: delete unused function journal_file_move_to_entry_by_offset()

journal: replace contexts hashmap with a plain array try_context() is such a hot path that the hashmap lookup is expensive. The number of contexts is small - it is the number of object types. Using a hashmap is overkill. A plain array will do. Before: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m9.445s user 0m9.228s sys 0m0.213s After: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m5.438s user 0m5.266s sys 0m0.170s

journal: push type_to_context conversion down to journal_file_move_to()

journal: have a named enum ObjectType

journal: consistently use OBJECT_<type> names instead of numbers Note that numbers 0 and -1 are both replaced with OBJECT_UNUSED, because they are treated the same everywhere (e.g. type_to_context() translates them both to 0).

journal: move type_to_context() to journal-file.c It has no other callers. It does not need to be in the header file.

journal: remove journal_file_object_keep/release functions The only user is sd_journal_enumerate_unique() and, as explained in the previous commit (fed67c38e3 "journal: map objects to context set by caller, not by actual object type"), the use of them there is now superfluous. Let's remove them. This reverts major parts of commits: ae97089d49 journal: fix access to munmapped memory in sd_journal_enumerate_unique 06cc69d44c sd-journal: fix sd_journal_enumerate_unique skipping values Tested with an "--enable-debug" build and "journalctl --list-boots". It gives the expected number of results. Additionally, if I then revert the previous commit ("journal: map objects to context set by caller, not to actual object type"), it crashes with SIGSEGV, as expected.

journal: map objects to context set by caller, not by actual object type When the caller of journal_file_move_to_object() specifies type==0, the object header is at first mapped in context 0. Then after the header is checked, the whole object is mapped in a context determined by the actual object type (which is not even range-checked using type_to_context()). This looks wrong. It should map in the caller-specified context. An old comment in sd_journal_enumerate_unique() supports this view: /* We do not use the type context here, but 0 instead, * so that we can look at this data object at the same * time as one on another file */ Clearly the expectation was that the data object will remain mapped in context 0 without being pushed away by mapping other objects in context OBJECT_DATA. I suspect that this was the real bug that got fixed by ae97089d49 "journal: fix access to munmapped memory in sd_journal_enumerate_unique". In other words, journal_file_object_keep/release are superfluous after applying this patch.

journald: correct spacing near eol code comments

journal: Fix navigating backwards missing entries With DIRECTION_UP (i.e. navigating backwards) in generic_array_bisect() when the needle was found as the last item in the array, it wasn't actually processed as match, resulting in entries being missed. https://bugs.freedesktop.org/show_bug.cgi?id=86855

treewide: use log_*_errno whenever %m is in the format string If the format string contains %m, clearly errno must have a meaningful value, so we might as well use log_*_errno to have ERRNO= logged. Using: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\((".*%m.*")/log_\1_errno(errno, \2/' Plus some whitespace, linewrap, and indent adjustments.

journald: be nice to coverity, add an extra assert coverity otherwise assumes that the chain object might be NULL.

journal: make JournalFile::chain_cache an OrderedHashmap The order of entries may matter here. Oldest entries are evicted first when the cache is full. (Though I don't see anything to rejuvenate entries on cache hits.)

sd-journal: fix sd_journal_enumerate_unique skipping values sd_journal_enumerate_unique will lock its mmap window to prevent it from being released by calling mmap_cache_get with keep_always=true. This call may return windows that are wider, but compatible with the parameters provided to it. This can result in a mismatch where the window to be released cannot properly be selected, because we have more than one window matching the parameters of mmap_cache_release. Therefore, introduce a release_cookie to be used when releasing the window. https://bugs.freedesktop.org/show_bug.cgi?id=79380

hashmap: introduce hash_ops to make struct Hashmap smaller It is redundant to store 'hash' and 'compare' function pointers in struct Hashmap separately. The functions always comprise a pair. Store a single pointer to struct hash_ops instead. systemd keeps hundreds of hashmaps, so this saves a little bit of memory.

sd-journal: verify that object start with the field name If the journal is corrupted, we might return an object that does not start with the expected field name and/or is shorter than it should.

Fix misuse of uint64_t as size_t They have different size on 32 bit, so they are really not interchangable.

journal: allow files with no data whatsoever If a file was opened for writing, and then closed immediately without actually writing any entries, on subsequent opening, it would be considered "corrupted". This should be totally fine, and even in read mode, an empty file can become non-empty later on.

Fix build without any compression enabled

fix #ifdef

journal: add LZ4 as optional compressor Add liblz4 as an optional dependency when requested with --enable-lz4, and use it in preference to liblzma for journal blob and coredump compression. To retain backwards compatibility, XZ is used to decompress old blobs. Things will function correctly only with lz4-119. Based on the benchmarks found on the web, lz4 seems to be the best choice for "quick" compressors atm. For pkg-config status, see http://code.google.com/p/lz4/issues/detail?id=135.

build-sys: use glibc's xattr support instead of requiring libattr

util: replace close_nointr_nofail() by a more useful safe_close() safe_close() automatically becomes a NOP when a negative fd is passed, and returns -1 unconditionally. This makes it easy to write lines like this: fd = safe_close(fd); Which will close an fd if it is open, and reset the fd variable correctly. By making use of this new scheme we can drop a > 200 lines of code that was required to test for non-negative fds or to reset the closed fd variable afterwards.

journal: assume that next entry is after previous entry With a corrupted file, we can get in a situation where two entries in the entry array point to the same object. Then journal_file_next_entry will find the first one using generic_arrray_bisect, and try to move to the second one, but since the address is the same, generic_array_get will return the first one. journal_file_next_entry ends up in an infinite loop. https://bugzilla.redhat.com/show_bug.cgi?id=1047039

Remove dead lines in various places As pointed-out by clang -Wunreachable-code. No behaviour changes.

journal: fix compiler warning in journal_file_append_data() gcc (4.8.2, arm) does not understand that journal_file_append_field() will always set 'fo' when it returns 0, so this warning is bogus. Anyway, fix it by initialiting fo = NULL.

use memzero(foo, length); for all memset(foo, 0, length); calls In trying to track down a stupid linker bug, I noticed a bunch of memset() calls that should be using memzero() to make it more "obvious" that the options are correct (i.e. 0 is not the length, but the data to set). So fix up all current calls to memset(foo, 0, length) to memzero(foo, length).

journal: fix access to munmapped memory in sd_journal_enumerate_unique sd_j_e_u needs to keep a reference to an object while comparing it with possibly duplicate objects in other files. Because the size of mmap cache is limited, with enough files and object to compare to, at some point the object being compared would be munmapped, resulting in a segmentation fault. Fix this issue by turning keep_always into a reference count that can be increased and decreased. Other callers which set keep_always=true are unmodified: their references are never released but are ignored when the whole file is closed, which happens at some point. keep_always is increased in sd_j_e_u and later on released.

journal: Add missing byte order conversions Convert entry_array.items[0] to host byte order prior to passing it to chain_cache_put(). [zj: also use le64toh in journal-verify.c] https://bugs.freedesktop.org/show_bug.cgi?id=73194

shared: switch our hash table implementation over to SipHash SipHash appears to be the new gold standard for hashing smaller strings for hashtables these days, so let's make use of it.

journal-file.c: remove redundant assignment of variable we also do 'last_index = (uint64_t) -1;' at the end of the while loop so there is no reason to also do it here.

journal: fix against (theoretical) undefined behavior While all the libc implementations I know return NULL when memchr's size parameter is 0, without accessing any memory, passing NULL to memchr is still invalid: C11 7.24.1p2: Where an argument declared as "size_t n" specifies the length of the array for a function, n can have the value zero on a call to that function. Unless explicitly stated otherwise in the description of a particular function in this subclause, pointer arguments on such a call shall still have valid values, as described in 7.1.4. On such a call, a function that locates a character finds no occurrence, a function that compares two character sequences returns zero, and a function that copies characters copies zero characters. see http://llvm.org/bugs/show_bug.cgi?id=18247

journal-file: protect against alloca(0)

journal: simplify pre-allocation logic let's just do a single fallocate() as far as possible, and don't distuingish between allocated space and file size. This way we can save a syscall for each append, which makes quite some benefits.

journal: allow journal_file_copy_entry() to work on non-local files

journal: fix iteration when we go backwards from the beginning of an array chain element

journal: optimize bisection logic a bit by caching the last position This way we can do a quick restart limiting a bit how wildly we need to jump around during the bisection process.

journal: make table const

journal: when appending to journal file, allocate larger blocks at once

Remove dead code and unexport some calls "make check-api-unused" informs us about code that is not used anymore or that is exported but only used internally. Fix these all over the place.

Never call qsort on potentially NULL arrays This extends 62678ded 'efi: never call qsort on potentially NULL arrays' to all other places where qsort is used and it is not obvious that the count is non-zero.

journald: replace new+snprintf with asprintf

journald: add missing error check

journalctl: properly print headers of empty journals

journalctl: print monotonic timestamp in --header

journalctl: print proper IDs with --header The same buffer was used for two different IDs, messing up the output.

Use stdint.h macros instead of casts to print uint64_t values Casts are visually heavy, and can obscure unwanted truncations.

journald: be more careful when we try to flush the runtime journal to disk and the disk is close to being full Bump the minimal size of the journal so that we can be sure creating the journal file will always succeed. Previously the minimum size was smaller than a empty jounral file...

journal: as a µ-opt, use sizeof instead of strlen

journal: Set the default keep free value to 15% (up from 5%) As some SSDs are still seeing performance degredation when reaching 85% usage the default value of 5% seems a little low. Set this to 15% by default.

Add __attribute__((const, pure, format)) in various places I'm assuming that it's fine if a _const_ or _pure_ function calls assert. It is assumed that the assert won't trigger, and even if it does, it can only trigger on the first call with a given set of parameters, and we don't care if the compiler moves the order of calls.

Always use our own MAX/MIN definitions code in src/shared/macro.h only defined MAX/MIN in case they were not defined previously. however the MAX/MIN macros implemented in glibc are not of the "safe" kind but defined as: define MIN(a,b) (((a)<(b))?(a):(b)) define MAX(a,b) (((a)>(b))?(a):(b)) Avoid nasty side effects by using our own versions instead. Also fix the warnings derived from this change. [zj: - modify MAX3 macro to fix warning about _a shadowing _a, - do bootchart/svg.c too, - remove unused MIN3.]

journal: Add sync timer to journal server Add option to force journal sync with fsync. Default timeout is 5min. Interval configured via SyncIntervalSec option at journal.conf. Synced journal files will be marked as OFFLINE. Manual sync can be performed via sending SIGUSR1.

journal: by default do not decompress dat objects larger than 64K This introduces a new data threshold setting for sd_journal objects which controls the maximum size of objects to decompress. This is relieves the library from having to decompress full data objects even if a client program is only interested in the initial part of them. This speeds up "systemd-coredumpctl" drastically when invoked without parameters.

journal: special case the trivial cache chain cache entry

journal: introduce entry array chain cache When traversing entry array chains for a bisection or for retrieving an item by index we previously always started at the beginning of the chain. Since we tend to look at the same chains repeatedly, let's cache where we have been the last time, and maybe we can skip ahead with this the next time. This turns most bisections and index lookups from O(log(n)*log(n)) into O(log(n)). More importantly however, we seek around on disk much less, which is good to reduce buffer cache and seek times on rotational disks.

journal: suggest rotation when fields are not indexed

journal: add ability to list values a specified field can take in all entries of the journal The new 'unique' API allows listing all unique field values that a field specified by a field name can take in all entries of the journal. This allows answering queries such as "What units logged to the journal?", "What hosts have logged into the journal?", "Which boot IDs have logged into the journal?". Ultimately this allows implementation of tools similar to lastlog based on journal data. Note that listing these field values will not work for journal files created with older journald, as the field values are not indexed in older files.

journal: implement time-based rotation/vacuuming This also enables time-based rotation (but not vacuuming) after 1month, so that not more one month of journal is lost at a time per vacuuming.

journal: sort data items of entries by offset This should slightly optimize disk access patterns on rotating disks for simple readers.

journal: set seal even for readonly journalfiles journalctl needs to know wether the file has been sealed to be able to do verification.

journal: don't allow journal_file_open() to be called with ret being NULL

man: fix a bunch of typos in docs https://bugs.freedesktop.org/show_bug.cgi?id=54501

journal: never assert directly on data read from the journal

journald: don't reposition window if we don't have to

journal: don't set seal flag if we don't have gcrypt built-in

journal: don't try to compress without XZ otherwise the header contains the HEADER_INCOMPATIBLE_COMPRESSED flag even though the data is not compressed and reading the journal fails.

journald: detect invalid header pointers correctly

journal: add call to determine current journal file disk usage

journald: avoid logging to kmsg in the normal paths

journal: be more careful when keeping around mmaps we still need

journal: make libgcrypt dependency optional

journal: use a macro to check for file header flags

journal: fix tag ordering check

mmap: resize arrays dynamically

journal: even more simple static object tests

journal: fix verification without key

journald: add additional simple static tests to verifier

journal: be fine with opening rotated/corrupted journal files

journal: show new header fields in header dump

journal: rework terminology Let's clean up our terminology a bit. New terminology: FSS = Forward Secure Sealing FSPRG = Forward Secure Pseudo-Random Generator FSS is the combination of FSPRG and a HMAC. Sealing = process of adding authentication tags to the journal. Verification = process of checking authentication tags to the journal. Sealing Key = The key used for adding authentication tags to the journal. Verification Key = The key used for checking authentication tags of the journal. Key pair = The pair of Sealing Key and Verification Key Internally, the Sealing Key is the combination of the FSPRG State plus change interval/start time. Internally, the Verification Key is the combination of the FSPRG Seed plus change interval/start time.

journal: add FSPRG journal authentication

journal: parse fsprg seed

journal: count number of entry arrays in header

journal: rename 'mmap' to 'mmap_cache' to appease gcc warning: declaration of 'mmap' shadows a global declaration [-Wshadow]

journal: verify hashes only during actual verification, not all the time

journal: split up journal-file.c

journal: add superficial structure verifier

journal: implement basic journal file verification logic

journal: implement generic sharable mmap caching logic instead of having one simple per-file cache implement an more comprehensive one that works for multiple files and can actually maintain multiple maps per file and per object type.

journal: include tag object header in hmac

journal: add all objects we add to HMAC

journald: initial version of FSPRG hookup This adds forward-secure authentication of journal files. This patch includes key generation as well as tagging of journal files, Verification of journal files will be added in a later patch.

fix a couple of issues found with llvm-analyze

journal: rotate busy files away when we try to write to them

journal: when we suggest a rotation, explain why

journal: estimate data hash table size a bit larger by default

journal: actually set archived files to archived state

journalctl: decode more header files in --header

journal: size journal data hash table based on maximum file size metrics The default of 2047 hash table entries turned out to result in way too many collisions for bigger files, hence scale the hash table size by the estimated maximum file size.

journal: immediately rotate when the journal was previously not closed properly

journal: automatically rotate journal files if the data hash table is full > 75% Previously, when the main data hash table grows too full the performance simply started to decrease drastically. Instead, now simply rotate to a new journal file as the hash table gets to full, so that we can start with a new fresh empty hash table.

journal: use tail/head timestamps from header for cutoff logic We have them, they are faster to use them, so use them...

journald: don't enforce monotonicity of realtime clocks when copying entries After all the point of the realtime clock (in contrast to the monotonic clock) is that it does not have to be strictly monotonic, hence don't enforce this when flushing the journal from /run to /var.

journal: beef up journal matches considerably we now can take multiple matches, and they will apply as AND if they apply to different fields and OR if they apply to the same fields. Also, terms of this kind can be combined with an overreaching OR.

journal: fix seeking by realtime/seqnum

journal: fix bad memory access

journal: fix bisection logic for first entry

journal: fix monotonic seeking

journal: expose and make use of cutoff times of journal This helps explaining when the log output of "systemctl status" is incomplete because the logs got rotated since the service was started.

journal: correct list link up on hash collisions

journal: add basic object definition for signatures

journal: replace arena offset by header size

journal-file: fix mmap leak https://bugzilla.redhat.com/show_bug.cgi?id=831132

Fixed handling of posix_fallocate() returned value According to the man pages of posix_fallocate, it returns zero on success or an error number on failure; however, errno is not set on failure. If the kernel or a library other than glibc does not support the function for example, EOPNOTSUPP will be returned and the error will not be handled properly with original code.

fix a couple of things found with the llvm static analyzer

relicense to LGPLv2.1 (with exceptions) We finally got the OK from all contributors with non-trivial commits to relicense systemd from GPL2+ to LGPL2.1+. Some udev bits continue to be GPL2+ for now, but we are looking into relicensing them too, to allow free copy/paste of all code within systemd. The bits that used to be MIT continue to be MIT. The big benefit of the relicensing is that closed source code may now link against libsystemd-login.so and friends.

journal: decrease default mmap window size to allow a bigger number of journals to be traversed in parallel

add sparse support to detect endianness bug le16/32/64_t type should be used when storing little-endian value header to integrate with sparse from Josh Triplett <josh@joshtriplett.org>

journal: react with immediate rotation to a couple of more errors

man: document systemd-journald.conf(5)

journal: only use uint8_t for state

journald: be a bit more verbose

journald: fix calculation of disk space

journald: copy metrics/compression state from template when rotating

journald: increase max file size to 128MB

journal: vacuum corrupted files, too

journal: if we encounter a corrupted file, rotate and go on

journal: make sure to refresh window position and pointer after we linked up a data object

journal: fix endianness errors

systemd-journald: fix endianess bug

journal: increase compression threshold for objects from 64 to 512 Apparently the perfomance price for compression is to steep to apply it for all objects >= 64 and < 512 in size, as measured by Arjan Van De Ven, hence increase the threshold to 512 which yields better results.

util: introduce FORMAT_BYTES_MAX

journal: fix more 32/64 bit issues

journald: don't assume size_t and uint64_t are the same

journal: don't realign window twice

journald: remove inner loop debug message

journal: it's not a problem if the realtime jumps, hence don't ensure monotonicity of realtime for entries we write

journal: when increasing window, make sure to use the increased window

journal: apply seek check before resizing window, and refresh stat data if necessary

journal: add memory barrier before linking in newly created entries

journal: silence gcc warnings

journal: automatically deduce journal metrics from file system sizes

journald: flush /run to /var as soon as it becomes available

journal: never mmap beyond file size

journal: fix hash table lookup logic

journal: properly handle first inline bisect array entry

journal: add inline compression support with XZ

journal: fix space reservation limit enforcement

journal: fix matches

journal: implement inotify-based live logging logic

journal: implement multiple field matches

journal: unify code for up and for down traversal

journal: properly implement matching with multiple matches

journal: synchronize seqnum across files

journal: implement parallel file traversal

journal: implementation rotation

journal: only fallocate() what we really need to avoid slowness on file systems which do not support fallocate natively

journal: implement parallel traversal in client