tree-wide: remove Emacs lines from all files This should be handled fine now by .dir-locals.el, so need to carry that stuff in every file.

tree-wide: sort includes Sort the includes accoding to the new coding style.

treewide: use the negative error codes returned by our functions Our functions return negative error codes. Do not rely on errno being set after calling our own functions.

util-lib: split out allocation calls into alloc-util.[ch]

util-lib: move more file I/O related calls into fileio.[ch]

util-lib: split out fd-related operations into fd-util.[ch] There are more than enough to deserve their own .c file, hence move them over.

journal: s/Envalid/Invalid/

cgtop: underline table header Let's underline the header line of the table shown by cgtop, how it is customary for tables. In order to do this, let's introduce new ANSI underline macros, and clean up the existing ones as side effect.

tree-wide: make use of log_error_errno() return value Turns this: r = -errno; log_error_errno(errno, "foo"); into this: r = log_error_errno(errno, "foo"); and this: r = log_error_errno(errno, "foo"); return r; into this: return log_error_errno(errno, "foo");

journal: uppercase first character in verify error messages In the english language the first character of a sentence is supposed to be uppercase. Let's make sure this also applies to the journal verification error messages.

journal: explain the error when we find a non-DATA object that is compressed Only objects of type DATA may be compressed, generate a message about that, like we do for all other errros.

journal: when verifying journal files, handle empty ones nicely A journal file that carries no objects should be considered valid.

journal: avoid mapping empty data and field hash tables When a new journal file is created we write the header first, then sync and only then create the data and field hash tables in them. That means to other processes it might appear that the files have a valid header but not data and field hash tables. Our reader code should be able to deal with this. With this change we'll not map the two hash tables right-away after opening a file for reading anymore (because that will of course fail if the objects are missing), but delay this until the first time we access them. On top of that, when we want to look something up in the hash tables and we notice they aren't initialized yet, we consider them empty. This improves handling of some journal files reported in #487.

journal-verify: don't hit SIGFPE when determining progress If we determine the progress based on a number of objects available, don't blindly devide by the number of objects, given that it might be 0.

shared: add terminal-util.[ch]

tree-wide: there is no ENOTSUP on linux Replace ENOTSUP by EOPNOTSUPP as this is what linux actually uses.

remove unused includes This patch removes includes that are not used. The removals were found with include-what-you-use which checks if any of the symbols from a header is in use.

journal: consistently use OBJECT_<type> names instead of numbers Note that numbers 0 and -1 are both replaced with OBJECT_UNUSED, because they are treated the same everywhere (e.g. type_to_context() translates them both to 0).

journal: remove journal_file_object_keep/release functions The only user is sd_journal_enumerate_unique() and, as explained in the previous commit (fed67c38e3 "journal: map objects to context set by caller, not by actual object type"), the use of them there is now superfluous. Let's remove them. This reverts major parts of commits: ae97089d49 journal: fix access to munmapped memory in sd_journal_enumerate_unique 06cc69d44c sd-journal: fix sd_journal_enumerate_unique skipping values Tested with an "--enable-debug" build and "journalctl --list-boots". It gives the expected number of results. Additionally, if I then revert the previous commit ("journal: map objects to context set by caller, not to actual object type"), it crashes with SIGSEGV, as expected.

journald: correct spacing near eol code comments

treewide: use log_*_errno whenever %m is in the format string If the format string contains %m, clearly errno must have a meaningful value, so we might as well use log_*_errno to have ERRNO= logged. Using: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\((".*%m.*")/log_\1_errno(errno, \2/' Plus some whitespace, linewrap, and indent adjustments.

sd-journal: fix sd_journal_enumerate_unique skipping values sd_journal_enumerate_unique will lock its mmap window to prevent it from being released by calling mmap_cache_get with keep_always=true. This call may return windows that are wider, but compatible with the parameters provided to it. This can result in a mismatch where the window to be released cannot properly be selected, because we have more than one window matching the parameters of mmap_cache_release. Therefore, introduce a release_cookie to be used when releasing the window. https://bugs.freedesktop.org/show_bug.cgi?id=79380

journal: Do not count on the compiler initializing found_last to false There is a very unlikely case where this can happen since gcc usually does the sane thing. But let's make sure found_last is initialized anyway. Fixes: CID#996386

Fix misuse of uint64_t as size_t They have different size on 32 bit, so they are really not interchangable.

journal/verify: flush progress bar, print offset in more places Before, fragments of the progress bar would remain when errors or warnings were printed.

journal/compress: fix calls to decompress_blob

journal: add LZ4 as optional compressor Add liblz4 as an optional dependency when requested with --enable-lz4, and use it in preference to liblzma for journal blob and coredump compression. To retain backwards compatibility, XZ is used to decompress old blobs. Things will function correctly only with lz4-119. Based on the benchmarks found on the web, lz4 seems to be the best choice for "quick" compressors atm. For pkg-config status, see http://code.google.com/p/lz4/issues/detail?id=135.

util: replace close_nointr_nofail() by a more useful safe_close() safe_close() automatically becomes a NOP when a negative fd is passed, and returns -1 unconditionally. This makes it easy to write lines like this: fd = safe_close(fd); Which will close an fd if it is open, and reset the fd variable correctly. By making use of this new scheme we can drop a > 200 lines of code that was required to test for non-negative fds or to reset the closed fd variable afterwards.

Replace mkostemp+unlink with open(O_TMPFILE) This will only work on Linux >= 3.11, and probably not on all filesystems. Fallback code is provided.

journal: fix access to munmapped memory in sd_journal_enumerate_unique sd_j_e_u needs to keep a reference to an object while comparing it with possibly duplicate objects in other files. Because the size of mmap cache is limited, with enough files and object to compare to, at some point the object being compared would be munmapped, resulting in a segmentation fault. Fix this issue by turning keep_always into a reference count that can be increased and decreased. Other callers which set keep_always=true are unmodified: their references are never released but are ignored when the whole file is closed, which happens at some point. keep_always is increased in sd_j_e_u and later on released.

journal: Add missing byte order conversions Convert entry_array.items[0] to host byte order prior to passing it to chain_cache_put(). [zj: also use le64toh in journal-verify.c] https://bugs.freedesktop.org/show_bug.cgi?id=73194

journal-verify: Use proper printf placeholder

journal-verify: allow unlinked data entries Sometimes an entry is not successfully written, and we end up with data items which are "unlinked", not connected to, and not used by any entry. This will usually happen when we write to write a core dump, and the initial small data fields are written successfully, but the huge COREDUMP= field is not written. This situation is hard to avoid, but the results are mostly harmless. Thus only warn about unused data items. Also, be more verbose about why journal files failed verification. This should help diagnose journal failure modes without resorting to a hexadecimal editor. https://bugs.freedesktop.org/show_bug.cgi?id=65235 (esp. see system.journal attached to the bug report).

Use stdint.h macros instead of casts to print uint64_t values Casts are visually heavy, and can obscure unwanted truncations.

journal: by default do not decompress dat objects larger than 64K This introduces a new data threshold setting for sd_journal objects which controls the maximum size of objects to decompress. This is relieves the library from having to decompress full data objects even if a client program is only interested in the initial part of them. This speeds up "systemd-coredumpctl" drastically when invoked without parameters.

util: unify usage of on_tty() in util.c

journal-verify: get rid of an unused variable When compiling without gcrypt, gcc emits an annoying warning.

journald: always pass first entry timestamp back from journal_file_verify()

journald: don't reposition window if we don't have to

journal: add HAVE_XZ check to avoid build failure Without this, build fail with --disable-xz or xz not installed.

journalctl: be more friendly when informing about seal verification

journal: be more careful when keeping around mmaps we still need

verify: optimize entry search a bit by using bisection

journal: automatically evolve FSS key even when nothing is logged

journal: add missing endianess conversion

journal: make libgcrypt dependency optional

journal: use a macro to check for file header flags

journal: fix tag ordering check

journal: fix bisection algorithm

journal: validate timestamps as well

journal: even more simple static object tests

journal: refuse verification of files with unknown flags

jounral: write bit fiddling test This test goes through every single bit in a journal file, toggles it, and checks if this change is detected by the verification.

journal: fix verification without key

journald: add additional simple static tests to verifier

journal: after verification output validated time range

journal: reword verification messages a bit

journal: ensure that entries and tags are properly ordered

journal: show new header fields in header dump

journal: don't write tag objects if nothing has been written since the last time

man: add man pages for new FSS stuff

journal: rework terminology Let's clean up our terminology a bit. New terminology: FSS = Forward Secure Sealing FSPRG = Forward Secure Pseudo-Random Generator FSS is the combination of FSPRG and a HMAC. Sealing = process of adding authentication tags to the journal. Verification = process of checking authentication tags to the journal. Sealing Key = The key used for adding authentication tags to the journal. Verification Key = The key used for checking authentication tags of the journal. Key pair = The pair of Sealing Key and Verification Key Internally, the Sealing Key is the combination of the FSPRG State plus change interval/start time. Internally, the Verification Key is the combination of the FSPRG Seed plus change interval/start time.

journal: add FSPRG journal authentication

journal: fix tag sequence number verification

journalctl: immeidately terminate on invalid seed

journal: parse fsprg seed

journal: count number of entry arrays in header

journal: fix variable initialization

journal: fix unitialized var

journal: journal-send.h doesn't actually exist

journal: verify structural consistency

journal: add color to verification progress bar

journal: verify compressed objects

journal: verify hashes only during actual verification, not all the time

journal: split up journal-file.c