journal: add the "repeating sequence" test case This was the case that caused various problems that were fixed in preceding patches, so it is good to add a test that uses it directly. In "may_fail" test cases try again with a bigger buffer. Instead of allocating various buffers on the stack, malloc them. This is more reliable in case of big buffers, and allows tools like valgrind and address sanitizer to find overflows more easily.

journal: add "xfail" test for partial lz4 decompression Add a test that LZ4_decompress_safe_partial does (not) work as expected, so that if it starts to work at some point, we'll catch this and adjust our code.

journal: add dst_allocated_size parameter for compress_blob compress_blob took src, src_size, dst and *dst_size, but dst_size wasn't used as an input parameter with the size of dst, but only as an output parameter. dst was implicitly assumed to be at least src_size-1. This code wasn't *wrong*, because the only real caller in journal-file.c got it right. But it was misleading, and the tests in test-compress.c got it wrong, and worked only because the output buffer happened to be the same size as input buffer. So add a seperate dst_allocated_size parameter to make it explicit what the size of the buffer is, and to allow test to proceed with different output buffer sizes.

util-lib: split out allocation calls into alloc-util.[ch]

util-lib: move more file I/O related calls into fileio.[ch]

util-lib: split out fd-related operations into fd-util.[ch] There are more than enough to deserve their own .c file, hence move them over.

coredump: use lz4frame api to compress coredumps This converts the stream compression to use the new lz4frame api, compatible with lz4cat. Previous code used custom headers, so the compressed file was not compatible with lz4 command line tools. I considered this the last blocker to using lz4 by default. Speed seems to be reasonable, although a bit (a few percent) slower than the lz4 binary, even though compression is the same. I don't consider this important. It could be caused by the overhead of library calls, but is probably caused by slightly different buffer sizes or such. The code in this patch uses mmap, since since this allows the buffer to be reused while not making the code more complicated at all. In my testing, this version is noticably faster (~20%) than a naive single-buffered version. mmap can cause the program to be killed with SIGBUS, if the underlying file is truncated or a disk error occurs. We only use this from within coredump and coredumpctl, so I don't consider this an issue. Old decompression code is retained and is used if the new code fails indicating a format error. There have been reports of various smaller distributions using previous lz4 code, i.e. the old format, and it is nice to provide backwards compatibility. We can remove the legacy code in a few versions. The way that blobs are compressed in the journal is not affected.

tree-wide: never use the off_t unless glibc makes us use it off_t is a really weird type as it is usually 64bit these days (at least in sane programs), but could theoretically be 32bit. We don't support off_t as 32bit builds though, but still constantly deal with safely converting from off_t to other types and back for no point. Hence, never use the type anymore. Always use uint64_t instead. This has various benefits, including that we can expose these values directly as D-Bus properties, and also that the values parse the same in all cases.

shared: add random-util.[ch]

tests: use assert_se instead of assert Otherwise they can be optimized away with -DNDEBUG

tests: use assert_se instead of assert Otherwise they can be optimized away with -DNDEBUG

treewide: no need to negate errno for log_*_errno() It corrrectly handles both positive and negative errno values.

treewide: auto-convert the simple cases to log_*_errno() As a followup to 086891e5c1 "log: add an "error" parameter to all low-level logging calls and intrdouce log_error_errno() as log calls that take error numbers", use sed to convert the simple cases to use the new macros: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\("(.*)%s"(.*), strerror\(-([a-zA-Z_]+)\)\);/log_\1_errno(-\4, "\2%m"\3);/' Multi-line log_*() invocations are not covered. And we also should add log_unit_*_errno().

test-compress: also test with incompressible inputs

test-compress: make sure asserts with side effects use assert_se() Otherwise the test fails when built with CPPFLAGS='-DNDEBUG' which disables assertions. Tested: - make check TESTS='test-compress' CPPFLAGS='-DNDEBUG'

Fix misuse of uint64_t as size_t They have different size on 32 bit, so they are really not interchangable.

fix #ifdef

journal: add LZ4 as optional compressor Add liblz4 as an optional dependency when requested with --enable-lz4, and use it in preference to liblzma for journal blob and coredump compression. To retain backwards compatibility, XZ is used to decompress old blobs. Things will function correctly only with lz4-119. Based on the benchmarks found on the web, lz4 seems to be the best choice for "quick" compressors atm. For pkg-config status, see http://code.google.com/p/lz4/issues/detail?id=135.

coredump: make compression configurable Add Compression={none,xz} and CompressionLevel=0-9 settings. Defaults are xz/6. Compression=filesystem may be added later. I picked "xz" for the compression "type", since we might want to add different compressors later on. XZ is fairly memory and CPU intensive, and embedded users will likely want to use LZO or some other lightweight compression mechanism.

journal/compress: add stream compression/decompression functions

tests: add test-compress