tree-wide: remove Emacs lines from all files This should be handled fine now by .dir-locals.el, so need to carry that stuff in every file.

treewide: fix typos and spacing

sd-journal: minor optimization No need to store the object and offset data if we don't actually need it ever.

sd-journal: add an API to enumerate known field names of the journal This adds two new calls to get the list of all journal fields names currently in use. This is the low-level support to implement the feature requested in #2176 in a more optimized way.

sd-journal: introduce has_runtime_files and has_persistent_files Also introduce sd_journal_has_runtime_files() and sd_journal_has_persistent_files() to the public API. These functions can be used to easily find out if the open journal files are runtime and/or persistent.

journal: decompress_startswith can return an error The return value was used directly in an if, so an error was treated as success; we need to bail out instead. An error should not happen, unless we have a compression/decompression mismatch, so output a debug line.

journalctl: when we fail to open a journal file, print why When we enumerate journal files and encounter an invalid one, remember which this, and show it to the user. Note the possibly slightly surprising logic here: we store only one path per error code. This means we show all error kinds but not every actual error we encounter. This has the benefit of not requiring us to keep a potentially unbounded list of errors with their sources around, but can still provide a pretty complete overview on the errors we encountered. Fixes #1669.

sd-journal: various clean-ups and modernizations - Always print a debug log message about files and directories we cannot open right when it happens instead of the caller, thus reducing the number of places where we need to generate the debug message. - Always push the errors we encounter immediately into the error set, when we run into them, instead of in the caller. Thus, we never forget to push them in. - Use stack instead of heap memory where we can. - Make remove_file() void, since it cannot fail anyway and always returned 0. - Make local machine check of journal directories explicit in a function, to make things more readable. - Port to all directory listing loops FOREACH_DIRENT_ALL() - sd-daemon is library code, hence never log at higher log levels than LOG_DEBUG.

sd-journal.c: port to extract_first_word

util-lib: move inotify-related definitions to fs-util.[ch]

util-lib: split out allocation calls into alloc-util.[ch]

util-lib: split out printf() helpers to stdio-util.h

util-lib: introduce dirent-util.[ch] for directory entry calls Also, move a couple of more path-related functions to path-util.c.

util-lib: split out IO related calls to io-util.[ch]

util-lib: split out fd-related operations into fd-util.[ch] There are more than enough to deserve their own .c file, hence move them over.

util-lib: split our string related calls from util.[ch] into its own file string-util.[ch] There are more than enough calls doing string manipulations to deserve its own files, hence do something about it. This patch also sorts the #include blocks of all files that needed to be updated, according to the sorting suggestions from CODING_STYLE. Since pretty much every file needs our string manipulation functions this effectively means that most files have sorted #include blocks now. Also touches a few unrelated include files.

util: make machine_name_is_valid() a macro and move it to hostname-util.h As it turns out machine_name_is_valid() does the exact same thing as hostname_is_valid() these days, as it just invoked that and checked the name length was < 64. However, hostname_is_valid() checks the length against HOST_NAME_MAX anyway (which is 64 on Linux), hence any additional check is redundant. We hence replace machine_name_is_valid() by a macro that simply maps it to hostname_is_valid() but sets the allow_trailing_dot parameter to false. We also move this this call to hostname-util.h, to the same place as the hostname_is_valid() declaration.

journal: remove error check that never happens remove_directory will always return 0 so this can never happen. Besides that, d->path and d are freed so we would end up with a null pointer dereference anyway.

shared: add formats-util.h

sd-journal: return error when we cannot open a file Lack of this caused journalctl not to display a hint about missing groups properly when the user lacks permissions.

journal: make skipping of exhausted journal files effective again Commit 668c965af "journal: skipping of exhausted journal files is bad if direction changed" fixed a correctness issue, but it also significantly limited the cases where the optimization that skips exhausted journal files could apply. As a result, some journalctl queries are much slower in v219 than in v218. (e.g. queries where a "--since" cutoff should have quickly eliminated older journal files from consideration, but didn't.) If already in the initial iteration find_location_with_matches() finds no entry, the journal file's location is not updated. This is fine, except that: - We must update at least f->last_direction. The optimization relies on it. Let's separate that from journal_file_save_location() and update it immediately after the direction checks. - The optimization was conditional on "f->current_offset > 0", but it would always be 0 in this scenario. This check is unnecessary for the optimization.

include <poll.h> instead of <sys/poll.h> include-what-you-use automatically does this and it makes finding unnecessary harder to spot. The only content of poll.h is a include of sys/poll.h so should be harmless.

util: rework strappenda(), and rename it strjoina() After all it is now much more like strjoin() than strappend(). At the same time, add support for NULL sentinels, even if they are normally not necessary.

Add a snprinf wrapper which checks that the buffer was big enough If we scale our buffer to be wide enough for the format string, we should expect that the calculation was correct. char_array_0() invocations are removed, since snprintf nul-terminates the output in any case. A similar wrapper is used for strftime calls, but only in timedatectl.c.

Revert "journal: do not check for number of files" This reverts commit b914ea8d379b446c4c9fac4ba181771676ef38cd. We really need to put a limit on all our resources, everywhere, and in particular if we operate on external data. Hence, let's reintroduce the limit, but bump it substantially, so that it is guaranteed to be higher than any realistic RLIMIT_NOFILE setting.

journal: do not check for number of files Now that we bump rlimit, we do not really know how many files we can open. Remove the check. https://bugzilla.redhat.com/show_bug.cgi?id=1179980

util: fix strict aliasing violations in use of struct inotify_event v5 There is alot of cleanup that will have to happen to turn on -fstrict-aliasing, but I think our code should be "correct" to the rule.

journal: skipping of exhausted journal files is bad if direction changed EOF is meaningless if the direction of iteration changes. Move the EOF optimization under the direction check. This fixes test-journal-interleaving for me. Thanks to Filipe Brandenburger for telling me about the failure.

journal: make next_with_matches() always use f->current_offset next_with_matches() is odd in that its "unit64_t *offset" parameter is both input and output. In other it's purely for output. The function is called from two places in next_beyond_location(). In both of them "&cp" is used as the argument and in both cases cp is guaranteed to equal f->current_offset. Let's just have next_with_matches() ignore "*offset" on input and operate with f->current_offset. I did not investigate why it is, but it makes my usual benchmark run reproducibly faster: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m4.032s user 0m3.896s sys 0m0.135s (Compare to preceding commit, where real was 4.4s.)

journal: fix skipping of duplicate entries in iteration I accidentally broke the detection of duplicate entries in 7943f42275 "journal: optimize iteration by returning previously found candidate entry". When we have a known location of a candidate entry, we must not return from next_beyond_location() immediately. We must go through the duplicates detection to make sure the candidate differs from the already iterated entry. This fix slows down iteration a bit, but it's still faster than it was before the rework. $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m4.448s user 0m4.298s sys 0m0.149s (Compare with results from commit 7943f42275, where real was 5.3s before the rework.)

journal: next_with_matches() now does not need a mapped object as input Now that journal_file_next_entry() does not need a pointer to the current object, next_with_matches() does not need it either.

journal: journal_file_next_entry() does not need pointer to current Object The current offset is sufficient information.

journal: optimize iteration by returning previously found candidate entry In next_beyond_location() when the JournalFile's location type is LOCATION_SEEK, it means there's nothing to do, because we already have the location of the candidate entry. Do an early return. Note that now next_beyond_location() does not anymore guarantee on return that the entry is mapped, but previous patches made sure the caller does not care. This optimization is at least as good as "journal: optimize iteration: skip files that cannot improve current candidate entry" was. Timing results on my workstation, using: $ time ./journalctl -q --since=2014-06-01 --until=2014-07-01 > /dev/null Before "Revert "journal: optimize iteration: skip files that cannot improve current candidate entry": real 0m5.349s user 0m5.166s sys 0m0.181s Now: real 0m3.901s user 0m3.724s sys 0m0.176s

journal: optimize iteration by skipping exhausted files If from a previous iteration we know we are at the end of a journal file, don't bother looking into the file again. This is complicated by the fact that the EOF does not have to be permanent (think of "journalctl -f"). So we also check if the number of entries in the journal file changed. This optimization has a similar effect as "journal: optimize iteration: skip whole files behind current location" had.

journal: drop unnecessary parameters of next_beyond_location() offset is redundant, because the caller can rely on f->current_offset. The object pointer the function saves in *ret is thrown away by the caller.

journal: remove redundant variable new_offset The file's current_offset is already updated at this point, so let's use it.

journal: compare candidate entries using JournalFiles' locations When comparing the locations of candidate entries, we can rely on the location information stored in struct JournalFile.

journal: simplify set_location() set_location() is called from real_journal_next() when a winning entry has been picked from among the candidates in journal files. The location type is always set to LOCATION_DISCRETE. No need to pass it as a parameter. The per-JournalFile location information is already updated at this point. No need for having the direction and offset here.

journal: keep per-JournalFile location info during iteration In next_beyond_location() when we find a candidate entry in a journal file, save its location information in struct JournalFile. The purpose of remembering the locations of candidate entries is to be able to save work in the next iteration. This patch does only the remembering part. LOCATION_SEEK means the location identifies a candidate entry. When a winner is picked from among candidates, it becomes LOCATION_DISCRETE. LOCATION_TAIL here signifies we've iterated the file to the end (or the beginning in the case of reversed direction).

journal: abstract the resetting of JournalFile's location

Revert "journal: optimize iteration: skip whole files behind current location" This reverts commit b7c88ab8cc7d55a43450bf3dea750f95f2e910d6. This optimization will be made redundant by the following patches.

Revert "journal: optimize iteration: skip files that cannot improve current candidate entry" This reverts commit f8b5a3b75fb55f0acb85c21424b3893c822742e9. This optimization will be made redundant by the following patches.

journal: consistently use OBJECT_<type> names instead of numbers Note that numbers 0 and -1 are both replaced with OBJECT_UNUSED, because they are treated the same everywhere (e.g. type_to_context() translates them both to 0).

journal: remove journal_file_object_keep/release functions The only user is sd_journal_enumerate_unique() and, as explained in the previous commit (fed67c38e3 "journal: map objects to context set by caller, not by actual object type"), the use of them there is now superfluous. Let's remove them. This reverts major parts of commits: ae97089d49 journal: fix access to munmapped memory in sd_journal_enumerate_unique 06cc69d44c sd-journal: fix sd_journal_enumerate_unique skipping values Tested with an "--enable-debug" build and "journalctl --list-boots". It gives the expected number of results. Additionally, if I then revert the previous commit ("journal: map objects to context set by caller, not to actual object type"), it crashes with SIGSEGV, as expected.

core: unify how we iterate over inotify events Let's add some syntactic sugar for iterating through inotify events, and use it everywhere.

journal: optimize iteration: skip files that cannot improve current candidate entry Suppose that while iterating we have already looked into a journal file and got a candidate for the next entry. And we are considering to look into another journal file because it may contain an entry that is nearer to the current location than the candidate. We should skip the whole journal file if we can tell by looking at its header that none of its entries can precede the candidate. Before: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m20.518s user 0m19.989s sys 0m0.328s After: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m9.445s user 0m9.228s sys 0m0.213s

journal: optimize iteration: skip whole files behind current location Interleaving of entries from many journal files is expensive. But there is room for optimization. We can skip looking into journal files whose entries all lie before the current iterating location. We can tell if that's the case from looking at the journal file header. This saves a huge amount of work if one has many of mostly not interleaved journal files. On my workstation with 90 journal files in /var/log/journal/ID/ totalling 3.4 GB I get these results: Before: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 5m54.258s user 2m4.263s sys 3m48.965s After: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m20.518s user 0m19.989s sys 0m0.328s The high "sys" time in the original was caused by putting more stress on the mmap-cache than it could handle. With the patch the working set now consists of fewer mmap windows and mmap-cache is not thrashing.

treewide: use log_*_errno whenever %m is in the format string If the format string contains %m, clearly errno must have a meaningful value, so we might as well use log_*_errno to have ERRNO= logged. Using: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\((".*%m.*")/log_\1_errno(errno, \2/' Plus some whitespace, linewrap, and indent adjustments.

treewide: more log_*_errno() conversions, multiline calls Basically: find . -name '*.[ch]' | while read f; do perl -i.mmm -e \ 'local $/; local $_=<>; s/log_(debug|info|notice|warning|error|emergency)\("([^"]*)%s"([^;]*),\s*strerror\(-?([->a-zA-Z_]+)\)\);/log_\1_errno(\4, "\2%m"\3);/gms;print;' \ $f; done Plus manual indentation fixups.

treewide: no need to negate errno for log_*_errno() It corrrectly handles both positive and negative errno values.

treewide: auto-convert the simple cases to log_*_errno() As a followup to 086891e5c1 "log: add an "error" parameter to all low-level logging calls and intrdouce log_error_errno() as log calls that take error numbers", use sed to convert the simple cases to use the new macros: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\("(.*)%s"(.*), strerror\(-([a-zA-Z_]+)\)\);/log_\1_errno(-\4, "\2%m"\3);/' Multi-line log_*() invocations are not covered. And we also should add log_unit_*_errno().

journal: make sd_journal::files a OrderedHashmap Anything that uses hashmap_next() almost certainly cares about the order and needs to be an OrderedHashmap.

machine: validate machine names using machine_name_is_valid() instead of string_is_safe() After all, we know have this as generic validator, so let's be correct and use it wherver applicable.

sd-journal: consistently use ternary operator for all direction checks

sd-journal: fix sd_journal_enumerate_unique skipping values sd_journal_enumerate_unique will lock its mmap window to prevent it from being released by calling mmap_cache_get with keep_always=true. This call may return windows that are wider, but compatible with the parameters provided to it. This can result in a mismatch where the window to be released cannot properly be selected, because we have more than one window matching the parameters of mmap_cache_release. Therefore, introduce a release_cookie to be used when releasing the window. https://bugs.freedesktop.org/show_bug.cgi?id=79380

sd-journal: do not reset sd_j_enumerate_unique position on error systemctl would call sd_j_enumerate_unique() interleaved with sd_j_next(). But the latter can remove a file if it detects an error in it. In those circumstances sd_j_enumerate_unique would restart with the first file in hashmap. With many corrupted files sd_j_enumerate_unique might iterate over the list multiple times. Avoid this by jumping to the next file in unique list if possible, or setting a flag that tells sd_j_enumerate_unique that it is done otherwise.

hashmap: introduce hash_ops to make struct Hashmap smaller It is redundant to store 'hash' and 'compare' function pointers in struct Hashmap separately. The functions always comprise a pair. Store a single pointer to struct hash_ops instead. systemd keeps hundreds of hashmaps, so this saves a little bit of memory.

sd-journal: verify that object start with the field name If the journal is corrupted, we might return an object that does not start with the expected field name and/or is shorter than it should.

sd-journal: never log anything by default from a library

Fix misuse of uint64_t as size_t They have different size on 32 bit, so they are really not interchangable.

Reject invalid quoted strings String which ended in an unfinished quote were accepted, potentially with bad memory accesses. Reject anything which ends in a unfished quote, or contains non-whitespace characters right after the closing quote. _FOREACH_WORD now returns the invalid character in *state. But this return value is not checked anywhere yet. Also, make 'word' and 'state' variables const pointers, and rename 'w' to 'word' in various places. Things are easier to read if the same name is used consistently. mbiebl_> am I correct that something like this doesn't work mbiebl_> ExecStart=/usr/bin/encfs --extpass='/bin/systemd-ask-passwd "Unlock EncFS"' mbiebl_> systemd seems to strip of the quotes mbiebl_> systemctl status shows mbiebl_> ExecStart=/usr/bin/encfs --extpass='/bin/systemd-ask-password Unlock EncFS $RootDir $MountPoint mbiebl_> which is pretty weird

man,journal: add note about sd_journal_get_cutoff_monotonic_usec return value Also modify the function itself to be a bit simpler to read.

Fix build without any compression enabled

journal/compress: fix calls to decompress_blob

journal: add LZ4 as optional compressor Add liblz4 as an optional dependency when requested with --enable-lz4, and use it in preference to liblzma for journal blob and coredump compression. To retain backwards compatibility, XZ is used to decompress old blobs. Things will function correctly only with lz4-119. Based on the benchmarks found on the web, lz4 seems to be the best choice for "quick" compressors atm. For pkg-config status, see http://code.google.com/p/lz4/issues/detail?id=135.

Use %m instead of strerror(errno) where appropiate

Remove unnecessary casts in printfs No functional change expected :)

util: replace close_nointr_nofail() by a more useful safe_close() safe_close() automatically becomes a NOP when a negative fd is passed, and returns -1 unconditionally. This makes it easy to write lines like this: fd = safe_close(fd); Which will close an fd if it is open, and reset the fd variable correctly. By making use of this new scheme we can drop a > 200 lines of code that was required to test for non-negative fds or to reset the closed fd variable afterwards.

journal: forget file after encountering an error If we encounter an inconsistency in a file, let's just ignore it. Otherwise, after previous patch, we would try, and fail, to use this file in every invocation of sd_journal_next or sd_journal_previous that happens afterwards.

Remove dead lines in various places As pointed-out by clang -Wunreachable-code. No behaviour changes.

journal: fix compiler warning in real_journal_next() gcc (4.8.2, arm) does not understand that next_beyond_location() will always set 'p' when it returns > 0. Initialize p in order to fix this.

journal: don't clobber return parameters of sd_journal_get_cutoff_realtime_usec() on failure

make gcc shut up If -flto is used then gcc will generate a lot more warnings than before, among them a number of use-without-initialization warnings. Most of them without are false positives, but let's make them go away, because it doesn't really matter.

doc: update punctuation Resolve spotted issues related to missing or extraneous commas, dashes.

journal: fix access to munmapped memory in sd_journal_enumerate_unique sd_j_e_u needs to keep a reference to an object while comparing it with possibly duplicate objects in other files. Because the size of mmap cache is limited, with enough files and object to compare to, at some point the object being compared would be munmapped, resulting in a segmentation fault. Fix this issue by turning keep_always into a reference count that can be increased and decreased. Other callers which set keep_always=true are unmodified: their references are never released but are ignored when the whole file is closed, which happens at some point. keep_always is increased in sd_j_e_u and later on released.

journal: replace readdir_r with readdir This commit also adds error handling for failures during directory reading.

journald: cache cgroup root path, instead of querying it on every incoming log message

journal: when listing logs of a container make sure we don't accidentally show messages from host too

journal: add ability to browse journals of running OS containers This adds the new library call sd_journal_open_container() and a new "-M" switch to journalctl. Particular care is taken that journalctl's "-b" switch resolves to the current boot ID of the container, not the host.

Use assert_return in more of the public API

journald: keep statistics on how of we hit/miss the mmap cache

journal: fix minor memory leak

list: make our list macros a bit easier to use by not requring type spec on each invocation We can determine the list entry type via the typeof() gcc construct, and so we should to make the macros much shorter to use.

Rename F_TYPE_CMP() to F_TYPE_EQUAL()

journalctl,systemctl: fix tiny memleak

journal: return -ECHILD after a fork A few asserts are replaced with 'return -EINVAL'. I think that assert should not be used to check argument in public functions. Fields in struct sd_journal are rearranged to make it less swiss-cheesy.

journal: change direction tests to use the same convention (cp </> np) The order was different in various places, which makes it harder to read to code. Also consistently use ternany for all direction checks. Remove one free(NULL).

journal: remember last direction of search and keep offset cache The fields in JournalFile are moved around to avoid wasting 7 bytes because of alignment.

journal: add sd_journal_open_files This allows the caller to explicitly specify which journal files should be opened. The same functionality could be achieved before by creating a directory and playing around with symlinks. It is useful to debug stuff and explore the journal, and has been requested before. Waiting is supported, the journal will notice modifications on the files supplied when opening the journal, but will not add any new files.

Use stdint.h macros instead of casts to print uint64_t values Casts are visually heavy, and can obscure unwanted truncations.

journal: loop less in MATCH_AND_TERM conditionals AND term usually don't have many subterms (4 seems to be the maximum sensible number, e.g. _BOOT_ID && _SYSTEMD_UNIT && _PID && MESSAGE_ID). Nevertheless, the cost of checking each subterm can be relatively high, especially when the nested terms are compound, and it makes sense to minimize the number of checks. Instead of looping to the end and then again over the whole list once again after at least one term changed the offset, start the loop at the term which caused the change. This way ½ terms in the AND match are not checked unnecessarily again.

journal: add ability to filter by current user This is the just the library part. SD_JOURNAL_CURRENT_USER flags is added to sd_j_open(), to open files from current user. SD_JOURNAL_SYSTEM_ONLY is renamed to SD_JOURNAL_SYSTEM, and changed to mean to (also) open system files. This way various flags can be combined, which gives them nicer semantics, especially if other ones are added later. Backwards compatibility is kept, because SD_JOURNAL_SYSTEM_ONLY is equivalent to SD_JOURNAL_SYSTEM if used alone, and before there we no other flags.

journal: simplify match_free_if_empty

systemctl: limit logs in status to current boot Also reworded a few debug messages for brevity, and added a log statement which prints out the filter at debug level: Journal filter: (((UNIT=sys-module-configfs.device AND _PID=1) OR (COREDUMP_UNIT=sys-module-configfs.device AND MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1) OR _SYSTEMD_UNIT=sys-module-configfs.device) AND _BOOT_ID=4e3c518ab0474c12ac8de7896fe6b154)

sd-journal: check if the pointers passed are the same

systemd-python: add __version__ strings

Add __attribute__((const, pure, format)) in various places I'm assuming that it's fine if a _const_ or _pure_ function calls assert. It is assumed that the assert won't trigger, and even if it does, it can only trigger on the first call with a given set of parameters, and we don't care if the compiler moves the order of calls.

Reintroduce f_type comparison macro This reverts commit 4826f0b7b5c0aefa08b8cc7ef64d69027f84da2c. Because statfs.t_type can be int on some architecures, we have to cast the const magic to the type, otherwise the compiler warns about signed/unsigned comparison, because the magic can be 32 bit unsigned. statfs(2) man page is also wrong on some systems, because f_type is not __SWORD_TYPE on some architecures. The following program: int main(int argc, char**argv) { struct statfs s; statfs(argv[1], &s); printf("sizeof(f_type) = %d\n", sizeof(s.f_type)); printf("sizeof(__SWORD_TYPE) = %d\n", sizeof(__SWORD_TYPE)); printf("sizeof(long) = %d\n", sizeof(long)); printf("sizeof(int) = %d\n", sizeof(int)); if (sizeof(s.f_type) == sizeof(int)) { printf("f_type = 0x%x\n", s.f_type); } else { printf("f_type = 0x%lx\n", s.f_type); } return 0; } executed on s390x gives for a btrfs: sizeof(f_type) = 4 sizeof(__SWORD_TYPE) = 8 sizeof(long) = 8 sizeof(int) = 4 f_type = 0x9123683e

journal: when iterating through a file we might lose messages when changing direction. https://bugs.freedesktop.org/show_bug.cgi?id=63672

Revert f_type fixups This reverts commit a858b64dddf79177e12ed30f5e8c47a1471c8bfe. This reverts commit aea275c43194b6ac519ef907b62c5c995050fde0. This reverts commit fc6e6d245ee3989c222a2a8cc82a33475f9922f3. This reverts commit c4073a27c555aeceac87a3b02a83141cde641a1e. This reverts commit cddf148028f525be8176e7f1cbbf4f862bd287f6. This reverts commit 8c68a70170b31f93c287f29fd06c6c17edaf19ad. The constants are now casted to __SWORD_TYPE, which should resolve the compiler warnings about signed vs unsigned. After talking to Kay, we concluded: This should be fixed in the kernel, not worked around in userspace tools. Architectures cannot use int and expect magic constants lager than INT_MAX to work correctly. The kernel header needs to be fixed. Even coreutils cannot handle it: #define RAMFS_MAGIC 0x858458f6 # stat -f -c%t / ffffffff858458f6 #define BTRFS_SUPER_MAGIC 0x9123683E # stat -f -c%t /mnt ffffffff9123683e Although I found the perfect working macro to fix the thing :) __extension__ ({ \ bool _ret = false; \ switch(f) { case c: _ret=true; }; \ ( _ret ); \ })

move _cleanup_ attribute in front of the type http://lists.freedesktop.org/archives/systemd-devel/2013-April/010510.html

rename CMP_F_TYPE to F_TYPE_CMP

Add ugly CMP_F_TYPE() macro On some architectures (like s390x) the kernel has the type int for f_type, but long in userspace. Assigning the 32 bit magic constants from linux/magic.h to the 31 bit signed f_type in the kernel, causes f_type to be negative for some constants. glibc extends the int to long for those architecures in 64 bit mode, so the negative int becomes a negative long, which cannot be simply compared to the original magic constant, because the compiler would automatically cast the constant to long. To workaround this issue, we also compare to the (int)MAGIC value in a macro. Of course, we could do #ifdef with the architecure, but it has to be maintained, and the magic constants are 32 bit anyway. Someday, when the int is unsigned or long for all architectures, we can remove this macro again. Until then, keep it as simple as it can be.

fixup for cddf148028f52 Instead of making a type up, just use __SWORD_TYPE, after reading statfs(2). Too bad, this does not fix s390x because __SWORD_TYPE is (long int) and the kernel uses (int) to fill in the field!!!!!!

fixup 8c68a7017 and cast to (unsigned long)

fixed statfs.f_type signed vs unsigned comparisons statfs.f_type is signed but the filesystem magics are unsigned. Casting the magics to signed will not make the signed. Problem seen on big-endian 64bit s390x with __fsword_t 8 bytes. Casting statfs.f_type to unsigned on the other hand will get us what we need. https://bugzilla.redhat.com/show_bug.cgi?id=953217

journal: add one more level on top with AND When using "-p" and "-b" in combination with "-u", the output is not what you would expect. The reason is the sd_journal_add_disjunction() call in add_matches_for_unit() and add_matches_for_user_unit(), which adds two ORs without taking the other conditions to every OR. Adding another level on top with AND and sd_journal_add_conjunction() solves the problem. Output before: $ journalctl -o short-monotonic -ab -p 0 -u sshd.service -- Reboot -- [ 3.216305] lenovo systemd[1]: Starting OpenSSH server daemon... -- Reboot -- [ 3.168666] lenovo systemd[1]: Starting OpenSSH server daemon... [ 3.169639] lenovo systemd[1]: Started OpenSSH server daemon. [36285.635389] lenovo systemd[1]: Stopped OpenSSH server daemon. -- Reboot -- [ 10.838657] lenovo systemd[1]: Starting OpenSSH server daemon... [ 10.913698] lenovo systemd[1]: Started OpenSSH server daemon. [ 6881.035183] lenovo systemd[1]: Stopped OpenSSH server daemon. -- Reboot -- [ 6.636228] lenovo systemd[1]: Starting OpenSSH server daemon... [ 6.662573] lenovo systemd[1]: Started OpenSSH server daemon. [ 6.681148] lenovo sshd[397]: Server listening on 0.0.0.0 port 22. [ 6.681379] lenovo sshd[397]: Server listening on :: port 22. As we see, the output is from _every_ boot and priority 0 is not taken into account. Output after patch: $ journalctl -o short-monotonic -ab -p 0 -u sshd.service -- Logs begin at Sun 2013-02-24 20:54:44 CET, end at Tue 2013-03-19 14:58:21 CET. -- Increasing the priority: $ journalctl -o short-monotonic -ab -p 6 -u sshd.service -- Logs begin at Sun 2013-02-24 20:54:44 CET, end at Tue 2013-03-19 14:59:12 CET. -- [ 6.636228] lenovo systemd[1]: Starting OpenSSH server daemon... [ 6.662573] lenovo systemd[1]: Started OpenSSH server daemon. [ 6.681148] lenovo sshd[397]: Server listening on 0.0.0.0 port 22. [ 6.681379] lenovo sshd[397]: Server listening on :: port 22.

journal: Fix typo This would break backwards skipping. https://bugs.freedesktop.org/show_bug.cgi?id=63250

journal: add sd_journal_get_timeout() call to public API Let's do the wake-up logic on NFS internally, making things simpler for users.

journal: add public API call sd_journal_get_events() This function should be used when filling in "struct pollfd"'s .events field for watching the journal. It will always return POLLIN for now, but we should keep our options open to change this later on. This mimics libsystemd-bus' sd_bus_get_events() call with the same purpose.

catalog: open up catalog internals In order to write tests for the catalog functions, they are made non-static and start taking a 'database' parameter, which is the name of a file with the preprocessed catalog entries. This makes it possible to make test-catalog part of the normal test suite, since it now only operates on files in /tmp. Some more tests are added.

journal: don't access j->files after use https://bugs.freedesktop.org/show_bug.cgi?id=62605

journalctl: various fixes to the access check logic - Reword messages a bit - Correct check whether EACCES is in the set of errors - Don't complain if no journal files are found - allocate Set object for errors lazily since in the best case we don't need it at all. - don't consider it an error if /run/log/journal doesn't exist (because that's the usual case actually, if storage is enabled)

journalctl: be smarter about journal error checks There are many ways in which we can get those checks wrong, so it is better to warn and then error out on a real access failure. The error messages are wrapped to <80 lines, because their primary use is to be displayed in the terminal, and it is easier to read them this way. Reading them in the journal can be a bit trickier, but this is a bug in logs-show.c.

sd-journal: do not require path to be absolute Seems natural to be able to specify relative directory, e.g. with journalctl -D. And even if, this should be checked in front-end code, not in the library.

journal: use _cleanup_ One log_debug() moved to match order in other functions.

journal: use sd_journal_close on error in sd_journal_new

libsystemd-journal: return 0 on success in get_data() The man page says so. Right now 0 would be returned if the data was encrypted, 1 otherwise.

log-show: look at everything in /run/log/journal Logs written by journald from the initramfs may be written to a directory with the name created from a random machine-id. Afterwards, when the root filesystem has been mounted and machine-id reinitalized, logs will be written to the directory with a name created from the proper machine-id. When logs are flushed to /var/log/journal, everything is copied to one output directory. When journalctl without '-m' is run after the logs have been flushed to /var/log/journal, all messages are shown. However, when run while logs are still in /run/log/journal, those stored under the random machine-id will not be shown. Make journalctl behave the same regardless whether persistent storage has been enabled or not, and slurp all files from /run/log/journal even without '-m'.

journal: sd_journal_get_cursor should return 0 on success Documentation states that 0 is correct, and all other similar functions return 0 on success. Pointed-out-by: Steven Hiscocks <steven-systemd@hiscocks.me.uk>

journal: by default do not decompress dat objects larger than 64K This introduces a new data threshold setting for sd_journal objects which controls the maximum size of objects to decompress. This is relieves the library from having to decompress full data objects even if a client program is only interested in the initial part of them. This speeds up "systemd-coredumpctl" drastically when invoked without parameters.

journal: add sd_journal_get_catalog_for_message_id() as API to get catalog entry for any message ID without requiring an open journal file

journal, shared: fix warnings during compilation on 32 bits Some filesystem magics are too big to fit in 31 bits, and are wrapped to negative. f_type is an int on 32 bits, so it is signed, and we get a warning on comparison.

journal: implement message catalog The message catalog can be used to attach short help texts to log lines, keyed by their MESSAGE_ID= fields. This is useful to help the administrator understand the context and cause of a message, find possible solutions and find further related documentation. Since this is keyed off MESSAGE_ID= this will only work for native journal messages. The message catalog supports i18n, and is useful to augment english language system messages with explanations in the local language. This commit only includes short explanatory messages for a few example message IDs, we'll add more complete documentation for the relevant systemd messages later on.

journal: provide an API that allows client to figure out whether they need to recheck the journal manually for changes in regular intervals Network file systems generally do not offer inotify() that would work across the network. We hence cannot rely on inotify() exclusiely in those case. Provide an API to determine these cases, and suggest doing manual regular rechecks. Note that this is not complete yet, as we need to rescan journal dirs on network file systems explicitly to find new/removed files

journal: properly determine cutoff max date

journal: typo fix

journal: validate field name in sd_journal_query_unique()

libsystemd-journal: export sd_journal_restart_unique

journal: add ability to list values a specified field can take in all entries of the journal The new 'unique' API allows listing all unique field values that a field specified by a field name can take in all entries of the journal. This allows answering queries such as "What units logged to the journal?", "What hosts have logged into the journal?", "Which boot IDs have logged into the journal?". Ultimately this allows implementation of tools similar to lastlog based on journal data. Note that listing these field values will not work for journal files created with older journald, as the field values are not indexed in older files.

journalctl: properly track rotated journals in follow mode

sd-journal: fix bad memory access

journal: when browsing the journal via browse.html allow clicking on entries to show their details

journal: drop path name from cursor strings we had this mostly for debugging purposes and it was ignored when parsing anyway, so let's get rid of it

journal: make sure sd_journal_seek_cursor() seeks to the specified entry if it exists, not one after it

sd-journal: properly parse cursor strings

util: define union dirent_storage and make use of it everywhere Make sure to allocate enough space for readdir_r(). https://bugzilla.redhat.com/show_bug.cgi?id=858754

journal: when comparing two entries from separate files make sure we reposition the mmap window The mmap cache doesn't guarantee that we can look at two files at the same time. Hence make sure to look at the entries to compare one after the other, instead of at the same time when comparing them, and reposition the window in between.

journal: add call to determine current journal file disk usage

sd-journal: return null when mmap_cache_new fails

mmap: resize arrays dynamically

journal: implement generic sharable mmap caching logic instead of having one simple per-file cache implement an more comprehensive one that works for multiple files and can actually maintain multiple maps per file and per object type.

journald: initial version of FSPRG hookup This adds forward-secure authentication of journal files. This patch includes key generation as well as tagging of journal files, Verification of journal files will be added in a later patch.

journalctl: include corrupted files in output If a journal file was rotated away because it was corrupted or dirty we should still show its contents via "journalctl".

journal: allow watching symlinked journal dirs

journal: when watching directories actually watch the directories asked for

journal: follow symlinks when enumerating journals

journald: don't choke on journal files with no cutoff date

journal: size journal data hash table based on maximum file size metrics The default of 2047 hash table entries turned out to result in way too many collisions for bigger files, hence scale the hash table size by the estimated maximum file size.

journal: automatically rotate journal files if the data hash table is full > 75% Previously, when the main data hash table grows too full the performance simply started to decrease drastically. Instead, now simply rotate to a new journal file as the hash table gets to full, so that we can start with a new fresh empty hash table.

Spelling fixes.

journal: align byte-buffer that gets cased to an object On Sun, Jul 15, 2012 at 2:00 PM, Koen Kooi <koen@dominion.thruhere.net> wrote: > | src/journal/sd-journal.c: In function 'sd_journal_process': > | src/journal/sd-journal.c:1891:21: warning: cast increases required alignment of target type [-Wcast-align] > | src/journal/sd-journal.c:1900:29: warning: cast increases required alignment of target type [-Wcast-align]

util: rename join() to strjoin() This is to match strappend() and the other string related functions.

journal: beef up journal matches considerably we now can take multiple matches, and they will apply as AND if they apply to different fields and OR if they apply to the same fields. Also, terms of this kind can be combined with an overreaching OR.

journal: check fields we search for more carefully

journal: fix interleaving of files with different time sources

journalctl: add a marker to log output for reboots With this we'll print a marker "----- Reboot -----" between two subsequent lines with different boot IDs.

journal: rework directory enumeration/watch logic There's now sd_journal_new_directory() for watching specific journal directories. This is exposed in journalctl -D. sd_journal_wait() and sd_journal_process() now return whether changes in the journal are invalidating or just appending. We now create inotify kernel watches only when we actually need them

journal: introduce sd_journal_wait() to simplify writing synchronous clients

journal: fix iteration through journal if one file is corrupt

journal: fix SD_JOURNAL_SYSTEM_ONLY flag

journal: expose and make use of cutoff times of journal This helps explaining when the log output of "systemctl status" is incomplete because the logs got rotated since the service was started.

journal: don't allow adding invalid matches to the context

journalctl: for now complain if more than one match is provided since this is still broken

util: split-out path-util.[ch]

relicense to LGPLv2.1 (with exceptions) We finally got the OK from all contributors with non-trivial commits to relicense systemd from GPL2+ to LGPL2.1+. Some udev bits continue to be GPL2+ for now, but we are looking into relicensing them too, to allow free copy/paste of all code within systemd. The bits that used to be MIT continue to be MIT. The big benefit of the relicensing is that closed source code may now link against libsystemd-login.so and friends.

journal: properly handle if we interleave files with different boot ids If we try to locate a monotonic time in a file that doesn't have any entries with the matching boot id, then don't fail on it, simply fall back to calendar time.

add sparse support to detect endianness bug le16/32/64_t type should be used when storing little-endian value header to integrate with sparse from Josh Triplett <josh@joshtriplett.org>

journal: Don't hold pointers to journal while remapping Hi! I was trying out the journal and the journalctl utility sometimes crashed on me. After some debugging, I tracked it down to the fact that next_with_matches() holds the "c" object pointer through the journal_file_next_entry_for_data() call -- which apparently may re-map the journal file, invalidating the pointer. The attached patch fixes this crash for me, but being unfamiliar with the code, I don't know if I'm doing the right thing. This patch is also available from my github repository: git://github.com/intgr/systemd.git https://github.com/intgr/systemd Regards, Marti For the record, here's the original stack trace at the time of remapping: ret=0x7fff1d5cdec0) at src/journal/journal-file.c:330 ret=0x7fff1d5cdf28) at src/journal/journal-file.c:414 ret=0x7fff1d5ce0a0, offset=0x7fff1d5ce098) at src/journal/journal-file.c:1101 i=5705, ret=0x7fff1d5ce0a0, offset=0x7fff1d5ce098) at src/journal/journal-file.c:1147 p=6413608, data_offset=66600, direction=DIRECTION_DOWN, ret=0x7fff1d5ce0a0, offset=0x7fff1d5ce098) at src/journal/journal-file.c:1626 direction=DIRECTION_DOWN, ret=0x7fff1d5ce120, offset=0x7fff1d5ce128) at src/journal/sd-journal.c:533 direction=DIRECTION_DOWN, ret=0x7fff1d5ce170, offset=0x7fff1d5ce178) at src/journal/sd-journal.c:595 src/journal/sd-journal.c:651 From 9266fc6a58065a7c5dab67430fd78925e519dce9 Mon Sep 17 00:00:00 2001 From: Marti Raudsepp <marti@juffo.org> Date: Fri, 9 Mar 2012 16:23:00 +0200 Subject: [PATCH] journal: Don't hold pointers to journal while remapping This would cause a segfault otherwise.

journal: systemd v38 fails to build on big-endians Hi, during the builds for Fedora/s390x I've found that systemd v38 fails to build on big-endian platforms. ... make[2]: Entering directory `/root/systemd' CC src/journal/libsystemd_journal_la-sd-journal.lo src/journal/sd-journal.c: In function 'init_location': src/journal/sd-journal.c:69:22: error: incompatible types when initializing type 'long unsigned int' using type 'sd_id128_t' src/journal/sd-journal.c:69:20: error: incompatible types when assigning to type 'sd_id128_t' from type 'long unsigned int' make[2]: *** [src/journal/libsystemd_journal_la-sd-journal.lo] Error 1 I see the problem in using le64toh() on the 16 bytes boot_id structure in init_location() Please see http://s390.koji.fedoraproject.org/koji/taskinfo?taskID=544375 for a full build log and attachment for a proposed fix. With regards Dan

sd-journal: implement a number of non-implemented calls from the API for now

journal: beef up journal output of systemctl and journalctl

journalctl: fix counting of -n parameter

journal: fix reverse traversing of entries

journal: add missing error check

journal: don't mind too much if we can't find a monotonic timestamp

journal: add missing error check

journal: add _public_ to all sd-journal calls, and add parameter checks

journald: flush /run to /var as soon as it becomes available

journal: fix typo

journald: enforce some syntax restrictions on field names sent from the client side

journalctl: add json, export, short and verbose output modes

journal: add inline compression support with XZ

journal: fix matches

journal: implement inotify-based live logging logic

journal: enforce limits on open journal files

journal: implement multiple field matches

journal: automaticall reset location when the set of matches changes

journal: implement seek to head/tail

journal: unify code for up and for down traversal

journal: properly implement matching with multiple matches

journal: if two entries match with everything but seqnums, they are still identical

journal: move field index from file into journal object

journal: when the same entry is in two files, skip over them in sync

journal: synchronize seqnum across files

journal: fix field retrieval by name

journal: implement parallel file traversal

journal: implementation rotation

journal: implement parallel traversal in client

journal: split user logs into their own journal files

journal: replace linked list by hashmap when merging files

journal: store XOR combination of entry data object hashes to identify hash lines

journal: add preliminary incomplete implementation