tree-wide: remove Emacs lines from all files This should be handled fine now by .dir-locals.el, so need to carry that stuff in every file.

journal: remove template from open_journal args None of the callers take advantage of this parameter, it's always NULL, this is just a private helper function to simplify the call sites so drop the template parameter altogether. If a caller emerges later who needs it, it can be restored.

journald: add a couple of static asserts checking logging constants Whenever we include a log level or facility in a journal string field, make sure the compiler checks for us that that's actually the right thing to do.

journald: minor fixes This primarily contains some minor coding style fixups for 7a24f3bf2fb181243a1957a0cdd54cd919396793 and earlier changes. Specifically: * Don't log at log levels above LOG_DEBUG from "library" code like journal-file.c * Don't negate errno values before passing them to log_debug_errno(), as the call can handle this fine anyway * Cast some calls we knowingly ignore the return values of to (void) * Don't clobber function call-by-ref return values on failure * Don't mix function calls and variable declarations in one line There's also one more relevant change: when failing to enqueue a journal change fs event, we'll run it immediately.

journald: restore oom safety v2: - use xsprintf

journald: use structured message + catalog entry for disk usage The format of the journald disk usage log entry was changed back and forth a few times. It is annoying to have a very verbose message, but if it is short it is hard to understand. But we have a tool for this, the catalogue. $ journalctl -x -u systemd-journald Jan 23 18:48:50 rawhide systemd-journald[891]: Runtime journal (/run/log/journal/) is 8.0M, max 196.2M, 188.2M free. -- Subject: Disk space used by the journal -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Runtime journal (/run/log/journal/) is currently using 8.0M. -- Maximum allowed usage is set to 196.2M. -- Leaving at least 294.3M free (of currently available 1.9G of disk space). -- Enforced usage limit is thus 196.2M, of which 188.2M are still available. -- -- The limits controlling how much disk space is used by the journal may -- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, -- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in -- /etc/systemd/journald.conf. See journald.conf(5) for details. Jan 23 18:48:50 rawhide systemd-journald[891]: System journal (/var/log/journal/) is 480.1M, max 1.6G, 1.2G free. -- Subject: Disk space used by the journal -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- System journal (/var/log/journal/) is currently using 480.1M. -- Maximum allowed usage is set to 1.6G. -- Leaving at least 2.5G free (of currently available 5.8G of disk space). -- Enforced usage limit is thus 1.6G, of which 1.2G are still available. -- -- The limits controlling how much disk space is used by the journal may -- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, -- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in -- /etc/systemd/journald.conf. See journald.conf(5) for details.

journald: allow additional payload in server_driver_message The code to format the iovec is shared with log.c. All call sites to server_driver_message are changed to include the additional "MESSAGE=" part, but the new functionality is not used and change in functionality is not expected. iovec is preallocated, so the maximum number of messages is limited. In server_driver_message N_IOVEC_PAYLOAD_FIELDS is currently set to 1. New code is not oom safe, it will fail if memory cannot be allocated. This will be fixed in subsequent commit.

journald: do not free uninitialized pointer in error path

journal: coalesce ftruncate()s in 250ms windows Prior to this change every journal append causes an ftruncate() for the sake of inotify propagation of the mmap-based writes. With this change the notification is deferred up to ~250ms, coalescing any repeated journal writes during the deferred period into a single ftruncate(). The ftruncate() call isn't free and doing it on every append adds unnecessary overhead and latency in the journald event loop. Introduces journal_file_enable_post_change_timer() which manages a timer on the provided sd-event instance for scheduling coalesced ftruncates. The ftruncate() behavior is unchanged unless journal_file_enable_post_change_timer() is called on the JournalFile. While not a tremendous improvement, profiling systemd-journald event loop latencies using instrumentation as introduced by 34b8751 it was observed that coalescing the ftruncates was low-hanging fruit worth pursuing. Note orders 12 and 13 shifting left into order 11 and order 6 dipping into order 5: Unmodified: log2(us) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ----------------------------------------------------------- [10685.414572] 0 0 0 0 38 602 61 2 290 60 1643 2554 13 1 4 1 0 0 1 [10690.415114] 0 0 0 0 0 646 54 7 309 44 2073 2148 17 1 3 0 0 0 1 [10695.415509] 0 0 0 0 1 650 73 3 324 37 2071 2270 9 0 0 1 0 1 0 [10700.416297] 0 0 0 0 0 659 50 4 318 38 2111 2152 6 0 1 0 0 1 1 [10705.417136] 0 0 0 0 2 660 48 4 320 38 2129 2146 12 1 1 0 0 1 1 [10710.489114] 0 0 0 0 0 673 38 3 321 37 1925 2339 7 0 0 0 0 1 1 [10715.489613] 0 0 0 0 3 656 64 8 317 48 2365 2007 7 0 0 0 0 0 1 Coalesced: log2(us) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ----------------------------------------------------------- [ 6169.161360] 0 0 0 1 24 786 54 11 389 24 4192 771 6 4 0 0 1 0 1 [ 6174.161705] 0 0 0 1 18 800 35 6 380 27 3977 893 3 1 0 0 1 0 1 [ 6179.162741] 0 0 0 1 28 768 51 4 391 16 3998 831 5 3 0 0 0 0 2 [ 6184.162856] 0 0 0 0 19 770 60 2 376 26 3795 1004 9 5 1 0 1 0 1 [ 6189.163279] 0 0 0 0 28 761 49 7 372 27 3729 1056 3 2 0 0 1 0 1 [ 6194.164255] 0 0 0 0 25 785 49 7 394 19 3996 908 6 3 2 0 0 0 1 [ 6199.164658] 0 0 0 0 29 797 35 5 389 18 3995 898 3 4 1 1 1 0 1 The remaining high-order delays are a result of the synchronous fsyncs in systemd-journald, beyond the scope of this commit.

journal: move the gist of server_fix_perms to acl-util.[hc] Most of the function is moved to acl-util.c to make it possible to add tests in subsequent commit. Setting of the mode in server_fix_perms is removed: - we either just created the file ourselves, and the permission be better right, - or the file was already there, and we should not modify the permissions. server_fix_perms is renamed to server_fix_acls to better reflect new meaning, and made static because it is only used in one file.

selinux: split up mac_selinux_have() from mac_selinux_use() Let's distuingish the cases where our code takes an active role in selinux management, or just passively reports whatever selinux properties are set. mac_selinux_have() now checks whether selinux is around for the passive stuff, and mac_selinux_use() for the active stuff. The latter checks the former, plus also checks UID == 0, under the assumption that only when we run priviliged selinux management really makes sense. Fixes: #1941

tree-wide: group include of libudev.h with sd-*

journald: trivial simplification

tree-wide: use right cast macros for UIDs, GIDs and PIDs

tree-wide: sort includes Sort the includes accoding to the new coding style.

journald: fix accuracy of watchdog timer event Adding 3/4th of the watchdog frequency as accuracy on top of 1/2 of the watchdog frequency means we might end up at 5/4th of the frequency which means we might miss the message from time to time. Maybe fixes #1804

journald: rework --sync/--rotate logic to use CLOCK_MONOTONIC timestamp files Previously, we'd rely on the mtime timestamps of the touch files to see if our sync/rotation requests were already suppressed. This means we rely on CLOCK_REALTIME timestamps. With this patch we instead store the CLOCK_MONOTONIC timestamp *in* the touch files, and avoid relying on mtime. This should make things more reliable when the clock or underlying mtime granularity is not very good. This also adds warning messages if writing any of the flag files fails.

journalctl: make --rotate synchronous, too Of course, ideally we'd just use normal synchronous bus calls, but this is out of the question as long as we rely on dbus-daemon (which logs to journald, and thus cannot use to avoid cyclic sync loops). Hence, instead, reuse the wait logic already implemented for --sync, and use a signal in one direction, and a mtime watch file for the reply.

journalctl: add new --sync switch for syncing the journal to disk With this new "--sync" switch we add a synchronous way to sync everything queued to disk, and return only after that's complete. This command gives the guarantee that anything queued before has hit the disk before the command returns. While we are at it, also improve the man pages and help text for journalctl a bit.

journald: be less picky when receiving epoll events The event might be flagged with stuff we don't expect, hence don't be needlessly picky, just rely on the kernel passing us sensible events.

journald: dispatch SIGTERM/SIGINT with a low priority Let's make sure to process all queued log data before exiting, so that we don't unnecessary lose messages when shutting down. https://github.com/systemd/systemd/pull/1812#issuecomment-155149871

defs: rework CONF_DIRS_NULSTR() macro The macro is generically useful for putting together search paths, hence let's make it truly generic, by dropping the implicit ".d" appending it does, and leave that to the caller. Also rename it from CONF_DIRS_NULSTR() to CONF_PATHS_NULSTR(), since it's not strictly about dirs that way, but any kind of file system path. Also, mark CONF_DIR_SPLIT_USR() as internal macro by renaming it to _CONF_PATHS_SPLIT_USR() so that the leading underscore indicates that it's internal.

treewide: use the negative error codes returned by our functions Our functions return negative error codes. Do not rely on errno being set after calling our own functions.

journal: restore watchdog support

journald: never block when sending messages on NOTIFY_SOCKET socket Otherwise we might run into deadlocks, when journald blocks on the notify socket on PID 1, and PID 1 blocks on IPC to dbus-daemon and dbus-daemon blocks on logging to journald. Break this cycle by making sure that journald never ever blocks on PID 1. Note that this change disables support for event loop watchdog support, as these messages are sent in blocking style by sd-event. That should not be a big loss though, as people reported frequent problems with the watchdog hitting journald on excessively slow IO. Fixes: #1505.

util-lib: split out allocation calls into alloc-util.[ch]

io-util.h: move iovec stuff from macro.h to io-util.h

util-lib: move /proc/cmdline parsing code to proc-cmdline.[ch]

src/basic: rename audit.[ch] → audit-util.[ch] and capability.[ch] → capability-util.[ch] The files are named too generically, so that they might conflict with the upstream project headers. Hence, let's add a "-util" suffix, to clarify that this are just our utility headers and not any official upstream headers.

util-lib: move string table stuff into its own string-table.[ch]

util-lib: move a number of fs operations into fs-util.[ch]

util-lib: introduce dirent-util.[ch] for directory entry calls Also, move a couple of more path-related functions to path-util.c.

util-lib: split string parsing related calls from util.[ch] into parse-util.[ch]

journal: add missing include

journald-server: port to extract_first_word

util-lib: split out fd-related operations into fd-util.[ch] There are more than enough to deserve their own .c file, hence move them over.

util-lib: split our string related calls from util.[ch] into its own file string-util.[ch] There are more than enough calls doing string manipulations to deserve its own files, hence do something about it. This patch also sorts the #include blocks of all files that needed to be updated, according to the sorting suggestions from CODING_STYLE. Since pretty much every file needs our string manipulation functions this effectively means that most files have sorted #include blocks now. Also touches a few unrelated include files.

journald: make sure r is always initialized

tree-wide: add more void casts for various syscall invocations

Document journald sockets paths a bit more Adding them to the documentation makes it easier to find the right man page for people who are trying to understand where some socket in the filesystem is coming from.

journald: make audit socket optional If we were given some sockets through socket activation, and audit socket is not among them, do not try to open it. This way, if the socket unit is disabled, we will not receive audit events. https://bugzilla.redhat.com/show_bug.cgi?id=1227379

journal: rework vacuuming logic Implement a maximum limit on number of journal files to keep around. Enforcing a limit is useful on this since our performance when viewing pays a heavy penalty for each journal file to interleve. This setting is turned on now by default, and set to 100. Also, actully implement what 348ced909724a1331b85d57aede80a102a00e428 promised: use whatever we find on disk at startup as lower bound on how much disk space we can use. That commit introduced some provisions to implement this, but actually never did. This also adds "journalctl --vacuum-files=" to vacuum files on disk by their number explicitly.

journal: use automatic clenup for ACL types

journal: improve some messages Indicate that we are ignoring errors, when we ignore them, and log that at LOG_WARNING level. Use the right error code for the log message.

journal: prefer stack allocation

journal: make journal_file_close() return NULL The way it is customary everywhere else in our sources.

journald: add syslog fields for driver messages

tree-wide: remove a number of invocations of strerror() and replace by %m Let's clean up our tree a bit, and reduce invocations of the thread-unsafe strerror() by replacing it with printf()'s %m specifier.

Bug #944: Deletion of unnecessary checks before a few calls of systemd functions The following functions return immediately if a null pointer was passed. * calendar_spec_free * link_address_free * manager_free * sd_bus_unref * sd_journal_close * udev_monitor_unref * udev_unref It is therefore not needed that a function caller repeats a corresponding check. This issue was fixed by using the software Coccinelle 1.0.1.

journald: move server_restore_streams out of server_open_stdout_socket One has little to do with the other, so it's confusing that the second also calls the first.

journal: reword msg about enforced size limits a bit http://lists.freedesktop.org/archives/systemd-devel/2015-July/033574.html

journal: in persistent mode create /var/log/journal, with all parents. systemd-journald races with systemd-tmpfiles-setup, and hence both are started at about the same time. On a bare-bones system (e.g. with empty /var, or even non-existent /var), systemd-tmpfiles will create /var/log. But it can happen too late, that is systemd-journald already attempted to mkdir /var/log/journal, ignoring the error. Thus failing to create /var/log/journal. One option, without modifiying the dependency graph is to create /var/log/journal directory with parents, when persistent storage has been requested.

everywhere: port everything to sigprocmask_many() and friends This ports a lot of manual code over to sigprocmask_many() and friends. Also, we now consistly check for sigprocmask() failures with assert_se(), since the call cannot realistically fail unless there's a programming error. Also encloses a few sd_event_add_signal() calls with (void) when we ignore the return values for it knowingly.

journald: don't employ inner loop for reading from incoming sockets Otherwise, if the socket is constantly busy we will never return to the event loop, but we really need to to dispatch other (possibly more high-priority) events too. Hence, return after dispatching one message to the event handler, and rely on the event loop calling us back right-away. Fixes #125

util: introduce CMSG_FOREACH() macro and make use of it everywhere It's only marginally shorter then the usual for() loop, but certainly more readable.

util: split out signal-util.[ch] from util.[ch] No functional changes.

util: split all hostname related calls into hostname-util.c

shared: add process-util.[ch]

shared: add formats-util.h

util: rework rm_rf() logic - Move to its own file rm-rf.c - Change parameters into a single flags parameter - Remove "honour sticky" logic, it's unused these days

remove unused includes This patch removes includes that are not used. The removals were found with include-what-you-use which checks if any of the symbols from a header is in use.

util: rework strappenda(), and rename it strjoina() After all it is now much more like strjoin() than strappend(). At the same time, add support for NULL sentinels, even if they are normally not necessary.

Add a snprinf wrapper which checks that the buffer was big enough If we scale our buffer to be wide enough for the format string, we should expect that the calculation was correct. char_array_0() invocations are removed, since snprintf nul-terminates the output in any case. A similar wrapper is used for strftime calls, but only in timedatectl.c.

tmpfiles: add 'a' type to set ACLs

journald: allow zero length datagrams again This undoes a small part of 13790add4bf648fed816361794d8277a75253410 which was erroneously added, given that zero length datagrams are OK, and hence zero length reads on a SOCK_DGRAM be no means mean EOF.

journald: allow restarting journald without losing stream connections Making use of the fd storage capability of the previous commit, allow restarting journald by serilizing stream state to /run, and pushing open fds to PID 1.

journald: reuse IOVEC_TOTAL_SIZE() macros where possible

journald: when we detect the journal file we are about to write to has been deleted, rotate https://bugzilla.redhat.com/show_bug.cgi?id=1171719

journald: constify all things

journald: prefix exported calls with "server_", unexport unnecessary calls

journald: process SIGBUS for the memory maps we set up Even though we use fallocate() it appears that file systems like btrfs will trigger SIGBUS on certain low-disk-space situation. We should handle that, hence catch the signal, add it to a list of invalidated pages, and replace the page with an empty memory area. After each write check if SIGBUS was triggered, and consider the write invalid if it was. This should make journald a lot more robust with file systems where fallocate() is not reliable, for example all CoW file systems (btrfs...), where changing written data can fail with disk full errors. https://bugzilla.redhat.com/show_bug.cgi?id=1045810

journald: correct spacing near eol code comments

journald: close passed fds we cannot make sense of This is mostly likely the audit socket, and we really should close it if we cannot make sense of it, since as long as it is open the kernel might disable the kmsg forwarding of audit msgs, and we should avoid that, since audit msgs might get completely lost then. I also downgraded the log message we show a bit, after all things should really work fine, and we proceed fine with it.

journald: Support journald.conf.d directories in the usual search paths

treewide: another round of simplifications Using the same scripts as in f647962d64e "treewide: yet more log_*_errno + return simplifications".

treewide: use log_*_errno whenever %m is in the format string If the format string contains %m, clearly errno must have a meaningful value, so we might as well use log_*_errno to have ERRNO= logged. Using: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\((".*%m.*")/log_\1_errno(errno, \2/' Plus some whitespace, linewrap, and indent adjustments.

treewide: more log_*_errno + return simplifications

treewide: more log_*_errno() conversions, multiline calls Basically: find . -name '*.[ch]' | while read f; do perl -i.mmm -e \ 'local $/; local $_=<>; s/log_(debug|info|notice|warning|error|emergency)\("([^"]*)%s"([^;]*),\s*strerror\(-?([->a-zA-Z_]+)\)\);/log_\1_errno(\4, "\2%m"\3);/gms;print;' \ $f; done Plus manual indentation fixups.

treewide: no need to negate errno for log_*_errno() It corrrectly handles both positive and negative errno values.

treewide: auto-convert the simple cases to log_*_errno() As a followup to 086891e5c1 "log: add an "error" parameter to all low-level logging calls and intrdouce log_error_errno() as log calls that take error numbers", use sed to convert the simple cases to use the new macros: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\("(.*)%s"(.*), strerror\(-([a-zA-Z_]+)\)\);/log_\1_errno(-\4, "\2%m"\3);/' Multi-line log_*() invocations are not covered. And we also should add log_unit_*_errno().

log: fix order of log_unit_struct() to match other logging calls Also, while we are at it, introduce some syntactic sugar for creating ERRNO= and MESSAGE= structured logging fields.

journald: proceed even if some sockets are unknown systemd-journald would refuse to start if it received an unknown socket from systemd. This is annoying, because the failure more for systemd-journald is unpleasant: systemd will keep restarting journald, but most likely the same error will occur every time. It is better to continue. journald will try to open missing sockets on its own, so things should mostly work. One question is whether to close the sockets which cannot be parsed or to keep them open. Either way we might lose some messages. This failure is most likely for the audit socket (selinux issues), which can be opened multiple times so this not a problem, so I decided to keep them open because it makes it easier to debug the issue after the system is fully started.

util: simplify proc_cmdline() to reuse get_process_cmdline() Also, make all parsing of the kernel cmdline non-fatal.

journald: don't pass around SO_TIMESTAMP timestamp for audit, which we don't have anyway

journalctl: add new --vacuum-size= and --vacuum-time= commands to clean up journal files based on a size/time limit This is equivalent to the effect of SystemMaxUse= and RetentionSec=, however can be invoked directly instead of implicitly.

journald: fix minor memory leak

journald: if available pull audit messages from the kernel into journal logs

journald: constify all things!

journald: fix flushing Commit 74055aa762 'journalctl: add new --flush command and make use of it in systemd-journal-flush.service' broke flushing because journald checks for the /run/systemd/journal/flushed file before opening the permanent journal. When the creation of this file was postponed, flushing stoppage ensued.

journal: make Server::user_journals an OrderedHashmap Order matters here. It replaces oldest entries first when USER_JOURNALS_MAX is reached.

mac: also rename use_{smack,selinux,apparmor}() calls so that they share the new mac_{smack,selinux,apparmor}_xyz() convention

journalctl: add new --flush command and make use of it in systemd-journal-flush.service This new command will ask the journal daemon to flush all log data stored in /run to /var, and wait for it to complete. This is useful, so that in case of Storage=persistent we can order systemd-tmpfiles-setup afterwards, to ensure any possibly newly created directory in /var/log gets proper access mode and owners.

journal: do server_vacuum for sigusr1 runtime journal is migrated to system journal when only "/run/systemd/journal/flushed" exist. It's ok but according to this the system journal directory size(max use) can be over the config. If journal is not rotated during some time the journal directory can be remained as over the config(or default) size. To avoid, do server_vacuum just after the system journal migration from runtime.

Modernization

hashmap: introduce hash_ops to make struct Hashmap smaller It is redundant to store 'hash' and 'compare' function pointers in struct Hashmap separately. The functions always comprise a pair. Store a single pointer to struct hash_ops instead. systemd keeps hundreds of hashmaps, so this saves a little bit of memory.

Properly report invalid quoted strings $ systemd-analyze verify trailing-g.service [./trailing-g.service:2] Trailing garbage, ignoring. trailing-g.service lacks ExecStart setting. Refusing. Error: org.freedesktop.systemd1.LoadFailed: Unit trailing-g.service failed to load: Invalid argument. Failed to create trailing-g.service/start: Invalid argument

Reject invalid quoted strings String which ended in an unfinished quote were accepted, potentially with bad memory accesses. Reject anything which ends in a unfished quote, or contains non-whitespace characters right after the closing quote. _FOREACH_WORD now returns the invalid character in *state. But this return value is not checked anywhere yet. Also, make 'word' and 'state' variables const pointers, and rename 'w' to 'word' in various places. Things are easier to read if the same name is used consistently. mbiebl_> am I correct that something like this doesn't work mbiebl_> ExecStart=/usr/bin/encfs --extpass='/bin/systemd-ask-passwd "Unlock EncFS"' mbiebl_> systemd seems to strip of the quotes mbiebl_> systemctl status shows mbiebl_> ExecStart=/usr/bin/encfs --extpass='/bin/systemd-ask-password Unlock EncFS $RootDir $MountPoint mbiebl_> which is pretty weird

Let config_parse open file where applicable Special care is needed so that we get an error message if the file failed to parse, but not when it is missing. To avoid duplicating the same error check in every caller, add an additional 'warn' boolean to tell config_parse whether a message should be issued. This makes things both shorter and more robust wrt. to error reporting.

Constify ConfigTableItem tables

journald: turn ForwardToSyslog= off by default After all, rsyslog and friends nowadays read their data directly from the journal, hence the forwarding is unnecessary in most cases.

journald: make MaxFileSec really default to 1month journald.conf(5) states that the default for MaxFileSec is one month, but the code didn't respect that.

journald: make SplitMode=uid the default Now that we actually can distuingish system and normal users there's no point in taking session information into account anymore when splitting up logs. This has the beenfit with that coredump information will actually end up in each user's own journal.

coredump: optionally store coredumps on disk, not in the journal Introduce a new configuration file /etc/systemd/coredump.conf to configure when to place coredumps in the journal and when on disk. Since the coredumps are quite large, default to storing them only on disk.

journald: create /run/log/journal with the correct access modes

journald: move /dev/log socket to /run This way we can make the socket also available for sandboxed apps that have their own private /dev. They can now simply symlink the socket from /dev.

logind: don't apply RemoveIPC= to system users We shouldn't destroy IPC objects of system users on logout. http://lists.freedesktop.org/archives/systemd-devel/2014-April/018373.html This introduces SYSTEM_UID_MAX defined to the maximum UID of system users. This value is determined compile-time, either as configure switch or from /etc/login.defs. (We don't read that file at runtime, since this is really a choice for a system builder, not the end user.) While we are at it we then also update journald to use SYSTEM_UID_MAX when we decide whether to split out log data for a specific client.

Remove unnecessary casts in printfs No functional change expected :)

sd-event: rework API to support CLOCK_REALTIME_ALARM and CLOCK_BOOTTIME_ALARM, too

util: replace close_nointr_nofail() by a more useful safe_close() safe_close() automatically becomes a NOP when a negative fd is passed, and returns -1 unconditionally. This makes it easy to write lines like this: fd = safe_close(fd); Which will close an fd if it is open, and reset the fd variable correctly. By making use of this new scheme we can drop a > 200 lines of code that was required to test for non-negative fds or to reset the closed fd variable afterwards.

journal: extract duplicated code to a function

journal: extract duplicated code to a function

Use strlen even for constant strings GCC optimizes strlen("string constant") to a constant, even with -O0. Thus, replace patterns like sizeof("string constant")-1 with strlen("string constant") where possible, for clarity. In particular, for expressions intended to add up the lengths of components going into a string, this often makes it clearer that the expression counts the trailing '\0' exactly once, by putting the +1 for the '\0' at the end of the expression, rather than hidden in a sizeof in the middle of the expression.

journald: add support for wall forwarding This will let journald forward logs as messages sent to all logged in users (like wall). Two options are added: * ForwardToWall (default yes) * MaxLevelWall (default emerg) 'ForwardToWall' is overridable by kernel command line option 'systemd.journald.forward_to_wall'. This is used to emulate the traditional syslogd behaviour of sending emergency messages to all logged in users.

Make tables for DEFINE_STRING_TABLE_LOOKUP consistent Bring some arrays that are used for DEFINE_STRING_TABLE_LOOKUP() in the same order than the enums they reference. Also, pass the corresponding _MAX value to the array initalizer where appropriate.

Remove dead lines in various places As pointed-out by clang -Wunreachable-code. No behaviour changes.

journald: ignore failure to watch hostname_fd on older kernels Prior to 3.2, /proc/sys/kernel/hostname isn't a pollable file and sd_event_add_io will return EPERM. Ignore this failure, since it isn't critical to journald operation. Reported and tested by user sraue on IRC.

api: in constructor function calls, always put the returned object pointer first (or second) Previously the returned object of constructor functions where sometimes returned as last, sometimes as first and sometimes as second parameter. Let's clean this up a bit. Here are the new rules: 1. The object the new object is derived from is put first, if there is any 2. The object we are creating will be returned in the next arguments 3. This is followed by any additional arguments Rationale: For functions that operate on an object we always put that object first. Constructors should probably not be too different in this regard. Also, if the additional parameters might want to use varargs which suggests to put them last. Note that this new scheme only applies to constructor functions, not to all other functions. We do give a lot of freedom for those. Note that this commit only changes the order of the new functions we added, for old ones we accept the wrong order and leave it like that.

journald: log provenience of signals

journald: do not free space when disk space runs low Before, journald would remove journal files until both MaxUse= and KeepFree= settings would be satisfied. The first one depends (if set automatically) on the size of the file system and is constant. But the second one depends on current use of the file system, and a spike in disk usage would cause journald to delete journal files, trying to reach usage which would leave 15% of the disk free. This behaviour is surprising for the user who doesn't expect his logs to be purged when disk usage goes above 85%, which on a large disk could be some gigabytes from being full. In addition attempting to keep 15% free provides an attack vector where filling the disk sufficiently disposes of almost all logs. Instead, obey KeepFree= only as a limit on adding additional files. When replacing old files with new, ignore KeepFree=. This means that if journal disk usage reached some high point that at some later point start to violate the KeepFree= constraint, journald will not add files to go above this point, but it will stay (slightly) below it. When journald is restarted, it forgets the previous maximum usage value, and sets the limit based on the current usage, so if disk remains to be filled, journald might use one journal-file-size less on each restart, if restarts happen just after rotation. This seems like a reasonable compromise between implementation complexity and robustness.

journald/server: replace readdir_r with readdir The available_space function now returns 0 if reading the directory fails. Previously, such errors were silently ignored.

event: rework sd-event exit logic With this change a failing event source handler will not cause the entire event loop to fail. Instead, we just disable the specific event source, log a message at debug level and go on. This also introduces a new concept of "exit code" which can be stored in the event loop and is returned by sd_event_loop(). We also rename "quit" to "exit" everywhere else. Altogether this should make things more robus and keep errors local while still providing a way to return event loop errors in a clear way.

journald: cache cgroup root path, instead of querying it on every incoming log message

journald: cache hostname, boot_id and machine_id fields instead of generating them fresh for each log entry

journald: port to sd-event and enable watchdog support

Ensure unit is journaled for short-lived or oneshot processes In the time it takes to process incoming log messages, the process we are logging details for may exit. This means the cgroup data is no longer available from '/proc'. Unfortunately, the way the code was structured before, we never log _SYSTEMD_UNIT if we don't have this cgroup information. Add an else if case that allows the passed in unit_id to be logged even if we couldn't capture cgroup information. This ensures a command like `journalctl -u run-XXX` will return all log messages from a oneshot process.

journald: mention how long we needed to flush to /var in the logs

util: unify reading of /proc/cmdline Instead of individually checking for containers in each user do this once in a new call proc_cmdline() that read the file only if we are not in a container.

journald: use greedy_realloc in one place

security: missing header inclusions

security: rework selinux, smack, ima, apparmor detection logic Always cache the results, and bypass low-level security calls when the respective subsystem is not enabled.

journald: remove rotated file from hashmap when rotation fails Before, when the user journal file was rotated, journal_file_rotate could close the old file and fail to open the new file. In that case, we would leave the old (deallocated) file in the hashmap. On subsequent accesses, we could retrieve this stale entry, leading to a segfault. When journal_file_rotate fails with the file pointer set to 0, old file is certainly gone, and cannot be used anymore. https://bugzilla.redhat.com/show_bug.cgi?id=890463

Do not use unitialized variable and remove duplicated line

journald: accept EPOLLERR from /dev/kmsg Also print out unexpected epoll events explictly.

journald: avoid NSS in journald In order to avoid a deadlock between journald looking up the "systemd-journal" group name, and nscd (or anyother NSS backing daemon) logging something back to the journal avoid all NSS in journald the same way as we avoid it from PID 1. With this change we rely on the kernel file system logic to adjust the group of created journal files via the SETGID bit on the journal directory. To ensure that it is always set, even after the user created it with a simply "mkdir" on the shell we fix it up via tmpfiles on boot.

journald: log the slice of a process along with each message in _SYSTEMD_SLICE=

journald: Log error when failed to get machine-id on start Can help since the journal requires /etc/machine-id to exists in order to start, and will simply silently exit when it does not.

journald: be a bit more verbose when vacuuming Vacuuming behaviour is a bit confusing, and/or we have some bugs, so those additional messages should help to find out what's going on. Also, rotation of journal files shouldn't be happening too often, so the level of the messages is bumped to info, so that they'll be logged under normal operation.

journald: remove unused variable

journal: handle multiline syslog messages Since the journal can handle multiple lines just well natively, and rsyslog can be configured to handle them as well, there is no need to truncate messages from syslog() after the first newline. Reproducer: 1. Add following four lines to /etc/rsyslog.conf ---------- $EscapeControlCharactersOnReceive off $ActionFileDefaultTemplate RSYSLOG_SysklogdFileFormat $SpaceLFOnReceive on $DropTrailingLFOnReception off ---------- 3. Restart rsyslog # service rsyslog restart 4. Compile and run the following program ---------- #include <stdio.h> #include <syslog.h> int main() { syslog(LOG_INFO, "aaa%caaa", '\n'); return 0; } ---------- Actual results: Below message appears in /var/log/messages. ---------- 　　　Sep 7 19:19:39 localhost test2: aaa ---------- Expected results: Below message, which worked prior to systemd-journald appears in /var/log/messages. ---------- 　　　Sep 7 19:19:39 localhost test2: aaa aaa https://bugzilla.redhat.com/show_bug.cgi?id=855313

journal: immediately sync to disk as soon as we receieve an EMERG/ALERT/CRIT message

journald: after the cgroup rework processes may be in both user and system units at the same time

journal: Leave server_dispatch_message early when Storage is none When using Storage=none there is no point in collecting all the information just to throw them away. After this change journald consumes a lot less CPU time when only forwarding messages.

journal: add logging of effective capabilities _CAP_EFFECTIVE I think this is the most important of the capabilities bitmasks to log.

journald-server: r should be checked after journal_file_open_reliably

journald: fix space limits reporting Reporting of the free space was bogus, since the remaining space was compared with the maximum allowed, instead of the current use being compared with the maximum allowed. Simplify and fix by reporting limits directly at the point where they are calculated. Also, assign a UUID to the message.

journald: bump the journal per-unit ratelimit defaults Too many people kept hitting them, so let's increase the limits a bit. https://bugzilla.redhat.com/show_bug.cgi?id=965803

journal: allow callers to specify OBJECT_PID= When journald encounters a message with OBJECT_PID= set coming from a priviledged process (UID==0), additional fields will be added to the message: OBJECT_UID=, OBJECT_GID=, OBJECT_COMM=, OBJECT_EXE=, OBJECT_CMDLINE=, OBJECT_AUDIT_SESSION=, OBJECT_AUDIT_LOGINUID=, OBJECT_SYSTEMD_CGROUP=, OBJECT_SYSTEMD_SESSION=, OBJECT_SYSTEMD_OWNER_UID=, OBJECT_SYSTEMD_UNIT= or OBJECT_SYSTEMD_USER_UNIT=. This is for other logging daemons, like setroubleshoot, to be able to augment their logs with data about the process. https://bugzilla.redhat.com/show_bug.cgi?id=951627

journald: do not calculate free space too early Since the system journal wasn't open yet, available_space() returned 0. Before: systemd-journal[22170]: Allowing system journal files to grow to 4.0G. systemd-journal[22170]: Journal size currently limited to 0B due to SystemKeepFree. After: systemd-journal[22178]: Allowing system journal files to grow to 4.0G. systemd-journal[22178]: Journal size currently limited to 3.0G due to SystemKeepFree. Also, when failing to write a message, show how much space was needed: "Failed to write entry (26 items, 260123456 bytes) despite vacuuming, ignoring: ...".

journal: letting (interleaved) seqnums go In the following scenario: server creates system.journal server creates user-1000.journal both journals share the same seqnum_id. Then server writes to user-1000.journal first, and server writes to system.journal a bit later, and everything is fine. The server then terminates (crash, reboot, rsyslog testing, whatever), and user-1000.journal has entries which end with a lower seqnum than system.journal. Now server is restarted server opens user-1000.journal and writes entries to it... BAM! duplicate seqnums for the same seqnum_id. Now, we usually don't see that happen, because system.journal is closed last, and opened first. Since usually at least one message is written during boot and lands in the system.journal, the seqnum is initialized from it, and is set to a number higher than than anything found in user journals. Nevertheless, if system.journal is corrupted and is rotated, it can happen that an entry is written to the user journal with a seqnum that is a duplicate with an entry found in the corrupted system.journal~. When browsing the journal, journalctl can fall into a loop where it tries to follow the seqnums, and tries to go the next location by seqnum, and is transported back in time to to the older duplicate seqnum. There is not way to find out the maximum seqnum used in a multiple files, without actually looking at all of them. But we don't want to do that because it would be slow, and actually it isn't really possible, because a file might e.g. be temporarily unaccessible. Fix the problem by using different seqnum series for user journals. Using the same seqnum series for rotated journals is still fine, because we know that nothing will write to the rotated journal anymore. Likely related: https://bugs.freedesktop.org/show_bug.cgi?id=64566 https://bugs.freedesktop.org/show_bug.cgi?id=59856 https://bugs.freedesktop.org/show_bug.cgi?id=64296 https://bugs.archlinux.org/task/35581 https://bugzilla.novell.com/show_bug.cgi?id=817778 Possibly related: https://bugs.freedesktop.org/show_bug.cgi?id=64293

journalctl: print monotonic timestamp in --header

journal: take KeepFree into account when reporting maximum size When reporting the maximum journal size add a hint if it's limited by KeepFree.

journald: DO recalculate the ACL mask, but only if it doesn't exist Since 11ec7ce, journald isn't setting the ACLs properly anymore if the files had no ACLs to begin with: acl_set_fd fails with EINVAL. An ACL with ACL_USER or ACL_GROUP entries but no ACL_MASK entry is invalid, so make sure a mask exists before trying to set the ACL.

journal: correctly convert usec_t to timespec. Use timespec_store instead of (incorrectly) doing it inline.

journald: don't recalculate the ACL mask Otherwise we might end up with executable files if some default ACL is set for the journal directory.

journald: be more careful when we try to flush the runtime journal to disk and the disk is close to being full Bump the minimal size of the journal so that we can be sure creating the journal file will always succeed. Previously the minimum size was smaller than a empty jounral file...

conf-parser: restrict .include usage Disallow recursive .include, and make it unavailable in anything but unit files.

journal: remove build warning when SELinux is disabled A small patch to remove a build warnining when SELinux is disabled.

cgroup: make sure all our cgroup objects have a suffix and are properly escaped Session objects will now get the .session suffix, user objects the .user suffix, nspawn containers the .nspawn suffix. This also changes the user cgroups to be named after the numeric UID rather than the username, since this allows us the parse these paths standalone without requiring access to the cgroup file system. This also changes the mapping of instanced units to cgroups. Instead of mapping foo@bar.service to the cgroup path /user/foo@.service/bar we will now map it to /user/foo@.service/foo@bar.service, in order to ensure that all our objects are properly suffixed in the tree.

move _cleanup_ attribute in front of the type http://lists.freedesktop.org/archives/systemd-devel/2013-April/010510.html

Report about syntax errors with metadata The information about the unit for which files are being parsed is passed all the way down. This way messages land in the journal with proper UNIT=... or USER_UNIT=... attribution. 'systemctl status' and 'journalctl -u' not displaying those messages has been a source of confusion for users, since the journal entry for a misspelt setting was often logged quite a bit earlier than the failure to start a unit. Based-on-a-patch-by: Oleksii Shevchuk <alxchk@gmail.com>

nspawn: introduce the new /machine/ tree in the cgroup tree and move containers there Containers will now carry a label (normally derived from the root directory name, but configurable by the user), and the container's root cgroup is /machine/<label>. This label is called "machine name", and can cover both containers and VMs (as soon as libvirt also makes use of /machine/). libsystemd-login can be used to query the machine name from a process. This patch also includes numerous clean-ups for the cgroup code.

journal: fix broken tags _SOURCE_REALTIME_TIMESTAMP and _MACHINE_ID

journald: no need to free audit vars

journald: drop two more memory allocations

journald: get rid of one more memory allocation

journald: Do not dynamically allocate _UID/_GID/_PID strings Avoid the dynamic allocation for the _UID, _GID, and _PID strings. The maximum size of the string can be determined at compile time. The code has only been compile tested.

journald: Do not always record _AUDIT_SESSION and _AUDIT_LOGINUID When systemd was compiled without audit support, do not collect the audit session and loginuid in the journal. This is saving a couple of syscalls and memory allocations per log message.

Use initalization instead of explicit zeroing Before, we would initialize many fields twice: first by filling the structure with zeros, and then a second time with the real values. We can let the compiler do the job for us, avoiding one copy. A downside of this patch is that text gets slightly bigger. This is because all zero() calls are effectively inlined: $ size build/.libs/systemd text data bss dec hex filename before 897737 107300 2560 1007597 f5fed build/.libs/systemd after 897873 107300 2560 1007733 f6075 build/.libs/systemd … actually less than 1‰. A few asserts that the parameter is not null had to be removed. I don't think this changes much, because first, it is quite unlikely for the assert to fail, and second, an immediate SEGV is almost as good as an assert.

journal: Add sync timer to journal server Add option to force journal sync with fsync. Default timeout is 5min. Interval configured via SyncIntervalSec option at journal.conf. Synced journal files will be marked as OFFLINE. Manual sync can be performed via sending SIGUSR1.

journal,shared: add _cleanup_journal_close_

journal: pass the *pid* to sd_pid_get_owner_uid()

journald: introduce new "systemd-journal" group and make it own the journal files Previously all journal files were owned by "adm". In order to allow specific users to read the journal files without granting it access to the full "adm" powers, introduce a new specific group for this. "systemd-journal" has to be created by the packaging scripts manually at installation time. It's a good idea to assign a static UID/GID to this group, since /var/log/journal might be shared across machines via NFS. This commit also grants read access to the journal files by default to members of the "wheel" and "adm" groups via file system ACLs, since these "almost-root" groups should be able to see what's going on on the system. These ACLs are created by "make install". Packagers probably need to duplicate this logic in their postinst scripts. This also adds documentation how to grant access to the journal to additional users or groups via fs ACLs.

journald: stpcpy() + mempcpy() are awesome

journald: drop splitting-by-audit entirely Thinking about it we should probably not hide bugs by falling back to audit when we have our own session information anyway.

journald: be a bit more careful when spitting up journals by user id

journald: check session owner UID rather then audit ID when splitting up journal files We should always go by our own cgroup hierarchy before using foreign schemes such as audit, so let's do that for the split out logic too.

journal: split journal uid only when audit uid data is valid

journald: do not barf when setting RateLimitInterval=0 Assertion 'interval > 0 || burst == 0' failed at src/journal/journald-rate-limit.c:78, function journal_rate_limit_new(). Aborting.

honor SELinux labels, when creating and writing config files Also split out some fileio functions to fileio.c and provide a SELinux aware pendant in fileio-label.c see https://bugzilla.redhat.com/show_bug.cgi?id=881577

journal: Don't use loginuid if it's not valid Code above this attempted to load loginuid, if this failed for whatever reason, we'd still end up using that value (0) in place of realuid. Fix this by setting a bool when we know the loginuid is valid. This fixes journal messages showing up in per-user journals in gnome-ostree (not configured with loginuid, but I'll shortly fix that).

journal: log _SYSTEMD_USER_UNIT for user session units

systemctl,loginctl,cgls: do not ellipsize cgroup members when --full is specified New file output.h with output flags and modes. --full parameter also for cgls and loginctl. Include 'all' parameter in flags (show_cgroup_by_path, show_cgroup, show_cgroup_and_extra, show_cgroup_and_extra_by_spec). get_process_cmdline with max_length == 0 will not ellipsize output. Replace LINE_MAX with 0 in some calls of get_process_cmdline. [zj: Default to --full when under pager for clgs. Drop '-f' since it wasn't documented and didn't actually work. Reindent a bit. ]

journald: use automatic cleanup

journal: by default do not decompress dat objects larger than 64K This introduces a new data threshold setting for sd_journal objects which controls the maximum size of objects to decompress. This is relieves the library from having to decompress full data objects even if a client program is only interested in the initial part of them. This speeds up "systemd-coredumpctl" drastically when invoked without parameters.

journald: fix bad memory access https://bugzilla.redhat.com/show_bug.cgi?id=875653

build-sys: store journald code in a noinst library The point is to allow the use of journald functions by other binaries. Before, journald code was split into multiple files (journald-*.[ch]), but all those files all required functions from journald.c. And journald.c has its own main(). Now, it is possible to link against those functions, e.g. from test binaries. This constitutes a fix for https://bugzilla.redhat.com/show_bug.cgi?id=872638. The patch does the following: 1. rename journald.h to journald-server.h and move corresponding code to journald-server.c. 2. add journald-server.c and other journald-*.c parts to libsystemd-journal-internal. 3. remove journald-syslog.c from test_journal_syslog_SOURCES, since it is now contained in libsystemd-journal-internal. There are no code changes, apart from the removal of a few static's, to allow function calls between files.