tree-wide: remove Emacs lines from all files This should be handled fine now by .dir-locals.el, so need to carry that stuff in every file.

journal-remote: decrease microhttpd memory limit Set the MHD_OPTION_CONNECTION_MEMORY_LIMIT to 128KB. The precious value was DATA_SIZE_MAX, which was defined as 1024*1024*768. This caused journal-remote to allocate 756MB for each journal-upload connection, thus exhausting the available memory.

journal-remote: fix broken --getter option This commit fixes the following broken --getter option: when systemd-journal-remote is called with --getter option, it causes the error meesage "Zero sources specified" and the getter command will not be called.

journal-remote: output file name is determined by the remote hostname When --url option is specified, e.g. --url='http://some.host:19531/entries' retrieved remote journal entries will be stored to /var/log/journal/remote/remote-some.host.journal

journal-remote: make --url option support arbitrary url Currently, --url option supports the only form like http(s)://some.host:19531. This commit adds support to call systemd-journal-remote as follwos: systemd-journal-remote --url='http://some.host:19531' systemd-journal-remote --url='http://some.host:19531/' systemd-journal-remote --url='http://some.host:19531/entries' systemd-journal-remote --url='http://some.host:19531/entries?boot&follow' The first three example result the same and retrieve all entries. The last example retrieves only current boot entries and wait new events.

systemd-activate: Add support for datagram sockets core: Add flexible way to provide socket type the socket type should be a diffrent argumet in make_socket_fd . In this way we can set the socket type like SOCK_STREAM SOCK_DGRAM in the address. journal-remote: modify make_socket_fd

journal-remote: fix warning about deprecated µhttpd macro src/journal-remote/journal-remote.c:590:13: warning: Value MHD_HTTP_METHOD_NOT_ACCEPTABLE is deprecated, use MHD_HTTP_NOT_ACCEPTABLE return mhd_respond(connection, MHD_HTTP_METHOD_NOT_ACCEPTABLE, ^ The new define was added in 0.9.38. Instead of requiring the new libmicrohttpd version, provide the fallback, it is trivial.

Add Seal option in the configuration file for journald-remote While journal received remotely can be sealed, it can only be done on the command line using --seal, so for consistency, we will also permit to set it in the configuration file.

journal-remote: split-mode=host, remove port from journal filename When constructing the journal filename to store logs from a remote host, remove the port of the tcp connection, as the port will change with every reboot/connection loss between sender/reveiver machines. Having the port in the filename will cause a new journal file to be created for every reboot or connection loss. For the implementation, a new argument "bool include_port" is added to the getpeername_pretty() function. This is passed to the sockaddr_pretty() function. The value of the include_port argument is set to true in all calls of getpeername_pretty(), except for 2 calls in journal-remote.c, where it is set to false.

defs: rework CONF_DIRS_NULSTR() macro The macro is generically useful for putting together search paths, hence let's make it truly generic, by dropping the implicit ".d" appending it does, and leave that to the caller. Also rename it from CONF_DIRS_NULSTR() to CONF_PATHS_NULSTR(), since it's not strictly about dirs that way, but any kind of file system path. Also, mark CONF_DIR_SPLIT_USR() as internal macro by renaming it to _CONF_PATHS_SPLIT_USR() so that the leading underscore indicates that it's internal.

treewide: use the negative error codes returned by our functions Our functions return negative error codes. Do not rely on errno being set after calling our own functions.

util-lib: move CONF_DIRS_NULSTR definition to def.h After all, this is not some compiler or C magic, but something very specific to how systemd works, hence let's move it into def.h, and out of macro.h

journal-remote: remove unused variable warning when building without GNUTLS.

journal-remote: increase memory limit per connection to avoid errors with HTTPS Explicitly set MHD_OPTION_CONNECTION_MEMORY_LIMIT to a larger value, when setting up microhttpd, to give more memory per HTTP(S) connection. This way systemd-journal-remote can now prevent microhttpd from failing in creating response headers with messages like "Not enough memory for write", especially when lots of HTTPS requests arrive. That's precisely because MHD_OPTION_CONNECTION_MEMORY_LIMIT in libmicrohttpd defaults to 32768, which is in practice insufficient in this case. See also https://gnunet.org/bugs/view.php?id=4007 for more details. Fixes: https://github.com/coreos/bugs/issues/927

journal-remote: port to extract_first_word

util-lib: split out allocation calls into alloc-util.[ch]

util-lib: split out printf() helpers to stdio-util.h

util-lib: move string table stuff into its own string-table.[ch]

util-lib: split stat()/statfs()/stavfs() related calls into stat-util.[ch]

util-lib: split string parsing related calls from util.[ch] into parse-util.[ch]

util-lib: split out fd-related operations into fd-util.[ch] There are more than enough to deserve their own .c file, hence move them over.

util-lib: split our string related calls from util.[ch] into its own file string-util.[ch] There are more than enough calls doing string manipulations to deserve its own files, hence do something about it. This patch also sorts the #include blocks of all files that needed to be updated, according to the sorting suggestions from CODING_STYLE. Since pretty much every file needs our string manipulation functions this effectively means that most files have sorted #include blocks now. Also touches a few unrelated include files.

util: split out escaping code into escape.[ch] This really deserves its own file, given how much code this is now.

journal: make sure to set MHD_USE_PIPE_FOR_SHUTDOWN for libmicrohttpd servers This makes sure libmicrohttpd won't call shutdown() on our listening sockets, which make sure socket activation and re-activation will work cleanly. See: https://github.com/systemd/systemd/pull/1286 https://lists.gnu.org/archive/html/libmicrohttpd/2015-09/msg00014.html Fixes #1286

util: introduce common version() implementation and use it everywhere This also allows us to drop build.h from a ton of files, hence do so. Since we touched the #includes of those files, let's order them properly according to CODING_STYLE.

journal-remote: typo in log_error when no sources are specified

tree-wide: replace while(1) by for(;;) everywhere Another Coccinelle script.

tree-wide: update empty-if coccinelle script to cover empty-while and more Let's also clean up single-line while and for blocks.

tree-wide: make use of log_error_errno() return value Turns this: r = -errno; log_error_errno(errno, "foo"); into this: r = log_error_errno(errno, "foo"); and this: r = log_error_errno(errno, "foo"); return r; into this: return log_error_errno(errno, "foo");

strv: convert strv_split_quotes into a generic strv_split_extract strv_split_extract is to strv_split_quotes as extract_first_word was to unquote_first_word. Now there's extract_first_word for extracting a single argument, extract_many_words for extracting a bounded number of arguments, and strv_split_extract for extracting an arbitrary number of arguments.

socket-util: library calls shouldn't log on their own Instead, make sure that all callers log properly.

everywhere: port everything to sigprocmask_many() and friends This ports a lot of manual code over to sigprocmask_many() and friends. Also, we now consistly check for sigprocmask() failures with assert_se(), since the call cannot realistically fail unless there's a programming error. Also encloses a few sd_event_add_signal() calls with (void) when we ignore the return values for it knowingly.

tree-wide: whenever we fork off a foreign child process reset signal mask/handlers Also, when the child is potentially long-running make sure to set a death signal. Also, ignore the result of the reset operations explicitly by casting them to (void).

util: split out signal-util.[ch] from util.[ch] No functional changes.

fix gcc warnings about uninitialized variables like: src/shared/install.c: In function ‘unit_file_lookup_state’: src/shared/install.c:1861:16: warning: ‘r’ may be used uninitialized in this function [-Wmaybe-uninitialized] return r < 0 ? r : state; ^ src/shared/install.c:1796:13: note: ‘r’ was declared here int r; ^

util: rework word parsing and c unescaping code When parsing words from input files, optionally automatically unescape the passed strings, controllable via a new flags parameter. Make use of this in tmpfiles, and port everything else over, too. This improves parsing quite a bit, since we no longer have to process the same string multiple times with different calls, where an earlier call might corrupt the input for a later call.

µhttp-util: setup gnutls logs in one function

tree-wide: there is no ENOTSUP on linux Replace ENOTSUP by EOPNOTSUPP as this is what linux actually uses.

journal-remote: stop using EWOULDBLOCK There is no reason to ever use EWOULDBLOCK. It's equivalent to EAGAIN on all architectures on linux.

journal-remote: process events without delay journal-remote buffers input, and then parses it handling one journal entry at a time. It was possible for useful data to be left in the buffer after some entries were processesed. But all data would be already read from the fd, so there would be no reason for the event loop to call the handler again. After some new data came in, the handler would be called again, and would then process the "old" data in the buffer. Fix this by enabling a handler wherever we process input data and do not exhaust data from the input buffer (i.e. when EAGAIN was not encountered). The handler runs until we encounter EAGAIN. Looping over the input data is done in this roundabout way to allow the event loop to dispatch other events in the meanwhile. If the loop was inside the handler, a source which produced data fast enough could completely monopolize the process. https://bugs.freedesktop.org/show_bug.cgi?id=89516

journal-remote: check also for EWOULDBLOCK This matches similar code elsewhere.

journal-remote: downgrade routine messages to debug https://bugs.freedesktop.org/show_bug.cgi?id=89486

remove unused includes This patch removes includes that are not used. The removals were found with include-what-you-use which checks if any of the symbols from a header is in use.

util: rework strappenda(), and rename it strjoina() After all it is now much more like strjoin() than strappend(). At the same time, add support for NULL sentinels, even if they are normally not necessary.

Add a snprinf wrapper which checks that the buffer was big enough If we scale our buffer to be wide enough for the format string, we should expect that the calculation was correct. char_array_0() invocations are removed, since snprintf nul-terminates the output in any case. A similar wrapper is used for strftime calls, but only in timedatectl.c.

Assorted format fixes Types used for pids and uids in various interfaces are unpredictable. Too bad.

wrap a few *_FOREACH macros in curly braces cppcheck would give up with "syntax error" without them. This led to reports of syntax errors in unrelated locations and potentially hid other errors

journald-remote,journal-upload: Support .d directories in the usual search paths

treewide: another round of simplifications Using the same scripts as in f647962d64e "treewide: yet more log_*_errno + return simplifications".

treewide: use log_*_errno whenever %m is in the format string If the format string contains %m, clearly errno must have a meaningful value, so we might as well use log_*_errno to have ERRNO= logged. Using: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\((".*%m.*")/log_\1_errno(errno, \2/' Plus some whitespace, linewrap, and indent adjustments.

treewide: yet more log_*_errno + return simplifications Using: find . -name '*.[ch]' | while read f; do perl -i.mmm -e \ 'local $/; local $_=<>; s/(if\s*\([^\n]+\))\s*{\n(\s*)(log_[a-z_]*_errno\(\s*([->a-zA-Z_]+)\s*,[^;]+);\s*return\s+\g4;\s+}/\1\n\2return \3;/msg; print;' $f done And a couple of manual whitespace fixups.

treewide: more log_*_errno + return simplifications

treewide: simplify log_*_errno(r,...) immediately followed by "return r"

treewide: more log_*_errno() conversions, multiline calls Basically: find . -name '*.[ch]' | while read f; do perl -i.mmm -e \ 'local $/; local $_=<>; s/log_(debug|info|notice|warning|error|emergency)\("([^"]*)%s"([^;]*),\s*strerror\(-?([->a-zA-Z_]+)\)\);/log_\1_errno(\4, "\2%m"\3);/gms;print;' \ $f; done Plus manual indentation fixups.

treewide: no need to negate errno for log_*_errno() It corrrectly handles both positive and negative errno values.

treewide: auto-convert the simple cases to log_*_errno() As a followup to 086891e5c1 "log: add an "error" parameter to all low-level logging calls and intrdouce log_error_errno() as log calls that take error numbers", use sed to convert the simple cases to use the new macros: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\("(.*)%s"(.*), strerror\(-([a-zA-Z_]+)\)\);/log_\1_errno(-\4, "\2%m"\3);/' Multi-line log_*() invocations are not covered. And we also should add log_unit_*_errno().

strv: rework strv_split_quoted() to use unquote_first_word() This should make the unquoting scheme a bit less naive.

sd-event: implicitly set signal event source's descriptions to the signal name

sd-event: rename sd_event_source_set_name() to sd_event_source_get_name() To mirror the recent name change of the concept for sd_bus objects, follow the same logic for sd_event_source objects, too.

shared/log: add log_trace as compile-time optional debugging Repetetive messages can be annoying when running with SYSTEMD_LOG_LEVEL=debug, but they are sometimes very useful when debugging problems. Add log_trace which is like log_debug but becomes a noop unless LOG_TRACE is defined during compilation. This makes it easy to enable very verbose logging for a subset of programs when compiling from source.

journal-remote: add --split-mode to help

journal-remote: better error message on failure Return a proper code instead of simply NULL for failure.

systemd-upload: print paths in help()

journal-remote: give names to event sources This possibility was recently added, and it makes debugging much nicer.

journal-upload: fix socket activation

journal-remote: fix mem leak on error

journal-remote: fix handling of non-blocking sources In the conversion to sd-event loop, handling of normal files got broken. We do not want to perform non-blocking reads on them, but simply do read() in a loop. Install a statically-enabled "source" to do that.

journal-remote: initialize writer hashmap before use https://bugs.freedesktop.org/show_bug.cgi?id=83682 [zj: move the initalization even earlier, before any sockets are looked at.]

journal-remote: check return code of sd_event_default Handle sd_event_default returning error and bail out properly as done in every other caller of this function. Found by coverity. Fixes: CID#1238957

hashmap: introduce hash_ops to make struct Hashmap smaller It is redundant to store 'hash' and 'compare' function pointers in struct Hashmap separately. The functions always comprise a pair. Store a single pointer to struct hash_ops instead. systemd keeps hundreds of hashmaps, so this saves a little bit of memory.

notify: send STOPPING=1 from our daemons

journal-remote: remove unreachable code

cmdline: for new tools avoid introduce new negative switches, and properly align --help texts Negative switches are a bad un-normalized thing. We alerady have some, but we should try harder to avoid intrdoucing new ones. Hence, instead of adding two switches: --foobar --no-foobar Let's instead use the syntax --foobar --foobar=yes --foobar=no Where the first two are equivalent. The boolean argument is parsed following the usual rules. Change all new negative switches this way. This patch also properly aligns the --help table, so that single char switches always get a column separate of the long switches.

journal-remote: fix parsing of fd command line argument

Unify parse_argv style getopt is usually good at printing out a nice error message when commandline options are invalid. It distinguishes between an unknown option and a known option with a missing arg. It is better to let it do its job and not use opterr=0 unless we actually want to suppress messages. So remove opterr=0 in the few places where it wasn't really useful. When an error in options is encountered, we should not print a lengthy help() and overwhelm the user, when we know precisely what is wrong with the commandline. In addition, since help() prints to stdout, it should not be used except when requested with -h or --help. Also, simplify things here and there.

journal-remote: rename KEY_FILE to avoid confict with <linux/input.h>

Properly report invalid quoted strings $ systemd-analyze verify trailing-g.service [./trailing-g.service:2] Trailing garbage, ignoring. trailing-g.service lacks ExecStart setting. Refusing. Error: org.freedesktop.systemd1.LoadFailed: Unit trailing-g.service failed to load: Invalid argument. Failed to create trailing-g.service/start: Invalid argument

Reject invalid quoted strings String which ended in an unfinished quote were accepted, potentially with bad memory accesses. Reject anything which ends in a unfished quote, or contains non-whitespace characters right after the closing quote. _FOREACH_WORD now returns the invalid character in *state. But this return value is not checked anywhere yet. Also, make 'word' and 'state' variables const pointers, and rename 'w' to 'word' in various places. Things are easier to read if the same name is used consistently. mbiebl_> am I correct that something like this doesn't work mbiebl_> ExecStart=/usr/bin/encfs --extpass='/bin/systemd-ask-passwd "Unlock EncFS"' mbiebl_> systemd seems to strip of the quotes mbiebl_> systemctl status shows mbiebl_> ExecStart=/usr/bin/encfs --extpass='/bin/systemd-ask-password Unlock EncFS $RootDir $MountPoint mbiebl_> which is pretty weird

Let config_parse open file where applicable Special care is needed so that we get an error message if the file failed to parse, but not when it is missing. To avoid duplicating the same error check in every caller, add an additional 'warn' boolean to tell config_parse whether a message should be issued. This makes things both shorter and more robust wrt. to error reporting.

journal-remote: remove obsolete variable

journal-remote: rework fd and writer reference handling

journal-remote: improve some messages

Fix problem with allocating large buffers and log leftovers

journal-remote: allow splitting incoming logs by source host Previously existing scheme where the file name would be based on the source was just too ugly and unpredicatable. Now there are only two options: 1. just one file (until rotation), 2. one file per source host, using the hostname as filename part. For the cases where the source is specified by the user, only option one is allowed, and the full of the file must be specified.

Allow addresses to be specified for --listen-... args Hostnames still aren't accepted.

journal-remote: add units and read certs from default locations

journal-upload: add watchdog support

journal-upload: use journal as the source

journal-upload: HTTPS support

journal-remote: reject fields above maximum size Also fix an infinite loop on E2BIG. Remember what range we already scanned for '\n', to avoid quadratic behaviour on long "text" fields.

journal-remote: small fixes

Move network-related journal programs to src/journal-remote/ Directory src/journal has become one of the largest directories, and since systemd-journal-gatewayd, systemd-journal-remote, and forthcoming systemd-journal-upload are all closely related, create a separate directory for them.