Fix minor spelling mistakes in responder/* Reviewed-by: Lukáš Slebodník <lslebodn@redhat.com>

mmap_cache: make checks independent of input size Currently the consistency checks for the mmap_cache payload data on the client and the responder side include the length of the input string of the current request. Since there might be hash collisions which other much longer or much shorter names those checks might fail although there is no data corruption. This patch removes the checks using the length of the input and adds a check if the name found in the payload is zero-terminated inside of the payload data. Resolves https://pagure.io/SSSD/sssd/issue/3571 Reviewed-by: Michal Židek <mzidek@redhat.com> Reviewed-by: Lukáš Slebodník <lslebodn@redhat.com>

nss: rewrite nss responder so it uses cache_req Given the size of the current nss responder it was quite impossible to simply switch into using the cache_req interface, especially because most of the code was duplication of cache lookups. This patch completely rewrites the responder from scratch. The amount of code was reduced to less than a half lines of code with no code duplication, better documentation and better maintainability and readability. All functionality should be intact. *Code organization* All protocol (parsing input message and send a reply) is placed in nss_protocol.c. Functions that deals with creating a reply packet are placed into their specific nss_protocol_$object.c files. All supported commands are placed into nss_cmd.c. Functions that deals with cache req are in nss_get_object.c and nss_enum.c. *Code flow for non-enumeration* An nss_getby_$input-type is called for each non-enumeration command. This function parses the input message, creates a cache_req_data structure and issues nss_get_object that calls cache_req. When this request is done nss_getby_done make sure a reply is sent to the client. *Comments on enumeration* I made some effort to make sure enumeration shares the same code for users, groups, services and netgroups. Netgroups now uses nss negative cache instead of implementing its own. Resolves: https://fedorahosted.org/sssd/ticket/3151 Reviewed-by: Lukáš Slebodník <lslebodn@redhat.com>

Fix typos reported by lintian Reviewed-by: Pavel Březina <pbrezina@redhat.com>

NSS: Initgr memory cache should work with fq names We need to stored two versions of name to the initgroups memory cache. Otherwise it could be stored many times if sssd is configured with case_sensitive = false. It would be impossible to invalidate all version of names after user login. As a result of this wrong user groups could be returned from initgroups memory cache. Therefore we store raw name provided by glibc function and internal sanitized fully qualified name, which is unique for particular user. This patch also increase average space for initgroups because there are also stored two quite long names in case of fq names. Resolves: https://fedorahosted.org/sssd/ticket/2712 Reviewed-by: Michal Židek <mzidek@redhat.com>

mmap_cache: "Override" functions for initgr mmap cache Functions sss_mc_get_strs_offset and sss_mc_get_strs_len provides data about strings for individual memory caches (passwd, ...) Their are used in generic responder mmap cache code to find a record in mmap cache (sss_mc_find_record). Data provided from functions sss_mc_get_* are used for checking the validity of record. So in case of corrupted record the whole mmap cache can be invalidated. Functions sss_mc_get_strs_offset and sss_mc_get_strs_len did not provide data for initgroups mmap cache and therefore particular record could not be invalidated. Resolves: https://fedorahosted.org/sssd/ticket/2716 Reviewed-by: Michal Židek <mzidek@redhat.com>

mmap_cache: Rename variables Reviewed-by: Michal Židek <mzidek@redhat.com>

nss: Store entries in responder to initgr mmap cache Resolves: https://fedorahosted.org/sssd/ticket/2485 Reviewed-by: Michal Židek <mzidek@redhat.com>

sss_atomic_write_s() return value is signed Reviewed-by: Sumit Bose <sbose@redhat.com>

mmap: Get errno when unlink fails Reviewed-by: Lukáš Slebodník <lslebodn@redhat.com>

Make DEBUG macro invocations variadic Use a script to update DEBUG macro invocations to use it as a variadic macro, supplying format string and its arguments directly, instead of wrapping them in parens. This script was used to update the code: grep -rwl --include '*.[hc]' DEBUG . | while read f; do mv "$f"{,.orig} perl -e \ 'use strict; use File::Slurp; my $text=read_file(\*STDIN); $text=~s#(\bDEBUG\s*\([^(]+)\((.*?)\)\s*\)\s*;#$1$2);#gs; print $text;' < "$f.orig" > "$f" rm "$f.orig" done Reviewed-by: Jakub Hrozek <jhrozek@redhat.com> Reviewed-by: Stephen Gallagher <sgallagh@redhat.com> Reviewed-by: Simo Sorce <simo@redhat.com>

responder: Use SAFEALIGN macros where appropriate. https://fedorahosted.org/sssd/ticket/1359

mmap_cache: Use two chains for hash collision. struct sss_mc_rec had two hash members (hash1 and hash2) but only one next member. This was a big problem in case of higher probability of hash collision. structure sss_mc_rec will have two next members (next1, next2) with this patch. next1 is related to hash1 and next2 is related to hash1. Iterating over chains is changed, because we need to choose right next pointer. Right next pointer will be chosen after comparing record hashes. This behaviour is wrapped in function sss_mc_next_slot_with_hash. Adding new record to chain is also changed. The situation is very similar to iterating. We need to choose right next pointer (next1 or next2). Right next pointer will be chosen after comparing record hashes. Adding reference to next slot is wrapped in function sss_mc_chain_slot_to_record_with_hash Size of structure sss_mc_rec was increased from 32 bytes to 40 bytes. Resolves: https://fedorahosted.org/sssd/ticket/2049

Revert "mmap_cache: Skip records which doesn't have same hash" This reverts commit 4662725ffef62b3b2502481438effa7c8fef9f80.

Fix formating of variables with type: size_t

mmap_cache: Do not remove record from chain twice It is not very likely, that record will have the same hash1 and hash2, but it is possible. In this situation, it does not make sense to remove record twice. Function sss_mc_rm_rec_from_chain was not robust and sssd_nss could crash in this situation. It was only possible if record was alone in chain. Resolves: https://fedorahosted.org/sssd/ticket/2049

mmap_cache: Use stricter check for hash keys. ht_size is size of hash_table in bytes, but hash keys have type uint32_t

mmap_cache: Skip records which doesn't have same hash The code uses 2 hashes for each record, but only one hash table to index them both, furthermore each record has only one single 'next' pointer. This means that in certain conditions a record main end up being on a hash chain even though its hashes do not match the hash chain. This can happen when another record 'drags' it in from another hash chain where they both belong. If the record without matching hashes happens to be the second of the chain and the first record is removed, then the non matching record is left on the wrong chain. On removal of the non-matching record the hash chain will not be updated and the hash chain will end up pointing to an invalid slot. This slot may be later reused for another record and may not be the first slot of this new record. In this case the hash chain will point to arbitrary data and may cause issues if the slot is interpreted as the head of a record. By skipping any block that has no matching hashes upon removing the first record in a chain we insure that dangling references cannot be left in the hash table Resolves: https://fedorahosted.org/sssd/ticket/2049

mmap_cache: Use sss_atomic_write_s instead of write. Use sss_atomic_write_s() instead of write() in sss_mc_save_corrupted(). Also unlink() the file if no data were written. It is better to use sss_atomic_write_s instead of write

mmap_cache: Store corrupted mmap cache before reset This patch adds function to store corrupted mmap cache file to disk for further analysis.

mmap_cache: Use better checks for corrupted mc in responder We introduced new way to check integrity of memcache in the client code. We should use similiar checks in the responder.

mmap_cache: Off by one error. Removes off by one error when using macro MC_SIZE_TO_SLOTS and adds new macro MC_SLOT_WITHIN_BOUNDS.

mmap_cache: Check if slot and name_ptr are not invalid. This patch prevents jumping outside of allocated memory in case of corrupted slot or name_ptr values. It is not proper solution, just hotfix until we find out what is the root cause of ticket https://fedorahosted.org/sssd/ticket/2018

Inform about function duplication. sss_mc_set_recycled is a static function, that should not be used outside nsssrv_mmap_cache.c. The sss_cache tool is an exception, because in the case when sssd is not running, sss_cache must invalidate the memory cache file. That is why sss_mc_set_recycled was copied to the tools_mc_util.c (as helper function for sss_memcache_invalidate function). It was duplicated to allow this function to remain static (and invisible to any .h files), so that it is not used anywhere else. Wrong usage of this function might cause race conditions and corrupt the cache. I'll add comments about the duplication to the code.

Debug message in sss_mc_create_file. This patch adds debug message for the case if sssd fails to open old mc file for some other reason than the file does not exist.

File descriptor leak in nss responder. File descriptors leaked every time sss_mmap_cache_reinit was called and also the old memory cache was still maped in memory (munmap was not called). This patch adds destructor for memory cache context to call close() and munmap() automaticly. https://fedorahosted.org/sssd/ticket/1826

memcache: add macro that validates record length

mmap cache: invalidate cache on fatal error If a fatal EFAULT error is returned by the internal function that frees used memory invalidate the whole cache and reinit it. This way we avoid further corruption and insure clients see consistent data. Also insure we use the right context in init() and we use talloc_zfree() in reinit so that if the init() later fails we do not leave around a pointer to free memory in the callers.

Carefully check records when forcibly invalidating We should never try to invalidate an already invalid record as internal pointers will not be consistent. Carefully test that the record really is valid when we are fishing for free space, and properly invalidate records or return a fatal error if something goes wrong. In order to make the code more robust always invalidate the whole data space on initialization by setting all bits to 1, and make sure to invalidate the whole last allocated slot by converting rec->len to the number of slots instead of just the space used.

Update free table when records are invalidated. We were holding up slots when entries were invalidated directly an not through our primitive garbage collection scheme.

nss_mc: Add extra checks when dereferencing records Although it should enver happen that we pass in an invalid hash it is always better to just not do anything than access memory ouf of the hash table. It can lead to segfaults, or worse referencing memory that should not be touched.

Allow mmap calls to gracefully return absent ctx This is to allow to freely call mc functions even if initialization failed. They will now gracefully fail instead of segfaulting.

NSS: Fix the error handler in sss_mc_create_file https://fedorahosted.org/sssd/ticket/1704 The function is short enough so that we can simply stick with return and release resources before returning as appropriate.

mmap cache: public functions to invalidate records These functions can be called from the nss responder to invalidate records that have ceased to exist or that need to be refreshed the first time an application needs them.

Add helpers to set common mc record fields Les copy/paste and chance of errors when setting basic record fields that are shared among all object types.

sss_cache: Remove fastcache even if sssd is not running. https://fedorahosted.org/sssd/ticket/1584

sss_cache tool invalidates records in memory cache.

Allow fast memcache timeout to be configurable https://fedorahosted.org/sssd/ticket/1318

Convert read and write operations to sss_atomic_read https://fedorahosted.org/sssd/ticket/1209

NSS: Fix debug message

nsssrv: add handling of memory cache group map

nsssrv: add handling of memory cache passwd map

nsssrv: Add memory cache record handling utils

nsssrv: shared memory cache server initialization