cache: Check for max_id/min_id in cache_req The cache_req code doesn't check the min_id/max_id boundaries for requests by ID. Extending the .lookup_fn function in each plugin that searches by ID for a check that returns non-zero if the entry is out of the range and 0 if not. Resolves: https://pagure.io/SSSD/sssd/issue/3569 Reviewed-by: Jakub Hrozek <jhrozek@redhat.com> Reviewed-by: Fabiano Fidêncio <fidencio@redhat.com>

CACHE_REQ: Add a private request cache_req_locate_domain() Adds a new request cache_req_locate_domain_send/recv. This request, if the plugin that is being processed supports the locator, will call the plugin's dp_get_domain_send_fn(). On any error, the request returns just the error code. On success, the request returns the domain the object was found at. If the getAccountDomain() method returns that the back end does not support the locator method, all further getAccountDomain() calls are disabled for that domain. Related: https://pagure.io/SSSD/sssd/issue/3468 Reviewed-by: Pavel Březina <pbrezina@redhat.com> Reviewed-by: Sumit Bose <sbose@redhat.com>

CACHE_REQ: Export cache_req_search_ncache_add() as cache_req private interface Previously, it was enough to add an entry to the negative cache of the domain being processed in cache_req (cr->domain). But the locator plugin can return any domain from the processed domain's subdomain list as well. Therefore, this patch extends the internal API for the possibility of setting the negative cache in another domain as well. Reviewed-by: Pavel Březina <pbrezina@redhat.com> Reviewed-by: Sumit Bose <sbose@redhat.com>

CACHE_REQ_SEARCH: Check for filtered users/groups also on cache_req_send() cache_req_send() may take some shortcuts in case the object is found in the cache and it's still valid. This behaviour may lead to exposing filtered users and groups when they're searched by their uid/gid. A solution for this issue was proposed on 4ef0b19a but, unfortunately, didn't take into consideration that this shortcut could be taken. There are basically two really easy ways to test this issue: 1) Using enumeration: - Set "enumerate = True" in the domain section - restart SSSD cleaning up the cache; - getent passwd <uid of a user who is part of the filter_users> - Wait a little bit till the entry_negative_timeout is expired - getent passwd <same uid used above> 2) Not using enumeration: - getent passwd <uid of a user who is part of the filter_users> - Wait a little bit till the entry_negative_timeout is expired - getent passwd <same uid used above> A test covering this code path will be added in the follow-up commit. Resolves: https://pagure.io/SSSD/sssd/issue/3362 Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com> Reviewed-by: Pavel Březina <pbrezina@redhat.com>

CACHE_REQ: Simplify _search_ncache_filter() Let's make the result and input/output argument for _search_ncache_filter() and free it inside the function whenever it's needed instead of leaving this responsibility for the caller. Related: https://pagure.io/SSSD/sssd/issue/3362 Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com> Reviewed-by: Pavel Březina <pbrezina@redhat.com>

CACHE_REQ: Avoid using of uninitialized value Commit 4ef0b19a introduced the following warning, as "req" may be used without being initialized: src/responder/common/cache_req/cache_req_search.c: In function 'cache_req_search_done': src/responder/common/cache_req/cache_req_search.c:467:9: error: 'req' may be used uninitialized in this function [-Werror=maybe-uninitialized] tevent_req_error(req, ret); ^ src/responder/common/cache_req/cache_req_search.c:424:24: note: 'req' was declared here struct tevent_req *req; ^ cc1: all warnings being treated as errors In order to fix the issue above, let's just allocate tmp_ctx after "req" is already set. Related: https://pagure.io/SSSD/sssd/issue/3362 Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com> Co-Author: Lukáš Slebodník <lslebodn@redhat.com> Reviewed-by: Sumit Bose <sbose@redhat.com>

CACHE_REQ: Make use of cache_req_ncache_filter_fn() This patch makes use of cache_req_ncache_filter_fn() in order to process the result of a cache_req search and then filter out all the results that are present in the negative cache. The "post cache_req search" result processing is done basically in two different cases: - plugins which don't use name as an input token (group_by_id, user_by_id and object_by_id), but still can be affected by filter_{users,groups} options; - plugins responsible for groups and users enumeration (enum_groups and enum_users); Resolves: https://pagure.io/SSSD/sssd/issue/3362 Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com> Reviewed-by: Pavel Březina <pbrezina@redhat.com>

CACHE_REQ: shortcut if object is found If we get a cache-hit but the object is expired or needs a midpoint refresh we assume that this domain is the one the result should come from and go to data provider directly. Related: https://pagure.io/SSSD/sssd/issue/3001 Reviewed-by: Fabiano Fidêncio <fidencio@redhat.com>

CACHE_REQ: Check the caches first This patch introduces a new configurable option to define whether the responder should query all domains' caches before querying the Data Providers. This new option is called cache_first and, by default, it's disabled, meaning that, for each provider, the responder may contact the cache and the data provider in the same iteration. Co-Author: Pavel Březina <pbrezina@redhat.com> Related: https://pagure.io/SSSD/sssd/issue/3001 Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com> Reviewed-by: Pavel Březina <pbrezina@redhat.com>

cache_req: use rctx as memory context during midpoint refresh Otherwise the tevent request is freed when we return data from cache_req to caller. It is no big deal since the request is still finished on provider side but the reply wouldn't be processed. Reviewed-by: Fabiano Fidêncio <fidencio@redhat.com>

CACHE_REQ: Add cache_req_data_set_bypass_cache() This new cache_req_data method has been added because of the upcoming changes in the PAM responder. For deciding whether to contact the cache, or just query the data provider directly, PAM responder calls pam_initgr_check_timeout() which will return whether the cache entry may still be valid. The cache will be contacted only in case the cache entry is still valid, otherwise the data provider will be called. pam_initgr_check_timeout() basically checks whether the user (being looked up) is still a part of an in-memory hash table. Because the entry is a part of the hash table for really short period of time, and is automatically removed, the communication with the data provider is forced to happen quite often. As the follow-up changes should not modify this behaviour, this function was introduced so we can still call pam_initgr_check_timeout() and pass its result to the cache_req call that will perform the lookup. Related: https://fedorahosted.org/sssd/ticket/1126 Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com> Reviewed-by: Sumit Bose <sbose@redhat.com> Reviewed-by: Jakub Hrozek <jhrozek@redhat.com>

cache_req: move dp request to plugin This will allow to use cache req even for object that do not use account request such as hosts. Reviewed-by: Jakub Hrozek <jhrozek@redhat.com>

cache_req: do not set ncache if dp request fails We will only remember entry in the negative cache if the data provider requests succeeded because only then we can be sure that the entry does not exist. Resolves: https://fedorahosted.org/sssd/ticket/3151 Reviewed-by: Lukáš Slebodník <lslebodn@redhat.com>

cache_req: improve debugging The original debugging was unnecessarily large. I removed request name from all debug messages (keeping the request number). The name is now print only when a new request is created. I also added an explicit message to see what object are we currently looking up. For example: [cache_req_set_plugin] (0x2000): CR #4: Setting "Group by ID" plugin [cache_req_send] (0x0400): CR #4: New request 'Group by ID' [cache_req_select_domains] (0x0400): CR #4: Performing a multi-domain search [cache_req_set_domain] (0x0400): CR #4: Using domain [LDAP.PB] [cache_req_search_send] (0x0400): CR #4: Looking up GID:20001@LDAP.PB [cache_req_search_ncache] (0x0400): CR #4: Checking negative cache for [GID:20001@LDAP.PB] [cache_req_search_ncache] (0x0400): CR #4: [GID:20001@LDAP.PB] is not present in negative cache [cache_req_search_cache] (0x0400): CR #4: Looking up [GID:20001@LDAP.PB] in cache [cache_req_search_send] (0x0400): CR #4: Returning [GID:20001@LDAP.PB] from cache [cache_req_create_and_add_result] (0x0400): CR #4: Found 1 entries in domain LDAP.PB [cache_req_done] (0x0400): CR #4: Finished: Success Resolves: https://fedorahosted.org/sssd/ticket/3151 Reviewed-by: Lukáš Slebodník <lslebodn@redhat.com>

cache_req: skip first search on bypass cache If we always want to contact the data provider to fetch data, we don't need to search the cache prior to this call. Resolves: https://fedorahosted.org/sssd/ticket/3151 Reviewed-by: Lukáš Slebodník <lslebodn@redhat.com>

cache_req: move from switch to plugins; add logic cache_req grown quite big from the original code and it turned out that using switch statements to branch code for different cases makes the code quite hard to read and further extend and any modification to the logic itself is difficult. This patch changes the switch statements to plugins with small functions and separates logic into multiple modules. This gives us better control over the code and improves readability and maintainability while keeping code duplication to minimum. Reviewed-by: Jakub Hrozek <jhrozek@redhat.com> Reviewed-by: Petr Cech <pcech@redhat.com>