2N/A * The contents of this file are subject to the terms of the 2N/A * Common Development and Distribution License (the "License"). 2N/A * You may not use this file except in compliance with the License. 2N/A * See the License for the specific language governing permissions 2N/A * and limitations under the License. 2N/A * When distributing Covered Code, include this CDDL HEADER in each 2N/A * If applicable, add the following below this CDDL HEADER, with the 2N/A * fields enclosed by brackets "[]" replaced with your own identifying 2N/A * information: Portions Copyright [yyyy] [name of copyright owner] 2N/A * Copyright 2008 Sun Microsystems, Inc. All rights reserved. 2N/A * Use is subject to license terms. 2N/A * This file may contain confidential information of 2N/A * Mellanox Technologies, Ltd. and should not be distributed in source 2N/A * form without approval from Sun Legal. 2N/A * Function signatures 2N/A/* exported to other HCAs */ 2N/A * Note: The 64 bit doorbells need to written atomically. 2N/A * In 32 bit libraries we need to use the special assembly rtn 2N/A * because compiler generated code splits into 2 word writes 2N/A/* use a macro to ensure inlining on S10 amd64 compiler */ 2N/A * dapli_tavor_cq_doorbell() 2N/A * Takes the specified cq cmd and cq number and rings the cq doorbell 2N/A /* Build the doorbell from the parameters */ 2N/A /* Write the doorbell to UAR */ 2N/A /* 32 bit version */ 2N/A * For 32 bit intel we assign the doorbell in the order 2N/A * prescribed by the Tavor PRM, lower to upper addresses 2N/A * dapli_tavor_qp_send_doorbell() 2N/A * Takes the specified next descriptor information, qp number, opcode and 2N/A * rings the send doorbell 2N/A /* Build the doorbell from the parameters */ 2N/A /* Write the doorbell to UAR */ 2N/A * For 32 bit intel we assign the doorbell in the order 2N/A * prescribed by the Tavor PRM, lower to upper addresses 2N/A * dapli_tavor_qp_recv_doorbell() 2N/A * Takes the specified next descriptor information, qp number and 2N/A * rings the recv doorbell 2N/A /* Build the doorbell from the parameters */ 2N/A /* Write the doorbell to UAR */ 2N/A * For 32 bit intel we assign the doorbell in the order 2N/A * prescribed by the Tavor PRM, lower to upper addresses 2N/A * dapls_tavor_max_inline() 2N/A * Return the max inline value that should be used. 2N/A * Env variable DAPL_MAX_INLINE can override the default. 2N/A * If it's not set (or set to -1), default behavior is used. 2N/A * If it's zero or negative (except -1) inline is not done. 2N/A /* Check the env exactly once, otherwise return previous value. */ 2N/A * dapls_ib_max_request_iov(), aka, max send sgl size. 2N/A * By default, compute reasonable send queue size based on #iovs, #wqes, 2N/A * max_iovs, and max inline byte count. If the #wqes is large, then we 2N/A * limit how much the SGL (space for inline data) can take. The heuristic 2N/A * is to increase the memory for the send queue to a maximum of 32KB: 2N/A * < 128 wqes increase to at most 256 minus header 2N/A * < 256 wqes increase to at most 128 minus header 2N/A * >= 256 wqes use SGL unaltered 2N/A * If the env is supplied (max_inline >= 0), use it without checking. 2N/A * dapli_tavor_wqe_send_build() 2N/A * Constructs a WQE for a given ibt_send_wr_t 2N/A * RC is the only supported transport in UDAPL 2N/A * For RC requests, we allow "Send", "RDMA Read", "RDMA Write" 2N/A * If this is a Send request, then all we need is 2N/A * the Data Segment processing below. 2N/A * Initialize the information for the Data Segments 2N/A * If this is an RDMA Read or RDMA Write request, then fill 2N/A * in the "Remote Address" header fields. 2N/A * Build the Remote Address Segment for the WQE, using 2N/A * the information from the RC work request. 2N/A /* Update "ds" for filling in Data Segments (below) */ 2N/A * Generate a new R_key 2N/A * Increment the upper "unconstrained" bits and need to keep 2N/A * the lower "constrained" bits the same it represents 2N/A * Build the Bind Memory Window Segments for the WQE, 2N/A * using the information from the RC Bind memory 2N/A * window work request. 2N/A * Update the "ds" pointer. Even though the "bind" 2N/A * operation requires no SGLs, this is necessary to 2N/A * facilitate the correct descriptor size calculations 2N/A "dapli_tavor_wqe_send_build: invalid wr_opcode=%d\n",
2N/A * Now fill in the Data Segments (SGL) for the Send WQE based on 2N/A * the values setup above (i.e. "sgl", "nds", and the "ds" pointer 2N/A * Start by checking for a valid number of SGL entries 2N/A * For each SGL in the Send Work Request, fill in the Send WQE's data 2N/A * segments. Note: We skip any SGL with zero size because Tavor 2N/A * hardware cannot handle a zero for "byte_cnt" in the WQE. Actually 2N/A * the encoding for zero means a 2GB transfer. Because of this special 2N/A * encoding in the hardware, we mask the requested length with 2N/A * TAVOR_WQE_SGL_BYTE_CNT_MASK (so that 2GB will end up encoded as 2N/A /* Return the size of descriptor (in 16-byte chunks) */ 2N/A * Fill in the Data Segment(s) for the current WQE, 2N/A * using the information contained in the 2N/A * scatter-gather list of the work request. 2N/A /* Return the size of descriptor (in 16-byte chunks) */ 2N/A * dapli_tavor_wqe_send_linknext() 2N/A * Takes a WQE and links it to the prev WQE chain 2N/A /* Set the "c" (i.e. "signaled") bit appropriately */ 2N/A /* Set the "s" (i.e. "solicited") bit appropriately */ 2N/A /* Set the "e" (i.e. "event") bit if notification is needed */ 2N/A * The "i" bit is unused since uDAPL doesn't support 2N/A * the immediate data 2N/A /* initialize the ctrl and next fields of the current descriptor */ 2N/A * Calculate the "next" field of the prev descriptor. This amounts 2N/A * to setting up the "next_wqe_addr", "nopcode", "fence", and "nds" 2N/A * Determine the value for the Tavor WQE "nopcode" field 2N/A * by using the IBTF opcode from the work request 2N/A /* Unsupported opcodes in UDAPL */ 2N/A "dapli_tavor_wqe_send_linknext: invalid nopcode=%d\n",
2N/A * A send queue doorbell will be rung for the next 2N/A * WQE on the chain, set the current WQE's "dbd" bit. 2N/A * Note: We also update the "dbinfo" structure here to pass 2N/A * back information about what should (later) be included 2N/A * in the send queue doorbell. 2N/A * Send queue doorbell will be rung for the next WQE on 2N/A * the chain, update the prev WQE's "next" field and return. 2N/A * dapli_tavor_wqe_recv_build() 2N/A * Builds the recv WQE for a given ibt_recv_wr_t 2N/A /* Fill in the Data Segments (SGL) for the Recv WQE */ 2N/A /* Check for valid number of SGL entries */ 2N/A * For each SGL in the Recv Work Request, fill in the Recv WQE's data 2N/A * segments. Note: We skip any SGL with zero size because Tavor 2N/A * hardware cannot handle a zero for "byte_cnt" in the WQE. Actually 2N/A * the encoding for zero means a 2GB transfer. Because of this special 2N/A * encoding in the hardware, we mask the requested length with 2N/A * TAVOR_WQE_SGL_BYTE_CNT_MASK (so that 2GB will end up encoded as 2N/A * Fill in the Data Segment(s) for the receive WQE, using the 2N/A * information contained in the scatter-gather list of the 2N/A /* Return the size of descriptor (in 16-byte chunks) */ 2N/A * dapli_tavor_wqe_recv_linknext() 2N/A * Links a recv WQE to the prev chain 2N/A * Note: curr_addr is the last WQE (In uDAPL we manipulate 1 WQE 2N/A * at a time. If there is no next descriptor (i.e. if the current 2N/A * descriptor is the last WQE on the chain), then set "next" field 2N/A * to TAVOR_WQE_DBD_MASK. This is because the Tavor hardware 2N/A * requires the "dbd" bit to be set to one for all Recv WQEs. 2N/A * In either case, we must add a single bit in the "reserved" field 2N/A * (TAVOR_RCV_WQE_NDA0_WA_MASK) following the NDA. This is the 2N/A * workaround for a known Tavor errata that can cause Recv WQEs with 2N/A * zero in the NDA field to behave improperly. 2N/A * If notification suppression is not desired then we set 2N/A * the "E" bit in the ctrl field. 2N/A if (!
ns) {
/* notification needed - so set the "E" bit */ 2N/A /* update the WQE */ 2N/A * Calculate the "next" field of the descriptor. This amounts 2N/A * to setting up the "next_wqe_addr", "dbd", and "nds" fields 2N/A * If this WQE is supposed to be linked to the previous 2N/A * descriptor, then we need to update not only the previous 2N/A * WQE's "next" fields but we must not touch this WQE's 2N/A * dapli_tavor_wqe_srq_build() 2N/A * Builds the recv WQE for a given ibt_recv_wr_t 2N/A /* Fill in the Data Segments (SGL) for the Recv WQE */ 2N/A /* Check for valid number of SGL entries */ 2N/A * For each SGL in the Recv Work Request, fill in the Recv WQE's data 2N/A * segments. Note: We skip any SGL with zero size because Tavor 2N/A * hardware cannot handle a zero for "byte_cnt" in the WQE. Actually 2N/A * the encoding for zero means a 2GB transfer. Because of this special 2N/A * encoding in the hardware, we mask the requested length with 2N/A * TAVOR_WQE_SGL_BYTE_CNT_MASK (so that 2GB will end up encoded as 2N/A * Fill in the Data Segment(s) for the receive WQE, using the 2N/A * information contained in the scatter-gather list of the 2N/A * For SRQ, if the number of data segments is less than the maximum 2N/A * specified at alloc, then we have to fill in a special "key" entry in 2N/A * the sgl entry after the last valid one in this post request. We do 2N/A * dapli_tavor_wqe_srq_linknext() 2N/A * Links a srq recv WQE to the prev chain 2N/A * Note: curr_addr is the last WQE (In uDAPL we manipulate 1 WQE 2N/A * at a time. If there is no next descriptor (i.e. if the current 2N/A * descriptor is the last WQE on the chain), then set "next" field 2N/A * to TAVOR_WQE_DBD_MASK. This is because the Tavor hardware 2N/A * requires the "dbd" bit to be set to one for all Recv WQEs. 2N/A * In either case, we must add a single bit in the "reserved" field 2N/A * (TAVOR_RCV_WQE_NDA0_WA_MASK) following the NDA. This is the 2N/A * workaround for a known Tavor errata that can cause Recv WQEs with 2N/A * zero in the NDA field to behave improperly. 2N/A * If notification suppression is not desired then we set 2N/A * the "E" bit in the ctrl field. 2N/A if (!
ns) {
/* notification needed - so set the "E" bit */ 2N/A /* update the WQE */ 2N/A * Calculate the "next" field of the descriptor. This amounts 2N/A * to setting up the "next_wqe_addr", "dbd", and "nds" fields 2N/A * If this WQE is supposed to be linked to the previous 2N/A * descriptor, then we need to update not only the previous 2N/A * WQE's "next" fields but we must not touch this WQE's 2N/A * dapli_tavor_cq_peek() 2N/A * Peeks into a given CQ to check if there are any events that can be 2N/A * polled. It returns the number of CQEs that can be polled. 2N/A /* Get the consumer index */ 2N/A * Calculate the wrap around mask. Note: This operation only works 2N/A * because all Tavor completion queues have power-of-2 sizes 2N/A /* Calculate the pointer to the first CQ entry */ 2N/A * Count entries in the CQ until we find an entry owned by 2N/A /* Error CQE map to multiple work completions */ 2N/A /* Increment the consumer index */ 2N/A /* Update the pointer to the next CQ entry */ 2N/A * dapli_tavor_cq_poll() 2N/A * This routine polls CQEs out of a CQ and puts them into the ibt_wc_t 2N/A * array that is passed in. 2N/A /* Get the consumer index */ 2N/A * Calculate the wrap around mask. Note: This operation only works 2N/A * because all Tavor completion queues have power-of-2 sizes 2N/A /* Calculate the pointer to the first CQ entry */ 2N/A * Keep pulling entries from the CQ until we find an entry owned by 2N/A * the hardware. As long as there the CQE's owned by SW, process 2N/A * each entry by calling dapli_tavor_cq_cqe_consume() and updating the 2N/A * CQ consumer index. Note: We only update the consumer index if 2N/A * dapli_tavor_cq_cqe_consume() returns TAVOR_CQ_SYNC_AND_DB. 2N/A * Otherwise, it indicates that we are going to "recycle" the CQE 2N/A * (probably because it is a error CQE and corresponds to more than one 2N/A /* Reset entry to hardware ownership */ 2N/A /* Increment the consumer index */ 2N/A /* Update the pointer to the next CQ entry */ 2N/A * If we have run out of space to store work completions, 2N/A * then stop and return the ones we have pulled of the CQ. 2N/A * Now we only ring the doorbell (to update the consumer index) if 2N/A * we've actually consumed a CQ entry. If we have, for example, 2N/A * pulled from a CQE that we are still in the process of "recycling" 2N/A * for error purposes, then we would not update the consumer index. 2N/A * Post doorbell to update the consumer index. Doorbell 2N/A * value indicates number of entries consumed (minus 1) 2N/A * If the CQ is empty, we can try to free up some of the WRID 2N/A * dapli_tavor_cq_poll_one() 2N/A * This routine polls one CQE out of a CQ and puts ot into the ibt_wc_t 2N/A /* Get the consumer index */ 2N/A /* Calculate the pointer to the first CQ entry */ 2N/A * Keep pulling entries from the CQ until we find an entry owned by 2N/A * the hardware. As long as there the CQE's owned by SW, process 2N/A * each entry by calling dapli_tavor_cq_cqe_consume() and updating the 2N/A * CQ consumer index. Note: We only update the consumer index if 2N/A * dapli_tavor_cq_cqe_consume() returns TAVOR_CQ_SYNC_AND_DB. 2N/A * Otherwise, it indicates that we are going to "recycle" the CQE 2N/A * (probably because it is a error CQE and corresponds to more than one 2N/A /* Reset entry to hardware ownership */ 2N/A /* Increment the consumer index */ 2N/A * dapli_tavor_cq_cqe_consume() 2N/A * Converts a given CQE into a ibt_wc_t object 2N/A * Determine if this is an "error" CQE by examining "opcode". If it 2N/A * is an error CQE, then call dapli_tavor_cq_errcqe_consume() and return 2N/A * whatever status it returns. Otherwise, this is a successful 2N/A * Fetch the Work Request ID using the information in the CQE. 2N/A * Parse the CQE opcode to determine completion type. This will set 2N/A * not only the type of the completion, but also any flags that might 2N/A * be associated with it (e.g. whether immediate data is present). 2N/A * The following opcodes will not be generated in uDAPL 2N/A * case TAVOR_CQE_SND_RDMAWR_IMM: 2N/A * case TAVOR_CQE_SND_SEND_IMM: 2N/A * case TAVOR_CQE_SND_ATOMIC_CS: 2N/A * case TAVOR_CQE_SND_ATOMIC_FA: 2N/A * The following opcodes will not be generated in uDAPL 2N/A * case TAVOR_CQE_RCV_RECV_IMM: 2N/A * case TAVOR_CQE_RCV_RECV_IMM2: 2N/A * case TAVOR_CQE_RCV_RDMAWR_IMM: 2N/A * case TAVOR_CQE_RCV_RDMAWR_IMM2: 2N/A /* If we got here, completion status must be success */ 2N/A * dapli_tavor_cq_errcqe_consume() 2N/A * Fetch the Work Request ID using the information in the CQE. 2N/A * Parse the CQE opcode to determine completion type. We know that 2N/A * the CQE is an error completion, so we extract only the completion 2N/A * The following error codes are not supported in the Tavor driver 2N/A * as they relate only to Reliable Datagram completion statuses: 2N/A * case TAVOR_CQE_LOCAL_RDD_VIO_ERR: 2N/A * case TAVOR_CQE_REM_INV_RD_REQ_ERR: 2N/A * case TAVOR_CQE_EEC_REM_ABORTED_ERR: 2N/A * case TAVOR_CQE_INV_EEC_NUM_ERR: 2N/A * case TAVOR_CQE_INV_EEC_STATE_ERR: 2N/A * case TAVOR_CQE_LOC_EEC_ERR: 2N/A * Now we do all the checking that's necessary to handle completion 2N/A * queue entry "recycling" 2N/A * It is not necessary here to try to sync the WQE as we are only 2N/A * attempting to read from the Work Queue (and hardware does not 2N/A * We can get doorbell info, WQE address, size for the next WQE 2N/A * from the "wre" (which was filled in above in the call to the 2N/A * tavor_wrid_get_entry() routine) 2N/A * Get the doorbell count from the CQE. This indicates how many 2N/A * completions this one CQE represents. 2N/A * Determine if we're ready to consume this CQE yet or not. If the 2N/A * next WQE has size zero (i.e. no next WQE) or if the doorbell count 2N/A * is down to zero, then this is the last/only completion represented 2N/A * by the current CQE (return TAVOR_CQ_SYNC_AND_DB). Otherwise, the 2N/A * current CQE needs to be recycled (see below). 2N/A * Return status to indicate that doorbell and sync may be 2N/A * Recycle the CQE for use in the next PollCQ() call 2N/A * Decrement the doorbell count, modify the error status, 2N/A * and update the WQE address and size (to point to the 2N/A * next WQE on the chain. Put these update entries back 2N/A * Despite the fact that we have updated the CQE, it is not 2N/A * necessary for us to attempt to sync this entry just yet 2N/A * as we have not changed the "hardware's view" of the 2N/A * entry (i.e. we have not modified the "owner" bit - which 2N/A * is all that the Tavor hardware really cares about. 2N/A "errcqe_consume: recycling cqe.eth=%x, wqe=%x\n",
2N/A * dapli_tavor_cq_notify() 2N/A * This function is used for arming the CQ by ringing the CQ doorbell. 2N/A * Determine if we are trying to get the next completion or the next 2N/A * "solicited" completion. Then hit the appropriate doorbell. 2N/A * dapli_tavor_post_send() 2N/A /* Grab the lock for the WRID list */ 2N/A /* Save away some initial QP state */ 2N/A * Check for "queue full" condition. If the queue is already full, 2N/A * then no more WQEs can be posted, return an error 2N/A * Increment the "tail index" and check for "queue full" condition. 2N/A * If we detect that the current work request is going to fill the 2N/A * work queue, then we mark this condition and continue. 2N/A * Get the user virtual address of the location where the next 2N/A * Send WQE should be built 2N/A * Call tavor_wqe_send_build() to build the WQE at the given address. 2N/A * This routine uses the information in the ibt_send_wr_t and 2N/A * returns the size of the WQE when it returns. 2N/A * Get the descriptor (io address) corresponding to the location 2N/A * Send WQE was built. 2N/A * Add a WRID entry to the WRID list. Need to calculate the 2N/A * "wqeaddrsz" and "signaled_dbd" values to pass to 2N/A * dapli_tavor_wrid_add_entry() 2N/A * Now link the wqe to the old chain (if there was one) 2N/A * Now if the WRID tail entry is non-NULL, then this 2N/A * represents the entry to which we are chaining the 2N/A * new entries. Since we are going to ring the 2N/A * doorbell for this WQE, we want set its "dbd" bit. 2N/A * On the other hand, if the tail is NULL, even though 2N/A * we will have rung the doorbell for the previous WQE 2N/A * (for the hardware's sake) it is irrelevant to our 2N/A * purposes (for tracking WRIDs) because we know the 2N/A * request must have already completed. 2N/A /* Update some of the state in the QP */ 2N/A /* Ring the doorbell */ 2N/A * dapli_tavor_post_recv() 2N/A /* Grab the lock for the WRID list */ 2N/A /* Save away some initial QP state */ 2N/A * For the ibt_recv_wr_t passed in, parse the request and build a 2N/A * Recv WQE. Link the WQE with the previous WQE and ring the 2N/A * Check for "queue full" condition. If the queue is already full, 2N/A * then no more WQEs can be posted. So return an error. 2N/A * Increment the "tail index" and check for "queue 2N/A * full" condition. If we detect that the current 2N/A * work request is going to fill the work queue, then 2N/A * we mark this condition and continue. 2N/A /* Get the descriptor (IO Address) of the WQE to be built */ 2N/A /* The user virtual address of the WQE to be built */ 2N/A * Call tavor_wqe_recv_build() to build the WQE at the given 2N/A * address. This routine uses the information in the 2N/A * ibt_recv_wr_t and returns the size of the WQE. 2N/A * Add a WRID entry to the WRID list. Need to calculate the 2N/A * "wqeaddrsz" and "signaled_dbd" values to pass to 2N/A * dapli_tavor_wrid_add_entry(). 2N/A * Note: all Recv WQEs are essentially "signaled" 2N/A * Now link the chain to the old chain (if there was one) 2N/A * and ring the doorbel for the recv work queue. 2N/A * Now if the WRID tail entry is non-NULL, then this 2N/A * represents the entry to which we are chaining the 2N/A * new entries. Since we are going to ring the 2N/A * doorbell for this WQE, we want set its "dbd" bit. 2N/A * On the other hand, if the tail is NULL, even though 2N/A * we will have rung the doorbell for the previous WQE 2N/A * (for the hardware's sake) it is irrelevant to our 2N/A * purposes (for tracking WRIDs) because we know the 2N/A * request must have already completed. 2N/A /* Update some of the state in the QP */ 2N/A /* Ring the doorbell */ 2N/A * dapli_tavor_post_srq() 2N/A /* Grab the lock for the WRID list */ 2N/A * For the ibt_recv_wr_t passed in, parse the request and build a 2N/A * Recv WQE. Link the WQE with the previous WQE and ring the 2N/A * Check for "queue full" condition. If the queue is already full, 2N/A * ie. there are no free entries, then no more WQEs can be posted. 2N/A * So return an error. 2N/A /* Save away some initial SRQ state */ 2N/A /* Get the descriptor (IO Address) of the WQE to be built */ 2N/A /* The user virtual address of the WQE to be built */ 2N/A * Call dapli_tavor_wqe_srq_build() to build the WQE at the given 2N/A * address. This routine uses the information in the 2N/A * ibt_recv_wr_t and returns the size of the WQE. 2N/A * Add a WRID entry to the WRID list. 2N/A * Now link the chain to the old chain (if there was one) 2N/A * and ring the doorbell for the SRQ. 2N/A /* Update some of the state in the SRQ */ 2N/A /* Ring the doorbell - for SRQ nds = 0 */ 2N/A * dapli_tavor_wrid_add_entry() 2N/A * Find the entry in the container pointed to by the "tail" index. 2N/A * Add all of the relevant information to that entry, including WRID, 2N/A * Update the "wrid_old_tail" pointer to point to the entry we just 2N/A * inserted into the queue. By tracking this pointer (the pointer to 2N/A * the most recently inserted entry) it will possible later in the 2N/A * PostSend() and PostRecv() code paths to find the entry that needs 2N/A * its "doorbelled" flag set (see comment in tavor_post_recv() and/or 2N/A * tavor_post_send()). 2N/A /* Update the tail index */ 2N/A * If the "tail" index has just wrapped over into the "head" index, 2N/A * then we have filled the container. We use the "full" flag to 2N/A * indicate this condition and to distinguish it from the "empty" 2N/A * condition (where head and tail are also equal). 2N/A * dapli_tavor_wrid_add_entry_srq() 2N/A /* ASSERT on impossible wqe_index values */ 2N/A * Given the 'wqe_index' value, we store the WRID at this WRE offset. 2N/A * And we set the WRE to be signaled_dbd so that on poll CQ we can find 2N/A * this information and associate the WRID to the WQE found on the CQE. 2N/A * Note: all Recv WQEs are essentially "signaled" 2N/A * dapli_tavor_cq_srq_entries_flush() 2N/A /* ASSERT(MUTEX_HELD(&qp->qp_rq_cqhdl->cq_lock)); */ 2N/A /* Get the consumer index */ 2N/A * Calculate the wrap around mask. Note: This operation only works 2N/A * because all Tavor completion queues have power-of-2 sizes 2N/A /* Calculate the pointer to the first CQ entry */ 2N/A * Loop through the CQ looking for entries owned by software. If an 2N/A * entry is owned by software then we increment an 'outstanding_cqes' 2N/A * count to know how many entries total we have on our CQ. We use this 2N/A * value further down to know how many entries to loop through looking 2N/A * for our same QP number. 2N/A /* increment total cqes count */ 2N/A /* increment the consumer index */ 2N/A /* update the pointer to the next cq entry */ 2N/A * Using the 'tail_cons_indx' that was just set, we now know how many 2N/A * total CQEs possible there are. Set the 'check_indx' and the 2N/A * 'new_indx' to the last entry identified by 'tail_cons_indx' 2N/A /* Grab QP number from CQE */ 2N/A * If the QP number is the same in the CQE as the QP that we 2N/A * have on this SRQ, then we must free up the entry off the 2N/A * SRQ. We also make sure that the completion type is of the 2N/A * 'TAVOR_COMPLETION_RECV' type. So any send completions on 2N/A * this CQ will be left as-is. The handling of returning 2N/A * entries back to HW ownership happens further down. 2N/A /* Add back to SRQ free list */ 2N/A * Copy the CQE into the "next_cqe" 2N/A /* Move index to next CQE to check */ 2N/A /* Initialize removed cqes count */ 2N/A /* If an entry was removed */ 2N/A * Set current pointer back to the beginning consumer index. 2N/A * At this point, all unclaimed entries have been copied to the 2N/A * index specified by 'new_indx'. This 'new_indx' will be used 2N/A * as the new consumer index after we mark all freed entries as 2N/A * having HW ownership. We do that here. 2N/A /* Loop through all entries until we reach our new pointer */ 2N/A /* Reset entry to hardware ownership */ 2N/A * Update consumer index to be the 'new_indx'. This moves it past all 2N/A * removed entries. Because 'new_indx' is pointing to the last 2N/A * previously valid SW owned entry, we add 1 to point the cons_indx to 2N/A * the first HW owned entry. 2N/A * Now we only ring the doorbell (to update the consumer index) if 2N/A * we've actually consumed a CQ entry. If we found no QP number 2N/A * matches above, then we would not have removed anything. So only if 2N/A * something was removed do we ring the doorbell. 2N/A * Post doorbell to update the consumer index. Doorbell 2N/A * value indicates number of entries consumed (minus 1)