2N/A * The contents of this file are subject to the terms of the 2N/A * Common Development and Distribution License (the "License"). 2N/A * You may not use this file except in compliance with the License. 2N/A * See the License for the specific language governing permissions 2N/A * and limitations under the License. 2N/A * When distributing Covered Code, include this CDDL HEADER in each 2N/A * If applicable, add the following below this CDDL HEADER, with the 2N/A * fields enclosed by brackets "[]" replaced with your own identifying 2N/A * information: Portions Copyright [yyyy] [name of copyright owner] 2N/A * Copyright 2010 Sun Microsystems, Inc. All rights reserved. 2N/A * Use is subject to license terms. 2N/A/* handy macro, useful because of cq_resize dynamics */ 2N/A * Function signatures 2N/A * Note: The 64 bit doorbells need to written atomically. 2N/A * In 32 bit libraries we need to use the special assembly rtn 2N/A * because compiler generated code splits into 2 word writes 2N/A * dapli_hermon_cq_doorbell() 2N/A * Takes the specified cq cmd and cq number and rings the cq doorbell 2N/A /* Build the doorbell from the parameters */ 2N/A /* Write the doorbell to UAR */ 2N/A /* 32 bit version */ 2N/A * For 32 bit intel we assign the doorbell in the order 2N/A * prescribed by the Tavor PRM, lower to upper addresses 2N/A * dapli_hermon_qp_send_doorbell() 2N/A * Takes the specified qp number and rings the send doorbell. 2N/A /* Write the doorbell to UAR */ 2N/A * For 32 bit intel we assign the doorbell in the order 2N/A * prescribed by the Tavor PRM, lower to upper addresses 2N/A * dapli_hermon_wqe_send_build() 2N/A * Constructs a WQE for a given ibt_send_wr_t 2N/A * RC is the only supported transport in UDAPL 2N/A * For RC requests, we allow "Send", "RDMA Read", "RDMA Write" 2N/A * If this is a Send request, then all we need is 2N/A * the Data Segment processing below. 2N/A * Initialize the information for the Data Segments 2N/A * If this is an RDMA Read or RDMA Write request, then fill 2N/A * in the "Remote Address" header fields. 2N/A * Build the Remote Address Segment for the WQE, using 2N/A * the information from the RC work request. 2N/A /* Update "ds" for filling in Data Segments (below) */ 2N/A * Generate a new R_key 2N/A * Increment the upper "unconstrained" bits and need to keep 2N/A * the lower "constrained" bits the same it represents 2N/A /* XXX - need equiv of "hermon_wr_bind_check(state, wr);" */ 2N/A /* XXX - uses hermon_mr_keycalc - what about Sinai vs. Arbel??? */ 2N/A * Build the Bind Memory Window Segments for the WQE, 2N/A * using the information from the RC Bind memory 2N/A * window work request. 2N/A * Update the "ds" pointer. Even though the "bind" 2N/A * operation requires no SGLs, this is necessary to 2N/A * facilitate the correct descriptor size calculations 2N/A "dapli_hermon_wqe_send_build: invalid wr_opcode=%d\n",
2N/A * Now fill in the Data Segments (SGL) for the Send WQE based on 2N/A * the values setup above (i.e. "sgl", "nds", and the "ds" pointer 2N/A * Start by checking for a valid number of SGL entries 2N/A * For each SGL in the Send Work Request, fill in the Send WQE's data 2N/A * segments. Note: We skip any SGL with zero size because Tavor 2N/A * hardware cannot handle a zero for "byte_cnt" in the WQE. Actually 2N/A * the encoding for zero means a 2GB transfer. Because of this special 2N/A * encoding in the hardware, we mask the requested length with 2N/A * TAVOR_WQE_SGL_BYTE_CNT_MASK (so that 2GB will end up encoded as 2N/A /* need to reduce the length by dword "len" fields */ 2N/A /* if this sgl overflows the inline segment */ 2N/A }
else {
/* this sgl fully fits */ 2N/A /* Return the size of descriptor (in 16-byte chunks) */ 2N/A * Fill in the Data Segment(s) for the current WQE, 2N/A * using the information contained in the 2N/A * scatter-gather list of the work request. 2N/A /* Return the size of descriptor (in 16-byte chunks) */ 2N/A *
size = 0;
/* do not use Hermon Blueflame */ 2N/A * dapli_hermon_wqe_recv_build() 2N/A * Builds the recv WQE for a given ibt_recv_wr_t 2N/A /* Fill in the Data Segments (SGL) for the Recv WQE */ 2N/A /* Check for valid number of SGL entries */ 2N/A * For each SGL in the Recv Work Request, fill in the Recv WQE's data 2N/A * segments. Note: We skip any SGL with zero size because Tavor 2N/A * hardware cannot handle a zero for "byte_cnt" in the WQE. Actually 2N/A * the encoding for zero means a 2GB transfer. Because of this special 2N/A * encoding in the hardware, we mask the requested length with 2N/A * TAVOR_WQE_SGL_BYTE_CNT_MASK (so that 2GB will end up encoded as 2N/A * Fill in the Data Segment(s) for the receive WQE, using the 2N/A * information contained in the scatter-gather list of the 2N/A /* Return the size of descriptor (in 16-byte chunks) */ 2N/A * dapli_hermon_wqe_srq_build() 2N/A * Builds the recv WQE for a given ibt_recv_wr_t 2N/A /* Fill in the Data Segments (SGL) for the Recv WQE */ 2N/A /* Check for valid number of SGL entries */ 2N/A * For each SGL in the Recv Work Request, fill in the Recv WQE's data 2N/A * segments. Note: We skip any SGL with zero size because Tavor 2N/A * hardware cannot handle a zero for "byte_cnt" in the WQE. Actually 2N/A * the encoding for zero means a 2GB transfer. Because of this special 2N/A * encoding in the hardware, we mask the requested length with 2N/A * TAVOR_WQE_SGL_BYTE_CNT_MASK (so that 2GB will end up encoded as 2N/A * Fill in the Data Segment(s) for the receive WQE, using the 2N/A * information contained in the scatter-gather list of the 2N/A * For SRQ, if the number of data segments is less than the maximum 2N/A * specified at alloc, then we have to fill in a special "key" entry in 2N/A * the sgl entry after the last valid one in this post request. We do 2N/A * dapli_hermon_cq_peek() 2N/A * Peeks into a given CQ to check if there are any events that can be 2N/A * polled. It returns the number of CQEs that can be polled. 2N/A /* Get the consumer index */ 2N/A /* Calculate the pointer to the first CQ entry */ 2N/A * Count entries in the CQ until we find an entry owned by 2N/A /* Error CQE map to multiple work completions */ 2N/A /* Increment the consumer index */ 2N/A /* Update the pointer to the next CQ entry */ 2N/A * dapli_hermon_cq_resize_helper() 2N/A * This routine switches from the pre-cq_resize buffer to the new buffer. 2N/A return (
1);
/* FAILED */ 2N/A return (0);
/* SUCCESS */ 2N/A * dapli_hermon_cq_poll() 2N/A * This routine polls CQEs out of a CQ and puts them into the ibt_wc_t 2N/A * array that is passed in. 2N/A /* Get the consumer index */ 2N/A /* Calculate the pointer to the first CQ entry */ 2N/A * Keep pulling entries from the CQ until we find an entry owned by 2N/A * the hardware. As long as there the CQE's owned by SW, process 2N/A * each entry by calling dapli_hermon_cq_cqe_consume() and updating the 2N/A * CQ consumer index. Note: We only update the consumer index if 2N/A * dapli_hermon_cq_cqe_consume() returns TAVOR_CQ_SYNC_AND_DB. 2N/A * Otherwise, it indicates that we are going to "recycle" the CQE 2N/A * (probably because it is a error CQE and corresponds to more than one 2N/A /* Reset to hardware ownership is implicit in Hermon */ 2N/A /* Increment the consumer index */ 2N/A /* Update the pointer to the next CQ entry */ 2N/A * If we have run out of space to store work completions, 2N/A * then stop and return the ones we have pulled of the CQ. 2N/A * Now we only ring the doorbell (to update the consumer index) if 2N/A * we've actually consumed a CQ entry. If we have, for example, 2N/A * pulled from a CQE that we are still in the process of "recycling" 2N/A * for error purposes, then we would not update the consumer index. 2N/A * Update the consumer index in both the CQ handle and the 2N/A * If the CQ is empty, we can try to free up some of the WRID 2N/A * dapli_hermon_cq_poll_one() 2N/A * This routine polls one CQE out of a CQ and puts ot into the ibt_wc_t 2N/A /* Get the consumer index */ 2N/A /* Calculate the pointer to the first CQ entry */ 2N/A * Keep pulling entries from the CQ until we find an entry owned by 2N/A * the hardware. As long as there the CQE's owned by SW, process 2N/A * each entry by calling dapli_hermon_cq_cqe_consume() and updating the 2N/A * CQ consumer index. Note: We only update the consumer index if 2N/A * dapli_hermon_cq_cqe_consume() returns TAVOR_CQ_SYNC_AND_DB. 2N/A * Otherwise, it indicates that we are going to "recycle" the CQE 2N/A * (probably because it is a error CQE and corresponds to more than one 2N/A /* Reset to hardware ownership is implicit in Hermon */ 2N/A /* Increment the consumer index */ 2N/A * dapli_hermon_cq_cqe_consume() 2N/A * Converts a given CQE into a ibt_wc_t object 2N/A * Determine if this is an "error" CQE by examining "opcode". If it 2N/A * is an error CQE, then call dapli_hermon_cq_errcqe_consume() and 2N/A * return whatever status it returns. Otherwise, this is a successful 2N/A * Fetch the Work Request ID using the information in the CQE. 2N/A * Parse the CQE opcode to determine completion type. This will set 2N/A * not only the type of the completion, but also any flags that might 2N/A * be associated with it (e.g. whether immediate data is present). 2N/A * The following opcodes will not be generated in uDAPL 2N/A * case TAVOR_CQE_SND_RDMAWR_IMM: 2N/A * case TAVOR_CQE_SND_SEND_IMM: 2N/A * case TAVOR_CQE_SND_ATOMIC_CS: 2N/A * case TAVOR_CQE_SND_ATOMIC_FA: 2N/A * The following opcodes will not be generated in uDAPL 2N/A * case TAVOR_CQE_RCV_RECV_IMM: 2N/A * case TAVOR_CQE_RCV_RECV_IMM2: 2N/A * case TAVOR_CQE_RCV_RDMAWR_IMM: 2N/A * case TAVOR_CQE_RCV_RDMAWR_IMM2: 2N/A /* If we got here, completion status must be success */ 2N/A * dapli_hermon_cq_errcqe_consume() 2N/A * Fetch the Work Request ID using the information in the CQE. 2N/A * Parse the CQE opcode to determine completion type. We know that 2N/A * the CQE is an error completion, so we extract only the completion 2N/A * The following error codes are not supported in the Tavor driver 2N/A * as they relate only to Reliable Datagram completion statuses: 2N/A * case TAVOR_CQE_LOCAL_RDD_VIO_ERR: 2N/A * case TAVOR_CQE_REM_INV_RD_REQ_ERR: 2N/A * case TAVOR_CQE_EEC_REM_ABORTED_ERR: 2N/A * case TAVOR_CQE_INV_EEC_NUM_ERR: 2N/A * case TAVOR_CQE_INV_EEC_STATE_ERR: 2N/A * case TAVOR_CQE_LOC_EEC_ERR: 2N/A * Return status to indicate that doorbell and sync may be 2N/A * dapli_hermon_cq_notify() 2N/A * This function is used for arming the CQ by ringing the CQ doorbell. 2N/A * Note: there is something very subtle here. This code assumes a very 2N/A * specific behavior of the kernel driver. The cmd_sn field of the 2N/A * arm_dbr is updated by the kernel driver whenever a notification 2N/A * event for the cq is received. This code extracts the cmd_sn field 2N/A * from the arm_dbr to know the right value to use. The arm_dbr is 2N/A * always updated atomically so that neither the kernel driver nor this 2N/A * will get confused about what the other is doing. 2N/A * Note: param is not used here. It is necessary for arming a CQ for 2N/A * N completions (param is N), but no uDAPL API supports this for now. 2N/A * Thus, we declare ARGSUSED to make lint happy. 2N/A * Determine if we are trying to get the next completion or the next 2N/A * "solicited" completion. Then hit the appropriate doorbell. 2N/A }
/* else it's already armed */ 2N/A }
/* else it's already armed */ 2N/A * Since uDAPL posts 1 wqe per request, we 2N/A * only need to do stores for the last one. 2N/A * dapli_hermon_post_send() 2N/A /* Grab the lock for the WRID list */ 2N/A /* Save away some initial QP state */ 2N/A * Check for "queue full" condition. If the queue is already full, 2N/A * then no more WQEs can be posted, return an error 2N/A * Increment the "tail index" and check for "queue full" condition. 2N/A * If we detect that the current work request is going to fill the 2N/A * work queue, then we mark this condition and continue. 2N/A * Get the user virtual address of the location where the next 2N/A * Send WQE should be built 2N/A * Call tavor_wqe_send_build() to build the WQE at the given address. 2N/A * This routine uses the information in the ibt_send_wr_t and 2N/A * returns the size of the WQE when it returns. 2N/A * Get the descriptor (io address) corresponding to the location 2N/A * Send WQE was built. 2N/A * Add a WRID entry to the WRID list. Need to calculate the 2N/A * "wqeaddr" to pass to dapli_tavor_wrid_add_entry(). 2N/A * signaled_dbd is still calculated, but ignored. 2N/A * Now if the WRID tail entry is non-NULL, then this 2N/A * represents the entry to which we are chaining the 2N/A * new entries. Since we are going to ring the 2N/A * doorbell for this WQE, we want set its "dbd" bit. 2N/A * On the other hand, if the tail is NULL, even though 2N/A * we will have rung the doorbell for the previous WQE 2N/A * (for the hardware's sake) it is irrelevant to our 2N/A * purposes (for tracking WRIDs) because we know the 2N/A * request must have already completed. 2N/A /* Update some of the state in the QP */ 2N/A /* Ring the doorbell */ 2N/A * dapli_hermon_post_recv() 2N/A /* Grab the lock for the WRID list */ 2N/A /* Save away some initial QP state */ 2N/A * For the ibt_recv_wr_t passed in, parse the request and build a 2N/A * Recv WQE. Link the WQE with the previous WQE and ring the 2N/A * Check for "queue full" condition. If the queue is already full, 2N/A * then no more WQEs can be posted. So return an error. 2N/A * Increment the "tail index" and check for "queue 2N/A * full" condition. If we detect that the current 2N/A * work request is going to fill the work queue, then 2N/A * we mark this condition and continue. 2N/A /* The user virtual address of the WQE to be built */ 2N/A * Call tavor_wqe_recv_build() to build the WQE at the given 2N/A * address. This routine uses the information in the 2N/A * ibt_recv_wr_t and returns the size of the WQE. 2N/A * Add a WRID entry to the WRID list. Need to calculate the 2N/A * "wqeaddr" and "signaled_dbd" values to pass to 2N/A * dapli_tavor_wrid_add_entry(). 2N/A * Note: all Recv WQEs are essentially "signaled" 2N/A * Now if the WRID tail entry is non-NULL, then this 2N/A * represents the entry to which we are chaining the 2N/A * new entries. Since we are going to ring the 2N/A * doorbell for this WQE, we want set its "dbd" bit. 2N/A * On the other hand, if the tail is NULL, even though 2N/A * we will have rung the doorbell for the previous WQE 2N/A * (for the hardware's sake) it is irrelevant to our 2N/A * purposes (for tracking WRIDs) because we know the 2N/A * request must have already completed. 2N/A /* Update some of the state in the QP */ 2N/A /* Update the doorbell record */ 2N/A * dapli_hermon_post_srq() 2N/A /* Grab the lock for the WRID list */ 2N/A * For the ibt_recv_wr_t passed in, parse the request and build a 2N/A * Recv WQE. Link the WQE with the previous WQE and ring the 2N/A * Check for "queue full" condition. If the queue is already full, 2N/A * ie. there are no free entries, then no more WQEs can be posted. 2N/A * So return an error. 2N/A /* Save away some initial SRQ state */ 2N/A /* Get the descriptor (IO Address) of the WQE to be built */ 2N/A /* The user virtual address of the WQE to be built */ 2N/A * Call dapli_hermon_wqe_srq_build() to build the WQE at the given 2N/A * address. This routine uses the information in the 2N/A * ibt_recv_wr_t and returns the size of the WQE. 2N/A * Add a WRID entry to the WRID list. 2N/A * Now link the chain to the old chain (if there was one) 2N/A * and update the wqe_counter in the doorbell record. 2N/A /* Update some of the state in the SRQ */ 2N/A /* Update the doorbell record */ 2N/A * dapli_hermon_cq_srq_entries_flush() 2N/A /* ASSERT(MUTEX_HELD(&qp->qp_rq_cqhdl->cq_lock)); */ 2N/A /* Get the consumer index */ 2N/A /* Calculate the pointer to the first CQ entry */ 2N/A * Loop through the CQ looking for entries owned by software. If an 2N/A * entry is owned by software then we increment an 'outstanding_cqes' 2N/A * count to know how many entries total we have on our CQ. We use this 2N/A * value further down to know how many entries to loop through looking 2N/A * for our same QP number. 2N/A /* increment total cqes count */ 2N/A /* increment the consumer index */ 2N/A /* update the pointer to the next cq entry */ 2N/A * Using the 'tail_cons_indx' that was just set, we now know how many 2N/A * total CQEs possible there are. Set the 'check_indx' and the 2N/A * 'new_indx' to the last entry identified by 'tail_cons_indx' 2N/A /* Grab QP number from CQE */ 2N/A * If the QP number is the same in the CQE as the QP that we 2N/A * have on this SRQ, then we must free up the entry off the 2N/A * SRQ. We also make sure that the completion type is of the 2N/A * 'TAVOR_COMPLETION_RECV' type. So any send completions on 2N/A * this CQ will be left as-is. The handling of returning 2N/A * entries back to HW ownership happens further down. 2N/A /* Add back to SRQ free list */ 2N/A * Copy the CQE into the "next_cqe" 2N/A /* Move index to next CQE to check */ 2N/A /* Initialize removed cqes count */ 2N/A /* If an entry was removed */ 2N/A * Set current pointer back to the beginning consumer index. 2N/A * At this point, all unclaimed entries have been copied to the 2N/A * index specified by 'new_indx'. This 'new_indx' will be used 2N/A * as the new consumer index after we mark all freed entries as 2N/A * having HW ownership. We do that here. 2N/A /* Loop through all entries until we reach our new pointer */ 2N/A /* Reset entry to hardware ownership */ 2N/A * Update consumer index to be the 'new_indx'. This moves it past all 2N/A * removed entries. Because 'new_indx' is pointing to the last 2N/A * previously valid SW owned entry, we add 1 to point the cons_indx to 2N/A * the first HW owned entry. 2N/A * Now we only ring the doorbell (to update the consumer index) if 2N/A * we've actually consumed a CQ entry. If we found no QP number 2N/A * matches above, then we would not have removed anything. So only if 2N/A * something was removed do we ring the doorbell. 2N/A * Update the consumer index in both the CQ handle and the 2N/A p[
1] =
nds;
/* nds is 0 for SRQ */ 2N/A /* cq_resize -- needs testing */ 2N/A /* pre-link the whole shared receive queue */