9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * This file and its contents are supplied under the terms of the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Common Development and Distribution License ("CDDL"), version 1.0.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * You may only use this file in accordance with the terms of version
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * 1.0 of the CDDL.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * A full copy of the text of the CDDL should have accompanied this
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * source. A copy of the CDDL is also available via the Internet at
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Copyright 2015 OmniTI Computer Consulting, Inc. All rights reserved.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Copyright 2016 Joyent, Inc.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ---------------------------------------------------------
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Buffer and Memory Management, Receiving, and Transmitting
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ---------------------------------------------------------
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Each physical function (PF), which is what we think of as an instance of the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * device driver, has a series of associated transmit and receive queue pairs.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Effectively, what we think of in MAC as rings. Each of these has their own
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ring of descriptors which is used as part of doing DMA activity.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The transmit ring of descriptors are 16-byte entries which are used to send
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * packets, program filters, etc. The receive ring of descriptors are either
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * 16-byte or 32-bytes each. At the moment, we opt to use the larger descriptor
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * format so that we're in a better position if we ever want to leverage that
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * information later on.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * However, these rings are just for descriptors, they don't talk or deal with
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * how we actually store the memory that we need for DMA or the associated
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * information that we need for keeping track of message blocks. To correspond
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * to the hardware descriptor ring which is how we communicate with hardware, we
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * introduce a control block which keeps track of our required metadata like DMA
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * There are two main considerations that dictate how much memory and buffers
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we end up allocating. Those are:
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * o The size of the ring (controlled through the driver.conf file)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * o The maximum size frame we can receive.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The size of the rings currently defaults to 1024 descriptors and is stored in
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the i40e_t`i40e_rx_ring_size and i40e_t`i40e_tx_ring_size.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * While the size of the rings is controlled by the driver.conf, the maximum
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * size frame is informed primarily through the use of dladm and the setting of
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the MTU property on the device. From the MTU, we then go and do some
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * machinations. The first thing we do is we then have to add in space for the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Ethernet header, potentially a VLAN header, and the FCS check. This value is
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * what's stored as i40e_t`i40e_frame_max and is derived any time
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * i40e_t`i40e_sdu changes.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * This size is then rounded up to the nearest 1k chunk, which represents the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * actual amount of memory that we'll allocate for a single frame.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Note, that for rx, we do something that might be unexpected. We always add
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * an extra two bytes to the frame size that we allocate. We then offset the DMA
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * address that we receive a packet into by two bytes. This ensures that the IP
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * header will always be 4 byte aligned because the MAC header is either 14 or
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * 18 bytes in length, depending on the use of 802.1Q tagging, which makes IP's
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * and MAC's lives easier.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Both the rx and tx descriptor rings (which are what we use to communicate
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * with hardware) are allocated as a single region of DMA memory which is the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * size of the descriptor (4 bytes and 2 bytes respectively) times the total
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * number of descriptors for an rx and tx ring.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * While the rx and tx descriptors are allocated using DMA-based memory, the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * control blocks for each of them are allocated using normal kernel memory.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * They aren't special from a DMA perspective. We'll go over the design of both
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * receiving and transmitting separately, as they have slightly different
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * control blocks and different ways that we manage the relationship between
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * control blocks and descriptors.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ---------------------------------
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * RX Descriptors and Control Blocks
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ---------------------------------
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * For every descriptor in the ring that the driver has, we need some associated
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * memory, which means that we need to have the receive specific control block.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We have a couple different, but related goals:
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * o Once we've completed the mc_start GLDv3 endpoint (i40e_m_start), we do
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * not want to do any additional memory allocations or DMA allocations if
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we don't have to.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * o We'd like to try and do as much zero-copy as possible, while taking into
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * account the cost of mapping in DMA resources.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * o We'd like to have every receive descriptor available.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Now, these rules are a bit in tension with one another. The act of mapping in
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * is an exercise of trying to find the break-even point between page table
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * updates and bcopy. We currently start by using the same metrics that ixgbe
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * used; however, it should be known that this value has effectively been
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * cargo-culted across to yet another driver, sorry.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * If we receive a packet which is larger than our copy threshold, we'll create
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * a message block out of the DMA memory via desballoc(9F) and send that up to
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * MAC that way. This will cause us to be notified when the message block is
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * then freed because it has been consumed, dropped, or otherwise. Otherwise, if
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * it's less than the threshold, we'll try to use allocb and bcopy it into the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * block, thus allowing us to immediately reuse the DMA resource. Note, on debug
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * builds, we allow someone to whack the variable i40e_debug_rx_mode to override
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the behavior and always do a bcopy or a DMA bind.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * To try and ensure that the device always has blocks that it can receive data
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * into, we maintain two lists of control blocks, a working list and a free
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * list. Each list is sized equal to the number of descriptors in the rx ring.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * During the GLDv3 mc_start routine, we allocate a number of rx control blocks
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * equal to twice the number of descriptors in the ring and we assign them
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * equally to the free list and to the working list. Each control block also has
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * DMA memory allocated and associated with which it will be used to receive the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * actual packet data. All of a received frame's data will end up in a single
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * DMA buffer.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * During operation, we always maintain the invariant that each rx descriptor
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * has an associated rx control block which lives in the working list. If we
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * feel that we should loan up DMA memory to MAC in the form of a message block,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we can only do so if we can maintain this invariant. To do that, we swap in
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * one of the buffers from the free list. If none are available, then we resort
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * to using allocb(9F) and bcopy(9F) on the packet instead, regardless of the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Loaned message blocks come back to use when freemsg(9F) or freeb(9F) is
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * called on the block, at which point we restore the rx control block to the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * free list and are able to reuse the DMA memory again. While the scheme may
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * seem odd, it importantly keeps us out of trying to do any DMA allocations in
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the normal path of operation, even though we may still have to allocate
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * message blocks and copy.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The following state machine describes the life time of a rx control block. In
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the diagram we abbrviate the rx ring descriptor entry as rxd and the rx
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * control block entry as rcb.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * * ... 1/2 of all initial rcb's ... *
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * +------------------+ +------------------+
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | rcb on free list |---*---------->| rcb on work list |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * +------------------+ . +------------------+
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ^ . moved to |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | replace rcb * . . Frame received,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | loaned to | entry on free list
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | MAC + co. | available. rcb's
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | | memory made into mblk_t
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * * . freemsg(9F) | and sent up to MAC.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | called on |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | loaned rcb |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | and it is v
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | recycled. +-------------------+
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * +--------------------<-----| rcb loaned to MAC |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * +-------------------+
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Finally, note that every rx control block has a reference count on it. One
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * reference is added as long as the driver has had the GLDv3 mc_start endpoint
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * called. If the GLDv3 mc_stop entry point is called, IP has been unplumbed and
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * no other DLPI consumers remain, then we'll decrement the reference count by
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * one. Whenever we loan up the rx control block and associated buffer to MAC,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * then we bump the reference count again. Even though the device is stopped,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * there may still be loaned frames in upper levels that we'll want to account
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * for. Our callback from freemsg(9F)/freeb(9F) will take care of making sure
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * that it is cleaned up.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * --------------------
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Managing the RX Ring
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * --------------------
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The receive ring descriptors are arranged in a circular buffer with a head
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * and tail pointer. There are both the conventional head and tail pointers
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * which are used to partition the ring into two portions, a portion that we,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the operating system, manage and a portion that is managed by hardware. When
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * hardware owns a descriptor in the ring, it means that it is waiting for data
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * to be filled in. However, when a portion of the ring is owned by the driver,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * then that means that the descriptor has been consumed and we need to go take
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * a look at it.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The initial head is configured to be zero by writing it as such in the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * receive queue context in the FPM (function private memory from the host). The
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * initial tail is written to be the last descriptor. This is written to via the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * PCIe register I40E_QRX_TAIL(). Technically, hardware owns everything between
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the HEAD and TAIL, inclusive. Note that while we initially program the HEAD,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the only values we ever consult ourselves are the TAIL register and our own
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * state tracking. Effectively, we cache the HEAD register and then update it
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ourselves based on our work.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * When we iterate over the rx descriptors and thus the received frames, we are
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * either in an interrupt context or we've been asked by MAC to poll on the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ring. If we've been asked to poll on the ring, we have a maximum number of
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * bytes of mblk_t's to return. If processing an rx descriptor would cause us to
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * exceed that count, then we do not process it. When in interrupt context, we
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * don't have a strict byte count. However, to ensure liveness, we limit the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * amount of data based on a configuration value
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * (i40e_t`i40e_rx_limit_per_intr). The number that we've started with for this
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * is based on similar numbers that are used for ixgbe. After some additional
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * time in the field, we'll have a sense as to whether or not it should be
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * When processing, we start at our own HEAD pointer
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * (i40e_rx_data_t`rxd_desc_next), which indicates the descriptor to start
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * processing. Every RX descriptor has what's described as the DD bit. This bit
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * (the LSB of the second 8-byte word), indicates whether or not the descriptor
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * is done. When we give descriptors to the hardware, this value is always
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * zero. When the hardware has finished a descriptor, it will always be one.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The first thing that we check is whether the DD bit indicates that the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * current HEAD is ready. If it isn't, then we're done. That's the primary
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * invariant of processing a frame. If it's done, then there are a few other
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * things that we want to look at. In the same status word as the DD bit, there
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * are two other important bits:
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * o End of Packet (EOP)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * o Error bits
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The end of packet indicates that we have reached the last descriptor. Now,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * you might ask when would there be more than one descriptor. The reason for
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * that might be due to large receive offload (lro) or header splitting
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * functionality, which presently isn't supported in the driver. The error bits
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * in the frame are only valid when EOP is set.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * If error bits are set on the frame, then we still consume it; however, we
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * will not generate an mblk_t to send up to MAC. If there are no error bits
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * set, then we'll consume the descriptor either using bcopy or DMA binding. See
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the earlier section 'RX DESCRIPTORS AND CONTROL BLOCKS' for more information
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * on how that selection is made.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Regardless of whether we construct an mblk_t or encounter an error, we end up
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * resetting the descriptor. This re-arms the descriptor for hardware and in the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * process, we may end up assigning it a new receive control bock. After we do
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * this, we always update our HEAD pointer, no matter what.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Finally, once we've consumed as much as we will in a given window, we go and
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * update the TAIL register to indicate all the frames we've consumed. We only
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * do a single bulk write for the ring.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ---------------------------------
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * TX Descriptors and Control Blocks
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ---------------------------------
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * While the transmit path is similar in spirit to the receive path, it works
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * differently due to the fact that all data is originated by the operating
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * system and not by the device.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Like rx, there is both a descriptor ring that we use to communicate to the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * driver and which points to the memory used to transmit a frame. Similarly,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * there is a corresponding transmit control block. Each transmit control block
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * has a region of DMA memory allocated to it; however, the way we use it
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The driver is asked to process a single frame at a time. That message block
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * may be made up of multiple fragments linked together by the mblk_t`b_cont
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * member. The device has a hard limit of up to 8 buffers being allowed for use
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * for a single logical frame. For each fragment, we'll try and use an entry
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * from the tx descriptor ring and then we'll allocate a corresponding tx
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * control block. Depending on the size of the fragment, we may copy it around
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * or we might instead try to do DMA binding of the fragment.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * If we exceed the number of blocks that fit, we'll try to pull up the block
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * and then we'll do a DMA bind and send it out.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * If we don't have enough space in the ring or tx control blocks available,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * then we'll return the unprocessed message block to MAC. This will induce flow
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * control and once we recycle enough entries, we'll once again enable sending
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * on the ring.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We size the working list as equal to the number of descriptors in the ring.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We size the free list as equal to 1.5 times the number of descriptors in the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ring. We'll allocate a number of tx control block entries equal to the number
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * of entries in the free list. By default, all entries are placed in the free
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * list. As we come along and try to send something, we'll allocate entries from
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the free list and add them to the working list, where they'll stay until the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * hardware indicates that all of the data has been written back to us. The
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * reason that we start with 1.5x is to help facilitate having more than one TX
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * buffer associated with the DMA activity.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * --------------------
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Managing the TX Ring
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * --------------------
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The transmit descriptor ring is driven by us. We maintain our own notion of a
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * HEAD and TAIL register and we update the hardware with updates to the TAIL
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * register. When the hardware is done writing out data, it updates us by
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * writing back to a specific address, not by updating the individual
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * descriptors. That address is a 4-byte region after the main transmit
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * descriptor ring. This is why the descriptor ring has an extra descriptor's
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * worth allocated to it.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We maintain our notion of the HEAD in the i40e_trqpair_t`itrq_desc_head and
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the TAIL in the i40e_trqpair_t`itrq_desc_tail. When we write out frames,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we'll update the tail there and in the I40E_QTX_TAIL() register. At various
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * points in time, through both interrupts, and our own internal checks, we'll
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * sync the write-back head portion of the DMA space. Based on the index it
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * reports back, we'll free everything between our current HEAD and the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * indicated index and update HEAD to the new index.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * When a frame comes in, we try to use a number of transmit control blocks and
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we'll transition them from the free list to the work list. They'll get moved
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * to the entry on the work list that corresponds with the transmit descriptor
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * they correspond to. Once we are indicated that the corresponding descriptor
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * has been freed, we'll return it to the list.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The transmit control block free list is managed by keeping track of the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * number of entries in it, i40e_trqpair_t`itrq_tcb_free. We use it as a way to
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * index into the free list and add things to it. In effect, we always push and
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * pop from the tail and protect it with a single lock,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * i40e_trqpair_t`itrq_tcb_lock. This scheme is somewhat simplistic and may not
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * stand up to further performance testing; however, it does allow us to get off
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the ground with the device driver.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The following image describes where a given transmit control block lives in
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * its lifetime:
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * * ... Initial placement for all tcb's
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * +------------------+ +------------------+
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | tcb on free list |---*------------------>| tcb on work list |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * +------------------+ . +------------------+
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ^ . tcb allocated |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | to send frame v
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | or fragment on |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | wire, mblk from |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * | MAC associated. |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * +------*-------------------------------<----+
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * . Hardware indicates
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * entry transmitted.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * tcb recycled, mblk
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * from MAC freed.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ------------
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Blocking MAC
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ------------
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Wen performing transmit, we can run out of descriptors and ring entries. When
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * such a case happens, we return the mblk_t to MAC to indicate that we've been
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * blocked. At that point in time, MAC becomes blocked and will not transmit
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * anything out that specific ring until we notify MAC. To indicate that we're
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * in such a situation we set i40e_trqpair_t`itrq_tx_blocked member to B_TRUE.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * When we recycle tx descriptors then we'll end up signaling MAC by calling
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * mac_tx_ring_update() if we were blocked, letting it know that it's safe to
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * start sending frames out to us again.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We set our DMA alignment requests based on the smallest supported page size
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * of the corresponding platform.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * This structure is used to maintain information and flags related to
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * transmitting a frame. The first member is the set of flags we need to or into
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the command word (generally checksumming related). The second member controls
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the word offsets which is required for IP and L4 checksumming.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Toggles on debug builds which can be used to override our RX behaviour based
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * on thresholds.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchitypedef enum {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_debug_rx_t i40e_debug_rx_mode = I40E_DEBUG_RX_DEFAULT;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi#endif /* DEBUG */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Notes on the following pair of DMA attributes. The first attribute,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * i40e_static_dma_attr, is designed to be used for both the descriptor rings
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * and the static buffers that we associate with control blocks. For this
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * reason, we force an SGL length of one. While technically the driver supports
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * a larger SGL (5 on rx and 8 on tx), we opt to only use one to simplify our
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * management here. In addition, when the Intel common code wants to allocate
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * memory via the i40e_allocate_virt_mem osdep function, we have it leverage
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the static dma attr.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The second set of attributes, i40e_txbind_dma_attr, is what we use when we're
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * binding a bunch of mblk_t fragments to go out the door. Note that the main
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * difference here is that we're allowed a larger SGL length -- eight.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Note, we default to setting ourselves to be DMA capable here. However,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * because we could have multiple instances which have different FMA error
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * checking capabilities, or end up on different buses, we make these static
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * and const and copy them into the i40e_t for the given device with the actual
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * values that reflect the actual capabilities.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchistatic const ddi_dma_attr_t i40e_g_static_dma_attr = {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi 0x00000000FFFFFFFFull, /* dma counter max */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi 0x00000000FFFFFFFFull, /* maximum transfer size */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi 0xFFFFFFFFFFFFFFFFull, /* maximum segment size */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchistatic const ddi_dma_attr_t i40e_g_txbind_dma_attr = {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi 0x00000000FFFFFFFFull, /* dma counter max */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi 0x00000000FFFFFFFFull, /* maximum transfer size */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi 0xFFFFFFFFFFFFFFFFull, /* maximum segment size */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi I40E_TX_MAX_COOKIE, /* scatter/gather list length */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Next, we have the attributes for these structures. The descriptor rings are
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * all strictly little endian, while the data buffers are just arrays of bytes
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * representing frames. Because of this, we purposefully simplify the driver
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * programming life by programming the descriptor ring as little endian, while
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * for the buffer data we keep it as unstructured.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Note, that to keep the Intel common code operating in a reasonable way, when
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we allocate DMA memory for it, we do not use byte swapping and thus use the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * standard i40e_buf_acc_attr.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchistatic const ddi_device_acc_attr_t i40e_g_desc_acc_attr = {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchistatic const ddi_device_acc_attr_t i40e_g_buf_acc_attr = {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The next two functions are designed to be type-safe versions of macros that
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * are used to increment and decrement a descriptor index in the loop. Note,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * these are marked inline to try and keep the data path hot and they were
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * effectively inlined in their previous life as macros.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchistatic inline int
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_next_desc(int base, int count, int size)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchistatic inline int
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_prev_desc(int base, int count, int size)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Free DMA memory that is represented by a i40e_dma_buffer_t.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_free_dma_buffer(i40e_dma_buffer_t *dmap)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi (void) ddi_dma_unbind_handle(dmap->dmab_dma_handle);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ddi_dma_free_handle(&dmap->dmab_dma_handle);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * These should only be set if we have valid handles allocated and
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * therefore should always be NULLed out due to the above code. This
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * is here to catch us acting sloppy.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Allocate size bytes of DMA memory based on the passed in attributes. This
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * fills in the information in dmap and is designed for all of our single cookie
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * allocations.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_alloc_dma_buffer(i40e_t *i40e, i40e_dma_buffer_t *dmap,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ddi_dma_attr_t *attrsp, ddi_device_acc_attr_t *accp, boolean_t stream,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Step one: Allocate the DMA handle
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ret = ddi_dma_alloc_handle(i40e->i40e_dip, attrsp, DDI_DMA_DONTWAIT,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate dma handle for I/O "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Step two: Allocate the DMA memory
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ret = ddi_dma_mem_alloc(dmap->dmab_dma_handle, size, accp, flags,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi DDI_DMA_DONTWAIT, NULL, &dmap->dmab_address, &len,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate %ld bytes of DMA for I/O "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Step three: Optionally zero
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Step four: Bind the memory
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ret = ddi_dma_addr_bind_handle(dmap->dmab_dma_handle, NULL,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi dmap->dmab_address, len, DDI_DMA_RDWR | flags, DDI_DMA_DONTWAIT,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate %ld bytes of DMA for I/O "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi dmap->dmab_dma_address = cookie.dmac_laddress;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * This function is called once the last pending rcb has been freed by the upper
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * levels of the system.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi (rxd->rxd_free_list_size + rxd->rxd_ring_size));
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_alloc_rx_data(i40e_t *i40e, i40e_trqpair_t *itrq)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rxd = kmem_zalloc(sizeof (i40e_rx_data_t), KM_NOSLEEP);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rxd->rxd_ring_size = i40e->i40e_rx_ring_size;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rxd->rxd_free_list_size = i40e->i40e_rx_ring_size;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rxd->rxd_rcb_free = rxd->rxd_free_list_size;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rxd->rxd_work_list = kmem_zalloc(sizeof (i40e_rx_control_block_t *) *
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate rx work list for a ring "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi "of %d entries for ring %d", rxd->rxd_ring_size,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rxd->rxd_free_list = kmem_zalloc(sizeof (i40e_rx_control_block_t *) *
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate a %d entry rx free list "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi "for ring %d", rxd->rxd_free_list_size, itrq->itrq_index);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rxd->rxd_rcb_area = kmem_zalloc(sizeof (i40e_rx_control_block_t) *
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi (rxd->rxd_free_list_size + rxd->rxd_ring_size), KM_NOSLEEP);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate a %d entry rcb area for "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi "ring %d", rxd->rxd_ring_size + rxd->rxd_free_list_size,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Free all of the memory that we've allocated for DMA. Note that we may have
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * buffers that we've loaned up to the OS which are still outstanding. We'll
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * always free up the descriptor ring, because we no longer need that. For each
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * rcb, we'll iterate over it and if we send the reference count to zero, then
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we'll free the message block and DMA related resources. However, if we don't
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * take the last one, then we'll go ahead and keep track that we'll have pending
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * data and clean it up when we get there.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_free_rx_dma(i40e_rx_data_t *rxd, boolean_t failed_init)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi count = rxd->rxd_ring_size + rxd->rxd_free_list_size;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * If we're cleaning up from a failed creation attempt, then an
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * entry may never have been assembled which would mean that
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * it's reference count is zero. If we find that, we leave it
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * be, because nothing else should be modifying it at this
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * point. We're not at the point that any more references can be
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * added, just removed.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (failed_init == B_TRUE && rcb->rcb_ref == 0)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Initialize the DMA memory for the descriptor ring and for each frame in the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * control block list.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * First allocate the rx descriptor ring.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi dmasz = sizeof (i40e_rx_desc_t) * rxd->rxd_ring_size;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_alloc_dma_buffer(i40e, &rxd->rxd_desc_area,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi &i40e->i40e_static_dma_attr, &i40e->i40e_desc_acc_attr, B_FALSE,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate DMA resources "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi "for rx descriptor ring");
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi (i40e_rx_desc_t *)(uintptr_t)rxd->rxd_desc_area.dmab_address;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi count = rxd->rxd_ring_size + rxd->rxd_free_list_size;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rxd->rxd_free_list[i - rxd->rxd_ring_size] = rcb;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi &i40e->i40e_static_dma_attr, &i40e->i40e_buf_acc_attr,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate rx dma buffer");
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Initialize the control block and offset the DMA address. See
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the note in the big theory statement that explains how this
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * helps IP deal with alignment. Note, we don't worry about
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * whether or not we successfully get an mblk_t from desballoc,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * it's a common case that we have to handle later on in the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi dmap->dmab_size -= I40E_BUF_IPHDR_ALIGNMENT;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi dmap->dmab_address += I40E_BUF_IPHDR_ALIGNMENT;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi dmap->dmab_dma_address += I40E_BUF_IPHDR_ALIGNMENT;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rcb->rcb_free_rtn.free_func = i40e_rx_recycle;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rcb->rcb_mp = desballoc((unsigned char *)dmap->dmab_address,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_tx_control_block_t *tcb = itrq->itrq_tcb_area;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi for (i = 0; i < itrq->itrq_tx_free_list_size; i++, tcb++) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_free_dma_buffer(&itrq->itrq_desc_area);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_tx_ring_size = i40e->i40e_tx_ring_size;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_tx_free_list_size = i40e->i40e_tx_ring_size +
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Allocate an additional tx descriptor for the writeback head.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi dmasz = sizeof (i40e_tx_desc_t) * itrq->itrq_tx_ring_size;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_alloc_dma_buffer(i40e, &itrq->itrq_desc_area,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi &i40e->i40e_static_dma_attr, &i40e->i40e_desc_acc_attr,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate DMA resources for tx "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi "descriptor ring");
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi (i40e_tx_desc_t *)(uintptr_t)itrq->itrq_desc_area.dmab_address;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_desc_wbhead = (uint32_t *)(itrq->itrq_desc_ring +
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_desc_free = itrq->itrq_tx_ring_size;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_tcb_work_list = kmem_zalloc(itrq->itrq_tx_ring_size *
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi sizeof (i40e_tx_control_block_t *), KM_NOSLEEP);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate a %d entry tx work list "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi "for ring %d", itrq->itrq_tx_ring_size, itrq->itrq_index);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_tcb_free_list = kmem_zalloc(itrq->itrq_tx_free_list_size *
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi sizeof (i40e_tx_control_block_t *), KM_SLEEP);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate a %d entry tx free list "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi "for ring %d", itrq->itrq_tx_free_list_size,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We allocate enough tx control blocks to cover the free list.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_tcb_area = kmem_zalloc(sizeof (i40e_tx_control_block_t) *
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate a %d entry tcb area for "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi "ring %d", itrq->itrq_tx_free_list_size, itrq->itrq_index);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * For each tcb, allocate DMA memory.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi for (i = 0; i < itrq->itrq_tx_free_list_size; i++, tcb++) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Allocate both a DMA buffer which we'll use for when we copy
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * packets for transmission and allocate a DMA handle which
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we'll use when we bind data.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi &i40e->i40e_txbind_dma_attr, DDI_DMA_DONTWAIT, NULL,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate DMA handle for tx "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi "data binding on ring %d: %d", itrq->itrq_index,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_alloc_dma_buffer(i40e, &tcb->tcb_dma,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi &i40e->i40e_static_dma_attr, &i40e->i40e_buf_acc_attr,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_error(i40e, "failed to allocate %ld bytes of "
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi "DMA for tx data binding on ring %d", dmasz,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_tcb_free = itrq->itrq_tx_free_list_size;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Free all memory associated with all of the rings on this i40e instance. Note,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * this is done as part of the GLDv3 stop routine.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_free_ring_mem(i40e_t *i40e, boolean_t failed_init)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi for (i = 0; i < i40e->i40e_num_trqpairs; i++) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_rx_data_t *rxd = i40e->i40e_trqpairs[i].itrq_rxdata;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Clean up our rx data. We have to free DMA resources first and
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * then if we have no more pending RCB's, then we'll go ahead
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * and clean things up. Note, we can't set the stopped flag on
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the rx data until after we've done the first pass of the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * pending resources. Otherwise we might race with
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * i40e_rx_recycle on determining who should free the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * i40e_rx_data_t above.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Allocate all of the resources associated with all of the rings on this i40e
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * instance. Note this is done as part of the GLDv3 start routine and thus we
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * should not use blocking allocations. This takes care of both DMA and non-DMA
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * related resources.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi for (i = 0; i < i40e->i40e_num_trqpairs; i++) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_alloc_rx_data(i40e, &i40e->i40e_trqpairs[i]) ==
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_alloc_rx_dma(i40e->i40e_trqpairs[i].itrq_rxdata) ==
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_alloc_tx_dma(&i40e->i40e_trqpairs[i]) == B_FALSE)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Because every instance of i40e may have different support for FMA
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * capabilities, we copy the DMA attributes into the i40e_t and set them that
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * way and use them for determining attributes.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_init_dma_attrs(i40e_t *i40e, boolean_t fma)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi bcopy(&i40e_g_static_dma_attr, &i40e->i40e_static_dma_attr,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi bcopy(&i40e_g_txbind_dma_attr, &i40e->i40e_txbind_dma_attr,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi bcopy(&i40e_g_desc_acc_attr, &i40e->i40e_desc_acc_attr,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi bcopy(&i40e_g_buf_acc_attr, &i40e->i40e_buf_acc_attr,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e->i40e_static_dma_attr.dma_attr_flags |= DDI_DMA_FLAGERR;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e->i40e_txbind_dma_attr.dma_attr_flags |= DDI_DMA_FLAGERR;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e->i40e_static_dma_attr.dma_attr_flags &= ~DDI_DMA_FLAGERR;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e->i40e_txbind_dma_attr.dma_attr_flags &= ~DDI_DMA_FLAGERR;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_rcb_free(i40e_rx_data_t *rxd, i40e_rx_control_block_t *rcb)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ASSERT(rxd->rxd_rcb_free < rxd->rxd_free_list_size);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ASSERT(rxd->rxd_free_list[rxd->rxd_rcb_free] == NULL);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rxd->rxd_free_list[rxd->rxd_rcb_free] = rcb;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rcb = rxd->rxd_free_list[rxd->rxd_rcb_free];
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rxd->rxd_free_list[rxd->rxd_rcb_free] = NULL;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * This is the callback that we get from the OS when freemsg(9F) has been called
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * on a loaned descriptor. In addition, if we take the last reference count
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * here, then we have to tear down all of the rx data.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi /* LINTED: E_BAD_PTR_CAST_ALIGN */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * It's possible for this to be called with a reference count of zero.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * That will happen when we're doing the freemsg after taking the last
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * reference because we're tearing down everything and this rcb is not
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * outstanding.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Don't worry about failure of desballoc here. It'll only become fatal
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * if we're trying to use it and we can't in i40e_rx_bind().
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rcb->rcb_mp = desballoc((unsigned char *)rcb->rcb_dma.dmab_address,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rcb->rcb_dma.dmab_size, 0, &rcb->rcb_free_rtn);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * It's possible that the rcb was being used while we are shutting down
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the device. In that case, we'll take the final reference from the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * device here.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * If this was the last block and it's been indicated that we've
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * passed the shutdown point, we should clean up.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (rxd->rxd_shutdown == B_TRUE && rxd->rxd_rcb_pending == 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_rx_bind(i40e_trqpair_t *itrq, i40e_rx_data_t *rxd, uint32_t index,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if ((rep_rcb = i40e_rcb_alloc(rxd)) == NULL) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_rx_bind_norcb.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Check to make sure we have a mblk_t. If we don't, this is our last
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * chance to try and get one.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi desballoc((unsigned char *)rcb->rcb_dma.dmab_address,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi rcb->rcb_dma.dmab_size, 0, &rcb->rcb_free_rtn);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_rx_bind_nomp.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi I40E_DMA_SYNC(&rcb->rcb_dma, DDI_DMA_SYNC_FORKERNEL);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_check_dma_handle(rcb->rcb_dma.dmab_dma_handle) != DDI_FM_OK) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ddi_fm_service_impact(i40e->i40e_dip, DDI_SERVICE_DEGRADED);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi atomic_or_32(&i40e->i40e_state, I40E_ERROR);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Note, we've already accounted for the I40E_BUF_IPHDR_ALIGNMENT.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We're going to allocate a new message block for this frame and attempt to
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * receive it. See the big theory statement for more information on when we copy
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * versus bind.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_rx_copy(i40e_trqpair_t *itrq, i40e_rx_data_t *rxd, uint32_t index,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi I40E_DMA_SYNC(&rcb->rcb_dma, DDI_DMA_SYNC_FORKERNEL);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_check_dma_handle(rcb->rcb_dma.dmab_dma_handle) != DDI_FM_OK) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ddi_fm_service_impact(i40e->i40e_dip, DDI_SERVICE_DEGRADED);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi atomic_or_32(&i40e->i40e_state, I40E_ERROR);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi mp = allocb(plen + I40E_BUF_IPHDR_ALIGNMENT, 0);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_rx_copy_nomem.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi bcopy(rcb->rcb_dma.dmab_address, mp->b_rptr, plen);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Determine if the device has enabled any checksum flags for us. The level of
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * checksum computed will depend on the type packet that we have, which is
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * contained in ptype. For example, the checksum logic it does will vary
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * depending on whether or not the packet is considered tunneled, whether it
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * recognizes the L4 type, etc. Section 8.3.4.3 summarizes which checksums are
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * While there are additional checksums that we could recognize here, we'll need
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * to get some additional GLDv3 enhancements to be able to properly describe
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_rx_hcksum(i40e_trqpair_t *itrq, mblk_t *mp, uint64_t status, uint32_t err,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * If the ptype isn't something that we know in the driver, then we
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * shouldn't even consider moving forward.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_hck_unknown.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * If hardware didn't set the L3L4P bit on the frame, then there is no
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * checksum offload to consider.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if ((status & (1 << I40E_RX_DESC_STATUS_L3L4P_SHIFT)) == 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_hck_nol3l4p.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The device tells us that IPv6 checksums where a Destination Options
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Header or a Routing header shouldn't be trusted. Discard all
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * checksums in this case.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (pinfo.outer_ip == I40E_RX_PTYPE_OUTER_IP &&
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi pinfo.outer_ip_ver == I40E_RX_PTYPE_OUTER_IPV6 &&
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi (status & (1 << I40E_RX_DESC_STATUS_IPV6EXADD_SHIFT))) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_hck_v6skip.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The hardware denotes three kinds of possible errors. Two are reserved
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * for inner and outer IP checksum errors (IPE and EIPE) and the latter
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * is for L4 checksum errors (L4E). If there is only one IP header, then
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the only thing that we care about is IPE. Note that since we don't
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * support inner checksums, we will ignore IPE being set on tunneled
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * packets and only care about EIPE.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (pinfo.outer_ip == I40E_RX_PTYPE_OUTER_IP &&
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi pinfo.outer_ip_ver == I40E_RX_PTYPE_OUTER_IPV4) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (pinfo.tunnel_type == I40E_RX_PTYPE_OUTER_NONE) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if ((err & (1 << I40E_RX_DESC_ERROR_IPE_SHIFT)) != 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_hck_iperr.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_hck_v4hdrok.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if ((err & (1 << I40E_RX_DESC_ERROR_EIPE_SHIFT)) != 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_hck_eiperr.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_hck_v4hdrok.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We only have meaningful L4 checksums in the case of IP->L4 and
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * IP->IP->L4. There is not outer L4 checksum data available in any
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * other case. Further, we don't bother reporting the valid checksum in
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the case of IP->IP->L4 set.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (pinfo.outer_ip == I40E_RX_PTYPE_OUTER_IP &&
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi pinfo.tunnel_type == I40E_RX_PTYPE_TUNNEL_NONE &&
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi (pinfo.inner_prot == I40E_RX_PTYPE_INNER_PROT_UDP ||
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi pinfo.inner_prot == I40E_RX_PTYPE_INNER_PROT_TCP ||
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi pinfo.inner_prot == I40E_RX_PTYPE_INNER_PROT_ICMP ||
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi pinfo.inner_prot == I40E_RX_PTYPE_INNER_PROT_SCTP)) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ASSERT(pinfo.payload_layer == I40E_RX_PTYPE_PAYLOAD_LAYER_PAY4);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if ((err & (1 << I40E_RX_DESC_ERROR_L4E_SHIFT)) != 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_hck_l4err.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_hck_l4hdrok.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_hck_set.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_hck_miss.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_ring_rx(i40e_trqpair_t *itrq, int poll_bytes)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Before we do anything else, we have to make sure that all of the DMA
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * buffers are synced up and then check to make sure that they're
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * actually good from an FM perspective.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi I40E_DMA_SYNC(&rxd->rxd_desc_area, DDI_DMA_SYNC_FORKERNEL);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_check_dma_handle(rxd->rxd_desc_area.dmab_dma_handle) !=
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ddi_fm_service_impact(i40e->i40e_dip, DDI_SERVICE_DEGRADED);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi atomic_or_32(&i40e->i40e_state, I40E_ERROR);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Prepare our stats. We do a limited amount of processing in both
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * polling and interrupt context. The limit in interrupt context is
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * based on frames, in polling context based on bytes.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * At this point, the descriptor ring is available to check. We'll try
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * and process until we either run out of poll_bytes or descriptors.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi stword = LE64_TO_CPU(cur_desc->wb.qword1.status_error_len);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Note, the primary invariant of this loop should be that cur_head,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * cur_desc, and stword always point to the currently processed
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * descriptor. When we leave the loop, it should point to a descriptor
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * that HAS NOT been processed. Meaning, that if we haven't consumed the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * frame, the descriptor should not be advanced.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi while ((stword & (1 << I40E_RX_DESC_STATUS_DD_SHIFT)) != 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The DD, PLEN, and EOP bits are the only ones that are valid
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * in every frame. The error information is only valid when EOP
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * is set in the same frame.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * At this time, because we don't do any LRO or header
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * splitting. We expect that every frame should have EOP set in
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * it. When later functionality comes in, we'll want to
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * re-evaluate this.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi eop = stword & (1 << I40E_RX_DESC_STATUS_EOF_SHIFT);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi error = (stword & I40E_RXD_QW1_ERROR_MASK) >>
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_rx_desc_error.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi plen = (stword & I40E_RXD_QW1_LENGTH_PBUF_MASK) >>
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ptype = (stword & I40E_RXD_QW1_PTYPE_MASK) >>
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * This packet contains valid data. We should check to see if
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we're actually going to consume it based on its length (to
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * ensure that we don't overshoot our quota). We determine
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * whether to bcopy or bind the DMA resources based on the size
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * of the frame. However, if on debug, we allow it to be
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * overridden for testing purposes.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We should be smarter about this and do DMA binding for
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * larger frames, but for now, it's really more important that
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we actually just get something simple working.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Ensure we don't exceed our polling quota by reading this
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * frame. Note we only bump bytes now, we bump frames later.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi mp = i40e_rx_bind(itrq, rxd, cur_head, plen);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi mp = i40e_rx_copy(itrq, rxd, cur_head, plen);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi i40e_rx_hcksum(itrq, mp, stword, error, ptype);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Now we need to prepare this frame for use again. See the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * discussion in the big theory statements.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * However, right now we're doing the simple version of this.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Normally what we'd do would depend on whether or not we were
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * doing DMA binding or bcopying. But because we're always doing
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * bcopying, we can just always use the current index as a key
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * for what to do and reassign the buffer based on the ring.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi CPU_TO_LE64((uintptr_t)rcb->rcb_dma.dmab_dma_address);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Finally, update our loop invariants.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi cur_head = i40e_next_desc(cur_head, 1, rxd->rxd_ring_size);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi stword = LE64_TO_CPU(cur_desc->wb.qword1.status_error_len);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * To help provide liveness, we limit the amount of data that
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we'll end up counting. Note that in these cases, an interrupt
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * is not dissimilar from a polling request.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (rx_frames > i40e->i40e_rx_limit_per_intr) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_rx_intr_limit.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * As we've modified the ring, we need to make sure that we sync the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * descriptor ring for the device. Next, we update the hardware and
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * update our notion of where the head for us to read from hardware is
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi I40E_DMA_SYNC(&rxd->rxd_desc_area, DDI_DMA_SYNC_FORDEV);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_check_dma_handle(rxd->rxd_desc_area.dmab_dma_handle) !=
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ddi_fm_service_impact(i40e->i40e_dip, DDI_SERVICE_DEGRADED);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi atomic_or_32(&i40e->i40e_state, I40E_ERROR);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ddi_acc_handle_t rh = i40e->i40e_osdep_space.ios_reg_handle;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi tail = i40e_prev_desc(cur_head, 1, rxd->rxd_ring_size);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi I40E_WRITE_REG(hw, I40E_QRX_TAIL(itrq->itrq_index), tail);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_check_acc_handle(rh) != DDI_FM_OK) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi atomic_or_32(&i40e->i40e_state, I40E_ERROR);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_bytes.value.ui64 += rx_bytes;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_rxstat.irxs_packets.value.ui64 += rx_frames;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * This function is called by the GLDv3 when it wants to poll on a ring. The
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * only primary difference from when we call this during an interrupt is that we
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * have a limit on the number of bytes that we should consume.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_ring_rx_poll(void *arg, int poll_bytes)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * This is a structure I wish someone would fill out for me for dorking with the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * checksums. When we get some more experience with this, we should go ahead and
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * consider adding this to MAC.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi uint8_t meoi_l2hlen; /* How long is the Ethernet header? */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi uint16_t meoi_l3proto; /* What's the Ethertype */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi uint8_t meoi_l3hlen; /* How long is the header? */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi uint8_t meoi_l4proto; /* What is the payload type? */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi uint8_t meoi_l4hlen; /* How long is the L4 header */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi mblk_t *meoi_l3ckmp; /* Which mblk has the l3 checksum */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi off_t meoi_l3ckoff; /* What's the offset to it */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi mblk_t *meoi_l4ckmp; /* Which mblk has the L4 checksum */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi off_t meoi_l4off; /* What is the offset to it? */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * This is something that we'd like to make a general MAC function. Before we do
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * that, we should add support for TSO.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We should really keep track of our offset and not walk everything every
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * time. I can't imagine that this will be kind to us at high packet rates;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * however, for the moment, let's leave that.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * This walks a message block chain without pulling up to fill in the context
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * information. Note that the data we care about could be hidden across more
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * than one mblk_t.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_meoi_get_uint8(mblk_t *mp, off_t off, uint8_t *out)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi /* Check for overflow */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_meoi_get_uint16(mblk_t *mp, off_t off, uint16_t *out)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi /* Check for overflow */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Data is in network order. Note the second byte of data might be in
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * the next mp.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchimac_ether_offload_info(mblk_t *mp, mac_ether_offload_info_t *meoi)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi bzero(meoi, sizeof (mac_ether_offload_info_t));
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi off = offsetof(struct ether_header, ether_type);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_meoi_get_uint16(mp, off, ðer) != 0)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi off = offsetof(struct ether_vlan_header, ether_type);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_meoi_get_uint16(mp, off, ðer) != 0)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi maclen = sizeof (struct ether_vlan_header);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * For IPv4 we need to get the length of the header, as it can
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * be variable.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi off = offsetof(ipha_t, ipha_version_and_hdr_length) + maclen;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_meoi_get_uint8(mp, off, &iplen) != 0)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi off = offsetof(ipha_t, ipha_protocol) + maclen;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_meoi_get_uint8(mp, off, &ipproto) == -1)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_meoi_get_uint8(mp, off, &ipproto) == -1)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi off = offsetof(tcph_t, th_offset_and_rsrvd) + maclen + iplen;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_meoi_get_uint8(mp, off, &l4len) == -1)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Attempt to put togther the information we'll need to feed into a descriptor
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * to properly program the hardware for checksum offload as well as the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * generally required flags.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * The i40e_tx_context_t`itc_cmdflags contains the set of flags we need to or
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * into the descriptor based on the checksum flags for this mblk_t and the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * actual information we care about.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_tx_context(i40e_t *i40e, i40e_trqpair_t *itrq, mblk_t *mp,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi mac_hcksum_get(mp, &start, NULL, NULL, NULL, &flags);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if ((ret = mac_ether_offload_info(mp, &meo)) != 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Have we been asked to checksum an IPv4 header. If so, verify that we
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * have sufficient information and then set the proper fields in the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * command structure.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if ((meo.meoi_flags & MEOI_L2INFO_SET) == 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if ((meo.meoi_flags & MEOI_L3INFO_SET) == 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi tctx->itc_cmdflags |= I40E_TX_DESC_CMD_IIPT_IPV4_CSUM;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi tctx->itc_offsets |= (meo.meoi_l2hlen >> 1) <<
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi tctx->itc_offsets |= (meo.meoi_l3hlen >> 2) <<
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We've been asked to provide an L4 header, first, set up the IP
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * information in the descriptor if we haven't already before moving
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * onto seeing if we have enough information for the L4 checksum
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if ((meo.meoi_flags & MEOI_L4INFO_SET) == 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if ((meo.meoi_flags & MEOI_L2INFO_SET) == 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if ((meo.meoi_flags & MEOI_L3INFO_SET) == 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi } else if (meo.meoi_l3proto == ETHERTYPE_IPV6) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi tctx->itc_offsets |= (meo.meoi_l2hlen >> 1) <<
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi tctx->itc_offsets |= (meo.meoi_l3hlen >> 2) <<
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi tctx->itc_cmdflags |= I40E_TX_DESC_CMD_L4T_EOFT_TCP;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi tctx->itc_cmdflags |= I40E_TX_DESC_CMD_L4T_EOFT_UDP;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi tctx->itc_cmdflags |= I40E_TX_DESC_CMD_L4T_EOFT_SCTP;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi tctx->itc_offsets |= (meo.meoi_l4hlen >> 2) <<
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_tcb_free(i40e_trqpair_t *itrq, i40e_tx_control_block_t *tcb)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ASSERT(itrq->itrq_tcb_free < itrq->itrq_tx_free_list_size);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_tcb_free_list[itrq->itrq_tcb_free] = tcb;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ret = itrq->itrq_tcb_free_list[itrq->itrq_tcb_free];
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_tcb_free_list[itrq->itrq_tcb_free] = NULL;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * This should be used to free any DMA resources, associated mblk_t's, etc. It's
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * used as part of recycling the message blocks when we have either an interrupt
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * or other activity that indicates that we need to take a look.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchii40e_tcb_reset(i40e_tx_control_block_t *tcb)
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi (void) ddi_dma_unbind_handle(tcb->tcb_dma_handle);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi /* Cast to pacify lint */
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi panic("trying to free tcb %p with bad type none", (void *)tcb);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi panic("unknown i40e tcb type: %d", tcb->tcb_type);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * This is called as part of shutting down to clean up all outstanding
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * descriptors. Similar to recycle, except we don't re-arm anything and instead
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * just return control blocks to the free list.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ASSERT(itrq->itrq_desc_free <= itrq->itrq_tx_ring_size);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Because we should have shut down the chip at this point, it should be
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * safe to just clean up all the entries between our head and tail.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi index = I40E_READ_REG(&itrq->itrq_i40e->i40e_hw_space,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi VERIFY0(index & (I40E_QTX_ENA_QENA_REQ_MASK |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi while (itrq->itrq_desc_free < itrq->itrq_tx_ring_size) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi bzero(&itrq->itrq_desc_ring[index], sizeof (i40e_tx_desc_t));
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi index = i40e_next_desc(index, 1, itrq->itrq_tx_ring_size);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We're here either by hook or by crook. We need to see if there are transmit
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * descriptors available for us to go and clean up and return to the hardware.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We may also be blocked, and if so, we should make sure that we let it know
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we're good to go.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ASSERT(itrq->itrq_desc_free <= itrq->itrq_tx_ring_size);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (itrq->itrq_desc_free == itrq->itrq_tx_ring_size) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_txstat.itxs_num_unblocked.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Now we need to try and see if there's anything available. The driver
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * will write to the head location and it guarantees that it does not
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * use relaxed ordering.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi VERIFY0(ddi_dma_sync(itrq->itrq_desc_area.dmab_dma_handle,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi sizeof (uint32_t), DDI_DMA_SYNC_FORKERNEL));
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_check_dma_handle(itrq->itrq_desc_area.dmab_dma_handle) !=
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ddi_fm_service_impact(i40e->i40e_dip, DDI_SERVICE_DEGRADED);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi atomic_or_32(&i40e->i40e_state, I40E_ERROR);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We zero this out for sanity purposes.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi bzero(&itrq->itrq_desc_ring[toclean], sizeof (i40e_tx_desc_t));
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi toclean = i40e_next_desc(toclean, 1, itrq->itrq_tx_ring_size);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_txstat.itxs_recycled.value.ui64 += count;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ASSERT(itrq->itrq_desc_free <= itrq->itrq_tx_ring_size);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_desc_free > i40e->i40e_tx_block_thresh) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi mac_tx_ring_update(i40e->i40e_mac_hdl, itrq->itrq_mactxring);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_txstat.itxs_num_unblocked.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Now clean up the tcb.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi DTRACE_PROBE2(i40e__recycle, i40e_trqpair_t *, itrq, uint32_t, count);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We've been asked to send a message block on the wire. We'll only have a
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * single chain. There will not be any b_next pointers; however, there may be
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * multiple b_cont blocks.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We may do one of three things with any given mblk_t chain:
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * 2) Transmit it
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * 3) Return it
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * If we return it to MAC, then MAC will flow control on our behalf. In other
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * words, it won't send us anything until we tell it that it's okay to send us
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi (i40e->i40e_link_state != LINK_STATE_UP)) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Figure out the relevant context about this frame that we might need
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * for enabling checksum, lso, etc. This also fills in information that
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we might set around the packet type, etc.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_tx_context(i40e, itrq, mp, &tctx) < 0) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_txstat.itxs_err_context.value.ui64++;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * For the primordial driver we can punt on doing any recycling right
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * now; however, longer term we need to probably do some more pro-active
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * recycling to cut back on stalls in the tx path.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Do a quick size check to make sure it fits into what we think it
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * should for this device. Note that longer term this will be false,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * particularly when we have the world of TSO.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi for (nmp = mp; nmp != NULL; nmp = nmp->b_cont) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * First we allocate our tx control block and prepare the packet for
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * transmit before we do a final check for descriptors. We do it this
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * way to minimize the time under the tx lock.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * For transmitting a block, we're currently going to use just a
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * single control block and bcopy all of the fragments into it. We
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * should be more intelligent about doing DMA binding or otherwise, but
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * for getting off the ground this will have to do.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi for (nmp = mp; nmp != NULL; nmp = nmp->b_cont) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi void *coff = tcb->tcb_dma.dmab_address + tcb->tcb_dma.dmab_len;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * While there's really no need to keep the mp here, but let's just do
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * it to help with our own debugging for now.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi I40E_DMA_SYNC(&tcb->tcb_dma, DDI_DMA_SYNC_FORDEV);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (itrq->itrq_desc_free < i40e->i40e_tx_block_thresh) {
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Build up the descriptor and send it out. Thankfully at the moment
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * we only need a single desc, because we're not doing anything fancy
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi txdesc = &itrq->itrq_desc_ring[itrq->itrq_desc_tail];
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_tcb_work_list[itrq->itrq_desc_tail] = tcb;
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi itrq->itrq_desc_tail = i40e_next_desc(itrq->itrq_desc_tail, 1,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Note, we always set EOP and RS which indicates that this is the last
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * data frame and that we should ask for it to be transmitted. We also
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * must always set ICRC, because that is an internal bit that must be
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * set to one for data descriptors. The remaining bits in the command
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * descriptor depend on checksumming and are determined based on the
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * information set up in i40e_tx_context().
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi CPU_TO_LE64((uintptr_t)tcb->tcb_dma.dmab_dma_address);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi txdesc->cmd_type_offset_bsz = CPU_TO_LE64(((uint64_t)type |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ((uint64_t)tctx.itc_offsets << I40E_TXD_QW1_OFFSET_SHIFT) |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ((uint64_t)cmd << I40E_TXD_QW1_CMD_SHIFT) |
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ((uint64_t)tcb->tcb_dma.dmab_len << I40E_TXD_QW1_TX_BUF_SZ_SHIFT)));
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Now, finally, sync the DMA data and alert hardware.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi I40E_DMA_SYNC(&itrq->itrq_desc_area, DDI_DMA_SYNC_FORDEV);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi I40E_WRITE_REG(hw, I40E_QTX_TAIL(itrq->itrq_index),
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi if (i40e_check_acc_handle(i40e->i40e_osdep_space.ios_reg_handle) !=
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Note, we can't really go through and clean this up very well,
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * because the memory has been given to the device, so just
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * indicate it's been transmitted.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi ddi_fm_service_impact(i40e->i40e_dip, DDI_SERVICE_DEGRADED);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi atomic_or_32(&i40e->i40e_state, I40E_ERROR);
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * We ran out of resources. Return it to MAC and indicate that we'll
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * need to signal MAC. If there are allocated tcb's, return them now.
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * Make sure to reset their message block's, since we'll return them
9d26e4fc021e249c93c2861629cc665e4f5bd4d6Robert Mustacchi * back to MAC.