1450N/A * Copyright (c) 2006, 2013, Oracle and/or its affiliates. All rights reserved. 1450N/A * Copyright (c) 2009, 2013, Intel Corporation. 1450N/A * Permission is hereby granted, free of charge, to any person obtaining a 1450N/A * copy of this software and associated documentation files (the "Software"), 1450N/A * to deal in the Software without restriction, including without limitation 1450N/A * the rights to use, copy, modify, merge, publish, distribute, sublicense, 1450N/A * and/or sell copies of the Software, and to permit persons to whom the 1450N/A * Software is furnished to do so, subject to the following conditions: 1450N/A * The above copyright notice and this permission notice (including the next 1450N/A * paragraph) shall be included in all copies or substantial portions of the 1450N/A * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 1450N/A * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 1450N/A * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL 1450N/A * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 1450N/A * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 1450N/A * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 1450N/A * Eric Anholt <eric@anholt.net> 1450N/A /* As we do not have an associated fence register, we will force 1450N/A * a tiling change if we ever need to acquire one. 1450N/A * Only wait 10 seconds for the gpu reset to complete to avoid hanging 1450N/A * userspace. If it takes that long something really bad is going on and 1450N/A * we should simply try to bail out and fail as gracefully as possible. 1450N/A /* fix me mutex_lock_interruptible */ 1450N/A /* GEM with user mode setting was never supported on ilk and later. */ 1450N/A /* Allocate the new object */ 1450N/A /* drop reference from allocate - handle holds it now */ 1450N/A * Creates a new mm object and returns a handle to it. 1450N/A /* Use the unswizzled path if this page isn't affected. */ 1450N/A /* Copy the data, XORing A6 with A17 (1). The user already knows he's 1450N/A * XORing with the other bits (A9 for Y, A9 and A10 for X) 1450N/A /* If we're not in the cpu read domain, set ourself into the gtt 1450N/A * read domain and manually flush cachelines (if required). This 1450N/A * optimizes for the case when the gpu will dirty the data 1450N/A * anyway again before the next pread happens. */ 1450N/A * shmem_page_index = page number within shmem file 1450N/A * shmem_page_offset = offset within page in shmem file 1450N/A * data_page_index = page number in get_user_pages return 1450N/A * data_page_offset = offset with data_page_index page. 1450N/A * page_length = bytes to copy for this page 1450N/A * Reads data from the object referenced by handle. 1450N/A * On error, the contents of *data are undefined. 1450N/A /* Pin the user pages containing the data. We can't fault while 1450N/A * holding the struct mutex, and all of the pwrite implementations 1450N/A * want to hold it while dereferencing the user data. 1450N/A /* If we're not in the cpu write domain, set ourself into the gtt 1450N/A * write domain and manually flush cachelines (if required). This 1450N/A * optimizes for the case when the gpu will use the data 1450N/A * right away and we therefore have to clflush anyway. */ 1450N/A /* Same trick applies for invalidate partially written cachelines before 1450N/A * shmem_page_index = page number within shmem file 1450N/A * shmem_page_offset = offset within page in shmem file 1450N/A * data_page_index = page number in get_user_pages return 1450N/A * data_page_offset = offset with data_page_index page. 1450N/A * page_length = bytes to copy for this page 1450N/A * Writes data to the object referenced by handle. 1450N/A * On error, the contents of the buffer that were to be modified are undefined. 1450N/A /* Bounds check destination. */ 1450N/A /* We can only do the GTT pwrite on untiled buffers, as otherwise 1450N/A * it would end up going through the fenced access, and we'll get 1450N/A * different detiling behavior between reading and writing. 1450N/A * perspective, requiring manual detiling by the client. 1450N/A /* Note that the gtt paths might fail with non-page-backed user 1450N/A * pointers (e.g. gtt mappings when moving data between 1450N/A * textures). Fallback to the shmem path in that case. */ 1450N/A /* Flushing cursor object */ 1450N/A /* Non-interruptible callers can't handle -EAGAIN, hence return 1450N/A * -EIO unconditionally for these. */ 1450N/A /* Recovery complete, but the reset failed ... */ 1450N/A * Compare seqno against outstanding lazy request. Emit a request if they are 1450N/A * __wait_seqno - wait until execution of seqno has finished 1450N/A * @ring: the ring expected to report seqno 1450N/A * @reset_counter: reset sequence associated with the given seqno 1450N/A * @interruptible: do an interruptible wait (normally yes) 1450N/A * @timeout: in - how long to wait (NULL forever); out - how much time remaining 1450N/A * Note: It is of utmost importance that the passed in seqno and reset_counter 1450N/A * values have been read by the caller in an smp safe manner. Where read-side 1450N/A * locks are involved, it is sufficient to read the reset_counter before 1450N/A * unlocking the lock that protects the seqno. For lockless tricks, the 1450N/A * reset_counter _must_ be read before, and an appropriate smp_rmb must be 1450N/A * Returns 0 if the seqno was found within the alloted time. Else returns the 1450N/A * errno with remaining time filled in timeout argument. 1450N/A /* busy check is faster than cv wait on gen6+ */ 1450N/A * Frequently read CS register may cause my GEN7 platform hang, 1450N/A * but it's crucial for missed IRQ issue. 1450N/A * So the first wait busy check the seqno, 1450N/A * the second wait force correct ordering 1450N/A * between irq and seqno writes then check again. 1450N/A /* We need to check whether any gpu reset happened in between 1450N/A * the caller grabbing the seqno and now ... */ 1450N/A /* ... but upgrade the -EGAIN to an -EIO if the gpu is truely 1450N/A * Waits for a sequence number to be signaled, and cleans up the 1450N/A * request and object lists appropriately for that event. 1450N/A /* Manually manage the write flush as we may have not yet 1450N/A * Note that the last_write_seqno is always the earlier of 1450N/A * we know we have passed the last write. 1450N/A * Ensures that all rendering to the object has completed and the object is 1450N/A * safe to unbind from the GTT or access from the CPU. 1450N/A/* A nonblocking variant of the above wait. This is a highly dangerous routine 1450N/A * as the object state may change during this call. 1450N/A * Called when user space prepares to use an object with the CPU, either 1450N/A * through the mmap ioctl's mapping or a GTT mapping. 1450N/A /* Only handle setting domains to types used by the CPU. */ 1450N/A /* Having something in the write domain implies it's in the read 1450N/A * domain, and only that read domain. Enforce that in the request. 1450N/A /* Try to flush the object off the GPU without holding the lock. 1450N/A * We will repeat the flush holding the lock in the normal manner 1450N/A * to catch cases where we are gazumped. 1450N/A /* Silently promote "you're not bound, there was nothing to do" 1450N/A * to success, since the client was just asking us to 1450N/A * make sure everything was done. 1450N/A * Called when user space has done writes to this buffer 1450N/A /* Pinned buffers may be scanout, so flush the cache */ 1450N/A * Maps the contents of an object, returning the address it is mapped 1450N/A * While the mapping holds a reference on the contents of the object, it doesn't 1450N/A * imply a ref on the object itself. 1450N/A /* prime objects have no backing filp to GEM mmap 1450N/A /* Now bind it into the GTT if needed */ 1450N/A /* Access to snoopable pages through the GTT is incoherent. */ 1450N/A /* Finally, remap it using the new GTT offset */ 1450N/A * i915_gem_create_mmap_offset - create a fake mmap offset for an object 1450N/A * GEM memory mapping works by handing back to userspace a fake mmap offset 1450N/A * it can use in a subsequent mmap(2) call. The DRM core code then looks 1450N/A * up the object based on the offset and sets up the various memory mapping 1450N/A * This routine allocates and attaches a fake offset for @obj. 1450N/A /* user_token is the fake offset 1450N/A * which create in drm_map_handle at alloc time 1450N/A * i915_gem_release_mmap - remove physical page mappings 1450N/A * Preserve the reservation of the mmaping with the DRM core code, but 1450N/A * relinquish ownership of the pages back to the system. 1450N/A * It is vital that we remove the page mapping if we have mapped a tiled 1450N/A * object through the GTT and then lose the fence register due to 1450N/A * resource pressure. Similarly if the object has been moved out of the 1450N/A * aperture, than pages mapped into userspace must be revoked. Removing the 1450N/A * mapping will then trigger a page fault on the next user access, allowing 1450N/A * fixup by i915_gem_fault(). 1450N/A /* Previous chips need a power-of-two fence region when tiling */ 1450N/A * i915_gem_get_gtt_alignment - return required GTT alignment for an object 1450N/A * Return the required GTT alignment for an object, taking into account 1450N/A * potential fence register mapping if needed. 1450N/A * Minimum alignment is 4k (GTT page size), but might be greater 1450N/A * if a fence register is needed for the object. 1450N/A * Previous chips need to be aligned to the size of the smallest 1450N/A * fence register that can contain the object. 1450N/A * i915_gem_mmap_gtt_ioctl - prepare an object for GTT mmap'ing 1450N/A * @data: GTT mapping ioctl data 1450N/A * @file_priv: GEM object info 1450N/A * Simply returns the fake offset to userspace so it can mmap it. 1450N/A * The mmap call will end up in drm_gem_mmap(), which will set things 1450N/A * up so we can get faults in the handler above. 1450N/A * The fault handler will take care of binding the object into the GTT 1450N/A * (since it may have been evicted to make room for something), allocating 1450N/A * a fence register, and mapping the appropriate aperture address into 1450N/A /* In the event of a disaster, abandon all caches and 1450N/A/* Ensure that the associated pages are gathered from the backing storage 1450N/A * and pinned into our object. i915_gem_object_get_pages() may be called 1450N/A * multiple times before they are released by a single call to 1450N/A * i915_gem_object_put_pages() - once the pages are no longer referenced 1450N/A * either as a result of memory pressure (reaping pages under the shrinker) 1450N/A * or as the object is itself released. 1450N/A /* Keep the seqno relative to the current ring */ 1450N/A /* Add a reference if we're newly entering the active list. */ 1450N/A /* Move from whatever list we were on to the tail of execution. */ 1450N/A /* Bump MRU to take account of the delayed flush */ 1450N/A /* Carefully retire all requests without writing to the rings */ 1450N/A /* Finally reset hw state */ 1450N/A /* HWS page needs to be set less than what we 1450N/A /* Carefully set the last_seqno value so that wrap 1450N/A /* reserve 0 for non-seqno */ 1450N/A * Emit any outstanding flushes - execbuf can fail to emit the flush 1450N/A * after having emitted the batchbuffer command. Hence we need to fix 1450N/A * things up similar to emitting the lazy request. The difference here 1450N/A * is that the flush _must_ happen before the next request, no matter 1450N/A /* Record the position of the start of the request so that 1450N/A * should we detect the updated seqno part-way through the 1450N/A * GPU processing the request, we never over-estimate the 1450N/A /* Whilst this request exists, batch_obj will be on the 1450N/A * active_list, and so will hold the active reference. Only when this 1450N/A * request is retired will the the batch_obj be moved onto the 1450N/A * inactive_list and lose its active reference. Hence we do not need 1450N/A * to explicitly hold another reference here. 1450N/A /* change to delay HZ and then run work (not insert to workqueue of Linux) */ 1450N/A /* There is a possibility that unmasked head address 1450N/A * pointing inside the ring, matches the batch_obj address range. 1450N/A * However this is extremely unlikely. 1450N/A /* Innocent until proven guilty */ 1450N/A /* If contexts are disabled or this is the default context, use 1450N/A * Commit delayed tiling changes if we have an object still 1450N/A * attached to the fence, otherwise just clear the fence. 1450N/A /* Move everything out of the GPU domains to ensure we do any 1450N/A * necessary invalidation upon reuse. 1450N/A * This function clears the request list as sequence numbers are passed. 1450N/A /* We know the GPU must have read the request to have 1450N/A * sent us the seqno + interrupt, so use the position 1450N/A * of tail of the request to update the last known position 1450N/A /* Move any buffers on the active list that are no longer referenced 1450N/A /* Come back later if the device is busy... */ 1450N/A /* Send a periodic flush down the ring so we don't hold onto GEM 1450N/A * Ensures that an object will eventually get non-busy by flushing any required 1450N/A * write domains, emitting any outstanding lazy request and retiring and 1450N/A * i915_gem_wait_ioctl - implements DRM_IOCTL_I915_GEM_WAIT 1450N/A * @DRM_IOCTL_ARGS: standard ioctl arguments 1450N/A * Returns 0 if successful, else an error is returned with the remaining time in 1450N/A * -ETIME: object is still busy after timeout 1450N/A * -ERESTARTSYS: signal interrupted the wait 1450N/A * -ENONENT: object doesn't exist 1450N/A * -ENODEV: Internal IRQ fail 1450N/A * -E?: The add request failed 1450N/A * The wait ioctl with a timeout of 0 reimplements the busy ioctl. With any 1450N/A * non-zero timeout parameter the wait ioctl will wait for the given number of 1450N/A * nanoseconds on an object becoming unbusy. Since the wait itself does so 1450N/A * without holding struct_mutex the object may become re-busied before this 1450N/A * function completes. A similar but shorter * race condition exists in the busy 1450N/A /* Need to make sure the object gets inactive eventually. */ 1450N/A /* Do this after OLR check to make sure we make forward progress polling 1450N/A * on this IOCTL with a 0 timeout (like busy ioctl) 1450N/A * i915_gem_object_sync - sync an object to a ring. 1450N/A * @obj: object which may be in use on another ring. 1450N/A * @to: ring we wish to use the object on. May be NULL. 1450N/A * This code is meant to abstract object synchronization with the GPU. 1450N/A * Calling with NULL implies synchronizing the object with the CPU 1450N/A * rather than a particular GPU ring. 1450N/A * Returns 0 if successful, else propagates up the lower layer error. 1450N/A /* We use last_read_seqno because sync_to() 1450N/A * might have just caused seqno wrap under 1450N/A /* Force a pagefault for domain tracking on next user access */ 1450N/A * Unbinds an object from the GTT aperture. 1450N/A /* Continue on if we fail due to EIO, the GPU is hung so we 1450N/A * should be safe and we need to cleanup or else we might 1450N/A * cause memory corruption through use-after-free. 1450N/A /* release the fence reg _after_ flushing */ 1450N/A /* Avoid an unnecessary call to unbind on rebind. */ 1450N/A /* Flush everything onto the inactive list. */ 1450N/A /* To w/a incoherency with non-atomic 64-bit register updates, 1450N/A * we split the 64-bit update into two 32-bit writes. In order 1450N/A * for a partial fence not to be evaluated between writes, we 1450N/A * precede the update with write to turn off the fence register, 1450N/A * and only enable the fence as the last step. 1450N/A * For extra levels of paranoia, we make sure each step lands 1450N/A * before applying the next step. 1450N/A DRM_ERROR(
"object 0x%08x [fenceable? %d] not 1M or pot-size (0x%08x) aligned\n",
1450N/A /* Note: pitch better be a power of two tile widths */ 1450N/A /* Ensure that all CPU reads are completed before installing a fence 1450N/A * and all writes before removing the fence. 1450N/A /* And similarly be paranoid that no direct access to this region 1450N/A * is reordered to before the fence is installed. 1450N/A /* First try to find a free reg */ 1450N/A /* None available, try to steal one or wait for a user to finish */ 1450N/A * i915_gem_object_get_fence_reg - set up a fence reg for an object 1450N/A * @obj: object to map through a fence reg 1450N/A * When mapping objects through the GTT, userspace wants to be able to write 1450N/A * to them without having to worry about swizzling if the object is tiled. 1450N/A * This function walks the fence regs looking for a free one for @obj, 1450N/A * stealing one if it can't find any. 1450N/A * It then sets up the reg based on the object's properties: address, pitch 1450N/A * For an untiled surface, this removes any existing fence. 1450N/A /* Have we updated the tiling parameters upon the object and so 1450N/A * will need to serialise the write to the associated fence register? 1450N/A /* Just update our place in the LRU if our fence is getting reused. */ 1450N/A /* On non-LLC machines we have to be careful when putting differing 1450N/A * types of snoopable memory together to avoid the prefetcher 1450N/A * crossing memory domains and dieing. 1450N/A DRM_ERROR(
"object reserved space [%08lx, %08lx] with wrong color, cache_level=%x, color=%lx\n",
1450N/A * Finds free space in the GTT aperture and binds the object there. 1450N/A /* If the object is bigger than the entire aperture, reject it early 1450N/A * before evicting everything in a vain attempt to find space. 1450N/A DRM_ERROR(
"Attempting to bind an object larger than the aperture: object=%zd > %s aperture=%zu\n",
1450N/A /* If we don't have a page list set up, then we're not pinned 1450N/A * to GPU, and we can ignore the cache flush because it'll happen 1450N/A * Stolen memory is always coherent with the GPU as it is explicitly 1450N/A * marked as wc by the system, or the system is cache-coherent. 1450N/A /* If the GPU is snooping the contents of the CPU cache, 1450N/A * we do not need to manually clear the CPU cache lines. However, 1450N/A * the caches are only snooped when the render cache is 1450N/A * and flushes when moving into and out of the RENDER domain, correct 1450N/A * snooping behaviour occurs naturally as the result of our domain 1450N/A/** Flushes the GTT write domain for the object if it's dirty. */ 1450N/A /* No actual flushing is required for the GTT write domain. Writes 1450N/A * to it immediately go to main memory as far as we know, so there's 1450N/A * no chipset flush. It also doesn't land in render cache. 1450N/A * However, we do have to enforce the order so that all writes through 1450N/A * the GTT land before any writes to the device, such as updates to 1450N/A/** Flushes the CPU write domain for the object if it's dirty. */ 1450N/A * Moves a single object to the GTT read, and possibly write domain. 1450N/A * This function returns when the move is complete, including waiting on 1450N/A /* Not valid to be called on unbound objects. */ 1450N/A /* Serialise direct access to this object with the barriers for 1450N/A * coherent writes from the GPU, by effectively invalidating the 1450N/A * GTT domain upon first access. 1450N/A /* It should now be out of any other write domains, and we can update 1450N/A * the domain values for our changes. 1450N/A /* GPU reset can handle this error */ 1450N/A// BUG_ON((obj->base.write_domain & ~I915_GEM_DOMAIN_GTT) != 0); 1450N/A /* And bump the LRU for this access */ 1450N/A /* Before SandyBridge, you could not use tiling or fence 1450N/A * registers with snooped memory, so relinquish any fences 1450N/A * currently pointing to our region in the aperture. 1450N/A /* If we're coming from LLC cached, then we haven't 1450N/A * actually been tracking whether the data is in the 1450N/A * CPU cache or not, since we only allow one bit set 1450N/A * in obj->write_domain and have been skipping the clflushes. 1450N/A * Just set it to the CPU cache for now. 1450N/A * Prepare buffer for display plane (scanout, cursors, etc). 1450N/A * Can be called from an uninterruptible phase (modesetting) and allows 1450N/A * any flushes to be pipelined (for pageflips). 1450N/A /* The display engine is not coherent with the LLC cache on gen6. As 1450N/A * a result, we make sure that the pinning that is about to occur is 1450N/A * done with uncached PTEs. This is lowest common denominator for all 1450N/A * However for gen6+, we could do better by using the GFDT bit instead 1450N/A * of uncaching, which would allow us to flush all the LLC-cached data 1450N/A * with that bit in the PTE to main memory with just one PIPE_CONTROL. 1450N/A /* As the user may map the buffer once pinned in the display plane 1450N/A * (e.g. libkms for the bootup splash), we have to ensure that we 1450N/A * always use map_and_fenceable for all scanout buffers. 1450N/A /* It should now be out of any other write domains, and we can update 1450N/A * the domain values for our changes. 1450N/A /* Ensure that we invalidate the GPU's caches and TLBs. */ 1450N/A * Moves a single object to the CPU read, and possibly write domain. 1450N/A * This function returns when the move is complete, including waiting on 1450N/A /* Flush the CPU cache if it's still invalid. */ 1450N/A /* It should now be out of any other write domains, and we can update 1450N/A * the domain values for our changes. 1450N/A /* If we're writing through the CPU, then the GPU read domains will 1450N/A * need to be invalidated at next use. 1450N/A/* Throttle our rendering by waiting until the ring has completed our requests 1450N/A * emitted over 20 msec ago. 1450N/A * Note that if we were to use the current jiffies each time around the loop, 1450N/A * we wouldn't escape the function with any frames outstanding if the time to 1450N/A * render a frame was over 20ms. 1450N/A * This should get us reasonable parallelism between CPU and GPU but also 1450N/A * relatively low latency when blocking on a particular request to finish. 1450N/A " offset=%x, req.alignment=%x, req.map_and_fenceable=%d," 1450N/A " obj->map_and_fenceable=%d\n",
1450N/A /* XXX - flush the CPU caches for pinned objects 1450N/A * as the X server doesn't manage domains yet 1450N/A /* Count all active objects as busy, even if they are currently not used 1450N/A * by the gpu. Users of this interface expect objects to eventually 1450N/A * become non-busy without any further actions, therefore emit any 1450N/A /* Don't enable buffer catch */ 1450N/A /* Avoid an unnecessary call to unbind on the first bind. */ 1450N/A /* On Gen6, we can have the GPU use the LLC (the CPU 1450N/A * cache) for about a 10% performance improvement 1450N/A * compared to uncached. Graphics requests other than 1450N/A * display scanout are coherent with the CPU in 1450N/A * accessing this cache. This means in this mode we 1450N/A * don't need to clflush on the CPU side, and on the 1450N/A * GPU side we only need to flush internal caches to 1450N/A * get data visible to the CPU. 1450N/A * However, we maintain the display planes as UC, and so 1450N/A * need to rebind when first used as such. 1450N/A /* Stolen objects don't hold a ref, but do hold pin count. Fix that up 1450N/A// if (obj->base.import_attach) 1450N/A// drm_prime_gem_destroy(&obj->base, NULL); 1450N/A /* Under UMS, be paranoid and evict. */ 1450N/A /* Hack! Don't let anybody do execbuf while we don't control the chip. 1450N/A * We need to replace this with a semaphore, or something. 1450N/A /* Cancel the retire work handler, wait for it to finish if running 1450N/A /* Make sure all the writes land before disabling dop clock gating */ 1450N/A * XXX: There was some w/a described somewhere suggesting loading 1450N/A DRM_INFO(
"PPGTT enable failed. This is not fatal, but unexpected\n");
1450N/A /* VLVA0 (potential hack), BIOS isn't actually waking us */ 1450N/A * Some BIOSes fail to initialise the GTT, which will cause DMA faults when 1450N/A * the IOMMU is enabled. We need to clear the whole GTT. 1450N/A /* workaround: prealloc fb buffer, make sure the start address 0 */ 1450N/A /* copy old content to fb buffer */ 1450N/A /* Flush everything out, we'll be doing GTT only from now on */ 1450N/A /* Allow hardware batchbuffers unless told otherwise, but not for KMS. */ 1450N/A /* On GEN3 we really need to make sure the ARB C3 LP bit is set */ 1450N/A /* Old X drivers will take 0-2 for front, back, depth buffers */ 1450N/A /* Initialize fence registers to zero */ 1450N/A * Create a physically contiguous memory object for this object 1450N/A * e.g. for cursor + overlay regs 1450N/A /* i915_gpu_idle() generates warning message, so just ignore return */ 1450N/A /* Clean up our request list when the client is going away, so that 1450N/A * later retire_requests won't dereference our soon-to-be-gone