page.h revision e44bd21c11b4f4cc72e87048602add8046a3d30d
* p_iolock This is a binary semaphore lock that provides * exclusive access to the i/o list links in each * page structure. It is always held while the page * is on an i/o list (i.e., involved in i/o). That is, * even though a page may be only `shared' locked * while it is doing a write, the following fields may * change anyway. Normally, the page must be * `exclusively' locked to change anything in it. * The following fields are protected by the global page_llock: * The following lists are protected by the global page_freelock: * The following, for our purposes, are protected by * the global freemem_lock: * The following fields are protected by hat layer lock(s). When a page * structure is not mapped and is not associated with a vnode (after a call * to page_hashout() for example) the p_nrm field may be modified with out * holding the hat layer lock: * The following field is file system dependent. How it is used and * the locking strategies applied are up to the individual file system * The page structure is used to represent and control the system's * physical pages. There is one instance of the structure for each * page that is not permenately allocated. For example, the pages that * hold the page structures are permanently held by the kernel * and hence do not need page structures to track them. The array * of page structures is allocated early on in the kernel's life and * is based on the amount of available physical memory. * Each page structure may simultaneously appear on several linked lists. * The lists are: hash list, free or in i/o list, and a vnode's page list. * Each type of list is protected by a different group of mutexes as described * The hash list is used to quickly find a page when the page's vnode and * offset within the vnode are known. Each page that is hashed is * connected via the `p_hash' field. The anchor for each hash is in the * array `page_hash'. An array of mutexes, `ph_mutex', protects the * lists anchored by page_hash[]. To either search or modify a given hash * list, the appropriate mutex in the ph_mutex array must be held. * The free list contains pages that are `free to be given away'. For * efficiency reasons, pages on this list are placed in two catagories: * pages that are still associated with a vnode, and pages that are not * associated with a vnode. Free pages always have their `p_free' bit set, * free pages that are still associated with a vnode also have their * `p_age' bit set. Pages on the free list are connected via their * `p_next' and `p_prev' fields. When a page is involved in some sort * of i/o, it is not free and these fields may be used to link associated * pages together. At the moment, the free list is protected by a * single mutex `page_freelock'. The list of free pages still associated * with a vnode is anchored by `page_cachelist' while other free pages * are anchored in architecture dependent ways (to handle page coloring etc.). * Pages associated with a given vnode appear on a list anchored in the * vnode by the `v_pages' field. They are linked together with * `p_vpnext' and `p_vpprev'. The field `p_offset' contains a page's * offset within the vnode. The pages on this list are not kept in * offset order. These lists, in a manner similar to the hash lists, * are protected by an array of mutexes called `vph_hash'. Before * searching or modifying this chain the appropriate mutex in the * vph_hash[] array must be held. * Again, each of the lists that a page can appear on is protected by a * mutex. Before reading or writing any of the fields comprising the * list, the appropriate lock must be held. These list locks should only * be held for very short intervals. * In addition to the list locks, each page structure contains a * To modify one of these fields, the `p_selock' must be exclusively held. * To read a field with a degree of certainty, the lock must be at least * Removing a page structure from one of the lists requires holding * the appropriate list lock and the page's p_selock. A page may be * prevented from changing identity, being freed, or otherwise modified * by acquiring p_selock shared. * To avoid deadlocks, a strict locking protocol must be followed. Basically * there are two cases: In the first case, the page structure in question * is known ahead of time (e.g., when the page is to be added or removed * from a list). In the second case, the page structure is not known and * must be found by searching one of the lists. * When adding or removing a known page to one of the lists, first the * page must be exclusively locked (since at least one of its fields * will be modified), second the lock protecting the list must be acquired, * third the page inserted or deleted, and finally the list lock dropped. * The more interesting case occures when the particular page structure * is not known ahead of time. For example, when a call is made to * page_lookup(), it is not known if a page with the desired (vnode and * offset pair) identity exists. So the appropriate mutex in ph_mutex is * acquired, the hash list searched, and if the desired page is found * an attempt is made to lock it. The attempt to acquire p_selock must * not block while the hash list lock is held. A deadlock could occure * if some other process was trying to remove the page from the list. * The removing process (following the above protocol) would have exclusively * locked the page, and be spinning waiting to acquire the lock protecting * the hash list. Since the searching process holds the hash list lock * and is waiting to acquire the page lock, a deadlock occurs. * The proper scheme to follow is: first, lock the appropriate list, * search the list, and if the desired page is found either use * page_trylock() (which will not block) or pass the address of the * list lock to page_lock(). If page_lock() can not acquire the page's * lock, it will drop the list lock before going to sleep. page_lock() * returns a value to indicate if the list lock was dropped allowing the * calling program to react appropriately (i.e., retry the operation). * If the list lock was dropped before the attempt at locking the page * was made, checks would have to be made to ensure that the page had * not changed identity before its lock was obtained. This is because * the interval between dropping the list lock and acquiring the page * In addition, when both a hash list lock (ph_mutex[]) and a vnode list * lock (vph_mutex[]) are needed, the hash list lock must be acquired first. * The routine page_hashin() is a good example of this sequence. * This sequence is ASSERTed by checking that the vph_mutex[] is not held * just before each acquisition of one of the mutexs in ph_mutex[]. * So, as a quick summary: * pse_mutex[]'s protect the p_selock and p_cv fields. * p_selock protects the p_free, p_age, p_vnode, p_offset and p_hash, * ph_mutex[]'s protect the page_hash[] array and its chains. * vph_mutex[]'s protect the v_pages field and the vp page chains. * First lock the page, then the hash chain, then the vnode chain. When * this is not possible `trylocks' must be used. Sleeping while holding * any of these mutexes (p_selock is not a mutex) is not allowed. * field reading writing ordering * ====================================================================== * p_vnode p_selock(E,S) p_selock(E) * ===================================================================== * p_hash p_selock(E,S) p_selock(E) && p_selock, ph_mutex * ===================================================================== * p_vpnext p_selock(E,S) p_selock(E) && p_selock, vph_mutex * ===================================================================== * When the p_free bit is set: * p_next p_selock(E,S) p_selock(E) && p_selock, * p_prev page_freelock page_freelock * When the p_free bit is not set: * p_next p_selock(E,S) p_selock(E) && p_selock, p_iolock * ===================================================================== * p_selock pse_mutex[] pse_mutex[] can`t acquire any * ===================================================================== * p_lckcnt p_selock(E,S) p_selock(E) && * ===================================================================== * p_nrm hat layer lock hat layer lock * ===================================================================== * E----> exclusive version of p_selock. * S----> shared version of p_selock. * Global data structures and variable: * field reading writing ordering * ===================================================================== * page_hash[] ph_mutex[] ph_mutex[] can hold this lock * ===================================================================== * vp->v_pages vph_mutex[] vph_mutex[] can only acquire * ===================================================================== * page_cachelist page_freelock page_freelock can't acquire any * page_freelist page_freelock page_freelock * ===================================================================== * freemem freemem_lock freemem_lock can't acquire any * freemem_wait other mutexes while * freemem_cv holding this mutex. * ===================================================================== * Page relocation, PG_NORELOC and P_NORELOC. * Pages may be relocated using the page_relocate() interface. Relocation * involves moving the contents and identity of a page to another, free page. * To relocate a page, the SE_EXCL lock must be obtained. The way to prevent * a page from being relocated is to hold the SE_SHARED lock (the SE_EXCL * lock must not be held indefinitely). If the page is going to be held * SE_SHARED indefinitely, then the PG_NORELOC hint should be passed * to page_create_va so that pages that are prevented from being relocated * can be managed differently by the platform specific layer. * are guaranteed to be held in memory, but can still be relocated * providing the SE_EXCL lock can be obtained. * The P_NORELOC bit in the page_t.p_state field is provided for use by * the platform specific code in managing pages when the PG_NORELOC * Memory delete and page locking. * The set of all usable pages is managed using the global page list as * implemented by the memseg structure defined below. When memory is added * or deleted this list changes. Additions to this list guarantee that the * list is never corrupt. In order to avoid the necessity of an additional * lock to protect against failed accesses to the memseg being deleted and, * more importantly, the page_ts, the memseg structure is never freed and the * page_t virtual address space is remapped to a page (or pages) of * zeros. If a page_t is manipulated while it is p_selock'd, or if it is * locked indirectly via a hash or freelist lock, it is not possible for * memory delete to collect the page and so that part of the page list is * prevented from being deleted. If the page is referenced outside of one * of these locks, it is possible for the page_t being referenced to be * deleted. Examples of this are page_t pointers returned by * page_numtopp_nolock, page_first and page_next. Providing the page_t * is re-checked after taking the p_selock (for p_vnode != NULL), the * remapping to the zero pages will be detected. * Page size (p_szc field) and page locking. * p_szc field of free pages is changed by free list manager under freelist * locks and is of no concern to the rest of VM subsystem. * p_szc changes of allocated anonymous (swapfs) can only be done only after * exclusively locking all constituent pages and calling hat_pageunload() on * each of them. To prevent p_szc changes of non free anonymous (swapfs) large * pages it's enough to either lock SHARED any of constituent pages or prevent * hat_pageunload() by holding hat level lock that protects mapping lists (this * method is for hat code only) * To increase (promote) p_szc of allocated non anonymous file system pages * one has to first lock exclusively all involved constituent pages and call * hat_pageunload() on each of them. To prevent p_szc promote it's enough to * either lock SHARED any of constituent pages that will be needed to make a * large page or prevent hat_pageunload() by holding hat level lock that * protects mapping lists (this method is for hat code only). * To decrease (demote) p_szc of an allocated non anonymous file system large * page one can either use the same method as used for changeing p_szc of * anonymous large pages or if it's not possible to lock all constituent pages * exclusively a different method can be used. In the second method one only * has to exclusively lock one of constituent pages but then one has to * acquire further locks by calling page_szc_lock() and * hat_page_demote(). hat_page_demote() acquires hat level locks and then * demotes the page. This mechanism relies on the fact that any code that * needs to prevent p_szc of a file system large page from changeing either * locks all constituent large pages at least SHARED or locks some pages at * least SHARED and calls page_szc_lock() or uses hat level page locks. * Demotion using this method is implemented by page_demote_vp_pages(). * Please see comments in front of page_demote_vp_pages(), hat_page_demote() * and page_szc_lock() for more details. struct vnode *
p_vnode;
/* vnode that this page is named by */ struct page *
p_hash;
/* hash by [vnode, offset] */ void *
p_mapping;
/* hat specific translation info */ /* index of entry in p_map when p_embed is set */ * Page hash table is a power-of-two in size, externally chained * through the hash field. PAGE_HASHAVELEN is the average length * desired for this chain, from which the size of the page_hash * table is derived at boot time and stored in the kernel variable * page_hashsz. In the hash function it is given by PAGE_HASHSZ. * PAGE_HASH_FUNC returns an index into the page_hash[] array. This * index is also used to derive the mutex that protects the chain. * In constructing the hash function, first we dispose of unimportant bits * (page offset from "off" and the low 3 bits of "vp" which are zero for * struct alignment). Then shift and sum the remaining bits a couple times * in order to get as many source bits from the two source values into the * resulting hashed value. Note that this will perform quickly, since the * shifting/summing are fast register to register operations with no additional * The amount to use for the successive shifts in the hash function below. * The actual value is LOG2(PH_TABLE_SIZE), so that as many bits as * possible will filter thru PAGE_HASH_FUNC() and PAGE_HASH_MUTEX(). * The page hash value is re-hashed to an index for the ph_mutex array. * For 64 bit kernels, the mutex array is padded out to prevent false * sharing of cache sub-blocks (64 bytes) of adjacent mutexes. * For 32 bit kernels, we don't want to waste kernel address space with * padding, so instead we rely on the hash function to introduce skew of * adjacent vnode/offset indexes (the left shift part of the hash function). * Since sizeof (kmutex_t) is 8, we shift an additional 3 to skew to a different * Flags used while creating pages. #
define PG_NORELOC 0x0010 /* Non-relocatable alloc hint. */ /* Page must be PP_ISNORELOC */ #
define PG_PANIC 0x0020 /* system will panic if alloc fails */ * When p_selock has the SE_EWANTED bit set, threads waiting for SE_EXCL * access are given priority over all other waiting threads. * Variables controlling locking of physical memory. /* page_list_{add,sub} flags */ * Flags for setting the p_toxic flag when a page has errors * These flags may be OR'ed into the p_toxic page flag to * indicate that error(s) have occurred on a page, * (see page_settoxic()). If both PAGE_IS_TOXIC and * PAGE_IS_FAILING are set, PAGE_IS_FAILING takes precedence. * When an error happens on a page, the trap handler sets * PAGE_IS_FAULTY on the page to indicate that an error has been * seen on the page. The error could be really a memory error or * something else (like a datapath error). When it is determined * that it is a memory error, the page is marked as PAGE_IS_TOXIC * or PAGE_IS_FAILING depending on the type of error and then * We use the page's 'toxic' flag to determine whether the page * has just got a single error - PAGE_IS_TOXIC - or is being * retired due to multiple soft errors - PAGE_IS_FAILING. In * page_free(), a page that has been marked PAGE_IS_FAILING will * not be cleaned, it will always be retired. A page marked * PAGE_IS_TOXIC is cleaned and is retired only if this attempt at * When a page has been successfully retired, we set PAGE_IS_RETIRED. * Page relocation interfaces. page_relocate() is generic. * page_get_replacement_page() is provided by the PSM. * page_free_replacement_page() is generic. * Tell the PIM we are adding physical memory * hw_page_array[] is configured with hardware supported page sizes by * platform specific code. /* page_get_replacement page flags */ #
define PGR_SAMESZC 0x1 /* only look for page size same as orig */#
define PGR_NORELOC 0x2 /* allocate a P_NORELOC page */ * Constants used for the p_iolock_state * Constants used for page_release status * The p_state field holds what used to be the p_age and p_free * bits. These fields are protected by p_selock (see above). #
define P_FREE 0x80 /* Page on free list */#
define P_NORELOC 0x40 /* Page is non-relocatable */#
define P_MIGRATE 0x20 /* Migrate page on next touch */#
define P_SWAP 0x10 /* belongs to vnode that is V_ISSWAP */ * kpm large page description. * The virtual address range of segkpm is divided into chunks of * kpm_pgsz. Each chunk is controlled by a kpm_page_t. The ushort * is sufficient for 2^^15 * PAGESIZE, so e.g. the maximum kpm_pgsz * for 8K is 256M and 2G for 64K pages. It it kept as small as * possible to save physical memory space. * There are 2 segkpm mapping windows within in the virtual address * space when we have to prevent VAC alias conflicts. The so called * Alias window (mappings are always by PAGESIZE) is controlled by * kp_refcnta. The regular window is controlled by kp_refcnt for the * normal operation, which is to use the largest available pagesize. * When VAC alias conflicts are present within a chunk in the regular * window the large page mapping is broken up into smaller PAGESIZE * mappings. kp_refcntc is used to control the pages that are invoked * in the conflict and kp_refcnts holds the active mappings done * with the small page size. In non vac conflict mode kp_refcntc is * also used as "go" indication (-1) for the trap level tsbmiss short kp_refcnta;
/* pages mapped in Alias window */ short kp_refcntc;
/* TL-tsbmiss flag; #vac alias conflict pages */ short kp_refcnts;
/* vac alias: pages mapped small */ * Note: khl_lock offset changes must be reflected in sfmmu_asm.s * kpm small page description. * When kpm_pgsz is equal to PAGESIZE a smaller representation is used * to save memory space. Alias range mappings and regular segkpm * mappings are done in units of PAGESIZE and can share the mapping * information and the mappings are always distinguishable by their * virtual address. Other information neeeded for VAC conflict prevention * is already available on a per page basis. There are basically 3 states * a kpm_spage can have: not mapped (0), mapped in Alias range or virtually * uncached (1) and mapped in the regular segkpm window (-1). The -1 value * is also used as "go" indication for the segkpm trap level tsbmiss * handler for small pages (value is kept the same as it is used for large * Note: kshl_lock offset changes must be reflected in sfmmu_asm.s * Each segment of physical memory is described by a memseg struct. * Within a segment, memory is considered contiguous. The members * can be categorized as follows: * . Platform independent: * pages, epages, pages_base, pages_end, next, lnext. * . 64bit only but platform independent: * kpm_pbase, kpm_nkpmpgs, kpm_pages, kpm_spages. * . Really platform or mmu specific: * pagespa, epagespa, nextpa, kpm_pagespa. struct memseg *
lnext;
/* next segment in deleted list */ /* memseg union aliases */ /* memseg support macros */