# CDDL HEADER START

#

# The contents of this file are subject to the terms of the

# Common Development and Distribution License (the "License").

# You may not use this file except in compliance with the License.

#

# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE

# or http://www.opensolaris.org/os/licensing.

# See the License for the specific language governing permissions

# and limitations under the License.

#

# When distributing Covered Code, include this CDDL HEADER in each

# file and include the License file at usr/src/OPENSOLARIS.LICENSE.

# If applicable, add the following below this CDDL HEADER, with the

# fields enclosed by brackets "[]" replaced with your own identifying

# information: Portions Copyright [yyyy] [name of copyright owner]

#

# CDDL HEADER END

#

#

# Copyright 2008 Sun Microsystems, Inc. All rights reserved.

# Use is subject to license terms.

#

TITLE: Dynamic Memory Implementation Overview

DATE: 10/13/2000

AUTHOR: Jim Guerrera (james.guerrera@east)

1.0 Dynamic Memory Implementation in the SCM Module

The system memory allocation required by the Storage Cache Manager (SCM)

has been modified to more fully conform to the requirements of the Solaris

OS. The previous implementation required that the total memory requirements

of the package be allocated 'up front' during bootup and was never released.

The current implementation performs 'on demand' allocations at the time

memory is required in a piecemeal manner. In addition the requisitioned

memory will be released back to the system at some later time.

2.0 Implementation

2.1 Memory Allocation

The memory allocation involves modifications primarily to sd_alloc_buf()

in module sd_bcache.c. When a request is received for cache and system

resources it is broken down and each piece catagorized both as an

independent entity and as a member of a group with close neighbors. Cache

resources comprise cache control entries (ccent), write control entries

(wctrl for FWC support) and sytem memory. The current allocation algorithim

for ccent and wrctl remains the same. The memory allocation has been modified

and falls into two general catagories - single page and multi-page

allocations.

2.1.1 A single page allocation means exactly that - the ccent points to and

owns one page of system memory. If two or more ccent are requisitioned to

support the caching request then only the first entry in the group actually

owns the the allocated memory of two or more pages. The secondary entries

simply point to page boundaries within this larger piece of contiguous memory.

The first entry is termed a host and the secondaries are termed parasites.

The process for determining what is a host, a parasite or anything else is

done in three phases. Phase one simply determines whether the caching request

references a disk area already in cache and marks it as such. If it is not

in cache it is typed as eligible - i.e. needing memory allocation. Phase

two scans this list of typed cache entries and based on immediate neighbors

is catagorized as host, pest or downgraded to other. A host can only exist

if there is one or more eligible entries immediately following it and it

itself either starts the list or immediately follows a non-eligible entry.

If either condition proves false the catagory remains as eligible (i.e.

needs memory allocation) but the type is cleared to not host (i.e. other).

The next phase is simply a matter of scanning the cache entry list and

allocating multipage memory for hosts, single page entries for others or

simply setting up pointers in the parasitic entries into it's corresponding

host multipage memory allocation block.

2.1.2 The maximum number of parasitic entries following a host memory

allocation is adjustable by the system administrator. The details of this

are under the description of the KSTAT interface (Sec 3.0).

2.2 Memory Deallocation

Memory deallocation is implemented in sd_dealloc_dm() in module sd_io.c.

This possibly overly complicated routine works as follows:

In general the routine sleeps a specified amount of time then wakes and

examines the entire centry list. If an entry is available (i.e. not in use

by another thread and has memory which may be deallocated) it takes

possession and ages the centry by one tick. It then determines if the

centry has aged sufficiently to have its memory deallocated and for it to

be placed at the top of the lru.

2.3 There are two general deallocation schemes in place depending on

whether the centry is a single page allocation centry or it is a member

of a host/parasite multipage allocation chain.

2.3.1 The behavior for a single page allocation centry is as follows:

If the given centry is selected as a 'holdover' it will age normally

however at full aging it will only be placed at the head of the lru.

It's memory will not be deallocated until a further aging level has

been reached. The entries selected for this behavior are governed by

counting the number of these holdovers in existence on each wakeup

and comparing it to a specified percentage. This comparision is always

one cycle out of date and will float in the relative vicinity of the

specified number.

In addition there is a placeholder for centries identified as 'sticky

meta-data' with its own aging counter. It operates exactly as the holdover

entries as regards to aging but is absolute - i.e. no percentage governs

the number of such entries.

2.3.2 The percentage and additional aging count are adjustable by the

system administrator. The details of this are under the description of

the KSTAT interface (Sec. 3.0).

2.3.3 The behavior for a host/parasite chain is as follows:

The host/parasite subchain is examined. If all entries are fully aged the

entire chain is removed - i.e memory is deallocated from the host centry

and all centry fields are cleared and each entry requeued on to the lru.

There are three sleep times and two percentage levels specifiable by the

system administrator. A meaningful relationship between these variables

is:

sleeptime1 >= sleeptime2 >= sleeptime2 and

100% >= pcntfree1 >= pcntfree2 >= 0%

sleeptime1 is honored between 100% free and pcntfree1. sleeptime2 is

honored between pcntfree1 and pcntfree2. sleeptime3 is honored between

pcntfree2 and 0% free. The general thrust here is to automatically

adjust sleep time to centry load.

In addition there exist an accelerated aging flag which mimics hysterisis

behavior. If the available centrys fall between pcntfree1 and pcntfree2

an 8 bit counter is switched on. The effect is to keep the timer value

at sleeptime2 for 8 cycles even if the number available cache entries

drifts above pcntfree1. If it falls below pcntfree2 an additional 8 bit

counter is switched on. This causes the sleep timer to remain at sleeptime3

for at least 8 cycles even if it floats above pcntfree2 or even pcntfree1.

The overall effect of this is to accelerate the release of system resources

under what the thread thinks is a heavy load as measured by the number of

used cache entries.

3.0 Dynamic Memory Tuning

A number of behavior modification variables are accessible via system calls

to the kstat library. A sample program exercising the various features can

be found in ./src/cmd/ns/sdbc/sdbc_dynmem.c. In addition the behavior variable

identifiers can be placed in the sdbc.conf file and will take effect on bootup.

There is also a

number of dynamic memory statistics available to gauge its current state.

3.1 Behavior Variables

sdbc_monitor_dynmem --- D0=monitor thread shutdown in the console window

D1=print deallocation thread stats to the console

window

D2=print more deallocation thread stats to the console

window

(usage: setting a value of 6 = 2+4 sets D1 and D2)

sdbc_max_dyn_list ----- 1 to ?: sets the maximum host/parasite list length

(A length of 1 prevents any multipage allocations from

occuring and effectively removes the concept of

host/parasite.)

sdbc_cache_aging_ct1 -- 1 to 255: fully aged count (everything but meta and

holdover)

sdbc_cache_aging_ct2 -- 1 to 255: fully aged count for meta-data entries

sdbc_cache_aging_ct3 -- 1 to 255: fully aged count for holdovers

sdbc_cache_aging_sec1 - 1 to 255: sleep level 1 for 100% to pcnt1 free cache

entries

sdbc_cache_aging_sec2 - 1 to 255: sleep level 2 for pcnt1 to pcnt2 free cache

entries

sdbc_cache_aging_sec3 - 1 to 255: sleep level 3 for pcnt2 to 0% free cache

entries

sdbc_cache_aging_pcnt1- 0 to 100: cache free percent for transition from

sleep1 to sleep2

sdbc_cache_aging_pcnt2- 0 to 100: cache free percent for transition from

sleep2 to sleep3

sdbc_max_holds_pcnt --- 0 to 100: max percent of cache entries to be maintained

as holdovers

3.2 Statistical Variables

Cache Stats (per wake cycle) (r/w):

sdbc_alloc_ct --------- total allocations performed

sdbc_dealloc_ct ------- total deallocations performed

sdbc_history ---------- current hysterisis flag setting

sdbc_nodatas ---------- cache entries w/o memory assigned

sdbc_candidates ------- cache entries ready to be aged or released

sdbc_deallocs --------- cache entries w/memory deallocated and requeued

sdbc_hosts ------------ number of host cache entries

sdbc_pests ------------ number of parasitic cache entries

sdbc_metas ------------ number of meta-data cache entries

sdbc_holds ------------ number of holdovers (fully aged w/memory and requeued)

sdbc_others ----------- number of not [host, pests or metas]

sdbc_notavail --------- number of cache entries to bypass (nodatas+'in use by

other processes')

sdbc_process_directive- D0=1 wake thread

D1=1 temporaily accelerate aging (set the hysterisis

flag)

sdbc_simplect --------- simple count of the number of times the kstat update

routine has been called

3.3 Range Checks and Limits

Only range limits are checked. Internal inconsistencies are not checked

(e.g. pcnt2 > pcnt1). Inconsistencies won't break the system you just won't

get meaningful behavior.

The aging counter and sleep timer limits are arbitrarily limited to a byte

wide counter. This can be expanded. However max'ing the values under the

current implementation yields about 18 hours for full aging.

3.4 Kstat Lookup Name

The kstat_lookup() module name is "sdbc:dynmem" with an instance of 0.

3.5 Defaults

Default values are:

sdbc_max_dyn_list = 8

sdbc_monitor_dynmem = 0

sdbc_cache_aging_ct1 = 3

sdbc_cache_aging_ct2 = 3

sdbc_cache_aging_ct3 = 3

sdbc_cache_aging_sec1 = 10

sdbc_cache_aging_sec2 = 5

sdbc_cache_aging_sec3 = 1

sdbc_cache_aging_pcnt1 = 50

sdbc_cache_aging_pcnt2 = 25

sdbc_max_holds_pcnt = 0

To make the dynmem act for all intents and purposes like the static model

beyond the inital startup the appropriate values are:

sdbc_max_dyn_list = 1,

sdbc_cache_aging_ct1/2/3=255,

sdbc_cache_aging_sec1/2/3=255

The remaining variables are irrelevant.

4.0 KSTAT Implementation for Existing Statistics

The existing cache statistical reporting mechanism has been replaced by

the kstat library reporting mechanism. In general the statistics fall into

two general catagories - global and shared. The global stats reflect gross

behavior over all cached volumes and shared reflects behavior particular

to each cached volume.

4.1 Global KSTAT lookup_name

The kstat_lookup() module name is "sdbc:gstats" with an instance of 0. The

identifying ascii strings and associated values matching the sd_stats driver

structure are:

sdbc_dirty -------- net_dirty

sdbc_pending ------ net_pending

sdbc_free --------- net_free

sdbc_count -------- st_count - number of opens for device

sdbc_loc_count ---- st_loc_count - number of open devices

sdbc_rdhits ------- st_rdhits - number of read hits

sdbc_rdmiss ------- st_rdmiss - number of read misses

sdbc_wrhits ------- st_wrhits - number of write hits

sdbc_wrmiss ------- st_wrmiss - number of write misses

sdbc_blksize ------ st_blksize - cache block size

sdbc_num_memsize -- SD_MAX_MEM - number of defined blocks

(currently 6)

To find the size of each memory blocks append the numbers 0 to 5 to

'sdbc_memsize'.

sdbc_memsize0 ----- local memory

sdbc_memsize1 ----- cache memory

sdbc_memsize2 ----- iobuf memory

sdbc_memsize3 ----- hash memory

sdbc_memsize4 ----- global memory

sdbc_memsize5 ----- stats memory

sdbc_total_cmem --- st_total_cmem - memory used by cache structs

sdbc_total_smem --- st_total_smem - memory used by stat structs

sdbc_lru_blocks --- st_lru_blocks

sdbc_lru_noreq ---- st_lru_noreq

sdbc_lru_req ------ st_lru_req

sdbc_num_wlru_inq - MAX_CACHE_NET - number of net (currently 4)

To find the size of the least recently used write cache per net append

the numbers 0-3 to sdbc_wlru_inq

sdbc_wlru_inq0 ---- net 0

sdbc_wlru_inq1 ---- net 1

sdbc_wlru_inq2 ---- net 2

sdbc_wlru_inq3 ---- net 3

sdbc_cachesize ---- st_cachesize - cache size

sdbc_numblocks ---- st_numblocks - cache blocks

sdbc_num_shared --- MAXFILES*2 - number of shared structures (one for

each cached volume)

This number dictates the maximum

index size for shared stats and

names given below.

sdbc_simplect ----- simple count of the number of times the kstat update routine

has been called

All fields are read only.

4.2 Shared Structures KSTAT lookup_name

The kstat_lookup() module name is "sdbc:shstats" and "sdbc:shname" both with

an instance of 0. The identifying ascii strings and associated values matching

the sd_shared driver structure are:

sdbc:shstats module

sdbc_index ------- structure index number

sdbc_alloc ------- sh_alloc - is this allocated?

sdbc_failed ------ sh_failed - Disk failure status (0=ok,1= /o error

,2= open failed)

sdbc_cd ---------- sh_cd - the cache descriptor. (for stats)

sdbc_cache_read -- sh_cache_read - Number of bytes read from cache

sdbc_cache_write - sh_cache_write - Number of bytes written to cache

sdbc_disk_read --- sh_disk_read - Number of bytes read from disk

sdbc_disk_write -- sh_disk_write - Number of bytes written to disk

sdbc_filesize ---- sh_filesize - Filesize

sdbc_numdirty ---- sh_numdirty - Number of dirty blocks

sdbc_numio ------- sh_numio - Number of blocks on way to disk

sdbc_numfail ----- sh_numfail - Number of blocks failed

sdbc_flushloop --- sh_flushloop - Loops delayed so far

sdbc_flag -------- sh_flag - Flags visible to user programs

sdbc_simplect ---- simple count of the number of times the kstat update routine

has been called

sdbc:shname module

read in as raw bytes and interpreted as a nul terminated assci string.

These two modules operate hand in hand based on information obtained from the

"sdbc:gstats" module. "sdbc:gstats - sdbc_num_shared" gives the maximum number

possible of shared devices. It does not tell how many devices are actually

cached - just the maximum possible. In order to determine the number present

and retrieve the statistics for each device the user must:

1. open and read "sdbc:shstats"

2. set the index "sdbc_index" to a starting value (presumably 0)

3. write the kstat module ( the only item in the module is sdbc_index)

What this does is set a starting index for all subsequent reads.

4. to get the device count and associated statistics the user now simply

reads each module "sdbc:shstats" and "sdbc:shname" as a group repeatedly -

the index will auto increment

To reset the index set "sdbc:shstats - sdbc_index" to the required value

and write the module.

The first entry returning a nul string to "sdbc:shname" signifies no more

configured devices.