Copyright (C) 2004 BULL SA.
Written by Simon.Derr@bull.net
-Portions Copyright (c) 2004 Silicon Graphics, Inc.
+Portions Copyright (c) 2004-2006 Silicon Graphics, Inc.
Modified by Paul Jackson <pj@sgi.com>
+Modified by Christoph Lameter <clameter@sgi.com>
CONTENTS:
=========
1.3 How are cpusets implemented ?
1.4 What are exclusive cpusets ?
1.5 What does notify_on_release do ?
- 1.6 What is a marker_pid ?
- 1.7 What is memory_pressure ?
- 1.8 How do I use cpusets ?
+ 1.6 What is memory_pressure ?
+ 1.7 How do I use cpusets ?
2. Usage Examples and Syntax
2.1 Basic Usage
2.2 Adding/removing cpus
These subsets, or "soft partitions" must be able to be dynamically
adjusted, as the job mix changes, without impacting other concurrently
-executing jobs.
+executing jobs. The location of the running jobs pages may also be moved
+when the memory locations are changed.
The kernel cpuset patch provides the minimum essential kernel
mechanisms required to efficiently implement such subsets. It
1.3 How are cpusets implemented ?
---------------------------------
-Cpusets provide a Linux kernel (2.6.7 and above) mechanism to constrain
-which CPUs and Memory Nodes are used by a process or set of processes.
+Cpusets provide a Linux kernel mechanism to constrain which CPUs and
+Memory Nodes are used by a process or set of processes.
The Linux kernel already has a pair of mechanisms to specify on which
CPUs a task may be scheduled (sched_setaffinity) and on which Memory
The implementation of cpusets requires a few, simple hooks
into the rest of the kernel, none in performance critical paths:
- - in main/init.c, to initialize the root cpuset at system boot.
+ - in init/main.c, to initialize the root cpuset at system boot.
- in fork and exit, to attach and detach a task from its cpuset.
- in sched_setaffinity, to mask the requested CPUs by what's
allowed in that tasks cpuset.
and related changes in both sched.c and arch/ia64/kernel/domain.c
- in the mbind and set_mempolicy system calls, to mask the requested
Memory Nodes by what's allowed in that tasks cpuset.
- - in page_alloc, to restrict memory to allowed nodes.
+ - in page_alloc.c, to restrict memory to allowed nodes.
- in vmscan.c, to restrict page recovery to the current cpuset.
In addition a new file system, of type "cpuset" may be mounted,
- mem_exclusive flag: is memory placement exclusive?
- tasks: list of tasks (by pid) attached to that cpuset
- notify_on_release flag: run /sbin/cpuset_release_agent on exit?
- - marker_pid: pid of user task in co-ordinated operation sequence
- memory_pressure: measure of how much paging pressure in cpuset
In addition, the root cpuset only has the following file:
is the current value of their parents notify_on_release setting.
-1.6 What is a marker_pid ?
---------------------------
-
-The marker_pid helps manage cpuset changes safely from user space.
-
-The interface presented to user space for cpusets uses system wide
-numbering of CPUs and Memory Nodes. It is the responsibility of
-user level code, presumably in a library, to present cpuset-relative
-numbering to applications when that would be more useful to them.
-
-However if a task is moved to a different cpuset, or if the 'cpus' or
-'mems' of a cpuset are changed, then we need a way for such library
-code to detect that its cpuset-relative numbering has changed, when
-expressed using system wide numbering.
-
-The kernel cannot safely allow user code to lock kernel resources.
-The kernel could deliver out-of-band notice of cpuset changes by
-such mechanisms as signals or usermodehelper callbacks, however
-this can't be synchronously delivered to library code linked in
-applications without intruding on the IPC mechanisms available to
-the app. The kernel could require user level code to do all the work,
-tracking the cpuset state before and during changes, to verify no
-unexpected change occurred, but this becomes an onerous task.
-
-The "marker_pid" cpuset field provides a simple way to make this task
-less onerous on user library code. A task writes its pid to a cpusets
-"marker_pid" at the start of a sequence of queries and updates,
-and check as it goes that the cpusets marker_pid doesn't change.
-The pread(2) system call does a seek and read in a single call.
-If the marker_pid changes, the user code should retry the required
-sequence of operations.
-
-Anytime that a task modifies the "cpus" or "mems" of a cpuset,
-unless it's pid is in the cpusets marker_pid field, the kernel zeros
-this field.
-
-The above was inspired by the load linked and store conditional
-(ll/sc) instructions in the MIPS II instruction set.
-
-
-1.7 What is memory_pressure ?
+1.6 What is memory_pressure ?
-----------------------------
The memory_pressure of a cpuset provides a simple per-cpuset metric
of the rate that the tasks in a cpuset are attempting to free up in
times 1000.
-1.8 How do I use cpusets ?
+1.7 How do I use cpusets ?
--------------------------
In order to minimize the impact of cpusets on critical kernel
If the cpuset flag file 'memory_migrate' is set true, then when
tasks are attached to that cpuset, any pages that task had
allocated to it on nodes in its previous cpuset are migrated
-to the tasks new cpuset. Depending on the implementation,
-this migration may either be done by swapping the page out,
-so that the next time the page is referenced, it will be paged
-into the tasks new cpuset, usually on the node where it was
-referenced, or this migration may be done by directly copying
-the pages from the tasks previous cpuset to the new cpuset,
-where possible to the same node, relative to the new cpuset,
-as the node that held the page, relative to the old cpuset.
+to the tasks new cpuset. The relative placement of the page within
+the cpuset is preserved during these migration operations if possible.
+For example if the page was on the second valid node of the prior cpuset
+then the page will be placed on the second valid node of the new cpuset.
+
Also if 'memory_migrate' is set true, then if that cpusets
'mems' file is modified, pages allocated to tasks in that
cpuset, that were on nodes in the previous setting of 'mems',
-will be moved to nodes in the new setting of 'mems.' Again,
-depending on the implementation, this might be done by swapping,
-or by direct copying. In either case, pages that were not in
-the tasks prior cpuset, or in the cpusets prior 'mems' setting,
-will not be moved.
+will be moved to nodes in the new setting of 'mems.'
+Pages that were not in the tasks prior cpuset, or in the cpusets
+prior 'mems' setting, will not be moved.
There is an exception to the above. If hotplug functionality is used
to remove all the CPUs that are currently assigned to a cpuset,
# The next line should display '/Charlie'
cat /proc/self/cpuset
-In the case that a change of cpuset includes wanting to move already
-allocated memory pages, consider further the work of IWAMOTO
-Toshihiro <iwamoto@valinux.co.jp> for page remapping and memory
-hotremoval, which can be found at:
-
- http://people.valinux.co.jp/~iwamoto/mh.html
-
-The integration of cpusets with such memory migration is not yet
-available.
-
In the future, a C library interface to cpusets will likely be
available. For now, the only way to query or modify cpusets is
via the cpuset file system, using the various cd, mkdir, echo, cat,
projects / linux-2.6-omap-h63xx.git / blobdiff