notes on liang2kl's Blog

Operating Systems: Three Easy Pieces 学习笔记（五）

liang2kl@outlook.com (Liang Yesheng) — Wed, 25 May 2022 11:44:11 +0800

26 Concurrency: An Introduction

Each thread is very much like a separate process, except for one difference: they share the same address space and thus can access the same data.

State of a single thread:

program counter
private set of registers (context switch is needed)

which are stored in process control block.

Any stack-allocated variables, parameters, return values, and other things that we put on the stack will be placed in what is sometimes called thread-local storage.

26.1 Why Use Threads?

parallelism: speed up using multi-core processors
avoid blocking during I/O

Computers are hard enough to understand without concurrency. Unfortunately, with concurrency, it simply gets worse. Much worse.

26.4 The Heart Of The Problem: Uncontrolled Scheduling

Race condition: the results depend on the timing execution of the code.

Critical section: a piece of code that accesses a shared variable (or more generally, a shared resource) and must not be concurrently executed by more than one thread.

Mutal exclusion: if one thread is executing within the critical section, the others will be prevented from doing so.

27 Interlude: Thread API

27.4 Condition Variables

int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex);
int pthread_cond_signal(pthread_cond_t *cond);

To wait for a signal:

pthread_mutex_lock(&lock);
while (ready == 0)
 pthread_cond_wait(&cond, &lock); // will release the lock when sleep
 // and re-acquire before wakeup
pthread_mutex_unlock(&lock);

To send a signal:

pthread_mutex_lock(&lock);
ready = 1;
pthread_cond_signal(&cond);
pthread_mutex_unlock(&lock);

28 Locks

28.4 Evaluating Locks

Criteria:

mutal exclusion
fairness
performance

28.6 A Failed Attempt: Just Using Loads/Stores

Problems:

No mutal exclusion
Poor performance: spin-waiting

The problem: set after test (maybe after an interrupt).

28.7 Building Working Spin Locks with Test-And-Set

Test-and-set instruction (xchg on x86) is something like:

int testAndSet(int *old_ptr, int new) {
 int old = *old_ptr; // fetch old value at old_ptr
 *old_ptr = new; // store ’new’ into old_ptr
 return old; // return the old value
}

Lock and unlock:

void lock(lock_t *lock) {
 while (testAndSet(&lock->flag, 1) == 1)
 ; // spin-wait (do nothing)
}

void unlock(lock_t *lock) {
 lock->flag = 0;
}

28.8 Evaluating Spin Locks

correctness: yes
fairness: no, maybe spinning forever (starvation)
performance: bad in single CPU case

28.9 Compare-And-Swap

int compareAndSwap(int *ptr, int expected, int new) {
 int original = *ptr;
 if (original == expected)
 *ptr = new;
 return original;

28.10 Load-Linked and Store-Conditional

Store-conditional: only succeeds (and updates the value stored at the address just load-linked from) if no intervening store to the address has taken place.

void lock(lock_t *lock) {
 while (loadLinked(&lock->flag) == 1 ||
 storeConditional(&lock->flag, 1) == 0)
 ;
}

28.11 Fetch-And-Add

The pseudo C code:

int fetchAndAdd(int *ptr) {
 int old = *ptr;
 *ptr = old + 1;
 return old;
}

Lock and unlock:

void lock(lock_t *lock) {
 int myTurn = fetchAndAdd(&lock->ticket);
 while (lock->turn != myTurn)
 ;
}

void unlock(lock_t *lock) {
 lock->turn = lock->turn + 1;
}

This ensures progress for all threads.

28.13 A Simple Approach: Just Yield, Baby

Yield is simply a system call that moves the caller from the running state to the ready state, and thus promotes another thread to running. Thus, the yielding thread essentially deschedules itself.

Still costly: context switches.
Still unfair: endless yield loop.

28.14 Using Queues: Sleeping Instead Of Spinning

The real problem with our previous approaches is that they leave too much to chance.

Solaris provides two calls: park() to put a calling thread to sleep, and unpark(threadID) to wake a particular thread as designated by threadID.

typedef struct __lock_t {
 int flag;
 int guard;
 queue_t *q;
} lock_t;

void lock_init(lock_t *m) {
 m->flag = 0;
 m->guard = 0;
 queue_init(m->q);
}

void lock(lock_t *m) {
 while (testAndSet(&m->guard, 1) == 1)
 ; // acquire guard lock by spinning
 if (m->flag == 0) {
 m->flag = 1; // lock is acquired
 m->guard = 0;
 } else {
 queue_add(m->q, gettid());
 setpark();
 m->guard = 0;
 park();
 }
}

void unlock(lock_t *m) {
 while (testAndSet(&m->guard, 1) == 1)
 ; // acquire guard lock by spinning
 if (queue_empty(m->q))
 m->flag = 0; // let go of lock; no one wants it
 else
 unpark(queue_remove(m->q)); // hold lock
 // (for next thread!)
 m->guard = 0;
}

29 Lock-based Concurrent Data Structures

29.1 Concurrent Counters

Approximate counting: record counter values locally, and periodically transfer them to the global counter.

29.2 Concurrent Linked Lists

A hand-over-hand linked list may not be faster than the simple single lock approach, as the overhead of acquiring and releasing every node along the traversal path is high.

29.3 Concurrent Queues

29.4 Concurrent Hash Table

30 Condition Variables

30.1 Definition and Routines

A condition variable is an explicit queue that threads can put themselves on when some state of execution (i.e., some condition) is not as desired (by waiting on the condition); some other thread, when it changes said state, can then wake one (or more) of those waiting threads and thus allow them to continue (by signaling on the condition).

int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex);
int pthread_cond_signal(pthread_cond_t *cond);

wait() unlocks a mutex (assumed to be locked) and put the calling thread to sleep; it re-acquires the lock before returning.

int done = 0;
pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t c = PTHREAD_COND_INITIALIZER;

void thr_exit() {
 pthread_mutex_lock(&m);
 done = 1;
 pthread_cond_signal(&c);
 pthread_mutex_unlock(&m);
}

void thr_join() {
 pthread_mutex_lock(&m);
 while (done == 0)
 pthread_cond_wait(&c, &m);
 pthread_mutex_unlock(&m);
}

Some questions:

Why is the done state needed?

If the child runs immediately and calls thr_exit() immediately, then although the child will signal, there is no thread asleep. When the parent calls wait(), it will be stuck.

Why is the mutex needed?

The child might change the state before the parent goes to sleep, and the parent will sleep forever. If there is a mutex, the child can only alter the state after the parent goes to sleep (and thus releasing the mutex).

Always hold the lock while signaling!

30.2 The Producer/Consumer (Bounded Buffer) Problem

Thanks to Mesa semantics, a simple rule to remember with condition variables is to always use while loops.

30.3 Covering Conditions

pthread_cond_broadcast(): waking up all waiting threads; downside: negative performace impact.

Read Phenomena and Isolation Levels

liang2kl@outlook.com (Liang Yesheng) — Thu, 05 May 2022 23:11:19 +0800

Reference: https://en.wikipedia.org/wiki/Isolation_(database_systems)

Isolation is typically defined at database level as a property that defines how or when the changes made by one operation become visible to others.

Read phenomena

Dirty reads

A dirty read (aka uncommitted dependency) occurs when a transaction is allowed to read data from a row that has been modified by another running transaction and not yet committed.

Non-repetable reads

A non-repeatable read occurs when, during the course of a transaction, a row is retrieved twice and the values within the row differ between reads.

Phantom reads

A phantom read occurs when, in the course of a transaction, new rows are added or removed by another transaction to the records being read.

Isolation levels

Of the four ACID properties in a DBMS (Database Management System), the isolation property is the one most often relaxed.

For many database applications, the majority of database transactions can be constructed to avoid requiring high isolation levels (e.g. SERIALIZABLE level), thus reducing the locking overhead for the system.

Serializable

The execution of concurrent SQL-transactions at isolation level SERIALIZABLE is guaranteed to be serializable.

When using non-lock based concurrency control… And also range-locks must be acquired when a SELECT query uses a ranged WHERE clause, especially to avoid the phantom reads phenomenon.

Snapshot isolation

snapshot isolation is a guarantee that all reads made in a transaction will see a consistent snapshot of the database (in practice it reads the last committed values that existed at the time it started), and the transaction itself will successfully commit only if no updates it has made conflict with any concurrent updates made since that snapshot.

Repeatable reads

Range-locks are not managed, so phantom reads can occur.

Read committed

Read locks are released as soon as the SELECT operation is performed, so the non-repeatable reads phenomenon can occur in this isolation level.

Read uncommitted

No read locks. Dirty reads are allowed, so one transaction may see not-yet-committed changes made by other transactions.

Concurrency Control for Database Transactions

liang2kl@outlook.com (Liang Yesheng) — Mon, 11 Apr 2022 23:27:56 +0800

来自 DATABASE SYSTEMS: The Complete Book

Serial and Serializable Schedules

Serial schedule: no mixing of actions from different transactions.

Serializable schedule: if there is a serial schedule S’ such that for every initial database state, the effects of S and S’ are the same.

The details of the transactions do matter, but it is not realistic for the scheduler to concern itself with the details of computation undertaken by transactions.

We only care about the reads $r_T(X)$ and writes $w_T(X)$ performed by the transaction $T$.

Conflict-Serializability

Actions that conflict (i.e. their order may not be swapped):

Two actions of the same transaction.
Two writes of the same database element by different transactions.
A read and a write of the same database element by different transactions.

In other words, for actions from different transactions, they conflict when:

They involve the same database element, and
At least one is a write.

Conflict-equivalent: two schedules can be turned one into the other by a sequence of nonconflicting swaps of adjacent actions.

Conflict-serializable: the schedule is conflict-equivalent to a serial schedule

conflict-serializability $\Rightarrow$ serializability

Precedence: if there are actions $A_1$ of $T_1$ and $A_2$ of $T_2$, such that:

$A_1$ is ahead of $A_2$ in the schedule $S$,
Both $A_1$ and $A_2$ involve the same database element, and
At least one of $A_1$ and $A_2$ is a write action.

then we say that $T_1$ takes precedence over $T_2$ in $S$ ($T_1<_ST_2$).

Any conflict-equivalent serial schedule must have $T_1$ before $T_2$.

If the precedence graph is acyclic, then the schedule is conflict-equivalent.

Proof: for $n$, arrange the schedule by (Actions of $T_i$)(Actions of the other $n-1$ transactions) where $T_i$ is one node without an in arc.

Enforcing Serializability by Locks

The use of lock:

Consistency of Transactions: Actions and locks must relate in the expected ways:
1. A transaction can only read or write an element if it previously was granted a lock on that element and hasn’t yet released the lock.
2. If a transaction locks an element, it must later unlock that element.
Legality of Schedules: Locks must have their intended meaning: no two transactions may have locked the same element without one having first released the lock.

The two-phase locking condition than can ensure conflict-serializability:

In every transaction, all lock actions precede all unlock actions.

A transaction that obeys the 2PL condition is said to be a two-phase-locked transaction, or 2PL transaction.

Proof: for $n$, arrange the schedule by (Actions of $T_i$)(Actions of the other $n-1$ transactions) where $T_i$ is the transaction with the first unlock action $u_iX$.

The risk of deadlock:

Locking Systems With Several Lock Modes

Lock modes:

shared
exclusive

The compatibility matrix:

	Request S	Request X
Currently S	Yes	No
Currently X	No	No

The deadlock problem caused by upgrading locks:

Avoid the problem with a third lock mode: update locks.

An update lock $ul_i(X)$ gives transaction $T_j$ only the privilege to read X, not to write X;
Only the update lock can be upgraded to a write lock later; a read lock cannot be upgraded.

The compatibility matrix:

	Request S	Request X	Request U
Currently S	Yes	No	Yes
Currently X	No	No	No
Currently U	No	No	No

The columns for U and S locks are the same, and the rows for U and X locks are the same.

And now:

An Architecture for a Locking Scheduler

The architecture discussed here: the scheduler insert and release locks.

Part 1: inserts appropriate lock actions ahead of all database-access operations
Part 2: for received action,
1. if the issuing transcation is already delayed, then delay this action
2. otherwise,
  1. if the action is db access (the required locks are already been granted), execute it
  2. if is lock action
    1. if the lock can be granted: modify the lock table
    2. otherwise, request the lock and delay
When a transcation commits or aborts, Part 1 releases all locks held by the transaction; if any transactions are waiting for any of these locks, notify Part 2
Part 2 checks which actions can be executed now

The lock table entry:

The group mode: the most stringent conditions
The waiting bit: at least one transaction is waiting
A list of transactions that hold or waiting for the lock

Hierarchies of Database Elements

Warning locks: useful for hierarchical structure.

	IS	IX	S	X
IS	Yes	Yes	Yes	No
IX	Yes	Yes	No	No
S	Yes	No	Yes	No
X	No	No	No	No

The phantom tuple: one that should have been locked but wasn’t, because it didn’t exist at the time the locks were taken.

Solution: acquire an X lock on the entire relation.

The Tree Protocol

The problem: only one transaction that is not read-only can access the B-tree at any time, because the root node must be locked (at least with an update lock).

Solution: as soon as a transaction moves to a child of the root and observes the (quite usual) situation that rules out a rewrite of the root, we would like to release the lock on the root; but it violates 2PL, so a new protocol is needed.

The tree protocol:

The schedule is legal (i.e. does not violate lock schemes).
A transaction’s first lock may be at any node of the tree.
Subsequent locks may only be acquired if the transaction currently has a lock on the parent node.
Nodes may be unlocked at any time.
A transaction may not relock a node on which it has released a lock, even if it still holds a lock on the node’s parent.

Conclusion: the precedence graph must always be acyclic if the tree protocol is obeyed. Proof:

If two transactions lock several elements in common, then they are all locked in the same order.
If $T_i$ locks the root before $T_j$, then $T_i$ locks every node in common with $T_j$ before $T_j$ does.

Concurrency Control by Timestamps

Two approaches to generate timestamp:

use the system clock
the scheduler maintaining a counter

For the database element $X$:

$RT(X)$: the read time of $X$
$WT(X)$: the write time of $X$
$C(X)$: the commit bit fot $X$, true when the most recent transaction to write $X$ has already commited.

Two kinds of unrealizable behaviours:

The problem with dirty data:

$T$ better delays until $U$ commits.

The commit bit will be false.

The Thomas write rule: the writes can be skipped when a write with a later write-time is already in place (read intention for value written by T is too-late, as the value is written by a logically latter transaction).

The scheduler, in response to a read or write request from a transaction T has the choice of:

Granting the request,
Aborting T and restarting T with a new timestamp
Delaying T and later deciding whether to abort or grant the request.

The rules for timestamp-based scheduling:

The scheduler receives a request $r_T(X)$.
1. $TS(T)\ge WT(X)$: physically realizable.
  1. $C(X)$ is true: grant the request; set $RT(X)$ if $TS(T)>RT(X)$.
  2. $C(X)$ is false: delay until becomes true, or that transaction aborts (to avoid dirty reads).
2. $TS(T)<WT(X)$: physically unrealizable, rollback $T$.
The scheduler receives a request $w_T(X)$.
1. $TS(T) \ge RT(X)$ and $TS(T) > WT(X)$: physically realizable and must be performed (?).
  1. Write $X$,
  2. Set $WT(X) := TS(T)$, and
  3. Set $C(X) := \texttt{false}$.
2. $TS(T) \ge RT(X)$ but $TS(T) < WT(X)$: physically realizable, and
  1. If $C(X)$ is true, ignore the write by T (the thomas write rule);
  2. otherwise, delay T until the previous write operation commits or aborts.
3. $TS(T) < RT(X)$: physically unrealizable, roll back T.
The scheduler receives a request to commit T: find all elements written by T, and set $C(X) := \texttt{true}$; the delayed transactions are allowed to proceed.
The scheduler receives a request to abort T, or to rollback: check all delaying transcation waiting for any element that T wrote to see if they can proceed.

The use of multiversion timestamps: to provide an appropriate version for a read action (the version of X written immediately before T theoretically executed), even if the latest version is written after the transcation.

Comparision of locking and timestamp schemes:

Generally, timestamping is superior in situations where either most transactions are read-only, or it is rare that concurrent transactions will try to read and write the same element. In high-conflict situations, locking performs better.

Locking will frequently delay transactions as they wait for locks.
If concurrent transactions frequently read and write elements in common, then rollbacks will be frequent in a timestamp scheduler.

Concurrency Control by Validation

Validation differs from timestamping principally in that the scheduler maintains a record of what active transactions are doing, rather than keeping read and write times for all database elements.

Denote the write set and read set as $WS(T)$ and $RS(T)$. Transactions are executed in three phases:

Read. The transaction reads from the database all the elements in its read set, and also computes (locally) the results it is going to write.
Validate. The transaction compares its read and write sets with those of other transactions; rollback if fails.
Write. The transaction writes to the database the values for the elements in its write set.

Thus, the validation-based scheduler has an assumed serial order of the transactions to work with, and it bases its decision to validate or not on whether the transactions’ behaviors are consistent with this serial order.

The scheduler maintains three sets:

START: the set of transactions that have started, but not yet completed validation.
VAL: the set of transactions that have been validated but not yet finished the writing in phase 3.
FIN: the set of transactions that have completed phase 3.

Some examples:

A later transaction might read a value before an earlier transaction writes the value.

A later transaction might write before a validated, earlier transaction, as the write of the later one might occur earlier.

Summarization: to validate a transaction T:

Check that $RS(T) \cap WS(U) = \emptyset$ for any previously validated U that did not finish before T started, i.e., if $FIN(U) > START(T)$.
Check that $WS(T) \cap WS(U) = \emptyset$ for any previously validated U that did not finish before T validated, i.e., if $FIN(U) > VAL(T)$.

Comparison of Three Concurrency-Control Mechanisms

Storage: all schemes (can be optimized to) use storage space proportional to the sum over all active transactions of the number of database elements the transaction accesses.

Performace:

Locking delays transactions but avoids rollbacks, even when interaction is high.
Timestamps and validation do not delay transactions, but can cause them to rollback.
If interference is low, then neither timestamps nor validation will cause many rollbacks, and may be preferable to locking because they generally have lower overhead than a locking scheduler.
When a rollback is necessary, timestamps catch some problems earlier than validation, which always lets a transaction do all its internal work before considering whether the transaction must rollback.

总结

同时执行多个事务时，需要保证每个事务的原子性，同时还必须保持一定的先后次序。

对 schedule 最严格的要求是串行性（serial），当然这并不能做到并发。退一步，我们不需要严格保证不同事务处理的时间不互相重叠，但需要保证最终的效果和串行执行一致，即“可串行化”（serializable schedule）。当然，这个条件覆盖的范围太大，很难判断 schedule 是否满足可串行化条件。因此，我们需要增加更严格的条件，使得我们可以有简单的方法判断。当然，这样一来，满足条件的 schedule 只是可串行化 schedule 的一个真子集。

第一大类办法是使用锁，并加上 2PL 条件，使得 schedule 满足“冲突可串行化”（confilict-serializable）。上锁有两个需要解决的问题：锁的类型及其兼容性，以及锁的粒度。

第二大类办法是“乐观的”（optimistic），即假设不会发生不可串行化的事件，仅当的确出现违反可串行化条件时采取措施。Optimistic 方法包括两种：by timestamp，即通过比较事务开始的时间与元素被读或写的时间，判断某个操作是否可以进行；by validation，即在验证阶段比较与之前事务的读写集合判断是否可以提交，验证成功后再写入数据库。

上锁的方法避免了重新执行，但事务会受到延迟，在冲突严重时较优；乐观方法由于可以进行一定操作，只在出现问题时取消或重新执行，因此延迟较低，但在冲突严重时会造成大量的重新执行而影响性能。

Operating Systems: Three Easy Pieces 学习笔记（四）

liang2kl@outlook.com (Liang Yesheng) — Sun, 06 Mar 2022 17:24:23 +0800

13 The Abstraction: Address Spaces

13.3 The Address Space

The address space of a process contains all of the memory state of the running program (e.g. code, stack, heap…)

13.4 Goals

Goals of the design:

Transparency
Efficiency
Protection

14 Interlude: Memory API

15 Mechanism: Address Translation

15.3 Dynamic (Hardware-based) Relocation

When a program is loaded, the OS sets base and bounds registers.

physical address = virtual address + base

15.4 Hardware Support: A Summary

The required hardware support:

privileged mode
base/bounds registers
translate virtual address and check if within bounds
privileged instructions to update base/bounds registers
privileged instructions to register exception handlers
ability to raise exceptions

15.6 Summary

The problem: all the space between stack and heap is wasted (called internal fragmentation)

16 Segmentation

16.1 Segmentation: Generalized Base/Bounds

Segment: contiguous portion of the address space of a particular length.

Segmentation allows the OS to place segments in differnet parts of physical memory.

The hardware structure in the MMU: three base and bounds register pairs.

16.2 Which Segment Are We Referring To?

The explicit approach: chop up the address space into segments based on the top few bits of the virtual address.

The implicit approcah: the hardware determins the segment by noticing how the address was formed.

16.3 What About The Stack?

The difference: it grows backwords.

Use a bit in the hardware to indicate the sign of the offset.

16.5 Fine-grained vs. Coarse-grained Segmentation

16.6 OS Support

The operation to support segmentation:

On context switch: save and store segment registers
Support growing heap segment (sbrk())
Handle external fragmentation (physical memory quickly becomes full of little holes of free space, making it difficult to allocate new segments, or to grow existing ones)

17 Free-Space Management

17.1 Assumptions

Internal fragmentation: the allocator hands out chunks of memory bigger than that requested

17.2 Low-level Mechanisms

Splitting and Coalescing

Coalescing: coalesce spliited fragments when they are contiguous

Tracking The Size Of Allocated Regions

Most allocators store a little bit of extra information in a header block which is kept in memory to keep track of the size of the allocated region.

typedef struct {
 int size;
 int magic; /* for integrity checking */
} header_t;

Embedding A Free List

Growing The Heap

17.3 Basic Strategies

Unfortunately, because the stream of allocation and free requests can be arbitrary (after all, they are determined by the programmer), any particular strategy can do quite badly given the wrong set of inputs.

Best Fit

The best chunk: the smallest in candidates.

Cost: heavy performance overhead

Worst Fit

The opposite to the best fit strategy.

First Fit

Returns the first qualified block.

Problem: pollution at the beginning of the free list; approach: address-based ordering.

Advantage: speed.

Next Fit

Keep an eatra pointer to the last-searched location; the blocks are distributed more evenly than first-fit strategy.

17.4 Other Approaches

Segregated Lists

Keep a separate list just to manage objects of some popular sizes; all other requests are forwarded to a more general memory allocator.

Buddy Allocation

The binary buddy allocator: free memory is thought of as one big space of $2^N$; the search for free space recursively divides free space by two.

Suffers from internal fragmentation.

Free process: recursively coalescing blocks upwards.

Simple to determine the buddy of a particular block: each buddy pair (at the same level) only differs by a single bit.

18 Paging: Introduction

18.1 A Simple Example And Overview

Page table: a per-process data structure to store address translations.

18.3 What’s Actually In The Page Table?

Some bits:

valid bit (supporting a sparse address space; x86 doesn’t have it)
protection bit
present bit (whether a page has been swapped out)
dirty bit (whether a page has been modified)
refence bit (whether a page has been accessed)

Don’t worry: it deﬁnitely gets worse, because the mechanisms we are about to introduce only complicate this already complex machinery. Sorry!

19 Paging: Faster Translations (TLBs)

When we want to make things fast, the OS usually needs some help. And help often comes from the OS’s old friend: the hardware.

Translation-lookaside buffer (or TLB)

19.3 Who Handles The TLB Miss?

CISC: uses hardware-managed TLBs
RISC: uses software-managed TLBs

The software approach: more flexible and simple; need to avoid infinite chain of TLB misses.

19.4 TLB Contents: What’s In There?

The TLB typically has 32, 64 or 128 entries, and is fully associated (any given translation can be anywhere in the TLB).

19.5 TLB Issue: Context Switches

Some hardware systems provide an address space identifier (ASID) in the TLB to identify the associated process.

19.6 Issue: Replacement Policy

One common approach is to evict the least-recently-used or LRU entry.

Another typical approach is to use a random policy, which evicts a TLB mapping at random.

19.7 A Real TLB Entry

How to handle ‘ASID overflow’?

In MIPS, some TLB slots can be reserved for the OS.

20 Paging: Smaller Tables

Here, the problem should be clear: simple linear (array-based) page tables are too big.

20.2 Hybrid Approach: Paging and Segments

Whenever you have two reasonable but different approaches to something in life, you should always examine the combination of the two to see if you can obtain the best of both worlds.

Operating Systems: Three Easy Pieces 学习笔记（三）

liang2kl@outlook.com (Liang Yesheng) — Sat, 26 Feb 2022 13:26:41 +0800

4 The Abstraction: The Process

Process: a running program

4.2 Process API

create
destroy
wait
miscellaneous control
status

4.4 Process States

A process can be in one of three states:

running: executing instructions
ready: ready to run but the OS has chosen not to run at this moment
blocked: a process has performed some kind of operation that makes it not ready to run until some other event takes place

5 Interlude: Process API

5.4 Why? Motivating The API

The separation of fork() and exec() is essential in building a UNIX shell, because it lets the shell run code after the call to fork() but before the call to exec(); this code can alter the environment of the about-to-be-run program, and thus enables a variety of interesting features to be readily built.

5.5 Process Control And Users

The user, after entering a password to establish credentials, logs in to gain access to system resources (and as a result, can send signals to programs).

6 Mechanism: Limited Direct Execution

Use time sharing to achieve virtualization

Some challenges:

performance
control

6.1 Basic Technique: Limited Direct Execution

Direct execution: run the program directly on the CPU

6.2 Problem #1: Restricted Operations

The approach to restrict certain operations: introducing user mode (and kernel mode)

System calls allow the kernel to carefully expose certain key pieces of functionality to user programs, setting up a trap table for the hardware to jump to trap handlers

6.3 Problem #2: Switching Between Processes

The curx: how to regain control of the CPU?

A Cooperative Approach: Wait For System Calls

The OS regains control of the CPU by waiting for a system call or an illegal operation of some kind to take place.

A Non-Cooperative Approach: The OS Takes Control

A timer device can be programmed to raise an interrupt (timer interrupt) every so many milliseconds; when the interrupt is raised, the currently running process is halted, and a pre-configured interrupt handler in the OS runs.

Saving and Restoring Context

Two types of register saves/restores:

when timer interrupt occurs: the user registers are implicitly saved by the hardware to the kernel stack
when the OS decides to switch to another process: the kernel registers are explicitly saved by the software to the memory of the process structure (Still in the kernel mode so should kernel registers be saved/restored; the user registers are restored later during return-from-trap)

6.4 Worried About Concurrency?

OS might disable interrupts during interrupt processing (but can lead to lost of interrupts), and use locking to protect concurrent access to internal data structures.

7 Scheduling: Introduction

7.1 Workload Assumptions

Unrealistic assumptions:

each job runs for the same amount of time
all jobs arrive at the same time
once started, each job runs to completion
all jobs only use the CPU (i.e., they perform no I/O)
the run-time of each job is known (more unrealistic!)

7.2 Scheduling Metrics

Simplify with one single metric: turnaround time

$$ T_\mathrm{turnaround}=T_\mathrm{completion}-T_\mathrm{arrival} $$

In our case:

$$ T_\mathrm{turnaround}=T_\mathrm{completion} $$

7.3 First In, First Out (FIFO)

The First In, First Out scheduling, or First Come, First Served

Convoy effect: a number of relatively-short potential consumers of a resource get queued behind a heavyweight resource consumer.

7.4 Shortest Job First (SJF)

Shortest Job First: runs the shortest job first, then the next shortest, and so on; optimal.

Bad without assumption 2 (all jobs arrive at the same time)

7.5 Shortest Time-to-Completion First (STCF)

Relax assumption 3 (each job runs to completion)

The scheduler can preempt the current job; adding this to SJF, we get the Shortest Time-to-Completion First (STCF) or Preemptive Shortest Job First (PSJF) scheduler

7.6 A New Metric: Response Time

The response time:

$$ T_\mathrm{response}=T_\mathrm{firstrun}-T_\mathrm{arrival} $$

7.7 Round Robin

Round-Robin: instead of running jobs to completion, RR runs a job for a time slice (sometimes called a scheduling quantum) and then switches to the next job in the run queue

The length of the time slice is critical for RR; a trade-off between response time and the cost of context switching

The cost of context switch

When programs run, they build up a great deal of state in CPU caches, TLBs, branch predictors, and other on-chip hardware. Switching to another job causes this state to be flushed and new state relevant to the currently-running job to be brought in, which may exact a noticeable performance cost.

RR is pretty awful for turnaround time

More generally, any policy (such as RR) that is fair, i.e., that evenly divides the CPU among active processes on a small time scale, will perform poorly on metrics such as turnaround time.

7.8 Incorporating I/O

Relax assumption 4 (all jobs only use the CPU)

Overlap will occur if using STCF

7.9 No More Oracle

Relax our final assumption (the run-time of each job is known)

8 Scheduling: The Multi-Level Feedback Queue

The problem we try to address: given that we in general do not know anything about a process, how to

optimize turnaround time
minimize response time

8.1 MLFQ: Basic Rules

The MLFQ has a number of distinct queues, each assigned a different priority level. At any given time, a job that is ready to run is on a single queue.

A job with higher priority (i.e., a job on a higher queue) is chosen to run; use round-robin scheduling among jobs with the same priority.

rule 1: if Priority(A) > Priority(B), A runs (B doesn’t)
rule 2: if Priority(A) = Priority(B), A & B run in RR

To set the priority: MLFQ will try to learn about processes as they run, and thus use the history (long-running or interactive) of the job to predict its future behavior.

8.2 Attempt #1: How To Change Priority

rule 3: when a job enters the system, it is placed at the highest priority
rule 4
- a: if a job uses up an entire time slice while running, its priority is reduced
- b: if a job gives up the CPU before the time slice is up (therefore jobs with lower priority may have a chance to run), it stays at the same priority level

Example 1: A Single Long-Running Job

Example 2: Along Came A Short Job

The idea: first assumes a job to be short, otherwise move it down; this is an approximization of SJF

Example 3: What About I/O?

Problems With Our Current MLFQ

starvation: too many interactive jobs will consume all CPU time, thus long-running jobs will never receive any CPU time
can be attacked by first using most of the time and then perform an I/O
a program might change its behaviour

8.3 Attempt #2: The Priority Boost

rule 5: after some time period $S$, move all the jobs in the system to the topmost queue

If $S$ is set too high, long-running jobs could starve; too low, and interactive jobs may not get a proper share of the CPU.

8.4 Attempt #3: Better Accounting

rule 4: once a job uses up its time allotment (分配) at a given level (regardless of how many times it has given up the CPU), its priority is reduced (i.e., it moves down one queue)

8.5 Tuning MLFQ And Other Issues

The concept: instead of optimizing for turnaround or response time, a scheduler might instead try to guarantee that each job obtain a certain percentage of CPU time.

Tickets: the share of a resource that a process (or user or whatever) should receive.

The system picks a ticket (randomly) whose owner process can run at this point

If you are ever in need of a mechanism to represent a proportion of ownership, this concept just might be … (wait for it) … the ticket.

9.2 Ticket Mechanisms

ticket currency
ticket transfer
ticket inflation (通货膨胀)

9.3 Implementation

Best practice: order the list from the highest number of tickets to the lowest

9.4 An Example

Overall fairness is higher with higher job length

9.5 How To Assign Tickets?

Given a set of jobs, the “ticket-assignment problem” remains open.

9.6 Stride Scheduling

Stride: inverse in proportion to the number of tickets a job has

curr = remove_min(queue); // pick client with min pass
schedule(curr); // run for quantum
curr->pass += curr->stride; // update pass using stride
insert(queue, curr); // return curr to queue

Stride scheduling is a deterministic fair-share scheduler.

Lottery scheduling has no global state, so that new processes are easier to be incorporated

9.7 The Linux Completely Fair Scheduler (CFS)

Recent studies have shown that scheduler efficiency is surprisingly important; specifically, in a study of Google datacenters, Kanev et al. show that even after aggressive optimization, scheduling uses about 5% of overall datacenter CPU time.

Basic Operation

Goal: to fairly divide a CPU evenly among all competing processes

In the basic case, each process’s vruntime increases at the same rate, propotional to the physical time. Pick the process with the lowest vruntime to run next.

Divide sched_latency by process number to calculate the time slice; use min_granularity to constrain the minimum possible time slice.

Weighting (Niceness)

Use nice level to control the priority of a process. Positive nice values imply lower priority and negative values imply higher priority

When you’re too nice, you just don’t get as much (scheduling) attention, alas.

Calculation of the time slice:

$$ \texttt{time_slice}_k=\cfrac{\texttt{weight}_k}{\sum_{i=0}^{n-1}\texttt{weight}_i}\cdot\texttt{sched_latency} $$

Also adjusting the calculation of virtual runtime:

$$ \texttt{vruntime}_i=\texttt{vruntime}_i+\cfrac{\texttt{weight}_0}{\texttt{weight}_i}\cdot\texttt{runtime}_i $$

The table preserves CPU proportionality ratios when the difference in nice values is constant

Using Red-Black Trees

Use a red-black tree to maintain running (or runnable) processes; if a process goes to sleep (say, waiting fot an I/O to complete), it is removed.

Dealing With I/O And Sleeping Processes

A just-awaken process will monopolize the CPU; solution: set the vruntime of that job to the min value found in the tree (which only keeps track of running jobs!).

10 Multiprocessor Scheduling (Advanced)

10.1 Background: Multiprocessor Architecture

Two kinds of locality:

temporal locality
spatial locality

Problem with multi-code processors: cache coherence; the hardware uses bus snooping to update/invalidate outdated cache

10.2 Don’t Forget Synchronization

10.3 One Final Issue: Cache Affinity

A multiprocessor scheduler should consider cache affinity when making its scheduling decisions, perhaps preferring to keep a process on the same CPU if at all possible.

10.4 Single-Queue Scheduling

Singlequeue multiprocessor scheduling (or SQMS): take an existing policy that picks the best job to run next and adapt it to work on more than one CPU

Problems:

performance: the overhead of locking is high
cache affinity: each job ends up bouncing around from CPU to CPU

10.5 Multi-Queue Scheduling

Each CPU has a queue of tasks to be performed.

To solve load imbalance: using migration; a basic approach is to use work stealing

Operating Systems: Three Easy Pieces 学习笔记（二）

liang2kl@outlook.com (Liang Yesheng) — Wed, 23 Feb 2022 21:37:07 +0800

42 Crash Consistency: FSCK and Journaling

One major challenge faced by a file system is how to update persistent data structures despite the presence of a power loss or system crash.

Crash-consistency problem: On-disk state may only partially get updated on system crash or power loss

42.1 A Detailed Example

Crash Scenarios

Three possible outcomes if an error occured when a data block is allocated and written for a file (which should allocate a data block (and write data to it), update the data bitmap, and update the file inode):

only the data block is written: no problem at all (from the perspective of file-system crash consistency)
only the updated inode is written: garbage data will be read; file-system inconsistency: the allocated status of the data block is different in the bitmap and the inode
only the bitmap is written: space leak
only the inode and the bitmap are written: garbage data
only the inode and the data block: inconsistency between the inode and the bitmap
only the bitmap and the data block: space leak again

The Crash Consistency Problem

Crash-consistency problem (or consistent-update problem): the disk can only commit one write at a time, so that an update cannot be done atomically

42.2 Solution #1: The File System Checker

Early systems’ approach: fix inconsistencies later (when rebooting)

fsck’s phases:

check superblock
scans inodes, indirect blocks, etc. to make consistency between inodes and bitmaps (by trusting inodes)
check inodes for corruption and other problems
verify link count
check for duplicate pointers (two different inodes refer to the same block) (clear some inodes or make copies of the duplicate data)
check for bad block pointers (and remove them)
check directories

Fundamental problems of fsck: too slow

At a higher level, the basic premise of fsck seems just a tad irrational. Consider our example above, where just three blocks are written to the disk; it is incredibly expensive to scan the entire disk to fix problems that occurred during an update of just three blocks.

42.3 Solution #2: Journaling (or Write-Ahead Logging)

Write-ahead logging (or journaling): write logs before overwriting the structures to be able to recover from errors

Data Journaling

Journal write:

transaction begin block
blocks containing the exact contents of the updating blocks (physical logging) or some more compact forms (logical logging)
transaction end block

Checkpointing: override the old structures in the file system

Problems if a crach occurs during the write to the journal: if writing all the log blocks at once, as the disk might perform scheduling, they might not complete in order, and thus some blocks might not be correct (e.g. invalid data blocks within valid TxB and TxE blocks).

Solution: write all blocks except the TxE at once, and write TxE block after these writes. With atomicity guatantee!

Performance problem: the transaction-end block can only be written after previous blocks are written.

Optimizing log writes: using checksum

Three phases:

journal write (TxB, metadata and data)
journal commit (TxE)
checkpoint (write the contents to final on-disk locations)

Recovery

Replay the transactions that are commited to the disk (redo logging)

Batching Log Updates

Buffer all updates into a global transaction to avoid writing some same blocks many times.

Making The Log Finite

Journaling file systems treat the log as a circular data structure (the journal is referred to as circular log).

Mark the transaction free in the journal by updating the journal superblock some time after checkpoint.

Now the four phases:

journal write
journal commit
checkpoint
free

Metadata journaling

Ordered journaling (or metadata journaling): the user data is not written to the journal

One possible order (to avoid garbage data):

data write (1)
journal metadata write (2)
journal commit
checkpoint metadata
free

(1) and (2) can be written concurrently

43 Log-structured File Systems

The motivation

system memories are growing so that more data can be cached, and disk traffic increasingly consists of writes

a large gap between random and sequential I/O performace

exising file systems perform poorly on many common workloads

file systems are not RAID-aware

An ideal file system would thus focus on write performance, and try to make use of the sequential bandwidth of the disk. Further, it would perform well on common workloads that not only write out data but also update on-disk metadata structures frequently. Finally, it would work well on RAIDs as well as single disks.

43.1 Writing To Disk Sequentially

The basic idea: writing all updates (such as data blocks, inodes, etc.) to the disk sequentially

43.2 Writing Sequentially And Effectively

Sequential writes is not enough; to achieve good performance we must issue contiguous writes (or one large write)

Write buffering: keeps track of updates in memory, and write the buffered data at once when gets sufficient updates (called a segment)

43.3 How Much To Buffer?

$$ D=\cfrac{F}{1-F}\times R_\mathrm{peak}\times T_\mathrm{position} $$

43.4 Problem: Finding Inodes

43.5 Solution Through Indirection: The Inode Map

The inode map: takes an inode number as input and returns the disk address of the most recent version of the inode

LFS places chunks of the inode map right next to where it is writing all of the other new information.

Inode maps are compact enough to keep the active portions cached in main memory: inode map lookups rarely require disk accesses.

(From The Design and Implementation of a Log-Structured File System)

43.6 Completing The Solution: The Checkpoint Region

The checkpoint region (or CR) contains pointers to the latest pieces of the inode map; it is only updated periodically for performance

43.7 Reading A File From Disk: A Recap

The steps of reading a file (from start):

read the CR and cache it in memory
lookup the CR to find the address of a given inode
(the rest is pretty much the same as a typical FS)

43.8 What About Directories?

Steps:

look in the imap for the location of the inode of the directory
find inode number of the file in the inode of the directory
look in the imap for the location of the inode of the file

The imap also solves the recursive update problem through indirection.

43.9 A New Problem: Garbage Collection

The cleaner assures to preserve contiguous free regions by compacting existing segments to new, contiguous segments

43.10 Determining Block Liveness

Use segment summary block at the head of the segment to record the inode number and offset (which block of the file) for each data block

To determine whether a block is dead:

look in the segment summary block and find its inode number and offset
look in the imap to find and read the inode
look in the inode using the offset to compare the disk location

LFS can also use version number to determine this, avoiding extra reads.

43.11 A Policy Question: Which Blocks To Clean, And When?

The time to clean can be:

perioducally
during idle time
when the disk is full

43.12 Crash Recovery And The Log

Use two CRs containing update timestamps to recover from crash during CR updates

The last consistent snapshot of the file system may be quite old as LFS only writes the CR every 30 seconds or so; use roll forward to read through the next segments of the end of the log to see if there are any valid updates, and updates the file system accordingly.

44 Flash-based SSDs

Solid-state storage: random access, retains information on power loss

44.1 Storing a Single Bit

single-level cell (SLC) flash: a single bit per transistor
multi-level cell (MLC) flash: two bits per transistor (00, 01, 10, 11 represented by different levels of charge)
triple-level cell (TLC) flash: three bits per cell

SLC chips achieve higher performance and are more expensive.

44.2 From Bits to Banks/Planes

Flash chips are organized into banks or planes which consist of a large number of cells.

A bank is accessed in two different sized units:

block (or erase block): typically of size 128KB or 256KB
page: a few KB in size (e.g. 4KB)

44.3 Basic Flash Operations

Three low-level operations:

read (a page): random access; fast, 10s of $\mathrm{\mu s}$ or so
erase (a block): setting each bit to 1; quite expensive, takes a few ms; need to copy data of care somewhere else
program (a page): After a block is erased, change some of the 1’s within a page to 0’s; usually taking around $100\mathrm{\mu s}$

44.4 Flash Performance And Reliability

Wear out: erasing and programming accure a little bit of extra charge, 0 and 1 are increasingly difficult to be differentiated.

The typical lifetime:

MLC: 10,000 P/E (Program/Erase) cycles
SLC: 100,000 P/E cycles
actual lifetimes are much longer

Disturbance: some bits get flipped in neighboring pages when accessing (read/program) a particular page

44.5 From Raw Flash to Flash-Based SSDs

Flash translation layer (or FTL): translates read and write requests on logical blocks into low-level read, erase and program commands on underlying physical blocks and physical pages

The FTL should accomplish this task with the goal of delivering excellent performance and high reliability.

To achieve performance:

parallel
reduce write amplification

Write amplification: total write traffic issued to the flash chips (by the FTL) divided by the total write traffic issued to the SSD (by the client)

To achieve high reliability:

for wear out: wear leveling
for program disturbance: a sequential-programming approach

44.6 FTL Organization: A Bad Approach

Direct mapped organization: a read to a logical page is mapped directly to the physical page with the same page number

The problems with the direct-mapped FTL:

performance: each write needs to
- read in the entire block (costly)
- erase the entire block (quite costly)
- program the block (costly)
reliability: the same block can be erased and programmed over and over (e.g., file system metadata is overritten repeatedly)

The direct mapped approach simply gives too much control over wear out to the client workload; if the workload does not spread write load evenly across its logical blocks, the underlying physical blocks containing popular data will quickly wear out.

44.7 A Log-Structured FTL

Use a mapping table to map the logical block to the physical address

The log-based approach improves performance in:

erases being required once in a while
spread writes across pages (wear leveling)

Downsides:

garbage collection may drives up write amplification
in-memory mapping tables might be large

44.8 Garbage Collection

To reduce GC costs, overprovisioning can be performed: adding extra flash capacity and delay cleaning, which will be performed in background.

44.9 Mapping Table Size

Block-Based Mapping

Hybrid Mapping

Two types of mapping table:

log table (per-page mappings)
data table (per-block mappings)

To read:

first find in the log table
if not found, find in the data table

The key to the hybrid mapping strategy is keeping the number of log blocks small. To do that, the FTL has to periodically examine log blocks and switch them into blocks that can be pointed to by only a single block pointer.

Three types of merge:

switch merge: all pointers of a block in the log table can be switched into a pointer pointing to this block
partial merge
full merge

Page Mapping Plus Caching

44.10 Wear Leveling

One problem of the log-structured approach: blocks filled with long-lived data do not get overwritten; solution: periodically read and rewrite elsewhere

Operating Systems: Three Easy Pieces 学习笔记（一）

liang2kl@outlook.com (Liang Yesheng) — Thu, 13 Jan 2022 20:38:28 +0800

36 I/O Devices

36.1 System Architecture

Devices are connected to the system via:

memory bus (CPU - main memory)
general I/O bus: PCI (graphics, high performance devices)
peripheral bus: SCSI, SATA, USB (slow devices)

In modern systems:

PCIe Graphics
memory interconnect
DMI (Direct Media Interface)
PCIe
eSATA

36.2 A Canonical Device

A device has two important components:

interface
- status
- command
- data
internal structure

36.3 The Canonical Protocol

Programmed I/O (PIO): the main CPU is involved with the data movement

36.4 Lowering CPU Overhead With Interrupts

Utilize CPU:

interrupt
a two-phased approach: polls first, using interrupts if not yet finished
coalescing

36.5 More Efficient Data Movement With DMA

Direct Memory Access (DMA): transfer data between devices and main memory without much CPU intervention

36.6 Methods Of Device Interaction

explicit I/O instructions (on x86, the in and out instructions), usually privileged
memory-mapped I/O (no new instructions are needed)

36.7 Fitting Into The OS: The Device Driver

Read/write blocks:

file system
raw interface (support low-level storage management)

36.8 Case Study: A Simple IDE Disk Driver

Registers are available by reading or writing to specific “I/O addresses” using (on x86) the in and out I/O instructions.

wait for drive to be ready
write parameters to command registers
start the I/O: issuing read/write to command register
data transfer (for writes): write data to data port
handle interrupts

37 Hard Disk Drives

37.1 The Interface

The drive consists of a large number of sectors (512-byte blocks), each of which can be read or written

View the disk as an array of sectors (扇区)

multi-sector operations are possible
only guarantee a single 512-byte write is atomic

Efficiency:

access closer blocks are faster than farther ones
sequential read or write are faster than the random access manner

37.2 Basic Geometry

platter: 磁盘
surface: the 2 sides of a platter
spindle: 轴
track: 道, concentric circles of sectors
disk head
disk arm

37.3 A Simple Disk Drive

I/O time:

rotational delay
seek time
- acceleration
- coasting
- deceleration
- settling (significant)
transfer

Track skew: sequential reads can be properly serviced when crossing track boundaries

Multi-zoned disk drives:

a zone is a consecutive set of tracks on a surface
each zone has the same number of sectors per track
outer zones have more sectors

Cache (track buffer):

read: to read in all of the sectors on that track and cache them in its memory
write
- write back: acknowledge when puts into memory
- write through: acknowledge after actual writes

37.4 I/O Time: Doing The Math

The rate of I/O:

$$ T_\mathrm{I/O} = T_\mathrm{seek}+T_\mathrm{rotation}+T_\mathrm{transfer} $$

$$ R_\mathrm{I/O}=\cfrac{Size_\mathrm{transfer}}{T_\mathrm{I/O}} $$

a huge gap in drive performance between random and sequential workloads
a large difference in performance between high-end “performance” drives and low-end “capacity” drives

37.5 Disk Scheduling

Principle: shortest job first, SJF

Shortest-seek-time-first (SSTF, also shortest-seek-first or SSF): picking requests on nearest track to complete first

problems:

the drive geometry is not available to the OS (fix: nearest-block-first, NBF)
starvation: a steady stream to the closest track will block the access to other tracks

Elevator algorithm

SCAN: simply moves back and forth across the disks

F-SCAN: freezes the queue when doing a sweep (avoid starvation of far-away requests)

C-SCAN (Circular SCAN): only sweeps from outer-to-inner (more fair to the inner and outer tracks)

Prevent fights from taking place on elevators!

SCAN and SSTF does not adhere to SJF closly: they ignore rotation

SPTF: Shortest positioning time first

Also called shortest access time first, SATF

In engineering, it turns out “it depends” is almost always the answer, reflecting that trade-offs are part of the life of the engineer.

On modern drives both seek and rotation are roughly equivalent

SPTF is usually performed inside a drive

Other scheduling issues

Where to perform scheduling: The OS scheduler usually picks (what it thinks) some best few requests (say 16) and issues them all to disk; the disk then uses its internal knowledge to service said requests in the best possible (SPTF) order.

I/O merging: merge I/O requests by the OS

Non-work-conserving: waiting for some time for a possible better request

38 Redundant Arrays of Inexpensive Disks (RAIDs)

RAID: a technique to use multiple disks in concert to build a faster, bigger, and more reliable disk system

externally looks like a disk
internally consisting multiple disks, memory and processor(s)

38.1 Interface And RAID Internals

To perform a logical I/O:

calculate the actual disk(s)
perform one or more physical I/Os

38.2 Fault Model

RAIDs are designed to detect and recover from certain kinds of disk faults.

The fail-stop fault model: a not-working disk is assumed permanently lost

38.3 How To Evaluate A RAID

Characteristics:

capacity
reliability
performance

38.4 RAID Level 0: Striping

RAID level 0, or striping as it is better known, serves as an excellent upper-bound on performance and capacity.

Striping: no redundancy; spread the blocks of the array across the disks in a round-robin fashion to get the most parrallelism.

Simple striping:

Disk 0	Disk 1	Disk 2	Disk 3
0	1	2	3
4	5	6	7
8	9	10	11
12	13	14	15

Stripe: blocks in the same row (0, 1, 2, 3)

Striping with a bigger chunk size (8KB, 2 blocks, a stripe consists of 32KB of data):

Disk 0	Disk 1	Disk 2	Disk 3
0	2	4	6
1	3	5	7
8	10	12	14
9	11	13	15

Chunk sizes

small chunk size: on large files, positioning more on a single disk
large chunk size: less parallelism, but also less posisioning time

Thus, determining the “best” chunk size is hard to do, as it requires a great deal of knowledge about the workload presented to the disk system.

RAID-0 Analysis

capacity: perfect
reliability: any disk failure will lead to data loss
performance: all disks are utilized, often in parallel

Evaluating RAID Performances

Two performance metrics:

single-request latency
steady-state throughput (critical in high-perfromance environments)

Two types of workloads (considered here):

sequential: large contiguous chunks
- searching through a large file for a keyword
random: samll requests, to different random locations on disk
- transactional workloads on a database management system

Denote the sequential and random workload’s throughput as:

$S$ for sequential

$R$ for random

38.5 RAID Level 1: Mirroring

RAID-10 (or RAID 1+0, stripe of mirrors):

Disk 0	Disk 1	Disk 2	Disk 3
0	0	1	1
2	2	3	3
4	4	5	5
6	6	7	7

RAID-01

Disk 0 Disk 1 Disk 2 Disk 3

0 1 0 1

2 3 2 3

4 5 4 5

6 7 6 7

Disk 0	Disk 1	Disk 2	Disk 3
0	1	0	1
2	3	2	3
4	5	4	5
6	7	6	7

read: either copy
write: update both

Analysis

Consistent-update problem

untimely failures on write: inconsistent copies

solution: write-ahead log and recovery procedure

capacity: expensive, $N\cdot B/2$ for mirroring level = 2
reliability: tolerate failure on any disk, up to $N/2$ disks can fail
single request performance
- read: sames as on a single disk ($R$, writes to two blocks)
- write: slightly higher than writing to a single disk (worst-case seek and rotational delay)
steady-state throughput
- write: ($\frac{N}{2}\cdot S$, writes to $N$ blocks)
- read: same as write ($\frac{N}{2}\cdot S$, each disk receives a request for every other block)
random reads/writes
- read: $N\cdot R$
- write: $\frac{N}{2}\cdot R$

38.6 RAID Level 4: Saving Space With Parity

Parity-based approaches attempt to use less capacity and thus overcome the huge space penalty paid by mirrored systems. They do so at a cost, however: performance.

Disk 0	Disk 1	Disk 2	Disk 3	Disk 4
0	1	2	3	P0
4	5	6	7	P1
8	9	10	11	P2
12	13	14	15	P3

Simple function: XOR, which makes a row having even number of 1

Analysis

capacity: $(N-1)\cdot B$
reliability: tolerates 1 disk failure
performance
- steady-state: $(N-1)\cdot S$ (full-stripe write: calculate parity bits first then write all disks in parallel)
- random
  - read: $(N-1)\cdot R$
  - write: $R/2$ (the parity disk is the bottleneck)
    - additive parity: read in parallel, calculate XOR, and write in parallel (larger RAIDs require a high number of reads)
    - subtractive parity: $P_\mathrm{new}=(C_\mathrm{old}\oplus C_\mathrm{new})\oplus P_\mathrm{old}$ (Why not always better?– because it has to read the parity disk?)

Small-write problem: the parity disk is a bottleneck under small writes which prevents parallelism

38.7 RAID Level 5: Rotating Parity

Disk 0	Disk 1	Disk 2	Disk 3	Disk 4
0	1	2	3	P0
5	6	7	P1	4
10	11	P2	8	9
15	P3	12	13	14
P4	16	17	18	19

Analysis

random write: $\frac{N}{4}\cdot R$ (four I/O operations) (aren’t the reads/writes parallel?–the data disk itself serves as parity disk as well)

39 Interlude: Files and Directories

Thus, the OS must take extra care with such a device: this is where users keep data that they really care about.

39.1 Files And Directories

file: linear array of bytes
- low-level name (inode number)
directory
- low-level name (inode number)
- a list of (user-readable name, low-level name) pairs

39.3 Creating Files

Create a new file: call open() and pass O_CREAT flag:

int fd = open("foo", O_CREAT|O_WRONLY|O_TRUNC, S_IRUSR|S_IWUSR);

O_CREAT: create the file if it does not exist
O_WRONLY: read only
O_TRUNC: if exists already, truncates it to a size of zero bytes

39.5 Reading And Writing, But Not Sequentially

Reading some random offsets within the document:

off_t lseek(int fildes, off_t offset, int whence);

fildes: file descriptor
offset: the file offset to a particular location
whence
- SEEK_SET: the offset is set to offset bytes
- SEEK_CUR: the offset is set to its current location plus offset bytes
- SEEK_END: the offset is set to the size of the file plus offset bytes

Calling lseek() does not perform a disk seek

The “current offset” tracked by the OS:

implicitly updates when read or write
explicitly updates with lseek

The file structure:

struct file {
 int ref;
 char readable;
 char writable;
 struct inode * ip;
 uint off;
};

The open file table:

struct {
 struct spinlock lock;
 struct file file[NFILE];
} ftable;

39.6 Shared File Table Entries: `fork()` And `dup()`

The open file table entries are shared by 1. the parent and child processes created by fork(); 2. one file descriptor and its duplicates created by dup(), with the reference count incremented.

39.7 Writing Immediately With `fsync()`

Calls fsync() to force all dirty (not yet written) data to disk.

int rc = write(fd, buffer, size); assert(rc == size);

In some cases, it is also needed to fsync() the containing directory to ensure that the file is part of the directory.

Not surprisingly, this type of detail is often overlooked, leading to many application-level bugs.

mmap() and persistent memory

By combining the persistence of files with the access semantics of memory, file-backed memory mappings support a software abstraction called persistent memory.

39.8 Renaming Files

rename(char *old, char *new)

rename() call is usually implemented as an atomic call.

An example to use this kind of guarantee:

int fd = open("foo.txt.tmp", O_WRONLY|O_CREAT|O_TRUNC, S_IRUSR|S_IWUSR);
write(fd, buffer, size);
fsync(fd);
close(fd);
rename("foo.txt.tmp", "foo.txt");

39.9 Getting Information About Files

Use stat() or fstat() to access metadata of files or directories.

39.10 Removing Files

Use unlink() to remove files.

39.11 Making Directories

Use mkdir() to create a directory.

An empty directory has two entries:

refer to itself
refer to its parent

39.12 Reading Directories

Use opendir(), readdir() and closedir() to read contents of a directory.

DIR * dp = opendir(".");
assert(dp != NULL);
struct dirent * d;
while ((d = readdir(dp)) != NULL) {
 printf("%lu %s\n", (unsigned long) d->d_ino, d->d_name);
}
closedir(dp);

39.13 Deleting Directories

Call rmdir() to delete a directory.

39.14 Hard Links

Call link() to make an entry in the file system tree, simply creates another name in the directory and refers it to the same inode number.

ln file file2

When creating a file, two things happened:

making a structure (the inode) that tracks relevent information

linking a human-readable name to that file, and put that link into a directory

When the file system unlinks a file, it checks the reference count (link count) within the inode number, and only free the node and related data blocks when it reaches zero.

39.15 Symbolic Links

Call symlink() to make an symbolic link (soft link) to a file.

ln -s file file2

Differences from hard links:

symbolic links is actually a file itself of a different type
possibility for dangling reference

39.16 Permission Bits And Access Control Lists

Permission bits consist of three groupings:

owner
group
other (anyone)

The owner of the file can change the permissions.

But now you have learned an even more important lesson: there can be no secrets in a good marriage, even within the file system.

The Time Of Check To Time Of Use (TOCTTOU) problem

The problem is caused by the gap between the check and the update.

39.17 Making And Mounting A File System

And mkfs said, let there be a file system!

Mount: to take an existing directory as a target mount point and paste a new file system onto the directory tree at that point.

Mount unfies all file systems into one tree, making naming uniform and convenient.

40 File System Implementation

vsfs (the Very Simple File System)

The file system is pure software; unlike our development of CPU and memory virtualization, we will not be adding hardware features to make some aspect of the file system work better (though we will want to pay attention to device characteristics to make sure the file system works well).

40.2 Overall Organization

The disk is divided into:

data blocks: user data
inodes
allocation structures: to track whether inodes or data blocks are free or allocated
superblock: contains information about the fie system

40.3 File Organization: The Inode

Inode (index node): the data structure that holds the metadata of a given file

i-number (low-level name): the number that the inode is implicitly referred to; disk position can be calculated by the number

Calculation:

$$ block = (inumber\times sizeof(inode)) / blockSize $$

$$ sector = ((block\times blockSize) + inodeStartAddr) / sectorSize $$

Metadata in a inode:

type: directory or file (kept in the directory entry and thus is not found in the inode)
size
number of blocks
protection information
time
data block position on disk

The Multi-Level Index

Indirect pointer: points to a block that contains pointers which each points to user data

An inode may have some fixed number of direct pointers and a single indirect pointer.

If a file grows large enough, an indirect block is allocated (from the data block) and the inode’s indirect pointer is set to point to it.

Extent-based approaches: a disk pointer plus a length; allow for more than one extent

Linked-based approaches: use a linked list to keep track all blocks of a file

Double indirect pointer: refer to a block that contains indirect pointers

Most files are small: a small number of direct pointers is sufficient

40.4 Directory Organization

Deleting a file can leave an empty space in the middle of the directory; a new entry may reuse an old, bigger, deleted entry.

File systems often streat directories as a special type of file, and thus they have inodes in the inode table.

40.5 Free Space Management

To track free inode/data blocks:

use bitmap
use free list
use more soplisticated ds, e.g. B-tree

Pre-allocation policy: reserve a sequence of blocks for a newly-created file, to guarantee that a portion of the file is contiguous on the disk.

40.6 Access Paths: Reading and Writing

Reading A File From Disk

The timeline of reading a file /foo/bar:

open("/foo/bar", O_RDONLY)

read the root inode (/) (whose i-number is well known; 2 in most UNIX FSs)
read / data (to look for an entry for foo)
read foo inode
read foo data (to look for an entry for bar)
read bar inode

read(...)

read bar inode
read specific block of bar’s data
write bar inode for last access time

Yes, life can get pretty bad when reading a file; as you’re about to find out, writing out a file (and especially, creating a new one) is even worse.

Reads don’t access allocation structures (e.g. bitmaps); they are only accessed when allocation is needed. There is no need to make sure a block is allocated when the inode already points to it

Writing A File To Disk

Each write to a file logically generates 5 I/Os:

read file inode
read data bitmap
write data bitmap (to mark the newly-allocated block as used)
write file data
write file inode

For file creation:

read inode and inode data traversly to find the containing directory
read containing directory data
read inode bitmap (find a free inode)
write inode bitmap (to mark the inode allocated)
write directory data (link)
read/write directory inode
read/write inode of the new file (initialize)
additional I/O might be needed if the directory needs to grow

40.7 Caching and Buffering

Most file systems aggressively use system memory (DRAM) to cache important blocks.

Early file systems use a fixed-size cache to hold popular blocks; this static partitioning method can be wasteful.

Modern systems use a dynamic partitioning approach.

Static vs. dynamic partitioning

static: predictable performance

dynamic: better utilization; more complex to implement

Buffering: delay writes to reduce I/O

In this case, laziness (in writing blocks to disk) is a virtue.

41 Locality and The Fast File System

41.1 The Problem: Poor Performance

Main issue of the old UNIX file system: data was spread all over the place, thus had expensive positioning costs

Some problems:

the inode and the data block are often far away
the file system is getting more and more fragmented
the block size is too small, so that transferring data is inefficient

41.3 Organizing Structure: The Cylinder Group

Cylinder (柱面): a set of tracks on different surfaces that are the same distance from the center

FFS divides the disk into a number of cylinder groups.

As modern drives only export a logical address space of blocks, modern file systems instead organize the drive into block groups, each of which is a consecutive portion of the disk’s address space. By placing two files within the same group, FFS can ensure that accessing one after the other will not result in long seek time.

FFS needs to have the ability to place files and directories into a groups and track relevant information, thus all the structures that a file system is expected to have is included in each group.

The components of a single cylinder group:

a copy of the super block
a per-group inode bitmap and data bitmap
inode region
data block region

41.4 Policies: How To Allocate Files and Directories

The basic mantra is simple: keep related stuff together (and its corollary, keep unrelated stuff far apart).

A few placement heuristics:

directories: find the group
- with a low number of allocated directories (to balance directories across groups)
- with a high number of free inodes (to be able to allocate a bunch of files)
files:
- allocate the data blocks in the same group as its inode
- places all files that are in the same directory in the cylinder group of the directory they are in

41.6 The Large-File Exception

Filling an entire block group with large files will prevents related files from being placed within the same group.

FFS’ policy for large files: after some number of blocks are allocated into the first block group, it places the next large chunk of the file in another block group.

To address the performance problem: choosing chunk size carefully. By amotization, if the chunk size is large enough, the seek time will be relatively little.

FFS’s approach

the first 12 blocks are placed in the same group as the inode
each subsquent indirect block and the blocks it points to are placed in a different group

Note that the trend in disk drives is that transfer rate improves fairly rapidly, as disk manufacturers are good at cramming more bits into the same surface, but the mechanical aspects of drives related to seeks (disk arm speed and the rate of rotation) improve rather slowly.

41.7 A Few Other Things About FFS

Internal fragmentation: 4KB blocks are used by many 2KB files

Sub-blocks: 512-byte blocks can be allocated to files; copy to a 4KB block when the size grows beyond 4KB

To address the performance issue caused by copies: buffer writes, to avoid this in most cases.

A refined disk layout (parameterization): skipping a block when issuing a sequential read (e.g. 0 6 1 7 2 8 3 9 4 10 5 11); to issue the performance problem: using track buffer.

FFS also supports:

long file names
symbolic links

Certainly all modern systems account for the main lesson of FFS: treat the disk like it’s a disk.

notes on liang2kl's Blog

Operating Systems: Three Easy Pieces 学习笔记（五）

26 Concurrency: An Introduction

26.1 Why Use Threads?

26.4 The Heart Of The Problem: Uncontrolled Scheduling

27 Interlude: Thread API

27.4 Condition Variables

28 Locks

28.4 Evaluating Locks

28.6 A Failed Attempt: Just Using Loads/Stores

28.7 Building Working Spin Locks with Test-And-Set

28.8 Evaluating Spin Locks

28.9 Compare-And-Swap

28.10 Load-Linked and Store-Conditional

28.11 Fetch-And-Add

28.13 A Simple Approach: Just Yield, Baby

28.14 Using Queues: Sleeping Instead Of Spinning

29 Lock-based Concurrent Data Structures

29.1 Concurrent Counters

29.2 Concurrent Linked Lists

29.3 Concurrent Queues

29.4 Concurrent Hash Table

30 Condition Variables

30.1 Definition and Routines

30.2 The Producer/Consumer (Bounded Buffer) Problem

30.3 Covering Conditions

Read Phenomena and Isolation Levels

Read phenomena

Dirty reads

Non-repetable reads

Phantom reads

Isolation levels

Serializable

Repeatable reads

Read committed

Read uncommitted

Concurrency Control for Database Transactions

Serial and Serializable Schedules

Conflict-Serializability

Enforcing Serializability by Locks

Locking Systems With Several Lock Modes

An Architecture for a Locking Scheduler

Hierarchies of Database Elements

The Tree Protocol

Concurrency Control by Timestamps

Concurrency Control by Validation

Comparison of Three Concurrency-Control Mechanisms

总结

Operating Systems: Three Easy Pieces 学习笔记（四）

13 The Abstraction: Address Spaces

13.3 The Address Space

13.4 Goals

14 Interlude: Memory API

15 Mechanism: Address Translation

15.3 Dynamic (Hardware-based) Relocation

15.4 Hardware Support: A Summary

15.6 Summary

16 Segmentation

16.1 Segmentation: Generalized Base/Bounds

16.2 Which Segment Are We Referring To?

16.3 What About The Stack?

16.4 Support for Sharing

16.5 Fine-grained vs. Coarse-grained Segmentation

16.6 OS Support

17 Free-Space Management

17.1 Assumptions

17.2 Low-level Mechanisms

Splitting and Coalescing

Tracking The Size Of Allocated Regions

Embedding A Free List

Growing The Heap

17.3 Basic Strategies

Best Fit

Worst Fit

First Fit

Next Fit

17.4 Other Approaches

Segregated Lists

Buddy Allocation

18 Paging: Introduction