study on liang2kl's Blog

中国主流媒体视角与游戏产业发展的割裂——以《人民日报》游戏相关报道（1989-2021）为例

liang2kl@outlook.com (Liang Yesheng) — Thu, 28 Jul 2022 23:42:15 +0800

本文为 2022 年夏季学期程祥钰老师《写作与沟通》课程游戏主题长文终稿

摘要主流媒体几乎参与了中国游戏产业发展的全过程，但其报道相较于游戏产业的发展而言存在着缺失和偏差，存在着明显的割裂。纵观中国游戏产业发展史，这种割裂长期存在，总体上在缓慢弥合，但最近又有重新加大的趋势。由于主流媒体宣传官方主张、传播主流声音的性质，其与游戏产业的发展存在着产生割裂的必然性，而游戏相关政策与主流舆论对割裂的产生具有重要作用。二者之间割裂的影响具有两面性，游戏产业的健康发展需要主流媒体保持适当的距离。

关键词 游戏；游戏产业；主流媒体；游戏报道；人民日报

主流媒体对游戏¹的报道自其进入中国以来就一直存在。《人民日报》对游戏的间接报道最早可追溯到 1981 年²；此后，随着中国游戏产业的不断发展，《人民日报》对游戏的直接报道数量基本呈递增趋势，目前游戏已成为其热门话题之一。可见，主流媒体几乎参与了中国游戏产业发展的全过程。

然而，尽管主流媒体似乎并未缺席中国游戏行业的发展历史，且主流媒体的视角与中国游戏产业的发展大体上步调一致，但其在某些历史阶段的报道忽略了游戏产业发展的重要历程，且立场存在明显的偏向性；主流媒体的视角与游戏产业的发展之间存在着缺失和偏差。例如，对于 20 世纪 80 年代末到 21 世纪初之间的历史时期，在游戏产业发展史的叙事中，“20 世纪 90 年代新出现的游戏杂志群开始频繁讨论游戏中的文化议题，其核心主张便是如何生产作为文化抵抗媒介的国产游戏。”，自此产生了第一批国产游戏³；而在《人民日报》的叙事中，没有一篇文章提及此历史进程，反之在重复讨论“电子游戏热中的负效应”⁴、“电子游戏的暴力内容对孩子不利”⁵、“青少年沉迷电子游戏现象”⁶等游戏的负面形象。主流媒体的视角并不总是与游戏产业的发展相统一，二者之间存在明显的割裂。

对于媒体报道对游戏的舆论评价所产生的影响，何威等在研究《人民日报》游戏报道的话语变迁时指出，“不论普罗大众、文化精英或政府官员，多少都依赖媒体报道提供素材、主题、框架、观点，来理解和对待游戏及游戏玩家”⁷。具体到主流媒体，汉娜·沃曼在研究西方媒体对中国游戏文化和商业的报道时指出，“官方媒体成为了一种精英象征，代表社会上‘最有发言权’的群体，也因此可以优先影响‘大众的思想’”⁸。正因为《人民日报》等主流媒体反映了官方的态度，且有着巨大的舆论影响力，主流媒体的视角与游戏产业发展之间的割裂，以及这种割裂对中国游戏产业带来的深刻影响，值得关注和反思。

本文将以《人民日报》以游戏为主题的文章⁹为例，从中国主流媒体游戏报道话语演变及游戏产业发展的历史入手，揭示二者之间的割裂性，分析这种关系背后的影响因素，进而探讨这种关系对中国游戏产业的影响。

中国游戏产业发展的过程中，主流媒体的视角看向何处

中国游戏产业发展的历史可以清晰地划分为若干个阶段，每个发展阶段既有标志性的事件作为开端，又有明确的主题贯穿其中。《人民日报》的视角是否能够覆盖中国游戏发展的重大事件，其报道是否能够呈现当时游戏产业发展阶段的现状？我们以中国游戏发展的标志性事件为点、以发展阶段中的主要趋势为线，对《人民日报》游戏报道的视角进行分析。

1981 年北京第一轻工业研究研制出 YQ-1 型游戏机，标志着当代中国游戏史的开端。虽然同年《人民日报》第一次在报道中提到了电子游戏，但距离《人民日报》第一篇以游戏为主题的文章发表还有 8 年时间。在这段时间里，中国游戏产业尚未开始全面发展；电子游戏以学习机的面目出现，其游戏的本质被压抑，与大众尚有一定距离，游戏并未产生足够的社会影响。因此，游戏产业并不足以得到主流媒体的关注，这导致了主流媒体叙事中这段历史中的缺失。

1980 年代末，中国东南沿海逐渐具备规模化的任天堂兼容机生产能力，电子游戏开始成为社会现象¹⁰，自然也逐渐被主流媒体所关注。《人民日报》对游戏的第一篇直接报道（1989 年）便出现在此时期：这篇报道聚焦上海大量游戏机涌现而导致学生沉迷的问题，呼吁有关部门加强管理¹¹，可见此时电子游戏已具有相当的社会影响。值得注意的是，此后 10 年《人民日报》对于游戏的 15 篇报道均停留在其消极的社会影响上，并未关注电子游戏本身或其背后产业的发展，直到1999 年才出现第一篇介绍电子游戏的文章¹²。可见，此时期《人民日报》的游戏报道虽然逐渐增多，但其视角极其局限。这一结论在同一发展阶段的另一个标志性事件及其带来的发展趋势中可以得到印证：1994 年 10 月，金盘公司游戏部发表首款国产游戏《神鹰突击队》；一年后，《电子游戏软件》一篇题为《乌鸦·乌鸦·叫：该谈大陆电子游戏业了》的文章¹³批评了电子游戏中中国文化元素的缺失，引发了游戏界的广泛讨论，从此直到 21 世纪初，游戏国产化成为了中国游戏产业发展的主旋律；然而，如前文所述，这一阶段《人民日报》的视角仍局限在电子游戏产生的影响上，对游戏产业这一关键发展历程完全没有涉及。在强调电子游戏的危害的同时忽视游戏本身及游戏行业的发展现状，主流媒体的报道几乎与游戏产业的发展完全割裂。

2000 年 7 月到 9 月，全国开展电子游戏经营场所的专项治理，出台了持续 14 年之久的“游戏机禁令”¹⁴，从此中国失去游戏软件产业升级的硬件基础，转向代理海外网游的新模式，从此进入长达十几年的网游时代¹⁵。与之前的缺位不同，《人民日报》在此次转变中不仅在场，还起到了十分重要的推动作用：从 2003 年 3 月 3 日开始，《人民日报》以前所未有的频率关注游戏，在专项治理开始前的 3 个月连续刊发 8 篇关于游戏厅现状及其危害的文章，从时间上看这些报道导致了政策的制定和实行；随后的 4 个月中，《人民日报》又刊发了 4 篇文章介绍专项治理的效果；这一年，《人民日报》的全部 13 篇游戏报道均与专项治理相关。值得关注的是，虽然主流媒体的报道仍然局限在游戏的负面影响上，其在游戏产业的此次转折中扮演了一个完全不同的角色，在以往的旁观者、批判者之外，短暂地成为了游戏产业发展的推动者，尽管游戏产业本身在其报道中并未在场。

2001 年开始，中国游戏厂商开始代理国外网络游戏，开启了中国的网游时代。《人民日报》在 2002 年和 2003 年分别对网易推出的国产新网游《大话西游》¹⁶和金山推出的新网游《剑侠情缘网络版》¹⁷进行的报道，是其最早对具体游戏的主题报道；2003 年，《人民日报》发表了一篇解读网络游戏的文章，详细介绍了网友相关的政策和产业发展情况，并强调辩证地看待游戏的影响¹⁸。虽然这只是这个时代开端的几篇报道，但却反映了这个阶段主流媒体与游戏产业关系的明显变化：主流媒体终于主动向游戏产业靠近，游戏产业开始作为主体在主流媒体的报道中出现；主流媒体与游戏产业的对立开始得到缓解，其对游戏产业发展报道的缺失逐渐得到弥补。尽管如此，2009 年之前的主流媒体报道仍然带有对游戏的严重偏见。例如，“网瘾治疗专家”杨永信作为“靠着一颗爱心，挽救深陷泥潭的学子”的“好人”出现在人民日报的报道中¹⁹，尽管此人在当时饱受争议。主流媒体报道立场偏向性严重的问题导致其对游戏产业的报道不够全面客观，主流媒体与游戏产业发展之间的割裂继续存在。

2009 年后，随着“转变经济发展方式”、“科技强国”、“中国文化走出去”等主流叙事的流行，游戏的科技与文化属性成为这一时期《人民日报》报道游戏的主要主题：2009 至 2011 年，“产业经济”框架在游戏报道中占据绝对主流；而 2012 年到 2017 年，“文娱新方式”框架占比陡增，维持在26%至36%之间波动²⁰。可以看到，《人民日报》在这个时期从以往的游戏产业的旁观者变成了参与者，利用自身的舆论影响力服务于国家发展需要塑造舆论话语，有意识地引导游戏产业与国家发展战略接轨。为了达到此目标，《人民日报》对电子游戏危害的讨论相比以往几乎销声匿迹，报道角度也更加多元。以上种种变化，在客观上促进了主流媒体与游戏产业之间的交流，主流舆论与游戏产业之间更加平衡，同时一定程度上缓解了由于立场偏向而导致的割裂。

2017 年，《王者荣耀》风靡全国，带来了新的一轮网络手游热，由此沉寂已久的游戏成瘾话题又重新成为主流议题；相比之下科技创新、文化输出等报道框架虽仍然存在，但明显有所减少。2018 年世卫组织首次将游戏成瘾列为疾病²¹后，《人民日报》在游戏成瘾以外的报道框架进一步被压缩，直到 2021 年仍是如此。主流媒体与游戏产业之间的关系在当下又重新变得紧张，二者之间的割裂有重新加大的趋势。

什么造成了主流媒体与游戏产业发展的割裂

如前文所述，主流媒体与游戏产业发展之间的关系在40多年的中国游戏史中经历了明显的起伏。主流媒体与游戏产业发展之间的割裂，一方面表现在主流媒体对游戏的报道中游戏产业作为主体的缺失，另一方面表现在主流媒体报道话语的偏向性；这种割裂长期存在，总体上在缓慢地弥合，但有重新加大的趋势。这种关系变化的背后，有什么因素对二者之间的割裂产生影响？

首先需要指出的是，以《人民日报》为首的官方媒体的报道角度与游戏产业的发展之间存在产生割裂的必然性，所有对主流媒体与游戏产业关系的影响因素的讨论都必须基于这一前提。《人民日报》是党中央的机关报，其作用是“上连党心，下接民心”²²；《人民日报》并不是为游戏产业服务的，其目的是宣传国家中央主张、传播社会的主流声音，这就导致一旦游戏产业的发展与中央政策导向或主流舆论发生冲突，《人民日报》必然会在一定程度上站在游戏产业的对立面，其报道与游戏产业自身的发展历程必然会产生割裂，这与持小众、个人观点的游戏媒体有着本质的不同。

在这一前提下，通过前文对游戏发展过程的归纳，不难发现游戏相关政策对二者关系的重要影响。一个鲜明的例子是《人民日报》对游戏产业在上世纪 90 年代以及 2012 至 2017 年这两段发展历史截然不同的态度：这两段时期中，游戏产业都致力于将中国文化意象融合到游戏中，然而《人民日报》在前一段时期中完全忽视了游戏产业发展进程，将游戏妖魔化，而在后一段时期中却积极地推动着此进程的发展，游戏的负面报道几乎不再存在。难道 90 年代中国游戏产业的探索没有意义，而 2012 年以后游戏的负面影响就完全消失了吗？显然不是。政策的因素在其中起到了至关重要的作用：第一部与促进国产游戏发展的政策“民族网络游戏出版工程”在 2004 年的网游时代才面世，90 年代主流媒体的报道更多基于社会主流舆论产生；而在 2010 年，“讲好中国故事”、“中国文化走出去”已成为中国文化工作的主流导向²³，同时文化部指导网游厂商发起“家长监护工程”，意在引导未成年人“玩健康的游戏”和“健康地玩游戏”，游戏产业的发展得到了国家战略与政策的支持。尽管此时网游成瘾问题依然严重，对游戏的主流舆论评价仍然呈负面，但显然政策导向对主流媒体话语塑造具有更大的影响力。

另外，由于主流媒体传播的是社会的主流声音，其对游戏产业的报道受社会主流舆论影响较大。这种影响不仅如前文所述体现在报道内容和价值取向上，更隐含在由不同群体舆论话语权的不同而导致的主流媒体报道立场偏向性中。根据何威等的研究，在其所统计的《人民日报》历年对游戏负面报道中，引用专家学者观点的有 49 篇，引用管理部门观点的有 40 篇，而引用玩家的观点仅有 11 篇²⁴。可以看到，在“游戏的负面影响”这一主题下，与游戏产业直接相关的主体的话语权是缺失的，无论是游戏从业人员还是游戏玩家；他们不满与自身话语无法得到关注，却又对此无能为力。游戏杂志《家用电脑与游戏机》在 2000 年 7 月刊登了一篇反驳《光明日报》将游戏称作“电子海洛因”的文章，作者在其中愤怒地抱怨道：

为什么作者宁愿“暗访”地下游戏室三次，却不愿意去一趟正大光明的游戏公司或游戏媒体，甚至多走访一些真正热爱电脑游戏的普通人呢？看来是“非不能也，实不愿也”，既然早就定下了“电子游戏是瞄准孩子的‘电子海洛因’”的标题（或者说是结论），一切就要围绕这个中心服务，如此而已。²⁵

这篇文章是主流媒体与游戏产业之间冲突的直接表达，虽然已有 20 多年的历史，但即使是现在，由舆论话语权不平等导致的主流媒体与游戏产业发展之间割裂的问题也还没有得到较好的改善。这是因为，中国电子游戏在上世纪 90 年代刚刚兴起，而其影响力进一步扩大则是在十年后开始的网游时代。绝大部分出生于游戏流行年代的“游戏原住民”出生于 90 年代或 21 世纪，目前大多为 20 多岁的青少年，对主流舆论的影响力还远未成型，而他们恰恰是与游戏产业联系最为密切、最为庞大的群体，因此长期作为无法发声的、被批评的对象出现在游戏报道中；另一方面，40 多年来塑造主流舆论的群体大多作为旁观者的身份目睹游戏产业的发展，与游戏产业之间存在着天然的距离；即使对于部分人而言这种距离可能随着了解的不断深入而缩短，但大多数人由于主流媒体对游戏形象持续的污名化构建，其与游戏产业的距离只会越来越远，从而进一步加剧主流媒体与游戏产业发展之间的割裂。

总结：如何看待主流媒体与游戏产业发展的割裂

如前所述，主流媒体与游戏产业发展之间的割裂主要表现在主流媒体对游戏的报道中游戏产业作为主体的缺失，以及主流媒体报道话语的偏向性。随着历史进程的推进，主流媒体报道逐渐向游戏产业靠近，报道的价值取向取向也逐渐多元化，但是主流媒体与游戏产业发展之间的割裂是必然存在，而且将长期存在的：游戏易成瘾等特性导致大众舆论关于游戏社会影响的警惕，以及国家意志关于游戏对国家发展的影响的考虑一直存在，而这两者是影响主流媒体报道话语的最重要因素。尽管舆论对游戏的警惕可能会随着主流话语权的变更而放松，有关政策可能会因为游戏与国家发展向契合的属性而对游戏发展更加宽容，但它们背后的决定因素——游戏的消极影响是始终存在的。

与游戏媒体的报道与游戏产业发展较为同步的关系相比，主流媒体与游戏产业之间保持适当的距离在某些方面是有益的。主流媒体对游戏消极影响的关注催生了游戏防沉迷、实名制等机制的产生，在客观上对规范游戏产业的发展起到了一定的作用；同时，主流媒体对游戏产业发展的参与程度越来越高，利用自身舆论影响力引导游戏产业与国家战略接轨，在一定程度上促进了游戏产业的发展。但是，主流媒体对游戏的报道需要避免重新回到以预设立场完全否定游戏价值、无视游戏产业发展历程的状态，这对游戏产业发展，以及游戏与人之间的关系存在着不可忽视的消极影响。

游戏作为复杂的文化、社会现象，其影响必定是多元的。游戏产业的发展需要具有社会和舆论影响力的主流媒体起到化解不同群体之间冲突与矛盾、促进游戏产业与社会发展相统一的作用。保持一定的距离，批判地看待游戏，给出有建设性的意见，不失为达到这一目标的一种行之有效的方法。

如无特别说明，本文中“游戏”专指“电子游戏”，即一切“依靠电子媒体平台而运行的互动游戏”。 ↩︎
戚休：《广州东方宾馆利用外资更新客房效果显著》，1981年05月25日《人民日报》第2版。 ↩︎
邓剑：《中国电子游戏文化的源流与考辨》，《上海文化》2020年第12期，第45页。 ↩︎
刘建华：《电子游戏热中的负效应》，《人民日报》1990年4月4日第6版。 ↩︎
夷水寒：《电子游戏的暴力内容对孩子不利》，《人民日报》1991年2月12日第5版。 ↩︎
马永梅，吴光志，周开荣：《游戏岂能“犯规”——青少年沉迷电子游戏现象扫描》，《人民日报》1993年12月21日第10版。 ↩︎
何威，曹书乐:《从“电子海洛因”到“中国创造”: 〈人民日报〉游戏报道（1981-2017）的话语变迁》，《国际新闻界》第40期，2018年，第58页。 ↩︎
汉娜·沃曼：《西方新闻媒体的“中国-东方主义”：被漫画化的游戏文化和商业》，宋雅文译，见何威、刘梦霏编：《游戏研究读本》，上海：华东师范大学出版社，2020年，第220页。 ↩︎
使用“人民日报图文数据库”对“游戏、网游、网瘾、手游、电竞、电子竞技”进行标题检索，包括新闻、评论、读者来信等所有类别，并剔除无关文章，得到1989年至2021年刊登的共194篇相关文章。 ↩︎
邓剑：《中国当代游戏史的思想谱系：从本土现代化到资本与市场逻辑》，《探索与争鸣》2022年第2期，第166页。 ↩︎
朱丹：《学生入迷，家长忧心——上海街头游戏机成灾》，《人民日报》1989年4月17日第2版。 ↩︎
《电子游戏的分类》，《人民日报》1999年3月14日第7版。 ↩︎
本刊评论员：《乌鸦·乌鸦·叫:该谈大陆电子游戏业了》，《电子游戏软件》1995 年第 4 期。 ↩︎
国务院办公厅等：《国务院办公厅转发文化部等部门关于开展电子游戏经营场所专项治理意见的通知》，2000年6月15日，http://www.gov.cn/gongbao/content/2000/content_60240.htm，2022年7月21日。 ↩︎
邓剑：《中国当代游戏史的思想谱系：从本土现代化到资本与市场逻辑》，《探索与争鸣》2022年第2期，第169页。 ↩︎
周萍：《网易推出网络游戏〈大话西游〉》，《人民日报》2002年1月1日第5版。 ↩︎
崔扬：《国产游戏新出网络版》，《人民日报》2003年7月30日第11版。 ↩︎
任建民：《网络游戏：超越电影票房之后（热点解读）》，《人民日报》2003年12月15日第5版。 ↩︎
刘成友：《道德力量汇聚孔孟之乡——全国道德模范与身边好人现场交流活动侧记》，《人民日报》2009年4月4日第3版。 ↩︎
何威，曹书乐:《从“电子海洛因”到“中国创造”: 〈人民日报〉游戏报道（1981-2017）的话语变迁》，《国际新闻界》第40期，2018年，第71-72页。 ↩︎
世界卫生组织：《国际疾病分类》第11次修订本，https://www.who.int/zh/news-room/spotlight/international-classification-of-diseases。 ↩︎
汪晓东：《让党心和民心贴得更紧——写在习近平总书记“2·19”重要讲话发表3周年之际》，《人民日报》2019年2月19日第1版。 ↩︎
温红彦，李舫：《向世界讲述好中国故事——文化部副部长赵少华谈中华文化“走出去”》，2010年9月30日。 ↩︎
何威，曹书乐:《从“电子海洛因”到“中国创造”: 〈人民日报〉游戏报道（1981-2017）的话语变迁》，《国际新闻界》第40期，2018年，第77页。 ↩︎
王亚晖：《中国游戏风云》，北京：中国发展出版社，2018年11月，第113至114页。 ↩︎

2022 春季学期总结

liang2kl@outlook.com (Liang Yesheng) — Mon, 20 Jun 2022 16:26:00 +0800

春季学期对我来说有一个定律，或者说一个魔咒：学业一塌糊涂，学业之外收获甚多。

20 年的春季学期在家里上课，鬼使神差般地接触了 iOS 开发，从此一整个学期都在写一个 app，上课和下课都在写代码，导致那个学期的绩点异常整齐，除了体育之外全部都是 3.6，排名刷刷地往下掉，还退了微积分和概统。21 年春季学期进行转系，同时在高强度写树洞 app，没什么心思听课，最后离散 (2) 喜提 3.0，绩点进一步雪崩，并且又退了一次微积分（于是大学总共修了三次微积分 A2，唉！）。

但是，这两个学期对于我大学四年有着非常深远的影响。最重要的当然是开始写程序之后逐渐喜欢上了 CS，并直接导致最终转到计算机系；写树洞是第一次接触正经的 C/S 架构的开发，积累了大量的网络、git、前后端等等知识，这也使得我现在的技术能力放在贵系也还算可以（？）。

与之相反，秋季学期在经过前一个春季学期的“惨痛”经历后，往往走向另一个极端：抑制自己在其他地方的兴趣，以 GPA 为首要目标。这当然是有效的，我秋季学期的 GPA 都挺不错；但同时也是无聊的，天天跑图书馆写作业刷题对某些同学来说可能是享受，但我真的不喜欢以提高分数为唯一目的的学习。

我也曾妄想过“小孩子才做选择，我全都要”，每个学期之前都想着如何能够更高效地利用时间，把学业和其他的所有事情都做到极致。然而，这种想法未曾成为现实，主要有几个原因：第一，我时间管理极差，无法利用正常作息下白天和晚上大部分可以利用的时间。第二，我对于我感兴趣的事情往往有着极高的热情，而这种热情往往导致时间安排上极不均衡的优先级，进而导致一些目前阶段可能（或者说表面上）较为重要的任务没有充足的时间完成（用计算机的专业术语来说，starvation /笑）。

尽管在这个学期开学前有想过避免以前春季学期的问题，但很不幸，课内学习不够投入的问题并没有得到解决。现在想来，主要的原因是软工占用了大部分时间，其他课程上得也不舒服：数电 tp 老师讲课风格实在是不喜欢，人智导讲的内容实在是没兴趣，搜索引擎实在是太水，而我又比较喜欢捣鼓软工所用的一些技术……因此很多课都用来写软工了（数电前几周的课、搜索引擎后面的课、人智导大部分的课）。微积分倒是坚持认真听了十几周，但耐不住记 P/F 这一重大利好（终于能在不影响 GPA 的情况下修完微积分了 /哭），在记了 P/F 之后就没听了。

另一个原因，与之前相似，是课外的工作占用了大量的时间，这个学期主要是科研。这个学期开始跟着学长做科研项目，从寒假开始一直在恶补操作系统的知识，后面换方向后又从零开始学习了数据库的知识。等到终于开始上手代码了，花了很多时间调试和修改以前论文的代码，然后考试周就来了，只能搁置。总的来说，这一段科研经历的开端并不是特别顺利，也还没有完全找到自己感兴趣的方向，但总算是开了个头，也算是迈出了一小步。

总之这个学期就这么不太明白地过去了。有很多不好的经历：软工开题后焦虑、软工进展不顺、数电听不懂、数电实验遇到各种问题、人智导没兴趣、科研进展缓慢……同时也有很多美好的回忆：软工助教和用户代表高度评价我们的项目、软工做了一个（自认为）非常精彩的展示、科研开了一个还算可以的头……

和一个学期前相比，我的乐观精神显然消退了不少。或许是因为第一次遇到了对于计算机不太感兴趣或不太擅长的方面，或许是发现在某些方面自己没办法做到某些同学的高度，或许是在读文章时感受到自己的水平与顶尖水平的巨大差距……总之，发现有些事情是自己做不到的，这对于我自己能力的认识，以及对未来的规划都产生了不小的影响。当然，我也时常想起软工被用户代表和 rls 夸上天的时候的那种兴奋感。或许，我还是能够在一些地方找到自己的独特之处的。

至于暑假和下学期，还是花多点时间在科研上吧。如果这条路能走通，那就坚定走下去；如果走不通，那就要及时考虑一下其他道路了。

音乐剧 Come From Away 赏析

liang2kl@outlook.com (Liang Yesheng) — Sat, 28 May 2022 14:27:27 +0800

本文为 2022 年春季学期《西方音乐剧史》课程期末作业

《Come From Away》的故事原型是，在 911 事件发生时，加拿大的一个小镇接纳了超过 7000 个因飞行禁令临时降落在此地的乘客。故事本身并不复杂，主要讲述了当地居民尽其所能为滞留乘客提供住宿等帮助、乘客从一开始的恐慌、焦虑到最终融入当地人的生活中，并获得了继续前行的动力。

下面主要从舞台设计和主题表达两个方面进行赏析。

舞台设计

这部音乐剧最令人印象深刻的是对转场、道具和灯光的令人惊叹的设计，可以说充分展现了音乐剧作为舞台表演艺术的魅力。

场景切换

本部音乐剧总体上是刻画群像，涉及到当地居民和乘客等多个群体，并且群体内部又分成很多不同的角色。如何在 Cast 有限的条件下（只有 12 人）演出这么多不同的角色，并且营造本剧“来的人很多”的背景氛围呢？方法很简单：一个演员演多个角色，并在不同场景中切换。但是这种方法有一个最大的问题：如何让观众清楚地知道这个时刻舞台上的演员演的是哪个角色、现在是在哪个场景？

《Come From Away》的方法是，利用服装、道具、灯光等，来主动地为人物提供“背景”，并结合演员的动作和台词，来进行快速的场景切换。

本剧角色和场景切换的速度之快令人瞠目结舌，同时又达到了非常好的效果。比如，在下面这短短三分钟中，一共表现了 6 个场景的情形。

首先，场景是在小镇的酒吧内，大家刚刚得知恐怖袭击的消息：

此时，灯光熄灭，演员打开手电筒，瞬间切换另一个场景：

这里演员使用手电筒营造了光线不强、主要发光源是屏幕的环境，结合台词观众很容易就知道现在场景切换到了塔台上。

接下来的场景是塔台和机长进行交流。首先是塔台上管制员在繁忙地向机长传达必须马上降落的信息：

然后，椅子一转，所有演员面朝观众。这时候，演员以机长的身份分别与塔台交流（这里同时有多架飞机），我们因此可以知道现在场景切换到了飞机的驾驶舱内。

突然，灯光一暗，演员向右转 90 度，瞬间切换到了机舱内乘客们的场景，同时乘务员开始讲话。我们可以看到这里其实还有原先的桌子，实际上是不符合机舱的情况的，但是并不会造成观众的误解。

过了一段时间，演员站起来，抬头向上看，开始在数降落的飞机的数量。这里，场景又切换到了机场的地面上。我们可以看到，后面的部分演员仍然保持之前的坐姿，但由于他们都在向上看，并逐渐站起来，因此这两个场景的过渡非常自然。之前的乘务员脱下帽子和外套，拿起麦克风，摇身一变成为在地面进行报道的记者，毫无违和感。

最后，灯光由冷变暖，场景重新回到室内的酒吧。这里有个细节，切换前各个演员是分散的，而一切换到酒吧后，演员们都朝着镇长聚集，这也加强了对场景的构建。

总的来看，本剧进行场景切换的主要方法有：

演员表演：通过台词、动作和表情演绎不同角色，如机舱内乘客拨动头顶的开关、管制员用严肃的语气进行无线电通话
服装：通过有职业/角色特点的服装区分同一个演员饰演的不同角色
灯光：通过灯光区分不同场景，如室内的酒吧是暖色的，机舱和地面是冷色的，而塔台和驾驶舱是暗的
道具：通过道具（这里主要是椅子）表现空间的特点

上面的片段使用非线性叙事的方法，将“下达降落命令”这一时刻前后不同地点、不同角色的情况，包括角色的身份、心情等串联起来，立体地表现了不同人群在这一时间对事件的反应，颇有概念音乐剧的影子。

同时，演员所表现的是群像特征，而不是特定角色的特点，因此成功地表现了各个群体总体在此刻的境遇。这也使得观众无需将演员与特定角色绑定（除了酒吧的场景），使得场景切换非常自然。

有意思的是，剧组在剧中还故意展示了他们通过这种方式“省人手”的技巧。同一个人戴上眼镜，长了胡子，或者戴上帽子，配上不同的表情，就是不同性格的、不同镇的镇长，引来阵阵笑声：

道具

这部剧除了十几张椅子外几乎没有任何道具。但就是这十几张椅子，剧组把它们玩出了花。这些椅子的最大作用，如前所述，是用于场景构建。下面这两张图非常明显地展示了这点：

将椅子拍成两排，就变成了机舱的座位。

又如，机长在讲述自己的故事时，其余的人围成一圈：

情侣在野外登高时，用凳子构成一条栈道：

这种方式非常有效，因为不同场景的设施具有迥异的几何排布特征，使用凳子表现这些几何特征，就可以让观众迅速识别并沉浸入场景中（同时非常省经费！）。

另外，灯光和服装等设计，以及演员的表演也对舞台效果起了非常大的作用，在这里限于篇幅不再进行深入分析。让我印象比较深刻的有这些场景：

众人进行默哀，此时一束光穿过树林投下来，渲染了庄严的气氛：

这个是乘客和司机坐在校车上，司机脸前有亮光，而乘客的区域是暗的；同时乘客头瘫向一边非常生动地表现了乘客此时是在车上（急转弯），并且非常疲惫：

你很难想象在舞台上能够表演出的比这个更生动形象的场景了。

主题表现

本剧最主要的主题是表现人们在面对灾难时互相帮助的精神，但又不只于此。

剧组使用相同的演员来饰演不同的角色。这里固然有演员人数的考虑，但我相信这其中蕴含了另一层深意。在剧中，同一个演员既是需要住宿的乘客，也是提供帮助的本地人；既是与事件有直接关系的遇难者家属，也是因看到电视画面而产生心理创伤的普通人……在恐怖主义面前，全体人类都是需要帮助的对象。你就是我，我就是你。帮助他人，从另一个角度来看，也是帮助自己。

通过将灾难下的所有人视为一个整体，恐怖主义被放置在了全体人类的对立面。这以一种非常温和，却非常有力的方式蔑视了恐怖主义，即使全剧几乎没有一句台词直接谴责恐怖主义行为。

这也是本剧的主调，即以普通人的视角来审视 911 事件给人们带来的创伤，以及我们从灾难后人们的互相帮助中所看到的人类本性中最珍贵的特质：善良，乐于助人，等等。本剧之所以有很强的感染力，在于它不仅仅赞颂了人类的珍贵特质——这样的话就成“主旋律”“唱颂歌”类型的剧了；它同时通过去世消防员的母亲等角色提醒我们恐怖主义给我们带来的巨大创伤，提醒我们不要忘记每一个亲历者以及背后无数个家庭的痛苦经历。正如镇长的唱段中所言：

Tonight, we honor what was lost, but we also commemorate what we found.

这里有个小插曲，在《Something’s Missing》中，消防员的母亲在得知儿子去世后开口唱“You are here / At the end of the moment…” 的时候，背后拉小提琴的小姐姐忍不住哭了：

大概，共情是人类能够站在一起最重要的原因吧。

其他

这部音乐剧有比较浓的商业音乐剧的气息，比如无处不在的政治正确，以及时不时的玩梗和抛笑话，非常的 entertaining；这部剧也非常“小清新”，并没有激烈的冲突，也没有非常深入的主题挖掘。另一方面，近乎完美的编排、令人眼花缭乱的舞台表现也反映了当下音乐剧商业化、产业化的趋势。当然，这些并不妨碍这部剧成为一部非常优秀的音乐剧作品。

后记

《Come From Away》实际上是我看过的第一部音乐剧。去年官摄版上架 Apple TV+，恰巧我有订阅，之前也或多或少感受过音乐剧的魅力，于是便找了一个周末看了，之后就爱上了音乐剧（并选了《西方音乐剧史》课程 ^_^）。现在感觉自己挺幸运，第一次就能看到这样一部能够将音乐剧的魅力展现得淋漓尽致的剧（并且有 4k 画质和 Dolby Atmos 音效，which is 非常难得）。

当然，上完课之后再重新看一遍这部音乐剧，获得了完全不一样的体验。与之前只能看剧情不同，现在能够看懂一些表现手法了，也能在上面有模有样地分析它究竟好在哪里。希望以后能够继续深入音乐剧的世界。感谢罗老师的指导！

Operating Systems: Three Easy Pieces 学习笔记（五）

liang2kl@outlook.com (Liang Yesheng) — Wed, 25 May 2022 11:44:11 +0800

26 Concurrency: An Introduction

Each thread is very much like a separate process, except for one difference: they share the same address space and thus can access the same data.

State of a single thread:

program counter
private set of registers (context switch is needed)

which are stored in process control block.

Any stack-allocated variables, parameters, return values, and other things that we put on the stack will be placed in what is sometimes called thread-local storage.

26.1 Why Use Threads?

parallelism: speed up using multi-core processors
avoid blocking during I/O

Computers are hard enough to understand without concurrency. Unfortunately, with concurrency, it simply gets worse. Much worse.

26.4 The Heart Of The Problem: Uncontrolled Scheduling

Race condition: the results depend on the timing execution of the code.

Critical section: a piece of code that accesses a shared variable (or more generally, a shared resource) and must not be concurrently executed by more than one thread.

Mutal exclusion: if one thread is executing within the critical section, the others will be prevented from doing so.

27 Interlude: Thread API

27.4 Condition Variables

int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex);
int pthread_cond_signal(pthread_cond_t *cond);

To wait for a signal:

pthread_mutex_lock(&lock);
while (ready == 0)
 pthread_cond_wait(&cond, &lock); // will release the lock when sleep
 // and re-acquire before wakeup
pthread_mutex_unlock(&lock);

To send a signal:

pthread_mutex_lock(&lock);
ready = 1;
pthread_cond_signal(&cond);
pthread_mutex_unlock(&lock);

28 Locks

28.4 Evaluating Locks

Criteria:

mutal exclusion
fairness
performance

28.6 A Failed Attempt: Just Using Loads/Stores

Problems:

No mutal exclusion
Poor performance: spin-waiting

The problem: set after test (maybe after an interrupt).

28.7 Building Working Spin Locks with Test-And-Set

Test-and-set instruction (xchg on x86) is something like:

int testAndSet(int *old_ptr, int new) {
 int old = *old_ptr; // fetch old value at old_ptr
 *old_ptr = new; // store ’new’ into old_ptr
 return old; // return the old value
}

Lock and unlock:

void lock(lock_t *lock) {
 while (testAndSet(&lock->flag, 1) == 1)
 ; // spin-wait (do nothing)
}

void unlock(lock_t *lock) {
 lock->flag = 0;
}

28.8 Evaluating Spin Locks

correctness: yes
fairness: no, maybe spinning forever (starvation)
performance: bad in single CPU case

28.9 Compare-And-Swap

int compareAndSwap(int *ptr, int expected, int new) {
 int original = *ptr;
 if (original == expected)
 *ptr = new;
 return original;

28.10 Load-Linked and Store-Conditional

Store-conditional: only succeeds (and updates the value stored at the address just load-linked from) if no intervening store to the address has taken place.

void lock(lock_t *lock) {
 while (loadLinked(&lock->flag) == 1 ||
 storeConditional(&lock->flag, 1) == 0)
 ;
}

28.11 Fetch-And-Add

The pseudo C code:

int fetchAndAdd(int *ptr) {
 int old = *ptr;
 *ptr = old + 1;
 return old;
}

Lock and unlock:

void lock(lock_t *lock) {
 int myTurn = fetchAndAdd(&lock->ticket);
 while (lock->turn != myTurn)
 ;
}

void unlock(lock_t *lock) {
 lock->turn = lock->turn + 1;
}

This ensures progress for all threads.

28.13 A Simple Approach: Just Yield, Baby

Yield is simply a system call that moves the caller from the running state to the ready state, and thus promotes another thread to running. Thus, the yielding thread essentially deschedules itself.

Still costly: context switches.
Still unfair: endless yield loop.

28.14 Using Queues: Sleeping Instead Of Spinning

The real problem with our previous approaches is that they leave too much to chance.

Solaris provides two calls: park() to put a calling thread to sleep, and unpark(threadID) to wake a particular thread as designated by threadID.

typedef struct __lock_t {
 int flag;
 int guard;
 queue_t *q;
} lock_t;

void lock_init(lock_t *m) {
 m->flag = 0;
 m->guard = 0;
 queue_init(m->q);
}

void lock(lock_t *m) {
 while (testAndSet(&m->guard, 1) == 1)
 ; // acquire guard lock by spinning
 if (m->flag == 0) {
 m->flag = 1; // lock is acquired
 m->guard = 0;
 } else {
 queue_add(m->q, gettid());
 setpark();
 m->guard = 0;
 park();
 }
}

void unlock(lock_t *m) {
 while (testAndSet(&m->guard, 1) == 1)
 ; // acquire guard lock by spinning
 if (queue_empty(m->q))
 m->flag = 0; // let go of lock; no one wants it
 else
 unpark(queue_remove(m->q)); // hold lock
 // (for next thread!)
 m->guard = 0;
}

29 Lock-based Concurrent Data Structures

29.1 Concurrent Counters

Approximate counting: record counter values locally, and periodically transfer them to the global counter.

29.2 Concurrent Linked Lists

A hand-over-hand linked list may not be faster than the simple single lock approach, as the overhead of acquiring and releasing every node along the traversal path is high.

29.3 Concurrent Queues

29.4 Concurrent Hash Table

30 Condition Variables

30.1 Definition and Routines

A condition variable is an explicit queue that threads can put themselves on when some state of execution (i.e., some condition) is not as desired (by waiting on the condition); some other thread, when it changes said state, can then wake one (or more) of those waiting threads and thus allow them to continue (by signaling on the condition).

int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex);
int pthread_cond_signal(pthread_cond_t *cond);

wait() unlocks a mutex (assumed to be locked) and put the calling thread to sleep; it re-acquires the lock before returning.

int done = 0;
pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t c = PTHREAD_COND_INITIALIZER;

void thr_exit() {
 pthread_mutex_lock(&m);
 done = 1;
 pthread_cond_signal(&c);
 pthread_mutex_unlock(&m);
}

void thr_join() {
 pthread_mutex_lock(&m);
 while (done == 0)
 pthread_cond_wait(&c, &m);
 pthread_mutex_unlock(&m);
}

Some questions:

Why is the done state needed?

If the child runs immediately and calls thr_exit() immediately, then although the child will signal, there is no thread asleep. When the parent calls wait(), it will be stuck.

Why is the mutex needed?

The child might change the state before the parent goes to sleep, and the parent will sleep forever. If there is a mutex, the child can only alter the state after the parent goes to sleep (and thus releasing the mutex).

Always hold the lock while signaling!

30.2 The Producer/Consumer (Bounded Buffer) Problem

Thanks to Mesa semantics, a simple rule to remember with condition variables is to always use while loops.

30.3 Covering Conditions

pthread_cond_broadcast(): waking up all waiting threads; downside: negative performace impact.

Concurrency Control for Database Transactions

liang2kl@outlook.com (Liang Yesheng) — Mon, 11 Apr 2022 23:27:56 +0800

来自 DATABASE SYSTEMS: The Complete Book

Serial and Serializable Schedules

Serial schedule: no mixing of actions from different transactions.

Serializable schedule: if there is a serial schedule S’ such that for every initial database state, the effects of S and S’ are the same.

The details of the transactions do matter, but it is not realistic for the scheduler to concern itself with the details of computation undertaken by transactions.

We only care about the reads $r_T(X)$ and writes $w_T(X)$ performed by the transaction $T$.

Conflict-Serializability

Actions that conflict (i.e. their order may not be swapped):

Two actions of the same transaction.
Two writes of the same database element by different transactions.
A read and a write of the same database element by different transactions.

In other words, for actions from different transactions, they conflict when:

They involve the same database element, and
At least one is a write.

Conflict-equivalent: two schedules can be turned one into the other by a sequence of nonconflicting swaps of adjacent actions.

Conflict-serializable: the schedule is conflict-equivalent to a serial schedule

conflict-serializability $\Rightarrow$ serializability

Precedence: if there are actions $A_1$ of $T_1$ and $A_2$ of $T_2$, such that:

$A_1$ is ahead of $A_2$ in the schedule $S$,
Both $A_1$ and $A_2$ involve the same database element, and
At least one of $A_1$ and $A_2$ is a write action.

then we say that $T_1$ takes precedence over $T_2$ in $S$ ($T_1<_ST_2$).

Any conflict-equivalent serial schedule must have $T_1$ before $T_2$.

If the precedence graph is acyclic, then the schedule is conflict-equivalent.

Proof: for $n$, arrange the schedule by (Actions of $T_i$)(Actions of the other $n-1$ transactions) where $T_i$ is one node without an in arc.

Enforcing Serializability by Locks

The use of lock:

Consistency of Transactions: Actions and locks must relate in the expected ways:
1. A transaction can only read or write an element if it previously was granted a lock on that element and hasn’t yet released the lock.
2. If a transaction locks an element, it must later unlock that element.
Legality of Schedules: Locks must have their intended meaning: no two transactions may have locked the same element without one having first released the lock.

The two-phase locking condition than can ensure conflict-serializability:

In every transaction, all lock actions precede all unlock actions.

A transaction that obeys the 2PL condition is said to be a two-phase-locked transaction, or 2PL transaction.

Proof: for $n$, arrange the schedule by (Actions of $T_i$)(Actions of the other $n-1$ transactions) where $T_i$ is the transaction with the first unlock action $u_iX$.

The risk of deadlock:

Locking Systems With Several Lock Modes

Lock modes:

shared
exclusive

The compatibility matrix:

	Request S	Request X
Currently S	Yes	No
Currently X	No	No

The deadlock problem caused by upgrading locks:

Avoid the problem with a third lock mode: update locks.

An update lock $ul_i(X)$ gives transaction $T_j$ only the privilege to read X, not to write X;
Only the update lock can be upgraded to a write lock later; a read lock cannot be upgraded.

The compatibility matrix:

	Request S	Request X	Request U
Currently S	Yes	No	Yes
Currently X	No	No	No
Currently U	No	No	No

The columns for U and S locks are the same, and the rows for U and X locks are the same.

And now:

An Architecture for a Locking Scheduler

The architecture discussed here: the scheduler insert and release locks.

Part 1: inserts appropriate lock actions ahead of all database-access operations
Part 2: for received action,
1. if the issuing transcation is already delayed, then delay this action
2. otherwise,
  1. if the action is db access (the required locks are already been granted), execute it
  2. if is lock action
    1. if the lock can be granted: modify the lock table
    2. otherwise, request the lock and delay
When a transcation commits or aborts, Part 1 releases all locks held by the transaction; if any transactions are waiting for any of these locks, notify Part 2
Part 2 checks which actions can be executed now

The lock table entry:

The group mode: the most stringent conditions
The waiting bit: at least one transaction is waiting
A list of transactions that hold or waiting for the lock

Hierarchies of Database Elements

Warning locks: useful for hierarchical structure.

	IS	IX	S	X
IS	Yes	Yes	Yes	No
IX	Yes	Yes	No	No
S	Yes	No	Yes	No
X	No	No	No	No

The phantom tuple: one that should have been locked but wasn’t, because it didn’t exist at the time the locks were taken.

Solution: acquire an X lock on the entire relation.

The Tree Protocol

The problem: only one transaction that is not read-only can access the B-tree at any time, because the root node must be locked (at least with an update lock).

Solution: as soon as a transaction moves to a child of the root and observes the (quite usual) situation that rules out a rewrite of the root, we would like to release the lock on the root; but it violates 2PL, so a new protocol is needed.

The tree protocol:

The schedule is legal (i.e. does not violate lock schemes).
A transaction’s first lock may be at any node of the tree.
Subsequent locks may only be acquired if the transaction currently has a lock on the parent node.
Nodes may be unlocked at any time.
A transaction may not relock a node on which it has released a lock, even if it still holds a lock on the node’s parent.

Conclusion: the precedence graph must always be acyclic if the tree protocol is obeyed. Proof:

If two transactions lock several elements in common, then they are all locked in the same order.
If $T_i$ locks the root before $T_j$, then $T_i$ locks every node in common with $T_j$ before $T_j$ does.

Concurrency Control by Timestamps

Two approaches to generate timestamp:

use the system clock
the scheduler maintaining a counter

For the database element $X$:

$RT(X)$: the read time of $X$
$WT(X)$: the write time of $X$
$C(X)$: the commit bit fot $X$, true when the most recent transaction to write $X$ has already commited.

Two kinds of unrealizable behaviours:

The problem with dirty data:

$T$ better delays until $U$ commits.

The commit bit will be false.

The Thomas write rule: the writes can be skipped when a write with a later write-time is already in place (read intention for value written by T is too-late, as the value is written by a logically latter transaction).

The scheduler, in response to a read or write request from a transaction T has the choice of:

Granting the request,
Aborting T and restarting T with a new timestamp
Delaying T and later deciding whether to abort or grant the request.

The rules for timestamp-based scheduling:

The scheduler receives a request $r_T(X)$.
1. $TS(T)\ge WT(X)$: physically realizable.
  1. $C(X)$ is true: grant the request; set $RT(X)$ if $TS(T)>RT(X)$.
  2. $C(X)$ is false: delay until becomes true, or that transaction aborts (to avoid dirty reads).
2. $TS(T)<WT(X)$: physically unrealizable, rollback $T$.
The scheduler receives a request $w_T(X)$.
1. $TS(T) \ge RT(X)$ and $TS(T) > WT(X)$: physically realizable and must be performed (?).
  1. Write $X$,
  2. Set $WT(X) := TS(T)$, and
  3. Set $C(X) := \texttt{false}$.
2. $TS(T) \ge RT(X)$ but $TS(T) < WT(X)$: physically realizable, and
  1. If $C(X)$ is true, ignore the write by T (the thomas write rule);
  2. otherwise, delay T until the previous write operation commits or aborts.
3. $TS(T) < RT(X)$: physically unrealizable, roll back T.
The scheduler receives a request to commit T: find all elements written by T, and set $C(X) := \texttt{true}$; the delayed transactions are allowed to proceed.
The scheduler receives a request to abort T, or to rollback: check all delaying transcation waiting for any element that T wrote to see if they can proceed.

The use of multiversion timestamps: to provide an appropriate version for a read action (the version of X written immediately before T theoretically executed), even if the latest version is written after the transcation.

Comparision of locking and timestamp schemes:

Generally, timestamping is superior in situations where either most transactions are read-only, or it is rare that concurrent transactions will try to read and write the same element. In high-conflict situations, locking performs better.

Locking will frequently delay transactions as they wait for locks.
If concurrent transactions frequently read and write elements in common, then rollbacks will be frequent in a timestamp scheduler.

Concurrency Control by Validation

Validation differs from timestamping principally in that the scheduler maintains a record of what active transactions are doing, rather than keeping read and write times for all database elements.

Denote the write set and read set as $WS(T)$ and $RS(T)$. Transactions are executed in three phases:

Read. The transaction reads from the database all the elements in its read set, and also computes (locally) the results it is going to write.
Validate. The transaction compares its read and write sets with those of other transactions; rollback if fails.
Write. The transaction writes to the database the values for the elements in its write set.

Thus, the validation-based scheduler has an assumed serial order of the transactions to work with, and it bases its decision to validate or not on whether the transactions’ behaviors are consistent with this serial order.

The scheduler maintains three sets:

START: the set of transactions that have started, but not yet completed validation.
VAL: the set of transactions that have been validated but not yet finished the writing in phase 3.
FIN: the set of transactions that have completed phase 3.

Some examples:

A later transaction might read a value before an earlier transaction writes the value.

A later transaction might write before a validated, earlier transaction, as the write of the later one might occur earlier.

Summarization: to validate a transaction T:

Check that $RS(T) \cap WS(U) = \emptyset$ for any previously validated U that did not finish before T started, i.e., if $FIN(U) > START(T)$.
Check that $WS(T) \cap WS(U) = \emptyset$ for any previously validated U that did not finish before T validated, i.e., if $FIN(U) > VAL(T)$.

Comparison of Three Concurrency-Control Mechanisms

Storage: all schemes (can be optimized to) use storage space proportional to the sum over all active transactions of the number of database elements the transaction accesses.

Performace:

Locking delays transactions but avoids rollbacks, even when interaction is high.
Timestamps and validation do not delay transactions, but can cause them to rollback.
If interference is low, then neither timestamps nor validation will cause many rollbacks, and may be preferable to locking because they generally have lower overhead than a locking scheduler.
When a rollback is necessary, timestamps catch some problems earlier than validation, which always lets a transaction do all its internal work before considering whether the transaction must rollback.

总结

同时执行多个事务时，需要保证每个事务的原子性，同时还必须保持一定的先后次序。

对 schedule 最严格的要求是串行性（serial），当然这并不能做到并发。退一步，我们不需要严格保证不同事务处理的时间不互相重叠，但需要保证最终的效果和串行执行一致，即“可串行化”（serializable schedule）。当然，这个条件覆盖的范围太大，很难判断 schedule 是否满足可串行化条件。因此，我们需要增加更严格的条件，使得我们可以有简单的方法判断。当然，这样一来，满足条件的 schedule 只是可串行化 schedule 的一个真子集。

第一大类办法是使用锁，并加上 2PL 条件，使得 schedule 满足“冲突可串行化”（confilict-serializable）。上锁有两个需要解决的问题：锁的类型及其兼容性，以及锁的粒度。

第二大类办法是“乐观的”（optimistic），即假设不会发生不可串行化的事件，仅当的确出现违反可串行化条件时采取措施。Optimistic 方法包括两种：by timestamp，即通过比较事务开始的时间与元素被读或写的时间，判断某个操作是否可以进行；by validation，即在验证阶段比较与之前事务的读写集合判断是否可以提交，验证成功后再写入数据库。

上锁的方法避免了重新执行，但事务会受到延迟，在冲突严重时较优；乐观方法由于可以进行一定操作，只在出现问题时取消或重新执行，因此延迟较低，但在冲突严重时会造成大量的重新执行而影响性能。

软件工程：The Missing Semester

liang2kl@outlook.com (Liang Yesheng) — Tue, 22 Mar 2022 23:01:03 +0800

要是让我总结一个“在清华体验不太好的课程列表”，软件工程一定位列其中。这并不意味着我不喜欢软工项目中的环境配置、CI/CD、单元测试、部署等等工作；相反，我在其中找到了很多乐趣——毕竟，我是一个爱折腾的人。

问题出在作用接近于 0 的文档上：对于很多依赖于详细的文档才能上手的工作，这门课几乎没有提供任何有用的文档。对于这些工作，课程组只提供了没有任何注释的“样例”让我们照猫画虎。虽然说面向 Google 是计算机专业最好的学习方式，但是将一堆东西扔给学生自学，还几乎不提供任何的引导，这是这门课最大的问题。虽然其中很多知识我以前都接触过，但是每当我站在没有开发经验的同学的角度上思考这些复杂的技术栈对于同学们来说意味着什么，就不禁感到愤怒。

在摸索了几天、踩了无数坑之后，我总算是把我们项目的代码部署到了软工提供的服务器上。在这里，我将把一些在软工平台上部署的经验写成文档，作为软工课程文档的补充。

By the way，“The Missing Semester”在这里有两个含义：一是致敬 CMU 的“The Missing Semester”课程，一是表达我对于软工试图仅用一学期就让我们成为全栈工程师的不满。在我看来，软工至少需要两个学期；在这两个学期中，教学团队的引导和同学们的开发应当是相辅相成的，而不能只是学生自己独自面对一个完整的、陌生的、无从下手的、令人恐惧的项目，即使可能有很多有开发经验的同学并不这么觉得。

感谢 @prnake 学长在 GitHub 上公开他们的项目 Quick-Red，下面的很多例子源于此项目。没有它，很多问题我根本无从下手。

创建项目

在软工平台上新建仓库

如果需要部署仓库内容到服务器上，必须要在软工平台（SECoder）上的 项目管理 - 仓库管理 中新建，否则之后无法使用 deployer 部署到提供的服务器上。

分支管理

仓库中应该有如下分支：

master（或 main）：主分支，用于部署生产环境
dev：开发分支，所有最新的改动合并到此分支，定期合并到 master 分支
其他开发分支（例如 feat/login）：开发某一新功能时新建的分支，开发完成后合并到 dev 分支

需要注意 master 和 dev 只能通过 merge request 由其他分支合并，不能直接 push。另外，新建开发分支时一定要从 dev 分支上新建，不能从其他开发分支上新建。

仓库配置

GitLab 中有几项设置对后续协作开发有不小的帮助，值得花点时间配置一下：

在 General - Merge request 中取消选择 Merge options 下的 Enable 'Delete source branch' option by default。课程需要保留开发过程中的分支，需要避免不小心删除分支的情况。
在 Repository - Default Branch 中将 Default Branch 设置为 dev，使得进行 merge request 时默认合并到 dev 分支。
在 Repository - Protected Branches 中添加 master 和 dev 都设置为受保护的分支，并禁止任何人直接 push 到这两个分支上。

增加 issue template 和 merge request template

在仓库目录下新建文件夹 .gitlab/issue_templates/，并在其中添加若干 Markdown 文件，这些文件将可用作新建 issue 时作为模版供选择。例如，针对 bug 的 issue：

<!-- Title 格式：[bug] Bug 描述 -->
## 复现过程

1.
2.

## 当前的行为
<!-- 当前版本存在的问题行为 -->

## 预期的行为
<!-- 正常应当有怎样的行为 -->

添加模版之后，在新建 issue 时可以选择其中之一作为模版，保证 issue 的规范和质量。

同理，可以为 merge request 添加模版，可以参考 Quick-Red 的模版。

CI/CD

CI/CD 简单来说就是在服务器上执行一系列命令，从而自动化地在每次提交代码之后完成测试、检查、构建等工作，并在需要时完成部署。

Stage 和 job

Stage 指的是 CI/CD 的不同阶段，用来完成不同阶段的工作，一般情况下顺序执行。你可以使用 GitLab 的 DAG 功能使得没有依赖关系的 stage 并行运行以加快速度，具体可以参考官方文档。

在软工项目中的 stage 通常包括：

pre：准备阶段，完成依赖库缓存的拉取和保存等
build：构建 Docker 镜像
test：单元测试、代码风格检查、上传代码质量报告（sonar）
deploy：将镜像部署到服务器上

在 .gitlab-ci.yml 开头声明 stage：

stages:
 - pre
 - build
 - test
 - deploy

需要注意的是，不同 stage 之间的环境是相互隔离的。比如，在前面的 stage 中如果安装了项目依赖（如 npm install），在后面的 stage 中仍然需要安装这些依赖。在后面将会介绍加快这一过程的方法。

Job 指的是 stage 内部的若干个任务单元，执行具体的命令，同一个 stage 的所有 jobs 是并行执行的，环境也是相互隔离的。

Job 一些常用的属性有：

stage：指定这个 job 所属的 stage；若不指定，默认为 test
script：这个 job 执行的命令行命令
before_script：在执行 script 之前执行的命令，一般用于环境配置
after_script：在执行 script 之后执行的指令，比如使用 sonar-scanner 上传在 script 中得到的覆盖率报告
only 和 except：用于指定运行此 job 的条件，比如生产环境的部署对应的 job 只在 master 分支上进行

具体的用法可以参考 GitLab 文档。

Hidden jobs 与继承

如果多个 job 有大量相同的配置，可以使用继承的方法避免重复编写相同配置。比如，我们对 Node.js 项目分别进行单元测试和代码风格检查，而它们都需要安装相同的依赖、都属于 test stage。

首先，定义一个以 . 开头的 job，这个 job 为 hidden job，不会被执行：

.test:
 stage: test
 before_script:
 - npm config set registry http://mirrors.cloud.tencent.com/npm/
 - npm install

第一条命令用于将 npm 使用的源设置为国内腾讯云镜像，这将大大加快加载速度。

然后，在真正的 job 中，使用 extends 继承此 job：

lint-test:
 extends: .test
 script:
 - npm run lint

unit-test:
 extends: .test
 script:
 - npm run test:ci

这两个 job 因此拥有了与 .test 相同的 stage 和 before_script 属性。

你还可以继承多个 hidden job，具体行为可以查阅有关文档。

加快速度：cache

对于使用如 Node.js 这样的需要安装大量依赖库的项目，如果在每个 stage 中、每次 CI/CD 都重新安装一次依赖，CI/CD 的效率将会非常低。对这个问题，我们可以使用 GitLab 的 cache 功能将这些依赖缓存起来供重复利用。

因为每个 job 的环境都是独立的，因此每个 job 都需要定义缓存。我们使用 hidden job 完成统一配置：

.cache:
 cache:
 key: ${CI_COMMIT_REF_SLUG}
 paths:
 - node_modules/
 - package-lock.json
 policy: pull

其中，key 使用分支名（${CI_COMMIT_REF_SLUG}），即不同的分支使用不同的缓存，而同一分支下的所有 job 共享这份缓存。paths 指定了缓存项的路径，这里我们将 Node.js 的依赖文件缓存起来。policy 设置为 pull 的目的是，每次只从服务器上拉取缓存，而不需要在结束之后将此缓存上传到服务器上。这很重要，因为我们不希望每个 job 都重复地上传相同的缓存。

我们在 pre 阶段中添加一个 job 用来更新缓存：

node-cache:
 stage: pre
 extends: .cache
 cache:
 policy: pull-push
 script:
 - npm config set registry http://mirrors.cloud.tencent.com/npm/
 - npm install

注意到，我们将 cache policy 更改为了 pull-push，这是因为我们需要将此缓存上传到服务器上供其他 job 和 pipeline 使用。

然后，在需要用到此缓存的 job 中，继承 .cache 即可：

build:
 stage: build
 extends: .cache
 script:
 - npm config set registry http://mirrors.cloud.tencent.com/npm/
 - npm install
 - ...

这个 job 将会从服务器上拉取最新的缓存，从而无需重新下载。

当然，以防万一，还是加上一句 npm install，虽然正常情况下这条命令并不会从网络上下载依赖，因为在 pre 阶段我们已经下载好了需要的依赖。

Sonar 与单元测试的配置

课程要求使用 Sonar 对源代码进行静态分析。既然是静态分析，那么 Sonar 需要测试工具执行测试生成测试报告，才能进行分析。因此，测试与报告的流程为：

测试工具执行测试，并输出测试结果
使用 sonar-scanner 对测试结果和源代码进行分析并上传分析结果

使用测试工具生成测试报告

Sonar 需要提供 xml 格式的测试报告，对于不同语言、框架的方法不同，可以到搜索引擎上搜索生成报告的方法。在这里分享一下我们项目使用的框架（Django、React（create-react-app）、Go）适用的方法。

Django

如果直接使用 pytest，可以参考 Quick-Red。

如果使用 Django 自带的测试框架 django.test：

需要安装以下依赖：

pytest
coverage
unittest-xml-reporting

在 settings.py 中，添加如下配置：

# 指定 test runner 为 xmlrunner
TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner'
# 指定报告输出目录
TEST_OUTPUT_DIR = 'xunit-reports/'
# 指定报告文件名
TEST_OUTPUT_FILE_NAME = 'xunit-result.xml'

然后，运行

python -m coverage run --source path1,path2,...,pathn manage.py test
python -m coverage xml -o coverage-reports/coverage.xml

其中，path1、path2 等替换为包括在覆盖率报告中的文件/文件夹路径。

这样，分别生成了测试报告 xunit-reports/xunit-result.xml 和覆盖报告 coverage-reports/coverage.xml。

这些命令仅完成了最基本的测试报告的配置。关于可选的命令行参数，请参考官方文档。

React（create-react-app）

create-react-app 预置了 Jest，无需手动安装。

安装 jest-sonar：

npm install --save-dev jest-sonar

在 package.json 的 scripts 中添加：

"test:ci": "CI=true react-scripts test --coverage --reporters=jest-sonar --collectCoverageFrom=src/**/*.js"

然后，执行：

npm run test:ci

这样，在 coverage 目录下会生成测试报告。

这条命令仅完成了最基本的测试报告的配置。关于可选的命令行参数，请参考官方文档。

Golang

执行

go test -cover -race -v -coverprofile=coverage.out ./...

配置 sonar-project.properties 文件

我们通过项目根目录下的 sonar-project.properties 文件对 Sonar 进行配置。一些配置如下：

# 源代码所在目录
sonar.sources=src
# 包含测试文件的路径，sonar 将会寻找其中的所有测试文件
sonar.tests=src
# 包含在测试分析内的测试文件
sonar.test.inclusions=src/**/__tests__/*.test.js
# 排除在所有分析之外的路径
sonar.exclusions=node_modules/**,coverage/**
# 覆盖率分析时排除在外的路径
sonar.coverage.exclusions=src/utils/**/*.js

另外，还需要提供已经得到的测试报告的路径。对于 JavaScript：

sonar.javascript.lcov.reportPaths=coverage/lcov.info
sonar.testExecutionReportPaths=coverage/sonar-report.xml

对于使用 coverage 进行分析的 Python 代码：

sonar.python.coverage.reportPaths=coverage-reports/coverage.xml
sonar.python.xunit.reportPath=xunit-reports/xunit-result.xml

对于 Go：

sonar.go.coverage.reportPaths=coverage.out

这里路径需要改成上面一步所设置的覆盖报告和测试报告的路径。

设置 CI

在 CI 中，定义一个 job 用来执行测试和上传报告：

unit-test:
 ...
 script:
 # 执行测试的命令
 after_script:
 - SUFFIX=$RANDOM
 - curl "http://10.0.0.11/sonar-scanner.tar.gz" -s -o "/tmp/sonar-$SUFFIX.tar.gz"
 - tar -xf "/tmp/sonar-$SUFFIX.tar.gz" -C /opt
 - /opt/sonar-scanner/bin/sonar-scanner

Dockerfile

将代码部署在服务器上需要配置环境、编译应用、安装应用、开启服务等一系列操作，而 Docker 使得我们可以预先完成或指定这些操作，从而能够实现自动部署；Dockerfile 就是我们定义这些操作的地方。

在软工平台上使用 deployer 构建、上传 Docker 镜像，并使用此镜像进行部署。大概的逻辑是：

通过 Dockerfile 内的一系列语句预先配置好所需的环境并打包成镜像
将镜像上传到软工平台的镜像站上
部署时，拉取之前构建好的镜像，并在一个容器中运行，这样就完成了部署

这个过程在 .gitlab-ci.yml 中使用多个 job 完成：

# 构建 Docker 镜像
build:
 stage: build
 script:
 - export BUILD_IMAGE_NAME=$CI_REGISTRY_IMAGE
 - export BUILD_IMAGE_TAG=$CI_COMMIT_REF_SLUG
 - export BUILD_IMAGE_USERNAME=$CI_REGISTRY_USER
 - export BUILD_IMAGE_PASSWORD=$CI_REGISTRY_PASSWORD
 - deployer build

# 使用镜像部署
deploy:
 stage: deploy
 script:
 - deployer dyno replace $CI_PROJECT_NAME "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" "$REGISTRY_USER" "$REGISTRY_PWD"

build 的多个 export 指定了上传时使用的镜像名、tag、使用的用户名和密码；deployer build 命令完成了 (1) 使用 docker build 根据 Dockerfile 进行镜像构建，(2) 将镜像上传到平台上；deploy 工作将 build 中上传的镜像部署到服务器上。这样就完成了整个部署过程。

所以，我们需要做的，就是在 Dockerfile 中描述配置容器的过程。

配置 Dockerfile

首先，指定一个“基础镜像”，即使用一个预先配置好的镜像，在这个镜像的基础上配置我们的镜像。比如，对于 Node.js 应用，我们使用官方的 node 镜像：

FROM node:latest

在这之后，我们就在一个配置好了 Node.js 的环境中了。

接下来，无非就是执行一系列语句来编译我们的应用，并将其拷贝到合适的位置。这部分内容可以参考软工课程给出的样例文件。

比如，编译 React 应用：

# 第一个阶段：使用 node 镜像编译 React 应用；将此阶段命名为 build
FROM node:latest as build

# 设置环境变量 HOME
ENV HOME=/opt/app

# 将工作目录设为 HOME，并拷贝当前目录（项目代码）到 HOME 中
WORKDIR $HOME
COPY . $HOME

# 安装依赖
RUN npm config set registry http://mirrors.cloud.tencent.com/npm/
RUN npm install
# 执行 React 的 build，获得网页文件
RUN npm run build

# 第二个阶段：配置使用 nginx 容器，使得可以通过 HTTP 访问
FROM nginx:1.19

# 从第一个阶段（build）拷贝文件
# 拷贝编译后的网页文件到 nginx 的目录下
COPY --from=build /opt/app/build /usr/share/nginx/html
# 拷贝 nginx 配置文件（储存在源代码中）
# 对于 nginx 的配置和使用，你可能需要查阅相关文档，也可以直接使用已有的样例
COPY --from=build /opt/app/nginx.conf /etc/nginx/nginx.conf

# 暴露容器的 80 端口
EXPOSE 80

# 这里定义了容器启动时执行的命令（开启 Nginx 服务），注意并不是在构建镜像时执行
CMD ["nginx", "-g", "daemon off;"]

有关 Dockerfile 的语句，可以参考 Docker - 从入门到实践（中文）或 Dockerfile 官方文档（英文）。

在 CI 中向 Dockerfile 传递环境变量

有时候，我们需要向 Dockerfile 中传递环境变量，比如使用不同的环境部署生产环境和开发环境的代码。但是，deployer 并不会向 docker build 传递环境变量，因此在 Dockerfile 中是无法获取到我们希望传递的环境变量的。

一种不太优雅的解决办法是：使用不同的 Dockerfile，直接在 Dockerfile 中用 ENV 语句设置环境变量。

比如，当前项目根目录下有两个文件：用于生产环境的 Dockerfile，以及用于开发环境的 Dockerfile-dev，它们的唯一区别是，Dockerfile-dev 设置了一个名为 REACT_APP_IS_DEV_SERVER 的环境变量，在我们的 React 应用中需要根据此环境变量判断所处的是否为生产环境：

ENV REACT_APP_IS_DEV_SERVER=true

但是，因为 deployer 无法指定构建镜像所使用的 Dockerfile 而只能使用文件名为 Dockerfile 的文件，我们需要将文件更名为 Dockerfile。在 CI 中，分别为生产环境和开发环境定义 build 的 job：

build:
 extends: .build
 only:
 - master

build-dev:
 extends: .build
 before_script:
 - rm Dockerfile
 - mv Dockerfile-dev Dockerfile
 except:
 - master

我们用 Dockerfile-dev 替代了 Dockerfile，实现了根据条件传递环境变量的目的。

数据库配置

对于需要使用 Client/Server 通信模式数据库的情况，需要部署一个独立的数据库服务器。下面以 MySQL 为例。

对于其他数据库，过程大同小异，只需要完成：

创建仓库并设置好 CI/CD
将持久卷挂载到数据库储存文件的位置
设置好数据库访问的用户名和密码、创建数据库、授予权限等初始化工作
在后端上配置好访问数据库的信息

建立仓库、配置持久卷和端口

在软工平台上创建数据库所用的仓库，并在软工平台 项目管理 - 部署管理 - 持久储存 中新建一个持久卷。

之后，在上方 容器 中打开数据库容器，在 挂载 一栏中选择此持久卷，挂载点设置为 /var/lib/mysql。这是因为 MySQL 所有的数据库文件均储存在此目录下，将持久卷挂载到此目录上可以保持数据库文件不因重新部署而丢失。

在容器的 端口 中添加 MySQL 使用的 3306 端口。

配置 CI/CD

.gitlab-ci.yml 和 Dockerfile 直接使用 Quick-Red 的配置即可。

初始化数据库

初次部署完成后，需要进入容器初始化 MySQL。在软工平台数据库容器下打开 管理 - 终端，进入容器。

然后，执行 mysql 命令进入 MySQL。

首先，设置数据库 root 用户的密码（<password_root> 设置为你想要设置的密码）：

ALTER USER 'root'@'localhost' IDENTIFIED BY '<password_root>';

接下来，创建供外部访问的账号（<password_remote> 设置为外部访问时使用的密码）：

CREATE USER 'django'@'%' IDENTIFIED BY '<password_remote>';

在这里，'django'@'%' 的意思是，创建一个从任意 IP 访问（%）、用户名为 django 的账户。django 可以设置为你想设置的用户名。注意用户名尽量不要使用 root，会产生冲突。然后，创建一个空的数据库（db_name 更改为你想要设置的名称）：

CREATE DATABASE db_name;

将此数据库的访问权限授予远程用户（注意修改 db_name 和 django）：

GRANT ALL PRIVILEGES ON db_name.* TO 'django'@'%';

最后，更新权限：

FLUSH PRIVILEGES;

这样，我们就可以在后端服务器上访问此数据库了。

配置 Django

这里以 Django 为例配置使用上面已经配置好的 MySQL 数据库。

在软工平台中，创建一个“配置”，并上传一个包含数据库访问信息的 JSON 文件：

{
 // 在 MySQL 中新建的数据库名
 "MYSQL_NAME": "db_name",
 // 配置数据库时设置的远程访问用户名
 "MYSQL_USER": "root",
 // 配置数据库时设置的远程访问密码
 "MYSQL_PASSWORD": "<password_remote>",
 // 数据库服务器的本地地址，在软工平台上可以找到
 "MYSQL_HOST": "example-app.secoder.local",
 // MySQL 端口
 "MYSQL_PORT": 3306
}

然后，在软工平台的后端容器中将此文件（配置）挂载到源代码所在的子目录下（这里你可能需要查看你的 Dockerfile 来获取容器中源代码所在的目录）。注意，这个操作会将此目录的所有内容替换为上面配置的内容，因此不要将其挂载到源代码根目录下，也不要将其挂载到任何一个原有的子目录下。

在 settings.py 中，读取配置文件（这里，我们把配置文件目录挂载到了代码目录/config，文件名为 ‌config.json）：

if os.path.isfile("config/config.json"):
 with open("config/config.json", "r") as file:
 env = json.load(file)

然后，将 DATABASES 更改为：

DATABASES = {
 'default': {
 'ENGINE': 'django.db.backends.mysql',
 'NAME': env["MYSQL_NAME"],
 'USER': env["MYSQL_USER"],
 'PASSWORD': env["MYSQL_PASSWORD"],
 'HOST': env["MYSQL_HOST"],
 'PORT': env["MYSQL_PORT"],
 'OPTIONS': {'charset': 'utf8mb4'},
 },
}

至此，Django 就可以使用 MySQL 服务器了。注意，你应该避免在开发环境中使用生产环境的 MySQL 服务器，这个问题可以通过挂载不同的配置文件解决。

Operating Systems: Three Easy Pieces 学习笔记（四）

liang2kl@outlook.com (Liang Yesheng) — Sun, 06 Mar 2022 17:24:23 +0800

13 The Abstraction: Address Spaces

13.3 The Address Space

The address space of a process contains all of the memory state of the running program (e.g. code, stack, heap…)

13.4 Goals

Goals of the design:

Transparency
Efficiency
Protection

14 Interlude: Memory API

15 Mechanism: Address Translation

15.3 Dynamic (Hardware-based) Relocation

When a program is loaded, the OS sets base and bounds registers.

physical address = virtual address + base

15.4 Hardware Support: A Summary

The required hardware support:

privileged mode
base/bounds registers
translate virtual address and check if within bounds
privileged instructions to update base/bounds registers
privileged instructions to register exception handlers
ability to raise exceptions

15.6 Summary

The problem: all the space between stack and heap is wasted (called internal fragmentation)

16 Segmentation

16.1 Segmentation: Generalized Base/Bounds

Segment: contiguous portion of the address space of a particular length.

Segmentation allows the OS to place segments in differnet parts of physical memory.

The hardware structure in the MMU: three base and bounds register pairs.

16.2 Which Segment Are We Referring To?

The explicit approach: chop up the address space into segments based on the top few bits of the virtual address.

The implicit approcah: the hardware determins the segment by noticing how the address was formed.

16.3 What About The Stack?

The difference: it grows backwords.

Use a bit in the hardware to indicate the sign of the offset.

16.5 Fine-grained vs. Coarse-grained Segmentation

16.6 OS Support

The operation to support segmentation:

On context switch: save and store segment registers
Support growing heap segment (sbrk())
Handle external fragmentation (physical memory quickly becomes full of little holes of free space, making it difficult to allocate new segments, or to grow existing ones)

17 Free-Space Management

17.1 Assumptions

Internal fragmentation: the allocator hands out chunks of memory bigger than that requested

17.2 Low-level Mechanisms

Splitting and Coalescing

Coalescing: coalesce spliited fragments when they are contiguous

Tracking The Size Of Allocated Regions

Most allocators store a little bit of extra information in a header block which is kept in memory to keep track of the size of the allocated region.

typedef struct {
 int size;
 int magic; /* for integrity checking */
} header_t;

Embedding A Free List

Growing The Heap

17.3 Basic Strategies

Unfortunately, because the stream of allocation and free requests can be arbitrary (after all, they are determined by the programmer), any particular strategy can do quite badly given the wrong set of inputs.

Best Fit

The best chunk: the smallest in candidates.

Cost: heavy performance overhead

Worst Fit

The opposite to the best fit strategy.

First Fit

Returns the first qualified block.

Problem: pollution at the beginning of the free list; approach: address-based ordering.

Advantage: speed.

Next Fit

Keep an eatra pointer to the last-searched location; the blocks are distributed more evenly than first-fit strategy.

17.4 Other Approaches

Segregated Lists

Keep a separate list just to manage objects of some popular sizes; all other requests are forwarded to a more general memory allocator.

Buddy Allocation

The binary buddy allocator: free memory is thought of as one big space of $2^N$; the search for free space recursively divides free space by two.

Suffers from internal fragmentation.

Free process: recursively coalescing blocks upwards.

Simple to determine the buddy of a particular block: each buddy pair (at the same level) only differs by a single bit.

18 Paging: Introduction

18.1 A Simple Example And Overview

Page table: a per-process data structure to store address translations.

18.3 What’s Actually In The Page Table?

Some bits:

valid bit (supporting a sparse address space; x86 doesn’t have it)
protection bit
present bit (whether a page has been swapped out)
dirty bit (whether a page has been modified)
refence bit (whether a page has been accessed)

Don’t worry: it deﬁnitely gets worse, because the mechanisms we are about to introduce only complicate this already complex machinery. Sorry!

19 Paging: Faster Translations (TLBs)

When we want to make things fast, the OS usually needs some help. And help often comes from the OS’s old friend: the hardware.

Translation-lookaside buffer (or TLB)

19.3 Who Handles The TLB Miss?

CISC: uses hardware-managed TLBs
RISC: uses software-managed TLBs

The software approach: more flexible and simple; need to avoid infinite chain of TLB misses.

19.4 TLB Contents: What’s In There?

The TLB typically has 32, 64 or 128 entries, and is fully associated (any given translation can be anywhere in the TLB).

19.5 TLB Issue: Context Switches

Some hardware systems provide an address space identifier (ASID) in the TLB to identify the associated process.

19.6 Issue: Replacement Policy

One common approach is to evict the least-recently-used or LRU entry.

Another typical approach is to use a random policy, which evicts a TLB mapping at random.

19.7 A Real TLB Entry

How to handle ‘ASID overflow’?

In MIPS, some TLB slots can be reserved for the OS.

20 Paging: Smaller Tables

Here, the problem should be clear: simple linear (array-based) page tables are too big.

20.2 Hybrid Approach: Paging and Segments

Whenever you have two reasonable but different approaches to something in life, you should always examine the combination of the two to see if you can obtain the best of both worlds.

Operating Systems: Three Easy Pieces 学习笔记（三）

liang2kl@outlook.com (Liang Yesheng) — Sat, 26 Feb 2022 13:26:41 +0800

4 The Abstraction: The Process

Process: a running program

4.2 Process API

create
destroy
wait
miscellaneous control
status

4.4 Process States

A process can be in one of three states:

running: executing instructions
ready: ready to run but the OS has chosen not to run at this moment
blocked: a process has performed some kind of operation that makes it not ready to run until some other event takes place

5 Interlude: Process API

5.4 Why? Motivating The API

The separation of fork() and exec() is essential in building a UNIX shell, because it lets the shell run code after the call to fork() but before the call to exec(); this code can alter the environment of the about-to-be-run program, and thus enables a variety of interesting features to be readily built.

5.5 Process Control And Users

The user, after entering a password to establish credentials, logs in to gain access to system resources (and as a result, can send signals to programs).

6 Mechanism: Limited Direct Execution

Use time sharing to achieve virtualization

Some challenges:

performance
control

6.1 Basic Technique: Limited Direct Execution

Direct execution: run the program directly on the CPU

6.2 Problem #1: Restricted Operations

The approach to restrict certain operations: introducing user mode (and kernel mode)

System calls allow the kernel to carefully expose certain key pieces of functionality to user programs, setting up a trap table for the hardware to jump to trap handlers

6.3 Problem #2: Switching Between Processes

The curx: how to regain control of the CPU?

A Cooperative Approach: Wait For System Calls

The OS regains control of the CPU by waiting for a system call or an illegal operation of some kind to take place.

A Non-Cooperative Approach: The OS Takes Control

A timer device can be programmed to raise an interrupt (timer interrupt) every so many milliseconds; when the interrupt is raised, the currently running process is halted, and a pre-configured interrupt handler in the OS runs.

Saving and Restoring Context

Two types of register saves/restores:

when timer interrupt occurs: the user registers are implicitly saved by the hardware to the kernel stack
when the OS decides to switch to another process: the kernel registers are explicitly saved by the software to the memory of the process structure (Still in the kernel mode so should kernel registers be saved/restored; the user registers are restored later during return-from-trap)

6.4 Worried About Concurrency?

OS might disable interrupts during interrupt processing (but can lead to lost of interrupts), and use locking to protect concurrent access to internal data structures.

7 Scheduling: Introduction

7.1 Workload Assumptions

Unrealistic assumptions:

each job runs for the same amount of time
all jobs arrive at the same time
once started, each job runs to completion
all jobs only use the CPU (i.e., they perform no I/O)
the run-time of each job is known (more unrealistic!)

7.2 Scheduling Metrics

Simplify with one single metric: turnaround time

$$ T_\mathrm{turnaround}=T_\mathrm{completion}-T_\mathrm{arrival} $$

In our case:

$$ T_\mathrm{turnaround}=T_\mathrm{completion} $$

7.3 First In, First Out (FIFO)

The First In, First Out scheduling, or First Come, First Served

Convoy effect: a number of relatively-short potential consumers of a resource get queued behind a heavyweight resource consumer.

7.4 Shortest Job First (SJF)

Shortest Job First: runs the shortest job first, then the next shortest, and so on; optimal.

Bad without assumption 2 (all jobs arrive at the same time)

7.5 Shortest Time-to-Completion First (STCF)

Relax assumption 3 (each job runs to completion)

The scheduler can preempt the current job; adding this to SJF, we get the Shortest Time-to-Completion First (STCF) or Preemptive Shortest Job First (PSJF) scheduler

7.6 A New Metric: Response Time

The response time:

$$ T_\mathrm{response}=T_\mathrm{firstrun}-T_\mathrm{arrival} $$

7.7 Round Robin

Round-Robin: instead of running jobs to completion, RR runs a job for a time slice (sometimes called a scheduling quantum) and then switches to the next job in the run queue

The length of the time slice is critical for RR; a trade-off between response time and the cost of context switching

The cost of context switch

When programs run, they build up a great deal of state in CPU caches, TLBs, branch predictors, and other on-chip hardware. Switching to another job causes this state to be flushed and new state relevant to the currently-running job to be brought in, which may exact a noticeable performance cost.

RR is pretty awful for turnaround time

More generally, any policy (such as RR) that is fair, i.e., that evenly divides the CPU among active processes on a small time scale, will perform poorly on metrics such as turnaround time.

7.8 Incorporating I/O

Relax assumption 4 (all jobs only use the CPU)

Overlap will occur if using STCF

7.9 No More Oracle

Relax our final assumption (the run-time of each job is known)

8 Scheduling: The Multi-Level Feedback Queue

The problem we try to address: given that we in general do not know anything about a process, how to

optimize turnaround time
minimize response time

8.1 MLFQ: Basic Rules

The MLFQ has a number of distinct queues, each assigned a different priority level. At any given time, a job that is ready to run is on a single queue.

A job with higher priority (i.e., a job on a higher queue) is chosen to run; use round-robin scheduling among jobs with the same priority.

rule 1: if Priority(A) > Priority(B), A runs (B doesn’t)
rule 2: if Priority(A) = Priority(B), A & B run in RR

To set the priority: MLFQ will try to learn about processes as they run, and thus use the history (long-running or interactive) of the job to predict its future behavior.

8.2 Attempt #1: How To Change Priority

rule 3: when a job enters the system, it is placed at the highest priority
rule 4
- a: if a job uses up an entire time slice while running, its priority is reduced
- b: if a job gives up the CPU before the time slice is up (therefore jobs with lower priority may have a chance to run), it stays at the same priority level

Example 1: A Single Long-Running Job

Example 2: Along Came A Short Job

The idea: first assumes a job to be short, otherwise move it down; this is an approximization of SJF

Example 3: What About I/O?

Problems With Our Current MLFQ

starvation: too many interactive jobs will consume all CPU time, thus long-running jobs will never receive any CPU time
can be attacked by first using most of the time and then perform an I/O
a program might change its behaviour

8.3 Attempt #2: The Priority Boost

rule 5: after some time period $S$, move all the jobs in the system to the topmost queue

If $S$ is set too high, long-running jobs could starve; too low, and interactive jobs may not get a proper share of the CPU.

8.4 Attempt #3: Better Accounting

rule 4: once a job uses up its time allotment (分配) at a given level (regardless of how many times it has given up the CPU), its priority is reduced (i.e., it moves down one queue)

8.5 Tuning MLFQ And Other Issues

The concept: instead of optimizing for turnaround or response time, a scheduler might instead try to guarantee that each job obtain a certain percentage of CPU time.

Tickets: the share of a resource that a process (or user or whatever) should receive.

The system picks a ticket (randomly) whose owner process can run at this point

If you are ever in need of a mechanism to represent a proportion of ownership, this concept just might be … (wait for it) … the ticket.

9.2 Ticket Mechanisms

ticket currency
ticket transfer
ticket inflation (通货膨胀)

9.3 Implementation

Best practice: order the list from the highest number of tickets to the lowest

9.4 An Example

Overall fairness is higher with higher job length

9.5 How To Assign Tickets?

Given a set of jobs, the “ticket-assignment problem” remains open.

9.6 Stride Scheduling

Stride: inverse in proportion to the number of tickets a job has

curr = remove_min(queue); // pick client with min pass
schedule(curr); // run for quantum
curr->pass += curr->stride; // update pass using stride
insert(queue, curr); // return curr to queue

Stride scheduling is a deterministic fair-share scheduler.

Lottery scheduling has no global state, so that new processes are easier to be incorporated

9.7 The Linux Completely Fair Scheduler (CFS)

Recent studies have shown that scheduler efficiency is surprisingly important; specifically, in a study of Google datacenters, Kanev et al. show that even after aggressive optimization, scheduling uses about 5% of overall datacenter CPU time.

Basic Operation

Goal: to fairly divide a CPU evenly among all competing processes

In the basic case, each process’s vruntime increases at the same rate, propotional to the physical time. Pick the process with the lowest vruntime to run next.

Divide sched_latency by process number to calculate the time slice; use min_granularity to constrain the minimum possible time slice.

Weighting (Niceness)

Use nice level to control the priority of a process. Positive nice values imply lower priority and negative values imply higher priority

When you’re too nice, you just don’t get as much (scheduling) attention, alas.

Calculation of the time slice:

$$ \texttt{time_slice}_k=\cfrac{\texttt{weight}_k}{\sum_{i=0}^{n-1}\texttt{weight}_i}\cdot\texttt{sched_latency} $$

Also adjusting the calculation of virtual runtime:

$$ \texttt{vruntime}_i=\texttt{vruntime}_i+\cfrac{\texttt{weight}_0}{\texttt{weight}_i}\cdot\texttt{runtime}_i $$

The table preserves CPU proportionality ratios when the difference in nice values is constant

Using Red-Black Trees

Use a red-black tree to maintain running (or runnable) processes; if a process goes to sleep (say, waiting fot an I/O to complete), it is removed.

Dealing With I/O And Sleeping Processes

A just-awaken process will monopolize the CPU; solution: set the vruntime of that job to the min value found in the tree (which only keeps track of running jobs!).

10 Multiprocessor Scheduling (Advanced)

10.1 Background: Multiprocessor Architecture

Two kinds of locality:

temporal locality
spatial locality

Problem with multi-code processors: cache coherence; the hardware uses bus snooping to update/invalidate outdated cache

10.2 Don’t Forget Synchronization

10.3 One Final Issue: Cache Affinity

A multiprocessor scheduler should consider cache affinity when making its scheduling decisions, perhaps preferring to keep a process on the same CPU if at all possible.

10.4 Single-Queue Scheduling

Singlequeue multiprocessor scheduling (or SQMS): take an existing policy that picks the best job to run next and adapt it to work on more than one CPU

Problems:

performance: the overhead of locking is high
cache affinity: each job ends up bouncing around from CPU to CPU

10.5 Multi-Queue Scheduling

Each CPU has a queue of tasks to be performed.

To solve load imbalance: using migration; a basic approach is to use work stealing

Operating Systems: Three Easy Pieces 学习笔记（二）

liang2kl@outlook.com (Liang Yesheng) — Wed, 23 Feb 2022 21:37:07 +0800

42 Crash Consistency: FSCK and Journaling

One major challenge faced by a file system is how to update persistent data structures despite the presence of a power loss or system crash.

Crash-consistency problem: On-disk state may only partially get updated on system crash or power loss

42.1 A Detailed Example

Crash Scenarios

Three possible outcomes if an error occured when a data block is allocated and written for a file (which should allocate a data block (and write data to it), update the data bitmap, and update the file inode):

only the data block is written: no problem at all (from the perspective of file-system crash consistency)
only the updated inode is written: garbage data will be read; file-system inconsistency: the allocated status of the data block is different in the bitmap and the inode
only the bitmap is written: space leak
only the inode and the bitmap are written: garbage data
only the inode and the data block: inconsistency between the inode and the bitmap
only the bitmap and the data block: space leak again

The Crash Consistency Problem

Crash-consistency problem (or consistent-update problem): the disk can only commit one write at a time, so that an update cannot be done atomically

42.2 Solution #1: The File System Checker

Early systems’ approach: fix inconsistencies later (when rebooting)

fsck’s phases:

check superblock
scans inodes, indirect blocks, etc. to make consistency between inodes and bitmaps (by trusting inodes)
check inodes for corruption and other problems
verify link count
check for duplicate pointers (two different inodes refer to the same block) (clear some inodes or make copies of the duplicate data)
check for bad block pointers (and remove them)
check directories

Fundamental problems of fsck: too slow

At a higher level, the basic premise of fsck seems just a tad irrational. Consider our example above, where just three blocks are written to the disk; it is incredibly expensive to scan the entire disk to fix problems that occurred during an update of just three blocks.

42.3 Solution #2: Journaling (or Write-Ahead Logging)

Write-ahead logging (or journaling): write logs before overwriting the structures to be able to recover from errors

Data Journaling

Journal write:

transaction begin block
blocks containing the exact contents of the updating blocks (physical logging) or some more compact forms (logical logging)
transaction end block

Checkpointing: override the old structures in the file system

Problems if a crach occurs during the write to the journal: if writing all the log blocks at once, as the disk might perform scheduling, they might not complete in order, and thus some blocks might not be correct (e.g. invalid data blocks within valid TxB and TxE blocks).

Solution: write all blocks except the TxE at once, and write TxE block after these writes. With atomicity guatantee!

Performance problem: the transaction-end block can only be written after previous blocks are written.

Optimizing log writes: using checksum

Three phases:

journal write (TxB, metadata and data)
journal commit (TxE)
checkpoint (write the contents to final on-disk locations)

Recovery

Replay the transactions that are commited to the disk (redo logging)

Batching Log Updates

Buffer all updates into a global transaction to avoid writing some same blocks many times.

Making The Log Finite

Journaling file systems treat the log as a circular data structure (the journal is referred to as circular log).

Mark the transaction free in the journal by updating the journal superblock some time after checkpoint.

Now the four phases:

journal write
journal commit
checkpoint
free

Metadata journaling

Ordered journaling (or metadata journaling): the user data is not written to the journal

One possible order (to avoid garbage data):

data write (1)
journal metadata write (2)
journal commit
checkpoint metadata
free

(1) and (2) can be written concurrently

43 Log-structured File Systems

The motivation

system memories are growing so that more data can be cached, and disk traffic increasingly consists of writes

a large gap between random and sequential I/O performace

exising file systems perform poorly on many common workloads

file systems are not RAID-aware

An ideal file system would thus focus on write performance, and try to make use of the sequential bandwidth of the disk. Further, it would perform well on common workloads that not only write out data but also update on-disk metadata structures frequently. Finally, it would work well on RAIDs as well as single disks.

43.1 Writing To Disk Sequentially

The basic idea: writing all updates (such as data blocks, inodes, etc.) to the disk sequentially

43.2 Writing Sequentially And Effectively

Sequential writes is not enough; to achieve good performance we must issue contiguous writes (or one large write)

Write buffering: keeps track of updates in memory, and write the buffered data at once when gets sufficient updates (called a segment)

43.3 How Much To Buffer?

$$ D=\cfrac{F}{1-F}\times R_\mathrm{peak}\times T_\mathrm{position} $$

43.4 Problem: Finding Inodes

43.5 Solution Through Indirection: The Inode Map

The inode map: takes an inode number as input and returns the disk address of the most recent version of the inode

LFS places chunks of the inode map right next to where it is writing all of the other new information.

Inode maps are compact enough to keep the active portions cached in main memory: inode map lookups rarely require disk accesses.

(From The Design and Implementation of a Log-Structured File System)

43.6 Completing The Solution: The Checkpoint Region

The checkpoint region (or CR) contains pointers to the latest pieces of the inode map; it is only updated periodically for performance

43.7 Reading A File From Disk: A Recap

The steps of reading a file (from start):

read the CR and cache it in memory
lookup the CR to find the address of a given inode
(the rest is pretty much the same as a typical FS)

43.8 What About Directories?

Steps:

look in the imap for the location of the inode of the directory
find inode number of the file in the inode of the directory
look in the imap for the location of the inode of the file

The imap also solves the recursive update problem through indirection.

43.9 A New Problem: Garbage Collection

The cleaner assures to preserve contiguous free regions by compacting existing segments to new, contiguous segments

43.10 Determining Block Liveness

Use segment summary block at the head of the segment to record the inode number and offset (which block of the file) for each data block

To determine whether a block is dead:

look in the segment summary block and find its inode number and offset
look in the imap to find and read the inode
look in the inode using the offset to compare the disk location

LFS can also use version number to determine this, avoiding extra reads.

43.11 A Policy Question: Which Blocks To Clean, And When?

The time to clean can be:

perioducally
during idle time
when the disk is full

43.12 Crash Recovery And The Log

Use two CRs containing update timestamps to recover from crash during CR updates

The last consistent snapshot of the file system may be quite old as LFS only writes the CR every 30 seconds or so; use roll forward to read through the next segments of the end of the log to see if there are any valid updates, and updates the file system accordingly.

44 Flash-based SSDs

Solid-state storage: random access, retains information on power loss

44.1 Storing a Single Bit

single-level cell (SLC) flash: a single bit per transistor
multi-level cell (MLC) flash: two bits per transistor (00, 01, 10, 11 represented by different levels of charge)
triple-level cell (TLC) flash: three bits per cell

SLC chips achieve higher performance and are more expensive.

44.2 From Bits to Banks/Planes

Flash chips are organized into banks or planes which consist of a large number of cells.

A bank is accessed in two different sized units:

block (or erase block): typically of size 128KB or 256KB
page: a few KB in size (e.g. 4KB)

44.3 Basic Flash Operations

Three low-level operations:

read (a page): random access; fast, 10s of $\mathrm{\mu s}$ or so
erase (a block): setting each bit to 1; quite expensive, takes a few ms; need to copy data of care somewhere else
program (a page): After a block is erased, change some of the 1’s within a page to 0’s; usually taking around $100\mathrm{\mu s}$

44.4 Flash Performance And Reliability

Wear out: erasing and programming accure a little bit of extra charge, 0 and 1 are increasingly difficult to be differentiated.

The typical lifetime:

MLC: 10,000 P/E (Program/Erase) cycles
SLC: 100,000 P/E cycles
actual lifetimes are much longer

Disturbance: some bits get flipped in neighboring pages when accessing (read/program) a particular page

44.5 From Raw Flash to Flash-Based SSDs

Flash translation layer (or FTL): translates read and write requests on logical blocks into low-level read, erase and program commands on underlying physical blocks and physical pages

The FTL should accomplish this task with the goal of delivering excellent performance and high reliability.

To achieve performance:

parallel
reduce write amplification

Write amplification: total write traffic issued to the flash chips (by the FTL) divided by the total write traffic issued to the SSD (by the client)

To achieve high reliability:

for wear out: wear leveling
for program disturbance: a sequential-programming approach

44.6 FTL Organization: A Bad Approach

Direct mapped organization: a read to a logical page is mapped directly to the physical page with the same page number

The problems with the direct-mapped FTL:

performance: each write needs to
- read in the entire block (costly)
- erase the entire block (quite costly)
- program the block (costly)
reliability: the same block can be erased and programmed over and over (e.g., file system metadata is overritten repeatedly)

The direct mapped approach simply gives too much control over wear out to the client workload; if the workload does not spread write load evenly across its logical blocks, the underlying physical blocks containing popular data will quickly wear out.

44.7 A Log-Structured FTL

Use a mapping table to map the logical block to the physical address

The log-based approach improves performance in:

erases being required once in a while
spread writes across pages (wear leveling)

Downsides:

garbage collection may drives up write amplification
in-memory mapping tables might be large

44.8 Garbage Collection

To reduce GC costs, overprovisioning can be performed: adding extra flash capacity and delay cleaning, which will be performed in background.

44.9 Mapping Table Size

Block-Based Mapping

Hybrid Mapping

Two types of mapping table:

log table (per-page mappings)
data table (per-block mappings)

To read:

first find in the log table
if not found, find in the data table

The key to the hybrid mapping strategy is keeping the number of log blocks small. To do that, the FTL has to periodically examine log blocks and switch them into blocks that can be pointed to by only a single block pointer.

Three types of merge:

switch merge: all pointers of a block in the log table can be switched into a pointer pointing to this block
partial merge
full merge

Page Mapping Plus Caching

44.10 Wear Leveling

One problem of the log-structured approach: blocks filled with long-lived data do not get overwritten; solution: periodically read and rewrite elsewhere

Operating Systems: Three Easy Pieces 学习笔记（一）

liang2kl@outlook.com (Liang Yesheng) — Thu, 13 Jan 2022 20:38:28 +0800

36 I/O Devices

36.1 System Architecture

Devices are connected to the system via:

memory bus (CPU - main memory)
general I/O bus: PCI (graphics, high performance devices)
peripheral bus: SCSI, SATA, USB (slow devices)

In modern systems:

PCIe Graphics
memory interconnect
DMI (Direct Media Interface)
PCIe
eSATA

36.2 A Canonical Device

A device has two important components:

interface
- status
- command
- data
internal structure

36.3 The Canonical Protocol

Programmed I/O (PIO): the main CPU is involved with the data movement

36.4 Lowering CPU Overhead With Interrupts

Utilize CPU:

interrupt
a two-phased approach: polls first, using interrupts if not yet finished
coalescing

36.5 More Efficient Data Movement With DMA

Direct Memory Access (DMA): transfer data between devices and main memory without much CPU intervention

36.6 Methods Of Device Interaction

explicit I/O instructions (on x86, the in and out instructions), usually privileged
memory-mapped I/O (no new instructions are needed)

36.7 Fitting Into The OS: The Device Driver

Read/write blocks:

file system
raw interface (support low-level storage management)

36.8 Case Study: A Simple IDE Disk Driver

Registers are available by reading or writing to specific “I/O addresses” using (on x86) the in and out I/O instructions.

wait for drive to be ready
write parameters to command registers
start the I/O: issuing read/write to command register
data transfer (for writes): write data to data port
handle interrupts

37 Hard Disk Drives

37.1 The Interface

The drive consists of a large number of sectors (512-byte blocks), each of which can be read or written

View the disk as an array of sectors (扇区)

multi-sector operations are possible
only guarantee a single 512-byte write is atomic

Efficiency:

access closer blocks are faster than farther ones
sequential read or write are faster than the random access manner

37.2 Basic Geometry

platter: 磁盘
surface: the 2 sides of a platter
spindle: 轴
track: 道, concentric circles of sectors
disk head
disk arm

37.3 A Simple Disk Drive

I/O time:

rotational delay
seek time
- acceleration
- coasting
- deceleration
- settling (significant)
transfer

Track skew: sequential reads can be properly serviced when crossing track boundaries

Multi-zoned disk drives:

a zone is a consecutive set of tracks on a surface
each zone has the same number of sectors per track
outer zones have more sectors

Cache (track buffer):

read: to read in all of the sectors on that track and cache them in its memory
write
- write back: acknowledge when puts into memory
- write through: acknowledge after actual writes

37.4 I/O Time: Doing The Math

The rate of I/O:

$$ T_\mathrm{I/O} = T_\mathrm{seek}+T_\mathrm{rotation}+T_\mathrm{transfer} $$

$$ R_\mathrm{I/O}=\cfrac{Size_\mathrm{transfer}}{T_\mathrm{I/O}} $$

a huge gap in drive performance between random and sequential workloads
a large difference in performance between high-end “performance” drives and low-end “capacity” drives

37.5 Disk Scheduling

Principle: shortest job first, SJF

Shortest-seek-time-first (SSTF, also shortest-seek-first or SSF): picking requests on nearest track to complete first

problems:

the drive geometry is not available to the OS (fix: nearest-block-first, NBF)
starvation: a steady stream to the closest track will block the access to other tracks

Elevator algorithm

SCAN: simply moves back and forth across the disks

F-SCAN: freezes the queue when doing a sweep (avoid starvation of far-away requests)

C-SCAN (Circular SCAN): only sweeps from outer-to-inner (more fair to the inner and outer tracks)

Prevent fights from taking place on elevators!

SCAN and SSTF does not adhere to SJF closly: they ignore rotation

SPTF: Shortest positioning time first

Also called shortest access time first, SATF

In engineering, it turns out “it depends” is almost always the answer, reflecting that trade-offs are part of the life of the engineer.

On modern drives both seek and rotation are roughly equivalent

SPTF is usually performed inside a drive

Other scheduling issues

Where to perform scheduling: The OS scheduler usually picks (what it thinks) some best few requests (say 16) and issues them all to disk; the disk then uses its internal knowledge to service said requests in the best possible (SPTF) order.

I/O merging: merge I/O requests by the OS

Non-work-conserving: waiting for some time for a possible better request

38 Redundant Arrays of Inexpensive Disks (RAIDs)

RAID: a technique to use multiple disks in concert to build a faster, bigger, and more reliable disk system

externally looks like a disk
internally consisting multiple disks, memory and processor(s)

38.1 Interface And RAID Internals

To perform a logical I/O:

calculate the actual disk(s)
perform one or more physical I/Os

38.2 Fault Model

RAIDs are designed to detect and recover from certain kinds of disk faults.

The fail-stop fault model: a not-working disk is assumed permanently lost

38.3 How To Evaluate A RAID

Characteristics:

capacity
reliability
performance

38.4 RAID Level 0: Striping

RAID level 0, or striping as it is better known, serves as an excellent upper-bound on performance and capacity.

Striping: no redundancy; spread the blocks of the array across the disks in a round-robin fashion to get the most parrallelism.

Simple striping:

Disk 0	Disk 1	Disk 2	Disk 3
0	1	2	3
4	5	6	7
8	9	10	11
12	13	14	15

Stripe: blocks in the same row (0, 1, 2, 3)

Striping with a bigger chunk size (8KB, 2 blocks, a stripe consists of 32KB of data):

Disk 0	Disk 1	Disk 2	Disk 3
0	2	4	6
1	3	5	7
8	10	12	14
9	11	13	15

Chunk sizes

small chunk size: on large files, positioning more on a single disk
large chunk size: less parallelism, but also less posisioning time

Thus, determining the “best” chunk size is hard to do, as it requires a great deal of knowledge about the workload presented to the disk system.

RAID-0 Analysis

capacity: perfect
reliability: any disk failure will lead to data loss
performance: all disks are utilized, often in parallel

Evaluating RAID Performances

Two performance metrics:

single-request latency
steady-state throughput (critical in high-perfromance environments)

Two types of workloads (considered here):

sequential: large contiguous chunks
- searching through a large file for a keyword
random: samll requests, to different random locations on disk
- transactional workloads on a database management system

Denote the sequential and random workload’s throughput as:

$S$ for sequential

$R$ for random

38.5 RAID Level 1: Mirroring

RAID-10 (or RAID 1+0, stripe of mirrors):

Disk 0	Disk 1	Disk 2	Disk 3
0	0	1	1
2	2	3	3
4	4	5	5
6	6	7	7

RAID-01

Disk 0 Disk 1 Disk 2 Disk 3

0 1 0 1

2 3 2 3

4 5 4 5

6 7 6 7

Disk 0	Disk 1	Disk 2	Disk 3
0	1	0	1
2	3	2	3
4	5	4	5
6	7	6	7

read: either copy
write: update both

Analysis

Consistent-update problem

untimely failures on write: inconsistent copies

solution: write-ahead log and recovery procedure

capacity: expensive, $N\cdot B/2$ for mirroring level = 2
reliability: tolerate failure on any disk, up to $N/2$ disks can fail
single request performance
- read: sames as on a single disk ($R$, writes to two blocks)
- write: slightly higher than writing to a single disk (worst-case seek and rotational delay)
steady-state throughput
- write: ($\frac{N}{2}\cdot S$, writes to $N$ blocks)
- read: same as write ($\frac{N}{2}\cdot S$, each disk receives a request for every other block)
random reads/writes
- read: $N\cdot R$
- write: $\frac{N}{2}\cdot R$

38.6 RAID Level 4: Saving Space With Parity

Parity-based approaches attempt to use less capacity and thus overcome the huge space penalty paid by mirrored systems. They do so at a cost, however: performance.

Disk 0	Disk 1	Disk 2	Disk 3	Disk 4
0	1	2	3	P0
4	5	6	7	P1
8	9	10	11	P2
12	13	14	15	P3

Simple function: XOR, which makes a row having even number of 1

Analysis

capacity: $(N-1)\cdot B$
reliability: tolerates 1 disk failure
performance
- steady-state: $(N-1)\cdot S$ (full-stripe write: calculate parity bits first then write all disks in parallel)
- random
  - read: $(N-1)\cdot R$
  - write: $R/2$ (the parity disk is the bottleneck)
    - additive parity: read in parallel, calculate XOR, and write in parallel (larger RAIDs require a high number of reads)
    - subtractive parity: $P_\mathrm{new}=(C_\mathrm{old}\oplus C_\mathrm{new})\oplus P_\mathrm{old}$ (Why not always better?– because it has to read the parity disk?)

Small-write problem: the parity disk is a bottleneck under small writes which prevents parallelism

38.7 RAID Level 5: Rotating Parity

Disk 0	Disk 1	Disk 2	Disk 3	Disk 4
0	1	2	3	P0
5	6	7	P1	4
10	11	P2	8	9
15	P3	12	13	14
P4	16	17	18	19

Analysis

random write: $\frac{N}{4}\cdot R$ (four I/O operations) (aren’t the reads/writes parallel?–the data disk itself serves as parity disk as well)

39 Interlude: Files and Directories

Thus, the OS must take extra care with such a device: this is where users keep data that they really care about.

39.1 Files And Directories

file: linear array of bytes
- low-level name (inode number)
directory
- low-level name (inode number)
- a list of (user-readable name, low-level name) pairs

39.3 Creating Files

Create a new file: call open() and pass O_CREAT flag:

int fd = open("foo", O_CREAT|O_WRONLY|O_TRUNC, S_IRUSR|S_IWUSR);

O_CREAT: create the file if it does not exist
O_WRONLY: read only
O_TRUNC: if exists already, truncates it to a size of zero bytes

39.5 Reading And Writing, But Not Sequentially

Reading some random offsets within the document:

off_t lseek(int fildes, off_t offset, int whence);

fildes: file descriptor
offset: the file offset to a particular location
whence
- SEEK_SET: the offset is set to offset bytes
- SEEK_CUR: the offset is set to its current location plus offset bytes
- SEEK_END: the offset is set to the size of the file plus offset bytes

Calling lseek() does not perform a disk seek

The “current offset” tracked by the OS:

implicitly updates when read or write
explicitly updates with lseek

The file structure:

struct file {
 int ref;
 char readable;
 char writable;
 struct inode * ip;
 uint off;
};

The open file table:

struct {
 struct spinlock lock;
 struct file file[NFILE];
} ftable;

39.6 Shared File Table Entries: `fork()` And `dup()`

The open file table entries are shared by 1. the parent and child processes created by fork(); 2. one file descriptor and its duplicates created by dup(), with the reference count incremented.

39.7 Writing Immediately With `fsync()`

Calls fsync() to force all dirty (not yet written) data to disk.

int rc = write(fd, buffer, size); assert(rc == size);

In some cases, it is also needed to fsync() the containing directory to ensure that the file is part of the directory.

Not surprisingly, this type of detail is often overlooked, leading to many application-level bugs.

mmap() and persistent memory

By combining the persistence of files with the access semantics of memory, file-backed memory mappings support a software abstraction called persistent memory.

39.8 Renaming Files

rename(char *old, char *new)

rename() call is usually implemented as an atomic call.

An example to use this kind of guarantee:

int fd = open("foo.txt.tmp", O_WRONLY|O_CREAT|O_TRUNC, S_IRUSR|S_IWUSR);
write(fd, buffer, size);
fsync(fd);
close(fd);
rename("foo.txt.tmp", "foo.txt");

39.9 Getting Information About Files

Use stat() or fstat() to access metadata of files or directories.

39.10 Removing Files

Use unlink() to remove files.

39.11 Making Directories

Use mkdir() to create a directory.

An empty directory has two entries:

refer to itself
refer to its parent

39.12 Reading Directories

Use opendir(), readdir() and closedir() to read contents of a directory.

DIR * dp = opendir(".");
assert(dp != NULL);
struct dirent * d;
while ((d = readdir(dp)) != NULL) {
 printf("%lu %s\n", (unsigned long) d->d_ino, d->d_name);
}
closedir(dp);

39.13 Deleting Directories

Call rmdir() to delete a directory.

39.14 Hard Links

Call link() to make an entry in the file system tree, simply creates another name in the directory and refers it to the same inode number.

ln file file2

When creating a file, two things happened:

making a structure (the inode) that tracks relevent information

linking a human-readable name to that file, and put that link into a directory

When the file system unlinks a file, it checks the reference count (link count) within the inode number, and only free the node and related data blocks when it reaches zero.

39.15 Symbolic Links

Call symlink() to make an symbolic link (soft link) to a file.

ln -s file file2

Differences from hard links:

symbolic links is actually a file itself of a different type
possibility for dangling reference

39.16 Permission Bits And Access Control Lists

Permission bits consist of three groupings:

owner
group
other (anyone)

The owner of the file can change the permissions.

But now you have learned an even more important lesson: there can be no secrets in a good marriage, even within the file system.

The Time Of Check To Time Of Use (TOCTTOU) problem

The problem is caused by the gap between the check and the update.

39.17 Making And Mounting A File System

And mkfs said, let there be a file system!

Mount: to take an existing directory as a target mount point and paste a new file system onto the directory tree at that point.

Mount unfies all file systems into one tree, making naming uniform and convenient.

40 File System Implementation

vsfs (the Very Simple File System)

The file system is pure software; unlike our development of CPU and memory virtualization, we will not be adding hardware features to make some aspect of the file system work better (though we will want to pay attention to device characteristics to make sure the file system works well).

40.2 Overall Organization

The disk is divided into:

data blocks: user data
inodes
allocation structures: to track whether inodes or data blocks are free or allocated
superblock: contains information about the fie system

40.3 File Organization: The Inode

Inode (index node): the data structure that holds the metadata of a given file

i-number (low-level name): the number that the inode is implicitly referred to; disk position can be calculated by the number

Calculation:

$$ block = (inumber\times sizeof(inode)) / blockSize $$

$$ sector = ((block\times blockSize) + inodeStartAddr) / sectorSize $$

Metadata in a inode:

type: directory or file (kept in the directory entry and thus is not found in the inode)
size
number of blocks
protection information
time
data block position on disk

The Multi-Level Index

Indirect pointer: points to a block that contains pointers which each points to user data

An inode may have some fixed number of direct pointers and a single indirect pointer.

If a file grows large enough, an indirect block is allocated (from the data block) and the inode’s indirect pointer is set to point to it.

Extent-based approaches: a disk pointer plus a length; allow for more than one extent

Linked-based approaches: use a linked list to keep track all blocks of a file

Double indirect pointer: refer to a block that contains indirect pointers

Most files are small: a small number of direct pointers is sufficient

40.4 Directory Organization

Deleting a file can leave an empty space in the middle of the directory; a new entry may reuse an old, bigger, deleted entry.

File systems often streat directories as a special type of file, and thus they have inodes in the inode table.

40.5 Free Space Management

To track free inode/data blocks:

use bitmap
use free list
use more soplisticated ds, e.g. B-tree

Pre-allocation policy: reserve a sequence of blocks for a newly-created file, to guarantee that a portion of the file is contiguous on the disk.

40.6 Access Paths: Reading and Writing

Reading A File From Disk

The timeline of reading a file /foo/bar:

open("/foo/bar", O_RDONLY)

read the root inode (/) (whose i-number is well known; 2 in most UNIX FSs)
read / data (to look for an entry for foo)
read foo inode
read foo data (to look for an entry for bar)
read bar inode

read(...)

read bar inode
read specific block of bar’s data
write bar inode for last access time

Yes, life can get pretty bad when reading a file; as you’re about to find out, writing out a file (and especially, creating a new one) is even worse.

Reads don’t access allocation structures (e.g. bitmaps); they are only accessed when allocation is needed. There is no need to make sure a block is allocated when the inode already points to it

Writing A File To Disk

Each write to a file logically generates 5 I/Os:

read file inode
read data bitmap
write data bitmap (to mark the newly-allocated block as used)
write file data
write file inode

For file creation:

read inode and inode data traversly to find the containing directory
read containing directory data
read inode bitmap (find a free inode)
write inode bitmap (to mark the inode allocated)
write directory data (link)
read/write directory inode
read/write inode of the new file (initialize)
additional I/O might be needed if the directory needs to grow

40.7 Caching and Buffering

Most file systems aggressively use system memory (DRAM) to cache important blocks.

Early file systems use a fixed-size cache to hold popular blocks; this static partitioning method can be wasteful.

Modern systems use a dynamic partitioning approach.

Static vs. dynamic partitioning

static: predictable performance

dynamic: better utilization; more complex to implement

Buffering: delay writes to reduce I/O

In this case, laziness (in writing blocks to disk) is a virtue.

41 Locality and The Fast File System

41.1 The Problem: Poor Performance

Main issue of the old UNIX file system: data was spread all over the place, thus had expensive positioning costs

Some problems:

the inode and the data block are often far away
the file system is getting more and more fragmented
the block size is too small, so that transferring data is inefficient

41.3 Organizing Structure: The Cylinder Group

Cylinder (柱面): a set of tracks on different surfaces that are the same distance from the center

FFS divides the disk into a number of cylinder groups.

As modern drives only export a logical address space of blocks, modern file systems instead organize the drive into block groups, each of which is a consecutive portion of the disk’s address space. By placing two files within the same group, FFS can ensure that accessing one after the other will not result in long seek time.

FFS needs to have the ability to place files and directories into a groups and track relevant information, thus all the structures that a file system is expected to have is included in each group.

The components of a single cylinder group:

a copy of the super block
a per-group inode bitmap and data bitmap
inode region
data block region

41.4 Policies: How To Allocate Files and Directories

The basic mantra is simple: keep related stuff together (and its corollary, keep unrelated stuff far apart).

A few placement heuristics:

directories: find the group
- with a low number of allocated directories (to balance directories across groups)
- with a high number of free inodes (to be able to allocate a bunch of files)
files:
- allocate the data blocks in the same group as its inode
- places all files that are in the same directory in the cylinder group of the directory they are in

41.6 The Large-File Exception

Filling an entire block group with large files will prevents related files from being placed within the same group.

FFS’ policy for large files: after some number of blocks are allocated into the first block group, it places the next large chunk of the file in another block group.

To address the performance problem: choosing chunk size carefully. By amotization, if the chunk size is large enough, the seek time will be relatively little.

FFS’s approach

the first 12 blocks are placed in the same group as the inode
each subsquent indirect block and the blocks it points to are placed in a different group

Note that the trend in disk drives is that transfer rate improves fairly rapidly, as disk manufacturers are good at cramming more bits into the same surface, but the mechanical aspects of drives related to seeks (disk arm speed and the rate of rotation) improve rather slowly.

41.7 A Few Other Things About FFS

Internal fragmentation: 4KB blocks are used by many 2KB files

Sub-blocks: 512-byte blocks can be allocated to files; copy to a 4KB block when the size grows beyond 4KB

To address the performance issue caused by copies: buffer writes, to avoid this in most cases.

A refined disk layout (parameterization): skipping a block when issuing a sequential read (e.g. 0 6 1 7 2 8 3 9 4 10 5 11); to issue the performance problem: using track buffer.

FFS also supports:

long file names
symbolic links

Certainly all modern systems account for the main lesson of FFS: treat the disk like it’s a disk.

2021 秋季学期总结

liang2kl@outlook.com (Liang Yesheng) — Sat, 08 Jan 2022 01:05:00 +0800

关于各课程

这个学期总共 17 学分，除了体育和信概，上了五门课：汇编、自动机、数据结构、网原、概统。

汇编的收获挺大。老师其实讲得很一般，但课程内容很有意思，计算机系统的知识对于之前只接触过上层软件的我来说是一个全新的世界。计算机系统各个部分、各个层次的运转和协调非常精妙，下学期应该有机会继续深入研究。

自动机属于个人感受先抑后扬的课程。前两节课上来就讲上下文无关文法无异于当头一棒，这部分内容过于抽象，而且老师并没有作很好的解释，体验极其不好。后来随着正规语言、DFA 这些更符合直觉的内容的引入，终于慢慢弄清楚了研究这些各种各样的语言有什么用，它们分别对应程序设计中的什么问题，也逐渐适应了其中的逻辑。老师非常温柔，也非常认真负责，但是讲课实在是有点无聊。

邓公的数据结构可以说是清华课程的天花板，上课时都是抱着“上一节少一节”的虔诚心态来享受的。PPT 的组织、老师的讲解、课堂的演示几乎都达到了完美，每堂课听完都有酣畅淋漓的感觉。当然，课讲得好，不代表学生上完课水平就高。上课感觉啥都会了，开始做 PA 还是一脸懵逼，最后的考试更是直接把“你并没有学懂”写在了试卷上。虽然但是，数据结构的确是一门深刻改变编程思想的课程，经过一个学期的学习，受益匪浅。

网原的学习体验并不好。我对网络还算挺感兴趣，计算机网络本身也很有意思，但是这课恨不得把每一层历经几十年发展积累的无论是过时的还是先进的技术都一股脑塞给你，知识过于繁杂，导致上课根本没有一个清晰的脉络。而且，自底向上的方法让人在前几章学数据通信、物理层和链路层时很难理解这些东西有什么用，这些协议这样设计的意义是什么。不过最后考试出的题目还挺有意思，这个出乎意料。

概统选的是胡家信老师的课，全程很水，特别是和之前上的唐宏岩的课相比，知识的广度和深度都差太多，统计竟然只讲到参数估计，连假设检验都没讲。老师上课讲得很详细，但是统计部分基本念 PPT，也没有什么原理性、启发性的解释。考试挺难的，有几道题不会做。

关于项目

这个学期做的项目最主要是一个静态网站生成器的博客插件，给 Mkdocs 增加了博客功能。维护用了不少时间，陆续加了不少功能，用的人也逐渐多了起来，感受还是挺不错的。不过用插件很难构建一个功能完善的博客，最后还是转向了 Hugo 来建博客。

另外，还写了一点 SwiftUI 教程，主要想对 SwiftUI 一些难懂的概念、原理作一下介绍，不过写了一半就停下来了（的确太花时间），写的效果也没有达到预期，可能寒假会继续写下去。

总结

前几天和 lqw 吃饭的时候聊到，感觉这个学期过得比以往任何一个学期都要快。前几个学期要么有繁忙的社工，要么在玩命写项目，要么在紧张准备转系，相比之下这个学期显得过于平淡了。

这个调子其实在开学前就定下了。我的 GPA 在机械还能勉强排到中上游，而在各路大神齐聚的贵系，这个分数对应的排名就变得非常难看，压力变得空前巨大。于是，暑假的时候我就先后退掉了宣中的社工、拒绝了一个创业团队的邀请。我给出的理由都是：成绩不好，之后只能专心学业。

结果就是，大多数日子里，我基本上都在宿舍-教室-食堂-图书馆的循环中度过，有时候想做一些好玩的东西，也的确写了几个玩具项目，但最终还是一次次地告诫自己：不要让这些事情干扰到学习。

现在回想，专心学习的确是有好处的。我在大一第一个学期之后第一次做到线下上课全勤，上课基本上都认真地听讲并做了笔记，每天课后也对内容做了回顾。这样坚持下来的最明显的好处是：我能够很清晰地掌握每个课程的脉络，清楚地知道自己在这些课程中学到了什么东西、这些课程的精髓在什么地方。之前的学习过于匆忙，没有回顾、思考的时间，囫囵吞枣，因此几乎没有对哪一门课有“学懂”了的感觉。这个学期，投入了更多的时间进行复习、思考，对知识的理解就有了深度。

但是，毕竟学习只是生活的一部分，阅读、游戏、游玩、写程序等等都不应该因此而荒废。以后应当提高学习效率，减少浪费在漫无目的的刷手机上的时间，把更多的时间放在这些有意思的事情上面。

总的来说，这个学期算是挺有收获。在贵系的第一个学期，（应该？但愿？）没有掉队，而且学习的内容都挺感兴趣，虽然学习的过程并不一直都那么顺利。下学期学习任务更重，希望到时候能够既保持课内学习的状态，同时在科研和其他方面有一定的发展吧。

更新

今天（1 月 21 日）不经意间看到机械班群里面赵导发的群公告，才知道出成绩了，来这里更新一下。

不得不说我的学习感觉和最终的成绩还是比较吻合的，几门比较用功并且学得比较透彻的专业课（汇编、自动机、网原）都得到了 4.0；花了比较多时间但并没有学得很透彻的数据结构，以及没怎么花时间的概统得了 3.6。

其实考试过程中和考完试之后一直挺担心一件事，就是我自认为学得很透彻的几门课可能由于我考试时经常犯的计算错误、审题错误、脑子短路等问题而不能够取得一个比较好的成绩；考完汇编和概统之后我悲观地认为这个学期，甚至以后的学期一个 4.0 都拿不到了。事实证明我还是多虑了，大多数的考试都是所学内容的子集，如果已经学得比较透彻，那已经超过（大部分的）考试要求了，也基本上不会出现上面这些所谓的问题。虽然这是一个高中就已经非常清楚的常识，但说实话，在没有什么自信的情况下（特别是在贵系中）是很难相信这一点，也很难相信“有付出就有回报”的。

这个学期的学习算是达到了预期，下学期继续努力吧。