最新公告
  • 欢迎您光临小鹏博客,本站秉承服务宗旨 履行“站长”责任,销售只是起点 服务永无止境!立即加入我们
  • 正文概述
  • 评价建议
  • 资源名称:Spark内核设计的艺术架构设计与实现

    内容简介:

    本书赞誉

    前言

    第1章 环境准备 ········································1

    1.1 运行环境准备 ···········································2

    1.1.1 安装JDK ·········································2

    1.1.2 安装Scala ········································2

    1.1.3 安装Spark ·······································3

    1.2 Spark初体验 ···································4

    1.2.1 运行spark-shell ·······························4

    1.2.2 执行word count ······························5

    1.2.3 剖析spark-shell ·······························9

    1.3 阅读环境准备 ·········································14

    1.3.1 安装SBT ·······································15

    1.3.2 安装Git ·········································15

    1.3.3 安装Eclipse Scala IDE插件 ········15

    1.4 Spark源码编译与调试 ·························17

    1.5 小结 ···························23

    第2章 设计理念与基本架构 ···············24

    2.1 初识Spark ··································25

    2.1.1 Hadoop MRv1的局限···················25

    2.1.2 Spark的特点 ·································26

    2.1.3 Spark使用场景 ·····························28

    2.2 Spark基础知识 ······································29

    2.3 Spark基本设计思想 ·····························31

    2.3.1 Spark模块设计 ·····························32

    2.3.2 Spark模型设计 ·····························34

    2.4 Spark基本架构 ···································36

    2.5 小结 ·································38

    第3章 Spark基础设施 ·························39

    3.1 Spark配置 ········································40

    3.1.1 系统属性中的配置 ·······················40

    3.1.2 使用SparkConf配置的API ·········41

    3.1.3 克隆SparkConf配置 ····················42

    3.2 Spark内置RPC框架 ····························42

    3.2.1 RPC配置TransportConf ··············45

    3.2.2 RPC客户端工厂Transport- ClientFactory ·······················47

    3.2.3 RPC服务端TransportServer ········53

    3.2.4 管道初始化 ···································56

    3.2.5 TransportChannelHandler详解 ·····57

    3.2.6 服务端RpcHandler详解 ··············63

    3.2.7 服务端引导程序Transport-ServerBootstrap ·····················68

    3.2.8 客户端TransportClient详解 ········71

    3.3 事件总线 ····································78

    3.3.1 ListenerBus的继承体系 ···············79

    3.3.2 SparkListenerBus详解 ··················80

    3.3.3 LiveListenerBus详解 ····················83

    3.4 度量系统 ···········································87

    3.4.1 Source继承体系 ···························87

    3.4.2 Sink继承体系 ·······························89

    3.5 小结 ·········································92

    第4章 SparkContext的初始化 ·········93

    4.1 SparkContext概述 ·································94

    4.2 创建Spark环境 ·····································97

    4.3 SparkUI的实现 ····································100

    4.3.1 SparkUI概述 ·······························100

    4.3.2 WebUI框架体系 ·························102

    4.3.3 创建SparkUI ·······························107

    4.4 创建心跳接收器 ··································111

    4.5 创建和启动调度系统··························112

    4.6 初始化块管理器BlockManager ·······114

    4.7 启动度量系统 ·······························114

    4.8 创建事件日志监听器··························115

    4.9 创建和启动ExecutorAllocation-Manager ··························116

    4.10 ContextCleaner的创建与启动 ········120

    4.10.1 创建ContextCleaner ·················120

    4.10.2 启动ContextCleaner ·················120

    4.11 额外的SparkListener与启动事件总线 ··························122

    4.12 Spark环境更新 ··································123

    4.13 SparkContext初始化的收尾 ···········127

    4.14 SparkContext提供的常用方法 ·······128

    4.15 SparkContext的伴生对象················130

    4.16 小结 ····································131

    第5章 Spark执行环境 ························132

    5.1 SparkEnv概述 ·································133

    5.2 安全管理器SecurityManager ············133

    5.3 RPC环境 ·········································135

    5.3.1 RPC端点RpcEndpoint ···············136

    5.3.2 RPC端点引用RpcEndpointRef ···139

    5.3.3 创建传输上下文TransportConf ···142

    5.3.4 消息调度器Dispatcher ···············142

    5.3.5 创建传输上下文Transport-Context ·························154

    5.3.6 创建传输客户端工厂Transport-ClientFactory ····················159

    5.3.7 创建TransportServer ···················160

    5.3.8 客户端请求发送 ·························162

    5.3.9 NettyRpcEnv中的常用方法 ·······173

    5.4 序列化管理器SerializerManager ·····175

    5.5 广播管理器BroadcastManager ·········178

    5.6 map任务输出跟踪器 ··························185

    5.6.1 MapOutputTracker的实现 ··········187

    5.6.2 MapOutputTrackerMaster的实现原理 ·······················191

    5.7 构建存储体系 ·······································199

    5.8 创建度量系统 ·······································201

    5.8.1 MetricsCon?g详解 ·····················203

    5.8.2 MetricsSystem中的常用方法 ····207

    5.8.3 启动MetricsSystem ····················209

    5.9 输出提交协调器 ··································211

    5.9.1 OutputCommitCoordinator-Endpoint的实现 ··················211

    5.9.2 OutputCommitCoordinator的实现 ··························212

    5.9.3 OutputCommitCoordinator的工作原理 ························216

    5.10 创建SparkEnv ····································217

    5.11 小结 ·····································217

    第6章 存储体系 ·····································219

    6.1 存储体系概述 ·······································220

    6.1.1 存储体系架构 ·····························220

    6.1.2 基本概念 ·····································222

    6.2 Block信息管理器 ································227

    6.2.1 Block锁的基本概念 ···················227

    6.2.2 Block锁的实现 ···························229

    6.3 磁盘Block管理器 ······························234

    6.3.1 本地目录结构 ·····························234

    6.3.2 DiskBlockManager提供的方法 ···························236

    6.4 磁盘存储DiskStore ·····························239

    6.5 内存管理器 ·····································242

    6.5.1 内存池模型 ·································243

    6.5.2 StorageMemoryPool详解 ···········244

    6.5.3 MemoryManager模型 ················247

    6.5.4 Uni?edMemoryManager详解 ····250

    6.6 内存存储MemoryStore ······················252

    6.6.1 MemoryStore的内存模型 ··········253

    6.6.2 MemoryStore提供的方法 ··········255

    6.7 块管理器BlockManager ····················265

    6.7.1 BlockManager的初始化 ·············265

    6.7.2 BlockManager提供的方法 ·········266

    6.8 BlockManagerMaster对Block-Manager的管理 ·················285

    6.8.1 BlockManagerMaster的职责 ······285

    6.8.2 BlockManagerMasterEndpoint详解 ·································286

    6.8.3 BlockManagerSlaveEndpoint详解 ·····························289

    6.9 Block传输服务 ····································290

    6.9.1 初始化NettyBlockTransfer-Service ···························291

    6.9.2 NettyBlockRpcServer详解 ·········292

    6.9.3 Shuf?e客户端 ·····························296

    6.10 DiskBlockObjectWriter详解 ···········305

    6.11 小结 ·······································308

    第7章 调度系统 ·····································309

    7.1 调度系统概述 ·······································310

    7.2 RDD详解 ·····································312

    7.2.1 为什么需要RDD ························312

    7.2.2 RDD实现的初次分析 ················313

    7.2.3 RDD依赖 ····································316

    7.2.4 分区计算器Partitioner················318

    7.2.5 RDDInfo ······································320

    7.3 Stage详解 ········································321

    7.3.1 ResultStage的实现 ·····················322

    7.3.2 Shuf?eMapStage的实现 ·············323

    7.3.3 StageInfo ······································324

    7.4 面向DAG的调度器DAGScheduler ···326

    7.4.1 JobListener与JobWaiter ·············326

    7.4.2 ActiveJob详解 ····························328

    7.4.3 DAGSchedulerEventProcessLoop的简要介绍 ·······················328

    7.4.4 DAGScheduler的组成 ················329

    7.4.5 DAGScheduler提供的常用方法 ···330

    7.4.6 DAGScheduler与Job的提交 ····334

    7.4.7 构建Stage····································337

    7.4.8 提交ResultStage ························341

    7.4.9 提交还未计算的Task ·················343

    7.4.10 DAGScheduler的调度流程 ······347

    7.4.11 Task执行结果的处理 ··············348

    7.5 调度池Pool ······································351

    7.5.1 调度算法 ·······························352

    7.5.2 Pool的实现 ·································354

    7.5.3 调度池构建器 ·····························357

    7.6 任务集合管理器TaskSetManager ···363

    7.6.1 Task集合 ·····································363

    7.6.2 TaskSetManager的成员属性 ······364

    7.6.3 调度池与推断执行 ·····················366

    7.6.4 Task本地性 ·································370

    7.6.5 TaskSetManager的常用方法 ······373

    7.7 运行器后端接口LauncherBackend ···383

    7.7.1 BackendConnection的实现 ········384

    7.7.2 LauncherBackend的实现 ···········386

    7.8 调度后端接口SchedulerBackend ····389

    7.8.1 SchedulerBackend的定义 ··········389

    7.8.2 LocalSchedulerBackend的实现分析 ································390

    7.9 任务结果获取器TaskResultGetter ···394

    7.9.1 处理成功的Task ·························394

    7.9.2 处理失败的Task ·························396

    7.10 任务调度器TaskScheduler ··············397

    7.10.1 TaskSchedulerImpl的属性 ·····397

    7.10.2 TaskSchedulerImpl的初始化 ···399

    7.10.3 TaskSchedulerImpl的启动 ·····399

    7.10.4 TaskSchedulerImpl与Task的提交 ·······················400

    7.10.5 TaskSchedulerImpl与资源分配 ···························402

    7.10.6 TaskSchedulerImpl的调度流程 ······························405

    7.10.7 TaskSchedulerImpl对执行结果的处理 ·····························406

    7.10.8 TaskSchedulerImpl的常用方法 ···409

    7.11 小结 ·······································412

    第8章 计算引擎 ·····································413

    第9章 部署模式 ········································520

    第10章 Spark API ································647

    资源截图:

    1.png

    小鹏博客是一个优秀的主题,极致后台体验,无插件,集成会员系统
    小鹏博客 » Spark内核设计的艺术架构设计与实现

    常见问题FAQ

    免费下载或者VIP会员专享资源能否直接商用?
    本站所有资源版权均属于原作者所有,这里所提供资源均只能用于参考学习用,请勿直接商用。若由于商用引起版权纠纷,一切责任均由使用者承担。更多说明请参考 VIP介绍。
    提示下载完但解压或打开不了?
    最常见的情况是下载不完整: 可对比下载完压缩包的与网盘上的容量,若小于网盘提示的容量则是这个原因。这是浏览器下载的bug,建议用百度网盘软件或迅雷下载。若排除这种情况,可在对应资源底部留言,或 联络我们.。
    找不到素材资源介绍文章里的示例图片?
    对于PPT,KEY,Mockups,APP,网页模版等类型的素材,文章内用于介绍的图片通常并不包含在对应可供下载素材包内。这些相关商业图片需另外购买,且本站不负责(也没有办法)找到出处。 同样地一些字体文件也是这种情况,但部分素材会在素材包内有一份字体下载链接清单。

    发表评论

    售后服务:

    • 售后服务范围 1、商业模板使用范围内问题免费咨询
      2、源码安装、模板安装(一般 ¥50-300)服务答疑仅限SVIP用户
      3、单价超过200元的模板免费一次安装,需提供服务器信息。
      付费增值服务 1、提供dedecms模板、WordPress主题、discuz模板优化等服务请详询在线客服
      2、承接 WordPress、DedeCMS、Discuz 等系统建站、仿站、开发、定制等服务
      3、服务器环境配置(一般 ¥50-300)
      4、网站中毒处理(需额外付费,500元/次/质保三个月)
      售后服务时间 周一至周日(法定节假日除外) 9:00-23:00
      免责声明 本站所提供的模板(主题/插件)等资源仅供学习交流,若使用商业用途,请购买正版授权,否则产生的一切后果将由下载用户自行承担,有部分资源为网上收集或仿制而来,若模板侵犯了您的合法权益,请来信通知我们(Email: 80027422@qq.com),我们会及时删除,给您带来的不便,我们深表歉意!

    Hi, 如果你对这款模板有疑问,可以跟我联系哦!

    联系作者
    • 37会员总数(位)
    • 13499资源总数(个)
    • 0本周发布(个)
    • 0 今日发布(个)
    • 92稳定运行(天)

    小鹏博客XIAOPBK.COM一个高级程序员模板开发平台提供最优质的资源集合

    开通SVIP 客服QQ