PageSourceSearch

https://xiaowantree.com/shu-ju-gong-cheng-ru-men-lu-xian-tu/

html xiaowantree.com collected 2026-10-04 08:09:12 UTC 105,560 bytes, 1,598 lines download raw bytes

1<!DOCTYPE html>
2<html lang="zh">
3<head>
4  <meta charset="utf-8">
5  <meta name="viewport" content="width=device-width, initial-scale=1">
6  <meta name="theme-color" content="#2F5BEA">
7  <title>数据工程入门路线图(必看)</title>
8  <meta name="description" content="『文字版 』:写在前面:本次文章主要分成三个部分【国内市场、海外市场、两者对比】">
9    <link rel="icon" href="https://xiaowantree.com/content/images/size/w256h256/format/jpeg/2025/08/8---9----1-.jpg" type="image/jpeg">
10    <link rel="canonical" href="https://xiaowantree.com/shu-ju-gong-cheng-ru-men-lu-xian-tu/">
11    <meta name="referrer" content="no-referrer-when-downgrade">
12    
13    <meta property="og:site_name" content="小万和小万朋友的知识成长营地">
14    <meta property="og:type" content="article">
15    <meta property="og:title" content="数据工程入门路线图(必看)">
16    <meta property="og:description" content="『文字版 』:写在前面:本次文章主要分成三个部分【国内市场、海外市场、两者对比】">
17    <meta property="og:url" content="https://xiaowantree.com/shu-ju-gong-cheng-ru-men-lu-xian-tu/">
18    <meta property="og:image" content="https://xiaowantree.com/content/images/2025/12/real_pixels_jpg-by-RohithDezinr.jpeg">
19    <meta property="article:published_time" content="2025-12-16T08:31:00.000Z">
20    <meta property="article:modified_time" content="2026-07-27T12:32:39.000Z">
21    <meta property="article:tag" content="数仓知识点">
22    <meta property="article:tag" content="路线图">
23    <meta property="article:tag" content="数据工程基础">
24    
25    <meta property="article:publisher" content="https://www.facebook.com/ghost">
26    <meta name="twitter:card" content="summary_large_image">
27    <meta name="twitter:title" content="数据工程入门路线图(必看)">
28    <meta name="twitter:description" content="『文字版 』:写在前面:本次文章主要分成三个部分【国内市场、海外市场、两者对比】">
29    <meta name="twitter:url" content="https://xiaowantree.com/shu-ju-gong-cheng-ru-men-lu-xian-tu/">
30    <meta name="twitter:image" content="https://xiaowantree.com/content/images/2025/12/real_pixels_jpg-by-RohithDezinr.jpeg">
31    <meta name="twitter:label1" content="Written by">
32    <meta name="twitter:data1" content="小万来了">
33    <meta name="twitter:label2" content="Filed under">
34    <meta name="twitter:data2" content="数仓知识点, 路线图, 数据工程基础">
35    <meta name="twitter:site" content="@ghost">
36    <meta property="og:image:width" content="736">
37    <meta property="og:image:height" content="588">
38    
39    
39<script type="application/ld+json">
40{
41    "@context": "https://schema.org",
42    "@type": "Article",
43    "publisher": {
44        "@type": "Organization",
45        "name": "小万和小万朋友的知识成长营地",
46        "url": "https://xiaowantree.com/",
47        "logo": {
48            "@type": "ImageObject",
49            "url": "https://xiaowantree.com/content/images/2025/12/Cat-Paw-Sticker---Cat-Paw-High-Five---Discover---Share-GIFs.gif",
50            "width": 60,
51            "height": 60
52        }
53    },
54    "author": {
55        "@type": "Person",
56        "name": "小万来了",
57        "image": {
58            "@type": "ImageObject",
59            "url": "https://xiaowantree.com/content/images/2025/07/------------_20250715062913_877-2.jpg",
60            "width": 1168,
61            "height": 960
62        },
63        "url": "https://xiaowantree.com/author/xiaowan/",
64        "sameAs": []
65    },
66    "headline": "数据工程入门路线图(必看)",
67    "url": "https://xiaowantree.com/shu-ju-gong-cheng-ru-men-lu-xian-tu/",
68    "datePublished": "2025-12-16T08:31:00.000Z",
69    "dateModified": "2026-07-27T12:32:39.000Z",
70    "image": {
71        "@type": "ImageObject",
72        "url": "https://xiaowantree.com/content/images/2025/12/real_pixels_jpg-by-RohithDezinr.jpeg",
73        "width": 736,
74        "height": 588
75    },
76    "keywords": "数仓知识点, 路线图, 数据工程基础",
77    "description": "『文字版 』:写在前面:本次文章主要分成三个部分【国内市场、海外市场、两者对比】",
78    "mainEntityOfPage": "https://xiaowantree.com/shu-ju-gong-cheng-ru-men-lu-xian-tu/"
79}
80    </script>
80
81
82    <meta name="generator" content="Ghost 6.53">
83    <link rel="alternate" type="application/rss+xml" title="小万和小万朋友的知识成长营地" href="https://xiaowantree.com/rss/">
84    
84<script defer src="https://cdn.jsdelivr.net/ghost/portal@~2.69/umd/portal.min.js" data-i18n="true" data-ghost="https://xiaowantree.com/" data-key="21a6b98a8f6e0cbb6f2373f31a" data-api="https://xiaowantree.com/ghost/api/content/" data-locale="zh" crossorigin="anonymous"></script>
84<style id="gh-members-styles">.gh-post-upgrade-cta-content,
85.gh-post-upgrade-cta {
86    display: flex;
87    flex-direction: column;
88    align-items: center;
89    font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;
90    text-align: center;
91    width: 100%;
92    color: #ffffff;
93    font-size: 16px;
94}
95
96.gh-post-upgrade-cta-content {
97    border-radius: 8px;
98    padding: 40px 4vw;
99}
100
101.gh-post-upgrade-cta h2 {
102    color: #ffffff;
103    font-size: 28px;
104    letter-spacing: -0.2px;
105    margin: 0;
106    padding: 0;
107}
108
109.gh-post-upgrade-cta p {
110    margin: 20px 0 0;
111    padding: 0;
112}
113
114.gh-post-upgrade-cta small {
115    font-size: 16px;
116    letter-spacing: -0.2px;
117}
118
119.gh-post-upgrade-cta a {
120    color: #ffffff;
121    cursor: pointer;
122    font-weight: 500;
123    box-shadow: none;
124    text-decoration: underline;
125}
126
127.gh-post-upgrade-cta a:hover {
128    color: #ffffff;
129    opacity: 0.8;
130    box-shadow: none;
131    text-decoration: underline;
132}
133
134.gh-post-upgrade-cta a.gh-btn {
135    display: block;
136    background: #ffffff;
137    text-decoration: none;
138    margin: 28px 0 0;
139    padding: 8px 18px;
140    border-radius: 4px;
141    font-size: 16px;
142    font-weight: 600;
143}
144
145.gh-post-upgrade-cta a.gh-btn:hover {
146    opacity: 0.92;
147}</style>
147<script async src="https://js.stripe.com/v3/"></script>
147
148    
148<script defer src="https://cdn.jsdelivr.net/ghost/sodo-search@~1.8/umd/sodo-search.min.js" data-key="21a6b98a8f6e0cbb6f2373f31a" data-styles="https://cdn.jsdelivr.net/ghost/sodo-search@~1.8/umd/main.css" data-sodo-search="https://xiaowantree.com/" data-locale="zh" crossorigin="anonymous"></script>
148
149    
149<script defer src="https://cdn.jsdelivr.net/ghost/announcement-bar@~1.1/umd/announcement-bar.min.js" data-announcement-bar="https://xiaowantree.com/" data-api-url="https://xiaowantree.com/members/api/announcement/" crossorigin="anonymous"></script>
149
150    <link href="https://xiaowantree.com/webmentions/receive/" rel="webmention">
151    
151<script defer src="/public/cards.min.js?v=e3e7e3db28"></script>
151
152    <link rel="stylesheet" type="text/css" href="/public/cards.min.css?v=e3e7e3db28">
153    
153<script defer src="/public/member-attribution.min.js?v=e3e7e3db28"></script>
153<style>:root {--ghost-accent-color: #16d1aa;}</style>
154    <!-- Ghost Code Injection: Site Header -->
155<!-- Fixed version of your current setup. Keep this in Site Header only. -->
vendor: 65 bytes, lines 155-157
155
156
157<script async src="https://www.googletagmanager.com/gtag/js?id=
157G-MJ2YRQ6SYD
vendor: 12 bytes, line 157
157"></script>
158<script>
159  
vendor: 133 bytes, lines 159-162
159window.dataLayer = window.dataLayer || [];
160  function gtag(){dataLayer.push(arguments);}
161  gtag('js', new Date());
162  gtag('config', '
162G-MJ2YRQ6SYD
vendor: 4 bytes, line 162
162');
163</script>
163
164
165<link rel="stylesheet" href="https://unpkg.com/tocbot@4/dist/tocbot.css">
166<script src="https://unpkg.com/tocbot@4/dist/tocbot.min.js"></script>
166
167
168<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/styles/a11y-light.min.css">
169
170<script>
171  window.themeGlobal = window.themeGlobal || {};
172  window.themeGlobal.autoloadResources = true;
173</script>
173
174
175<style>
176  :root {
177    --xw-font: -apple-system, BlinkMacSystemFont, "Segoe UI", "Roboto",
178      "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei",
179      "Helvetica Neue", Arial, sans-serif;
180    --xw-text: #1f1f1f;
181    --xw-muted: #5f6368;
182    --xw-blue: #0b57d0;
183    --xw-border: #e0e0e0;
184    --xw-code-bg: #f0f4f9;
185  }
186
187  body,
188  p,
189  li,
190  h1,
191  h2,
192  h3,
193  h4,
194  h5,
195  h6,
196  td,
197  th,
198  div,
199  span,
200  blockquote,
201  a,
202  button,
203  input,
204  textarea {
205    font-family: var(--xw-font) !important;
206    -webkit-font-smoothing: antialiased;
207    letter-spacing: 0 !important;
208  }
209
210  .footer__copy span:nth-child(2),
211  .footer__copy a[href*="ghost.org"],
212  .footer__copy a[href*="brightthemes.com"],
213  .footer-copyright__link,
214  .footer-copyright {
215    display: none !important;
216  }
217
218  /* Article content: current Gemini/Google-like reading style, fixed syntax. */
219  .post-template .content,
220  .page-template .content,
221  .post-template .gh-content,
222  .page-template .gh-content {
223    display: block !important;
224    max-width: 740px !important;
225    width: 100% !important;
226    margin-left: auto !important;
227    margin-right: auto !important;
228    color: var(--xw-text) !important;
229    font-size: 17px !important;
230    line-height: 1.75 !important;
231    font-weight: 400 !important;
232  }
233
234  .post-template .content p,
235  .page-template .content p,
236  .post-template .gh-content p,
237  .page-template .gh-content p {
238    margin-top: 0 !important;
239    margin-bottom: 0.8em !important;
240  }
241
242  .post-template .content p:empty,
243  .page-template .content p:empty,
244  .post-template .gh-content p:empty,
245  .page-template .gh-content p:empty {
246    display: none !important;
247  }
248
249  .content h1,
250  .content h2,
251  .content h3,
252  .gh-content h1,
253  .gh-content h2,
254
255      margin: 0;
256    padding-left: 0;
257  }
258
259  .toc-list {
260    counter-reset: none;
261    display: flex;
262    flex-direction: column;
263    gap: 6px;
264    line-height: 1.4;
265    margin: 0;
266    padding: 0;
267  }
268
269  .toc-list .toc-list { padding-left: 12px; }
270  .toc-list-item { padding-left: 0; }
271
272  .toc-link {
273    border-radius: 8px;
274    color: #374151;
275    display: block;
276    padding: 4px 6px;
277    text-decoration: none;
278    transition: all 0.2s ease;
279  }
280
281  .toc-link::before {
282    background-color: transparent;
283    max-height: 20px;
284  }
285
286  .toc-link:hover {
287    background: rgba(255, 102, 0, 0.08);
288    color: #ff6600;
289  }
290
291  .is-active-link {
292    background: rgba(255, 102, 0, 0.08);
293    color: var(--ghost-accent-color, #ff6600) !important;
294    font-weight: 600;
295  }
296
297  .toc-toggle {
298    align-items: center;
299    background: #fff;
300    border: 3px solid #111;
301    border-radius: 14px;
302    box-shadow: 6px 6px 0 #111;
303    color: #111;
304    cursor: pointer;
305    display: none;
306    height: 44px;
307    justify-content: center;
308    padding: 8px;
309    position: fixed;
310    right: 1.5rem !important;
311    top: 6rem !important;
312    width: 44px;
313    z-index: 100;
314  }
315
316  .toc-toggle .icon-close { display: none; }
317  .toc-toggle.active .icon-close { display: block; }
318  .toc-toggle.active .icon-menu { display: none; }
319
320  @media (max-width: 900px) {
321    .js-toc {
322      max-width: 320px;
323      right: 1rem !important;
324      top: 7rem !important;
325    }
326
327    .toc-toggle {
328      right: 1rem !important;
329      top: 5.5rem !important;
330    }
331  }
332
333  /* Roadmap embed */
334  article.content.post-access-public > .roadmap-stage,
335  article.gh-content.post-access-public > .roadmap-stage {
336    max-width: none !important;
337    width: 100vw !important;
338    margin-left: calc(50% - 50vw) !important;
339    margin-right: calc(50% - 50vw) !important;
340  }
341
342  .roadmap-stage {
343    background: #f3f4f6;
344    display: flex;
345    justify-content: center;
346    padding: 28px 0;
347  }
348
349  .roadmap-stage > .roadmap-embed {
350    background: #fff;
351    border: 1px solid rgba(15, 23, 42, 0.08);
352    border-radius: 16px;
353    box-shadow: 0 18px 50px rgba(15, 23, 42, 0.08);
354    height: min(86vh, 980px);
355    margin: 0 auto;
356    max-width: none !important;
357    overflow: hidden;
358    position: relative;
359    width: min(1680px, calc(100vw - 64px)) !important;
360  }
361
362  .roadmap-toolbar {
363    display: flex !important;
364    gap: 10px !important;
365    left: 300px !important;
366    pointer-events: auto !important;
367    position: absolute !important;
368    top: 16px !important;
369    z-index: 9999 !important;
370  }
371
372  .rm-btn {
373    background: rgba(255, 255, 255, 0.92);
374    backdrop-filter: blur(10px);
375    border: 1px solid rgba(0, 0, 0, 0.10);
376    border-radius: 12px;
377    cursor: pointer;
378    font-size: 14px;
379    line-height: 1;
380    padding: 10px 12px;
381  }
382
383  .rm-exit { display: none; }
384  .roadmap-embed.is-active .rm-enter { display: none; }
385  .roadmap-embed.is-active .rm-exit { display: inline-flex; }
386
387  .roadmap-iframe {
388    border: 0;
389    height: 100%;
390    inset: 0;
391    pointer-events: none;
392    position: absolute;
393    width: 100%;
394  }
395
396  .roadmap-embed.is-active .roadmap-iframe {
397    pointer-events: auto;
398  }
399
400  .rm-btn.rm-enter {
401    animation: rm-pulse 2.2s ease-in-out infinite;
402    background: linear-gradient(135deg, #ef4444, #dc2626);
403    border: none;
404    box-shadow: 0 6px 18px rgba(239, 68, 68, 0.35),
405      inset 0 -1px 0 rgba(0, 0, 0, 0.15);
406    color: #fff;
407    font-weight: 600;
408  }
409
410  @keyframes rm-pulse {
411    0% { box-shadow: 0 0 0 0 rgba(239, 68, 68, 0.45); }
412    70% { box-shadow: 0 0 0 10px rgba(239, 68, 68, 0); }
413    100% { box-shadow: 0 0 0 0 rgba(239, 68, 68, 0); }
414  }
415</style>
416    <link rel="preconnect" href="https://fonts.bunny.net"><link rel="stylesheet" href="https://fonts.bunny.net/css?family=fira-mono:400,700"><style>:root {--gh-font-heading: Fira Mono;}</style>
417  <link rel="stylesheet" href="/assets/vendor/phosphor/duotone.css?v=e3e7e3db28">
418  <link rel="stylesheet" href="/assets/css/screen.css?v=e3e7e3db28">
419  <link rel="stylesheet" href="/assets/css/ai-roadmap.css?v=e3e7e3db28">
420  <link rel="stylesheet" href="/assets/css/reading-colors.css?v=e3e7e3db28">
421  <link rel="stylesheet" href="/assets/css/interview-sidebar.css?v=e3e7e3db28">
422  <style>:root { --academy-accent: #2F5BEA; }</style>
423  
423<script defer src="/assets/js/theme.js?v=e3e7e3db28"></script>
423
424</head>
425<body class="post-template tag-knowledge-points tag-roadmap tag-data-engineering-foundations tag-hash-order-011-2 gh-font-heading-fira-mono">
426  <a class="skip-link" href="#academy-main">跳到主要内容</a>
427  <div class="app-shell">
428    <header class="topbar" data-header>
429  <div class="topbar__left">
430    <button class="icon-button mobile-only" type="button" data-sidebar-open aria-label="打开课程目录">
431      <span class="ui-icon ui-icon--menu" aria-hidden="true"></span>
432    </button>
433    <a class="brand" href="https://xiaowantree.com" aria-label="返回学习首页">
434        <img class="brand__logo" src="https://xiaowantree.com/content/images/2025/12/Cat-Paw-Sticker---Cat-Paw-High-Five---Discover---Share-GIFs.gif" alt="小万来咯">
435    </a>
436  </div>
437
438  <nav class="main-nav" aria-label="主导航">
439      <ul class="nav-list">
440      <li class="nav-xu-yan-shi-yong-qian-bi-kan is-current"><a href="https://xiaowantree.com/shu-ju-gong-cheng-ru-men-lu-xian-tu/">序言(使用前必看)</a></li>
441      <li class="nav-shu-cang-lu-xian-tu-guo-nei-qiu-zhi-ban"><a href="https://xiaowantree.com/shi-shi-roadmap/">数仓路线图(国内求职版)</a></li>
442      <li class="nav-shu-cang-lu-xian-tu-hai-wai-qiu-zhi-ban"><a href="https://xiaowantree.com/xue-xi-lu-jing-hai-wai-qiu-zhi-ban/">数仓路线图(海外求职版)</a></li>
443      <li class="nav-zi-liao"><a href="https://xiaowantree.com/">资料</a></li>
444      <li class="nav-dai-ma-ku-kai-fa-zhong"><a href="https://git.xiaowantree.com/">代码库(开发中)</a></li>
445  </ul>
446
447    <a class="main-nav__service-link" href="/services/">服务</a>
448  </nav>
449
450  <div class="topbar__actions">
451    <a class="mobile-service-link mobile-only" href="/services/">服务</a>
452    <button class="search-field" type="button" data-ghost-search aria-label="搜索课程与文章">
453      <span class="ui-icon ui-icon--search" aria-hidden="true"></span>
454      <span>搜索课程、文章、问答</span>
455      <kbd>⌘K</kbd>
456    </button>
457
458      <button class="topbar-signin-button" type="button" data-portal="signin">登录 / 注册</button>
459  </div>
460</header>
461    
462              <div class="app-grid lesson-grid">
463  <div class="sidebar-scrim" data-sidebar-scrim></div>
464<button class="desktop-sidebar-toggle" type="button" data-sidebar-collapse aria-controls="academy-course-sidebar" aria-expanded="true" aria-label="折叠左侧学习路径" title="折叠左侧学习路径">
465  <i class="ph-duotone ph-sidebar-simple" aria-hidden="true"></i>
466  <span class="desktop-sidebar-toggle__label">收起路径</span>
467</button>
468<aside class="course-sidebar" id="academy-course-sidebar" data-course-sidebar aria-label="课程导航">
469  <div class="sidebar-mobile-head mobile-only">
470    <strong>课程目录</strong>
471    <button class="icon-button" type="button" data-sidebar-close aria-label="关闭课程目录"><span class="ui-icon ui-icon--close" aria-hidden="true"></span></button>
472  </div>
473
474  <div class="course-sidebar__scroll" data-sidebar-scroll>
475    <nav class="feed-sidebar-nav" aria-label="小万动态">
476      <a class="learning-path-link feed-sidebar-link " href="/xiaowan-feed/">
477        <span class="learning-path-link__logo learning-path-link__logo--feed" aria-hidden="true"><i class="ph-duotone ph-chat-circle-dots"></i></span>
478        <span><strong>Feed</strong><small>小万动态</small></span>
479        <span class="feed-sidebar-link__count" data-feed-week-badge hidden></span>
480      </a>
481    </nav>
482
483    <div data-feed-tag-source hidden>
484          <span data-feed-tag-record
485            data-feed-published="2026-09-27T12:25:18.000+08:00"
486            data-feed-tags="ai|"
487            data-feed-tag-names="AI|"></span>
488          <span data-feed-tag-record
489            data-feed-published="2026-09-27T12:24:27.000+08:00"
490            data-feed-tags="system-design-2|hai-wai-qiu-zhi|"
491            data-feed-tag-names="system design|海外求职|"></span>
492          <span data-feed-tag-record
493            data-feed-published="2026-09-27T12:23:23.000+08:00"
494            data-feed-tags="ai|"
495            data-feed-tag-names="AI|"></span>
496          <span data-feed-tag-record
497            data-feed-published="2026-09-04T15:39:08.000+08:00"
498            data-feed-tags=""
499            data-feed-tag-names=""></span>
500          <span data-feed-tag-record
501            data-feed-published="2026-09-04T15:29:50.000+08:00"
502            data-feed-tags=""
503            data-feed-tag-names=""></span>
504          <span data-feed-tag-record
505            data-feed-published="2026-08-22T21:50:08.000+08:00"
506            data-feed-tags=""
507            data-feed-tag-names=""></span>
508          <span data-feed-tag-record
509            data-feed-published="2026-07-31T14:22:29.000+08:00"
510            data-feed-tags=""
511            data-feed-tag-names=""></span>
512          <span data-feed-tag-record
513            data-feed-published="2026-07-30T14:42:51.000+08:00"
514            data-feed-tags=""
515            data-feed-tag-names=""></span>
516          <span data-feed-tag-record
517            data-feed-published="2026-07-30T14:18:56.000+08:00"
518            data-feed-tags="system-design-2|"
519            data-feed-tag-names="system design|"></span>
520          <span data-feed-tag-record
521            data-feed-published="2026-07-30T01:33:27.000+08:00"
522            data-feed-tags="ai|"
523            data-feed-tag-names="AI|"></span>
524          <span data-feed-tag-record
525            data-feed-published="2026-07-30T00:29:28.000+08:00"
526            data-feed-tags="hai-wai-qiu-zhi|"
527            data-feed-tag-names="海外求职|"></span>
528    </div>
529
530    <nav class="learning-path-accordion" aria-label="选择学习路径">
531    <section class="learning-path-branch is-active is-expanded" data-path-branch>
532      <a class="learning-path-link is-active" href="/learning-path/" data-path-toggle aria-current="page" aria-expanded="true" aria-controls="path-content-data">
533        <span class="learning-path-link__logo learning-path-link__logo--data" aria-hidden="true"><i class="ph-duotone ph-database"></i></span>
534        <span>Data Engineering 学习路径</span>
535        <i class="ph-duotone ph-caret-down learning-path-link__caret" aria-hidden="true"></i>
536      </a>
537        <div class="learning-path-branch__content" id="path-content-data" data-path-content>
538          <div class="course-groups">
539              <section class="course-group-block" data-module="data-world-two-camps" data-module-name="OLTP vs OLAP">
540    <button class="course-group-toggle" type="button" data-module-toggle="data-world-two-camps" aria-expanded="true">
541      <span>OLTP vs OLAP</span>
542      <span><span class="module-count-badge is-empty" data-module-count>0</span><i class="group-caret" aria-hidden="true"></i></span>
543    </button>
544    <div class="lesson-list module-lesson-list" data-module-lessons="data-world-two-camps">
545        <a class="lesson-nav-item" href="/a-oltp-vs-olap-two-camps/" data-lesson-id="6a6d814351469b00017baa80" data-lesson-title="a. 两个阵营" data-lesson-summary="理解 OLTP 与 OLAP 的设计目标、典型查询、行列存储差异,以及分析系统为什么需要与业务数据库分开。" data-module-name="OLTP vs OLAP" data-course-tags="hash-order-010 ">
546          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
547          <span class="lesson-nav-item__copy">
548            <strong>a. 两个阵营</strong>
549            <small>6 分钟</small>
550          </span>
551            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
552        </a>
553    </div>
554  </section>
555              <section class="course-group-block" data-module="data-modeling" data-module-name="Data Modeling 数据建模">
556    <button class="course-group-toggle" type="button" data-module-toggle="data-modeling" aria-expanded="false">
557      <span>Data Modeling 数据建模</span>
558      <span><span class="module-count-badge is-empty" data-module-count>0</span><i class="group-caret" aria-hidden="true"></i></span>
559    </button>
560    <div class="lesson-list module-lesson-list" data-module-lessons="data-modeling" hidden>
561        <a class="lesson-nav-item" href="/a-fact-tables/" data-lesson-id="6a66d2f7df916c00013cbfaa" data-lesson-title="a. Fact Tables" data-lesson-summary="理解事实表的粒度、度量与三种典型类型:事务、周期快照和累积快照。" data-module-name="Data Modeling 数据建模" data-course-tags="hash-order-010 ">
562          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
563          <span class="lesson-nav-item__copy">
564            <strong>a. Fact Tables</strong>
565            <small>5 分钟</small>
566          </span>
567            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
568        </a>
569        <a class="lesson-nav-item" href="/b-dim-tables/" data-lesson-id="6a66d2f7df916c00013cbfb5" data-lesson-title="b. Dim Tables" data-lesson-summary="理解维度表与缓慢变化维度(SCD),掌握 Type 1、Type 2 和 Type 3 的差异。" data-module-name="Data Modeling 数据建模" data-course-tags="hash-order-020 ">
570          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
571          <span class="lesson-nav-item__copy">
572            <strong>b. Dim Tables</strong>
573            <small>3 分钟</small>
574          </span>
575            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
576        </a>
577        <a class="lesson-nav-item" href="/c-database-modeling/" data-lesson-id="6a66d2f9df916c00013cbfc0" data-lesson-title="c. Database Modeling" data-lesson-summary="理解 schema、事实表与维度表的分工,以及主外键关联和星型模型,建立数据仓库建模的基础。" data-module-name="Data Modeling 数据建模" data-course-tags="hash-order-030 ">
578          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
579          <span class="lesson-nav-item__copy">
580            <strong>c. Database Modeling</strong>
581            <small>4 分钟</small>
582          </span>
583            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
584        </a>
585        <a class="lesson-nav-item" href="/d-data-model-sop/" data-lesson-id="6aa7ed7a51469b00017bb12b" data-lesson-title="d. Data model 的 SOP" data-lesson-summary="用四步 SOP 提取业务要素、选择事实表、定义粒度与维度,并通过外卖平台案例练习完整的建模思路。" data-module-name="Data Modeling 数据建模" data-course-tags="hash-order-040 ">
586          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
587          <span class="lesson-nav-item__copy">
588            <strong>d. Data model 的 SOP</strong>
589            <small>8 分钟</small>
590          </span>
591            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
592        </a>
593    </div>
594  </section>
595              <section class="course-group-block" data-module="data-warehouse" data-module-name="Data Warehouse 数据仓库">
596    <button class="course-group-toggle" type="button" data-module-toggle="data-warehouse" aria-expanded="false">
597      <span>Data Warehouse 数据仓库</span>
598      <span><span class="module-count-badge is-empty" data-module-count>0</span><i class="group-caret" aria-hidden="true"></i></span>
599    </button>
600    <div class="lesson-list module-lesson-list" data-module-lessons="data-warehouse" hidden>
601        <a class="lesson-nav-item" href="/a-etl/" data-lesson-id="6a66d752df916c00013cbfd3" data-lesson-title="a. ETL" data-lesson-summary="从 Extraction、Transformation 到 Load,理解 ETL 的核心阶段、常见处理方式与 SCD 策略。" data-module-name="Data Warehouse 数据仓库" data-course-tags="hash-order-010 ">
602          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
603          <span class="lesson-nav-item__copy">
604            <strong>a. ETL</strong>
605            <small>5 分钟</small>
606          </span>
607            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
608        </a>
609        <a class="lesson-nav-item" href="/b-sql-database-and-nosql-database/" data-lesson-id="6a66d753df916c00013cbfde" data-lesson-title="b. SQL数据库与NoSQL数据库" data-lesson-summary="比较 SQL 与 NoSQL 数据库的适用场景,并理解 Redshift 的列式存储、distkey 与 sortkey。" data-module-name="Data Warehouse 数据仓库" data-course-tags="hash-order-020 ">
610          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
611          <span class="lesson-nav-item__copy">
612            <strong>b. SQL数据库与NoSQL数据库</strong>
613            <small>5 分钟</small>
614          </span>
615            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
616        </a>
617        <a class="lesson-nav-item" href="/c-data-warehouse-optimization/" data-lesson-id="6a66d753df916c00013cbfe9" data-lesson-title="c. 更多的数据仓库优化手段" data-lesson-summary="围绕压缩、分区、小文件、物化视图、宽表与 SQL 调优,系统掌握数据仓库性能优化。" data-module-name="Data Warehouse 数据仓库" data-course-tags="hash-order-030 ">
618          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
619          <span class="lesson-nav-item__copy">
620            <strong>c. 更多的数据仓库优化手段</strong>
621            <small>6 分钟</small>
622          </span>
623            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
624        </a>
625        <a class="lesson-nav-item" href="/de-faq-warehouse-cn/" data-lesson-id="6aafbc3351469b00017bb220" data-lesson-title="面试常考问题|数据仓库(国内)" data-lesson-summary="数据仓库面试常考问题,覆盖维度建模、ETL、SCD、分层与性能优化。" data-module-name="Data Warehouse 数据仓库" data-course-tags="hash-order-900 ">
626          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
627          <span class="lesson-nav-item__copy">
628            <strong>面试常考问题|数据仓库(国内)</strong>
629            <small>5 分钟</small>
630          </span>
631            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
632        </a>
633    </div>
634  </section>
635              <section class="course-group-block" data-module="data-lake-lakehouse" data-module-name="Data Lake &amp; Lakehouse">
636    <button class="course-group-toggle" type="button" data-module-toggle="data-lake-lakehouse" aria-expanded="false">
637      <span>Data Lake &amp; Lakehouse</span>
638      <span><span class="module-count-badge is-empty" data-module-count>0</span><i class="group-caret" aria-hidden="true"></i></span>
639    </button>
640    <div class="lesson-list module-lesson-list" data-module-lessons="data-lake-lakehouse" hidden>
641        <a class="lesson-nav-item" href="/c-lakehouse-hu-cang/" data-lesson-id="6a5f3ca7c4edad0001d48356" data-lesson-title="c. Lakehouse 湖仓" data-lesson-summary="本文介绍 Lakehouse 的核心概念,以及它如何结合 Data Lake 与 Data Warehouse 的能力。后半部分将通过 Delta Lake 实验,演示事务日志、数据版本、Time Travel 和 RESTORE 的实际操作。" data-module-name="Data Lake &amp; Lakehouse" data-course-tags="hash-order-030 ">
642          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
643          <span class="lesson-nav-item__copy">
644            <strong>c. Lakehouse 湖仓</strong>
645            <small>40 分钟</small>
646          </span>
647            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
648        </a>
649        <a class="lesson-nav-item" href="/a-delta-lake-from-files-to-table/" data-lesson-id="6a66dd4ddf916c00013cbff5" data-lesson-title="a. 数据湖基础" data-lesson-summary="理解 Delta Lake 如何通过 _delta_log 为 Parquet 文件补上事务、Schema 管理、Time Travel 与 DML 能力。" data-module-name="Data Lake &amp; Lakehouse" data-course-tags="hash-order-010 ">
650          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
651          <span class="lesson-nav-item__copy">
652            <strong>a. 数据湖基础</strong>
653            <small>7 分钟</small>
654          </span>
655            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
656        </a>
657        <a class="lesson-nav-item" href="/b-data-lake-data-swamp/" data-lesson-id="6a6c447a51469b00017baa20" data-lesson-title="b. 数据湖的问题(Data Swamp)" data-lesson-summary="理解数据湖为什么会因缺乏事务、Schema 管理、小文件治理与数据治理能力而退化为 Data Swamp。" data-module-name="Data Lake &amp; Lakehouse" data-course-tags="hash-order-025 ">
658          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
659          <span class="lesson-nav-item__copy">
660            <strong>b. 数据湖的问题(Data Swamp)</strong>
661            <small>9 分钟</small>
662          </span>
663            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
664        </a>
665        <a class="lesson-nav-item" href="/f-medallion-jia-gou/" data-lesson-id="6a6c552851469b00017baa2f" data-lesson-title="e. Medallion 架构" data-lesson-summary="理解湖仓中的 Bronze、Silver、Gold 分层,以及数据重算、排查和责任边界为什么需要 Medallion 架构。" data-module-name="Data Lake &amp; Lakehouse" data-course-tags="hash-order-060 ">
666          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
667          <span class="lesson-nav-item__copy">
668            <strong>e. Medallion 架构</strong>
669            <small>5 分钟</small>
670          </span>
671            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
672        </a>
673        <a class="lesson-nav-item" href="/g-metadata-catalog/" data-lesson-id="6a6c628351469b00017baa52" data-lesson-title="f. 元数据 &amp; Catalog" data-lesson-summary="理解 Catalog 如何连接存储、计算与表格式,掌握主流 Catalog 的能力差异和选型思路。" data-module-name="Data Lake &amp; Lakehouse" data-course-tags="hash-order-070 ">
674          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
675          <span class="lesson-nav-item__copy">
676            <strong>f. 元数据 &amp; Catalog</strong>
677            <small>10 分钟</small>
678          </span>
679            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
680        </a>
681        <a class="lesson-nav-item" href="/g-lakehouse-optimization/" data-lesson-id="6a6d71d551469b00017baa71" data-lesson-title="g. Lakehouse 优化" data-lesson-summary="掌握 Data Skipping、Z-Order、Liquid Clustering、OPTIMIZE、VACUUM 等湖仓优化手段及慢查询排查顺序。" data-module-name="Data Lake &amp; Lakehouse" data-course-tags="hash-order-080 ">
682          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
683          <span class="lesson-nav-item__copy">
684            <strong>g. Lakehouse 优化</strong>
685            <small>13 分钟</small>
686          </span>
687            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
688        </a>
689        <a class="lesson-nav-item" href="/de-faq-lakehouse-overseas/" data-lesson-id="6aafbc3451469b00017bb22d" data-lesson-title="面试常考问题|Lakehouse(海外)" data-lesson-summary="按十个主题整理湖仓面试问题,覆盖开放表格式、ETL、性能、治理与场景题。" data-module-name="Data Lake &amp; Lakehouse" data-course-tags="hash-order-900 ">
690          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
691          <span class="lesson-nav-item__copy">
692            <strong>面试常考问题|Lakehouse(海外)</strong>
693            <small>6 分钟</small>
694          </span>
695            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
696        </a>
697    </div>
698  </section>
699              <section class="course-group-block" data-module="spark" data-module-name="Spark 分布式计算引擎">
700    <button class="course-group-toggle" type="button" data-module-toggle="spark" aria-expanded="false">
701      <span>Spark 分布式计算引擎</span>
702      <span><span class="module-count-badge is-empty" data-module-count>0</span><i class="group-caret" aria-hidden="true"></i></span>
703    </button>
704    <div class="lesson-list module-lesson-list" data-module-lessons="spark" hidden>
705        <a class="lesson-nav-item" href="/a-spark-he-xin-gai-nian/" data-lesson-id="6a585696c4edad0001d4801e" data-lesson-title="a. Spark 核心概念" data-lesson-summary="从单机瓶颈出发,理解 Spark 的分布式计算模型,掌握 Driver、Executor、Partition、DataFrame、惰性求值以及 Transformation 与 Action。" data-module-name="Spark 分布式计算引擎" data-course-tags="hash-order-010 ">
706          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
707          <span class="lesson-nav-item__copy">
708            <strong>a. Spark 核心概念</strong>
709            <small>15 分钟</small>
710          </span>
711            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
712        </a>
713        <a class="lesson-nav-item" href="/b-zhi-xing-yuan-li/" data-lesson-id="6a5c3c08c4edad0001d4819c" data-lesson-title="b. 执行原理" data-lesson-summary="从一次 Action 出发,拆解 Spark 如何构建 DAG、识别宽窄依赖、按 Shuffle 划分 Stage,并把 Partition 转换为可并行执行的 Task。" data-module-name="Spark 分布式计算引擎" data-course-tags="hash-order-020 ">
714          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
715          <span class="lesson-nav-item__copy">
716            <strong>b. 执行原理</strong>
717            <small>13 分钟</small>
718          </span>
719            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
720        </a>
721        <a class="lesson-nav-item" href="/c-spark-sql-you-hua-qi/" data-lesson-id="6a5c3d83c4edad0001d481b4" data-lesson-title="c. Spark SQL &amp; 优化器" data-lesson-summary="理解 DataFrame 与 SQL 如何经过 Catalyst 生成物理计划,并掌握谓词下推、Join 策略选择及 AQE 运行时优化的核心机制。" data-module-name="Spark 分布式计算引擎" data-course-tags="hash-order-030 ">
722          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
723          <span class="lesson-nav-item__copy">
724            <strong>c. Spark SQL &amp; 优化器</strong>
725            <small>18 分钟</small>
726          </span>
727            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
728        </a>
729        <a class="lesson-nav-item" href="/d-zuo-ye-ceng-diao-you/" data-lesson-id="6a5c431fc4edad0001d481c8" data-lesson-title="d. 作业层调优" data-lesson-summary="以 Spark UI 为入口,系统排查数据倾斜、Shuffle 膨胀、小文件、Spill 与 OOM,并通过分区、广播和缓存优化慢作业。" data-module-name="Spark 分布式计算引擎" data-course-tags="hash-order-040 ">
730          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
731          <span class="lesson-nav-item__copy">
732            <strong>d. 作业层调优</strong>
733            <small>10 分钟</small>
734          </span>
735            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
736        </a>
737        <a class="lesson-nav-item" href="/e-bu-shu-yu-zi-yuan/" data-lesson-id="6a5c43adc4edad0001d481dd" data-lesson-title="e. 部署与资源" data-lesson-summary="理解 Spark 在 YARN 与 Kubernetes 上的部署方式,学会配置 Executor 资源,并掌握 Structured Streaming 的微批、Watermark 与 Checkpoint。" data-module-name="Spark 分布式计算引擎" data-course-tags="hash-order-050 ">
738          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
739          <span class="lesson-nav-item__copy">
740            <strong>e. 部署与资源</strong>
741            <small>7 分钟</small>
742          </span>
743            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
744        </a>
745        <a class="lesson-nav-item" href="/de-faq-spark-english/" data-lesson-id="6aafbc0751469b00017bb1ec" data-lesson-title="面试常考问题|Spark(英文,含部分答案)" data-lesson-summary="英文 Spark 问题清单,以及原文已有的基础与架构问答。" data-module-name="Spark 分布式计算引擎" data-course-tags="hash-order-900 ">
746          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
747          <span class="lesson-nav-item__copy">
748            <strong>面试常考问题|Spark(英文,含部分答案)</strong>
749            <small>9 分钟</small>
750          </span>
751            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
752        </a>
753        <a class="lesson-nav-item" href="/de-faq-spark-cn/" data-lesson-id="6aafbc3051469b00017bb1f9" data-lesson-title="面试常考问题|Spark(国内,60 题)" data-lesson-summary="60 道 Spark 面试常考问题,保留原有难度与标签,按难度分段复习。" data-module-name="Spark 分布式计算引擎" data-course-tags="hash-order-910 ">
754          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
755          <span class="lesson-nav-item__copy">
756            <strong>面试常考问题|Spark(国内,60 题)</strong>
757            <small>7 分钟</small>
758          </span>
759            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
760        </a>
761        <a class="lesson-nav-item" href="/de-faq-spark-sql-cn/" data-lesson-id="6aafbc3151469b00017bb206" data-lesson-title="面试常考问题|Spark SQL(国内)" data-lesson-summary="Spark SQL 面试常考问题,覆盖 DataFrame、Catalyst、Hive 集成与查询调优。" data-module-name="Spark 分布式计算引擎" data-course-tags="hash-order-920 ">
762          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
763          <span class="lesson-nav-item__copy">
764            <strong>面试常考问题|Spark SQL(国内)</strong>
765            <small>5 分钟</small>
766          </span>
767            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
768        </a>
769        <a class="lesson-nav-item" href="/de-faq-pyspark-cn/" data-lesson-id="6aafbc3251469b00017bb213" data-lesson-title="面试常考问题|PySpark(国内)" data-lesson-summary="PySpark 面试常考问题,覆盖 SparkSession、RDD、DataFrame 与性能优化。" data-module-name="Spark 分布式计算引擎" data-course-tags="hash-order-930 ">
770          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
771          <span class="lesson-nav-item__copy">
772            <strong>面试常考问题|PySpark(国内)</strong>
773            <small>4 分钟</small>
774          </span>
775            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
776        </a>
777    </div>
778  </section>
779              <section class="course-group-block" data-module="cloud-platform" data-module-name="云平台">
780    <button class="course-group-toggle" type="button" data-module-toggle="cloud-platform" aria-expanded="false">
781      <span>云平台</span>
782      <span><span class="module-count-badge is-empty" data-module-count>0</span><i class="group-caret" aria-hidden="true"></i></span>
783    </button>
784    <div class="lesson-list module-lesson-list" data-module-lessons="cloud-platform" hidden>
785        <a class="lesson-nav-item" href="/dbt/" data-lesson-id="6aa8d21d51469b00017bb154" data-lesson-title="dbt" data-lesson-summary="准备 Data Engineer 或者 Analytics Engineer 面试,dbt 到底需要学什么,学到什么程度?" data-module-name="云平台" data-course-tags="hash-order-010 ">
786          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
787          <span class="lesson-nav-item__copy">
788            <strong>dbt</strong>
789            <small>5 分钟</small>
790          </span>
791            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
792        </a>
793    </div>
794  </section>
795              <section class="course-group-block" data-module="data-governance" data-module-name="Data Governance 数据治理">
796    <button class="course-group-toggle" type="button" data-module-toggle="data-governance" aria-expanded="false">
797      <span>Data Governance 数据治理</span>
798      <span><span class="module-count-badge is-empty" data-module-count>0</span><i class="group-caret" aria-hidden="true"></i></span>
799    </button>
800    <div class="lesson-list module-lesson-list" data-module-lessons="data-governance" hidden>
801        <a class="lesson-nav-item" href="/a-what-is-data-governance/" data-lesson-id="6a99288051469b00017baf1a" data-lesson-title="a. Data Governance 全景:从数据资产到可信数据产品" data-lesson-summary="从定义出发,理解 Data Governance、Data Management 与 Data Quality 的区别,并掌握数据治理的六个核心目标。" data-module-name="Data Governance 数据治理" data-course-tags="hash-order-010 ">
802          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
803          <span class="lesson-nav-item__copy">
804            <strong>a. Data Governance 全景:从数据资产到可信数据产品</strong>
805            <small>52 分钟</small>
806          </span>
807            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
808        </a>
809    </div>
810  </section>
811              <section class="course-group-block" data-module="system-design" data-module-name="System Design 系统设计">
812    <button class="course-group-toggle" type="button" data-module-toggle="system-design" aria-expanded="false">
813      <span>System Design 系统设计</span>
814      <span><span class="module-count-badge is-empty" data-module-count>0</span><i class="group-caret" aria-hidden="true"></i></span>
815    </button>
816    <div class="lesson-list module-lesson-list" data-module-lessons="system-design" hidden>
817        <a class="lesson-nav-item" href="/schema-bian-geng-data-engineerying-gai-zen-yao-zuo/" data-lesson-id="6a60ad9b4875ab0001917c8e" data-lesson-title="Schema 变更-Data Engineer应该怎么做" data-lesson-summary="整理了schema变更的话,Data Engineer可以从几个方面去考虑,附Netflix、paypal Databricks Uber Airbnb等公司的做法,也粘贴了原文。" data-module-name="System Design 系统设计" data-course-tags="hash-featured-order-010 ">
818          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
819          <span class="lesson-nav-item__copy">
820            <strong>Schema 变更-Data Engineer应该怎么做</strong>
821            <small>20 分钟</small>
822          </span>
823            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
824        </a>
825        <a class="lesson-nav-item" href="/a-data-storage-s3/" data-lesson-id="6a66e326df916c00013cc003" data-lesson-title="a. 数据存储:S3" data-lesson-summary="理解 S3 作为数据湖存储的核心定位、容量单位、存储层级与成本权衡。" data-module-name="System Design 系统设计" data-course-tags="hash-order-010 ">
826          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
827          <span class="lesson-nav-item__copy">
828            <strong>a. 数据存储:S3</strong>
829            <small>3 分钟</small>
830          </span>
831            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
832        </a>
833        <a class="lesson-nav-item" href="/b-compute-resources/" data-lesson-id="6a66e327df916c00013cc00e" data-lesson-title="b. 计算资源(Compute)" data-lesson-summary="理解计算资源、CPU 负载、EC2、垂直与水平扩展、缓存以及 ETL 工具的底层关系。" data-module-name="System Design 系统设计" data-course-tags="hash-order-020 ">
834          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
835          <span class="lesson-nav-item__copy">
836            <strong>b. 计算资源(Compute)</strong>
837            <small>6 分钟</small>
838          </span>
839            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
840        </a>
841        <a class="lesson-nav-item" href="/d-system-design-case-study/" data-lesson-id="6a66e328df916c00013cc019" data-lesson-title="d. 系统设计——案例研究" data-lesson-summary="以 Robinhood 数据湖为案例,拆解摄入、存储、处理、查询、验证与用户交互层的架构取舍。" data-module-name="System Design 系统设计" data-course-tags="hash-order-040 ">
842          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
843          <span class="lesson-nav-item__copy">
844            <strong>d. 系统设计——案例研究</strong>
845            <small>9 分钟</small>
846          </span>
847            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
848        </a>
849        <a class="lesson-nav-item" href="/e-system-design-questions/" data-lesson-id="6a66e329df916c00013cc024" data-lesson-title="e. 系统设计问题(questions)" data-lesson-summary="通过 TinyURL 与音乐推荐两个题目,练习数据工程系统设计的需求分析与架构表达。" data-module-name="System Design 系统设计" data-course-tags="hash-order-050 ">
850          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
851          <span class="lesson-nav-item__copy">
852            <strong>e. 系统设计问题(questions)</strong>
853            <small>1 分钟</small>
854          </span>
855            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
856        </a>
857        <a class="lesson-nav-item" href="/f-system-design-interview-seven-step-framework/" data-lesson-id="6a66e32adf916c00013cc02f" data-lesson-title="f. 系统设计面试的 7 步框架" data-lesson-summary="掌握数据工程系统设计面试的七步框架:需求、规模、架构、模型、流水线、深挖与 SLA。" data-module-name="System Design 系统设计" data-course-tags="hash-order-060 ">
858          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
859          <span class="lesson-nav-item__copy">
860            <strong>f. 系统设计面试的 7 步框架</strong>
861            <small>6 分钟</small>
862          </span>
863            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
864        </a>
865    </div>
866  </section>
867              <section class="course-group-block" data-module="sql" data-module-name="SQL">
868    <button class="course-group-toggle" type="button" data-module-toggle="sql" aria-expanded="false">
869      <span>SQL</span>
870      <span><span class="module-count-badge is-empty" data-module-count>0</span><i class="group-caret" aria-hidden="true"></i></span>
871    </button>
872    <div class="lesson-list module-lesson-list" data-module-lessons="sql" hidden>
873        <a class="lesson-nav-item" href="/a-calculate-metrics-with-sql/" data-lesson-id="6a66e32cdf916c00013cc03a" data-lesson-title="a. 通过SQL计算指标" data-lesson-summary="通过六类业务题与输入输出练习,掌握聚合、连接、窗口函数、日期计算和 SQL 指标设计。" data-module-name="SQL" data-course-tags="hash-order-010 ">
874          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
875          <span class="lesson-nav-item__copy">
876            <strong>a. 通过SQL计算指标</strong>
877            <small>20 分钟</small>
878          </span>
879            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
880        </a>
881        <a class="lesson-nav-item" href="/b-create-tables/" data-lesson-id="6a66e32edf916c00013cc045" data-lesson-title="b. 表创建" data-lesson-summary="理解 Redshift 建表语句中的数据类型、DISTKEY 与 SORTKEY,以及它们对存储和查询的影响。" data-module-name="SQL" data-course-tags="hash-order-020 ">
882          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
883          <span class="lesson-nav-item__copy">
884            <strong>b. 表创建</strong>
885            <small>1 分钟</small>
886          </span>
887            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
888        </a>
889    </div>
890  </section>
891              <section class="course-group-block" data-module="product-sense" data-module-name="Product Sense 产品 &amp; 业务">
892    <button class="course-group-toggle" type="button" data-module-toggle="product-sense" aria-expanded="false">
893      <span>Product Sense 产品 &amp; 业务</span>
894      <span><span class="module-count-badge is-empty" data-module-count>0</span><i class="group-caret" aria-hidden="true"></i></span>
895    </button>
896    <div class="lesson-list module-lesson-list" data-module-lessons="product-sense" hidden>
897        <a class="lesson-nav-item" href="/a-tlq-product-sense/" data-lesson-id="6a66ecdfdf916c00013cc05a" data-lesson-title="a. TLQ" data-lesson-summary="通过 TLQ 工业设备案例,理解近实时数据架构如何在可靠性、更新频率与成本之间做出权衡。" data-module-name="Product Sense 产品 &amp; 业务" data-course-tags="hash-order-010 ">
898          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
899          <span class="lesson-nav-item__copy">
900            <strong>a. TLQ</strong>
901            <small>3 分钟</small>
902          </span>
903            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
904        </a>
905        <a class="lesson-nav-item" href="/b-auto-product-sense/" data-lesson-id="6a66ecdfdf916c00013cc065" data-lesson-title="b. Auto" data-lesson-summary="通过 Auto 全球化案例,理解数据管道自动化、数据质量校验与可扩展设计如何创造业务价值。" data-module-name="Product Sense 产品 &amp; 业务" data-course-tags="hash-order-020 ">
906          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
907          <span class="lesson-nav-item__copy">
908            <strong>b. Auto</strong>
909            <small>2 分钟</small>
910          </span>
911            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
912        </a>
913        <a class="lesson-nav-item" href="/c-caching-scalability-optimization/" data-lesson-id="6a66ece0df916c00013cc070" data-lesson-title="c. 缓存(扩展性优化)" data-lesson-summary="通过限量商品流量峰值案例,理解 Redis 缓存如何降低成本、提升弹性与用户体验。" data-module-name="Product Sense 产品 &amp; 业务" data-course-tags="hash-order-030 ">
914          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
915          <span class="lesson-nav-item__copy">
916            <strong>c. 缓存(扩展性优化)</strong>
917            <small>1 分钟</small>
918          </span>
919            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
920        </a>
921        <a class="lesson-nav-item" href="/d-machine-learning-real-time-application/" data-lesson-id="6a66ece0df916c00013cc07b" data-lesson-title="d. 机器学习(实时化应用)" data-lesson-summary="通过餐饮机器视觉案例,理解 S3、Airflow 与 SageMaker 如何支撑机器学习系统的规模化落地。" data-module-name="Product Sense 产品 &amp; 业务" data-course-tags="hash-order-040 ">
922          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
923          <span class="lesson-nav-item__copy">
924            <strong>d. 机器学习(实时化应用)</strong>
925            <small>2 分钟</small>
926          </span>
927            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
928        </a>
929    </div>
930  </section>
931              <section class="course-group-block" data-module="projects" data-module-name="实战项目">
932    <button class="course-group-toggle" type="button" data-module-toggle="projects" aria-expanded="false">
933      <span>实战项目</span>
934      <span><span class="module-count-badge is-empty" data-module-count>0</span><i class="group-caret" aria-hidden="true"></i></span>
935    </button>
936    <div class="lesson-list module-lesson-list" data-module-lessons="projects" hidden>
937          <p class="module-empty">尚未发布课程</p>
938    </div>
939  </section>
940              <section class="course-group-block" data-module="de-common-questions" data-module-name="面试常考问题 · 补充专题">
941    <button class="course-group-toggle" type="button" data-module-toggle="de-common-questions" aria-expanded="false">
942      <span>面试常考问题 · 补充专题</span>
943      <span><span class="module-count-badge is-empty" data-module-count>0</span><i class="group-caret" aria-hidden="true"></i></span>
944    </button>
945    <div class="lesson-list module-lesson-list" data-module-lessons="de-common-questions" hidden>
946        <a class="lesson-nav-item" href="/de-faq-hadoop-cn/" data-lesson-id="6aafbc3651469b00017bb23a" data-lesson-title="面试常考问题|Hadoop(国内)" data-lesson-summary="Hadoop 面试常考问题,保留原有难度标签,覆盖核心组件、存储、调度与容错。" data-module-name="面试常考问题 · 补充专题" data-course-tags="hash-order-010 ">
947          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
948          <span class="lesson-nav-item__copy">
949            <strong>面试常考问题|Hadoop(国内)</strong>
950            <small>5 分钟</small>
951          </span>
952            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
953        </a>
954        <a class="lesson-nav-item" href="/de-faq-flink-cn/" data-lesson-id="6aafbc3751469b00017bb247" data-lesson-title="面试常考问题|Flink(国内)" data-lesson-summary="Flink 面试常考问题,覆盖流处理、状态、窗口、Watermark 与容错。" data-module-name="面试常考问题 · 补充专题" data-course-tags="hash-order-020 ">
955          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
956          <span class="lesson-nav-item__copy">
957            <strong>面试常考问题|Flink(国内)</strong>
958            <small>7 分钟</small>
959          </span>
960            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
961        </a>
962        <a class="lesson-nav-item" href="/de-faq-kafka-cn/" data-lesson-id="6aafbc3851469b00017bb254" data-lesson-title="面试常考问题|Kafka(国内)" data-lesson-summary="Kafka 面试常考问题,覆盖 Topic、Partition、Consumer Group、可靠性与性能。" data-module-name="面试常考问题 · 补充专题" data-course-tags="hash-order-030 ">
963          <span class="lesson-nav-item__icon" aria-hidden="true"></span>
964          <span class="lesson-nav-item__copy">
965            <strong>面试常考问题|Kafka(国内)</strong>
966            <small>6 分钟</small>
967          </span>
968            <span class="access-state access-state--locked" aria-label="需要会员权限"></span>
969        </a>
970    </div>
971  </section>
972          </div>
973        </div>
974    </section>
975
976    <section class="ar-side" aria-label="AI + Data 学习路径">
977  <nav id="nav" aria-label="AI + Data 两条学习路径">
978    <a class="hub-parent" href="/ai-data-learning-path/"><span class="path-icon"><i class="ph-duotone ph-chart-line-up" aria-hidden="true"></i></span><span>AI + Data 学习路径<small>两条职业学习路径</small></span><i class="ph-duotone ph-caret-down" aria-hidden="true"></i></a>
979    <div class="hub-children">
980      <a class="path-button" href="/ai-data-learning-path/#ai-data"><span class="path-icon"><i class="ph-duotone ph-flow-arrow" aria-hidden="true"></i></span><span class="child-path-label">AI for Data Engineers<small>数据工程师的 AI 进阶 · 10 章</small></span><i class="ph-duotone ph-caret-right caret" aria-hidden="true"></i></a>
981      <a class="path-button" href="/ai-data-learning-path/#ai"><span class="path-icon"><i class="ph-duotone ph-cpu" aria-hidden="true"></i></span><span class="child-path-label">AI Engineer<small>AI 应用与 Agent 工程 · 17 章</small></span><i class="ph-duotone ph-caret-right caret" aria-hidden="true"></i></a>
982    </div>
983  </nav>
984</section>
985
986    <section class="learning-path-branch " data-path-branch>
987      <a class="learning-path-link " href="/ai-learning-path/" data-path-toggle>
988        <span class="learning-path-link__logo learning-path-link__logo--ai" aria-hidden="true"><i class="ph-duotone ph-brain"></i></span>
989        <span>AI 学习路径</span>
990        <i class="ph-duotone ph-caret-down learning-path-link__caret" aria-hidden="true"></i>
991      </a>
992    </section>
993    </nav>
994
995    <div class="sidebar-section-divider" role="separator" aria-label="职业实战资源">
996      <span>职业实战</span>
997    </div>
998
999    <nav class="sidebar-resource-links" aria-label="职业实战资源">
1000    <a class="sidebar-resource-link sidebar-resource-link--services" href="/services/?from=academy-sidebar" target="_blank" rel="noopener noreferrer">
1001      <span class="learning-path-link__logo learning-path-link__logo--services" aria-hidden="true"><i class="ph-duotone ph-handshake"></i></span>
1002      <span>
1003        <strong>求职服务</strong>
1004        <small>Mock Interview · DE 项目</small>
1005      </span>
1006      <span class="sidebar-resource-link__arrow" aria-hidden="true"><i class="ph-duotone ph-arrow-up-right"></i></span>
1007    </a>
1008
1009    <a class="sidebar-resource-link sidebar-resource-link--featured " href="/featured-articles/?from=academy-sidebar">
1010      <span class="learning-path-link__logo learning-path-link__logo--featured" aria-hidden="true"><i class="ph-duotone ph-book-open-text"></i></span>
1011      <span>
1012        <strong>深度精选</strong>
1013        <small>值得反复阅读的好文章</small>
1014      </span>
1015        <span class="sidebar-resource-link__count">5</span>
1016    </a>
1017
1018    <a class="sidebar-resource-link sidebar-resource-link--projects " href="/projects/?from=academy-sidebar">
1019      <span class="learning-path-link__logo learning-path-link__logo--projects" aria-hidden="true"><i class="ph-duotone ph-kanban"></i></span>
1020      <span>
1021        <strong>实战项目库</strong>
1022        <small>把知识变成作品</small>
1023      </span>
1024        <span class="sidebar-resource-link__count">7</span>
1025    </a>
1026
1027    <a class="sidebar-resource-link sidebar-resource-link--interviews " href="/interview-questions/?from=academy-sidebar">
1028      <span class="learning-path-link__logo learning-path-link__logo--interviews" aria-hidden="true"><i class="ph-duotone ph-briefcase"></i></span>
1029      <span>
1030        <strong>DE 面试题库</strong>
1031        <small>国内与海外公司真题</small>
1032      </span>
1033        <span class="sidebar-resource-link__count">54</span>
1034    </a>
1035
1036
1037    <a class="sidebar-resource-link sidebar-resource-link--skills " href="/skills/?from=academy-sidebar">
1038      <span class="learning-path-link__logo learning-path-link__logo--skills" aria-hidden="true"><i class="ph-duotone ph-toolbox"></i></span>
1039      <span>
1040        <strong>Skills 工具箱</strong>
1041        <small>可安装的 AI 求职工具</small>
1042      </span>
1043        <span class="sidebar-resource-link__count">1</span>
1044    </a>
1045    </nav>
1046  </div>
1047
1048  <button class="invite-card" type="button" data-portal="signup">
1049    <span class="invite-card__gift" aria-hidden="true">✦</span>
1050    <span><strong>注册免费会员</strong><small>保存课程并获取学习更新</small></span>
1051    <span aria-hidden="true">→</span>
1052  </button>
1053</aside>
1054
1055  <main class="page-main lesson-main" id="academy-main">
1056    <a class="back-link" href="/learning-path/"><span aria-hidden="true">←</span> 返回课程目录</a>
1057
1058    <article class="lesson-article post tag-knowledge-points tag-roadmap tag-data-engineering-foundations tag-hash-order-011-2 post-access-public" data-lesson-article data-lesson-id="68fa2955512f5e0001d71394">
1059      <header class="lesson-header" id="lesson-top">
1060        <span class="eyebrow">数仓知识点</span>
1061        <h1>数据工程入门路线图(必看)</h1>
1062        <p>『文字版 』:写在前面:本次文章主要分成三个部分【国内市场、海外市场、两者对比】</p>
1063        <div class="lesson-meta">
1064          <span>◷ 23 分钟</span>
1065          <span>● 中级</span>
1066          <span>♛ 免费课程</span>
1067        </div>
1068      </header>
1069
1070        <figure class="lesson-feature-image ">
1071          <img srcset="/content/images/size/w320/2025/12/real_pixels_jpg-by-RohithDezinr.jpeg 320w, /content/images/size/w640/2025/12/real_pixels_jpg-by-RohithDezinr.jpeg 640w, /content/images/size/w960/2025/12/real_pixels_jpg-by-RohithDezinr.jpeg 960w, /content/images/size/w1600/2025/12/real_pixels_jpg-by-RohithDezinr.jpeg 1600w" sizes="(max-width: 820px) calc(100vw - 32px), (max-width: 1240px) 720px, 880px" src="/content/images/size/w960/2025/12/real_pixels_jpg-by-RohithDezinr.jpeg" alt="数据工程入门路线图(必看)">
1072          
1073        </figure>
1074
1075      <div class="lesson-body gh-content" data-lesson-content><p>写在前面:本次文章主要分成三个部分【国内市场、海外市场、两者对比】,右侧目录的字体有点小,已经在修改css,但是目前还没找到合适的把他改掉念</p><h2 id="%E6%95%B0%E6%8D%AE%E5%B7%A5%E7%A8%8B%E5%85%A5%E9%97%A8%E8%B7%AF%E7%BA%BF%E5%9B%BE%EF%BC%88%E5%9B%BD%E5%86%85%E5%B8%82%E5%9C%BA%EF%BC%89"><strong>数据工程入门路线图(国内市场)</strong></h2><h3 id="%E5%BC%95%E8%A8%80"><strong>引言</strong></h3><p>在国内市场,数据工程岗位更强调<strong>理论与原理</strong>:要理解数据仓库的分层逻辑、大数据组件的协同关系,以及稳定、高效的数据处理流程。</p><p>相比云平台操作,企业更看重你是否理解<strong>Hadoop 生态体系的原理</strong>、能否设计清晰的数仓结构,以及是否能分析系统瓶颈。</p><hr><h3 id="%E5%85%88%E8%81%8A%E7%AD%96%E7%95%A5%EF%BC%9A%E5%88%AB%E8%BF%B7%E4%BF%A1%E2%80%9C%E5%88%B7-leetcode-%E6%8B%BF-offer%E2%80%9D"><strong>先聊策略:别迷信“刷 LeetCode 拿 Offer”</strong></h3><p>数据工程岗位并不是算法驱动的工作。国内面试更关心:</p><blockquote>你是否理解一条数据从采集到分析的全流程? 你能否解释清楚一个分布式任务是如何运行的? 出现延迟、倾斜或数据错误时,你怎么定位问题?</blockquote><p><a href="https://xiaowantree.com/">xiaowantree.com</a> 里面会定期更新优质文章,大家可以注册免费或者付费观看,如果需要了解产品,请点击这里,助力您的offer之路。</p><hr><h3 id="%E8%A6%81%E6%8B%BF%E5%88%B0%E6%95%B0%E6%8D%AE%E5%B7%A5%E7%A8%8B-offer%EF%BC%8C%E4%BD%A0%E9%9C%80%E8%A6%81%E8%BF%99%E4%BA%9B"><strong>要拿到数据工程 Offer,你需要这些</strong></h3><ul><li><strong>SQL 与 Hive 基础功</strong>:能熟练写多表 JOIN、窗口函数、分区分桶查询。</li><li><strong>分布式计算原理:</strong><ul><li><strong>Hadoop 基础</strong>:HDFS 存储原理(块、复制、NameNode、DataNode),MapReduce 执行过程(Map/Shuffle/Reduce 流ç¨
1075‹ï¼‰ã€‚</li><li><strong>Spark(离线批处理)</strong>:RDD → DataFrame → Catalyst 优化器 → Tungsten 执行引擎。理解 Shuffle、Skew、AQE、自适应执行。</li><li><strong>Flink(实时流处理)</strong>:事件时间与处理时间、窗口机制、水位线(Watermark)、状态管理、Checkpoint、反压机制。</li></ul></li><li><strong>调度与依赖管理</strong>:Airflow、DolphinScheduler 原理,DAG、幂等、任务重跑与数据回填。</li><li><strong>数据建模与质量治理</strong>:ODS→DWD→DWS→ADS 四层建模思想;数据准确性、完整性、一致性检查。</li></ul><hr><h3 id="%E4%B8%80%E3%80%81%E5%AD%A6%E4%B9%A0-sql%EF%BC%9A%E6%95%B0%E6%8D%AE%E5%B7%A5%E7%A8%8B%E7%9A%84%E2%80%9C%E6%AF%8D%E8%AF%AD%E2%80%9D"><strong>一、学习 SQL:数据工程的“母语”</strong></h3><h4 id="1-%E6%8E%A8%E8%8D%90%E5%AD%A6%E4%B9%A0%E8%B5%84%E6%BA%90">1. 推荐学习资源</h4><ul><li><a href="https://leetcode.com/?ref=xiaowantree.com">Leetcode</a>: 侧重sql题目的练习</li><li><a href="https://datalemur.com/?ref=xiaowantree.com">DataLemur</a>:侧重面试级SQL题,贴近实战</li><li><a href="https://www.stratascratch.com/?ref=xiaowantree.com">StrataScratch</a>:含真实业务场景的SQL练习</li></ul><h4 id="2-%E5%BF%85%E9%A1%BB%E6%8E%8C%E6%8F%A1%E7%9A%84%E6%A0%B8%E5%BF%83%E7%9F%A5%E8%AF%86%E7%82%B9">2. 必须掌握的核心知识点</h4><h5 id="%EF%BC%881%EF%BC%89%E5%9F%BA%E7%A1%80%E6%93%8D%E4%BD%9C%E4%B8%8E%E8%81%9A%E5%90%88">(1)基础操作与聚合</h5><ul><li>JOIN类型:重点掌握INNER JOIN、LEFT JOIN、FULL OUTER JOIN,<strong>几乎不用RIGHT JOIN</strong></li><li>GROUP BY聚合:明确COUNT与COUNT(DISTINCT)的区别(分布式环境中COUNT(DISTINCT)速度慢)</li></ul><p>基数缩减:将高基数维度转为低基数,例如按年龄分桶:</p><pre><code class="language-sql">SELECT
1076  CASE WHEN age &gt; 30 THEN 'old' ELSE 'young' END as age_bucket,
1077  COUNT(1)
1078FROM users
1079GROUP BY 1
1080
1081</code></pre><p>CASE WHEN+聚合:例如统计过期订单数:</p><pre><code class="language-sql">COUNT(CASE WHEN status = 'expired' THEN order_id END)
1082
1083</code></pre><h5 id="%EF%BC%882%EF%BC%89%E6%9F%A5%E8%AF%A2%E7%BB%93%E6%9E%84%E5%AF%B9%E6%AF%94%EF%BC%9Acte%E5%AD%90%E6%9F%A5%E8%AF%A2%E8%A7%86%E5%9B%BE%E4%B8%B4%E6%97%B6%E8%A1%A8">(2)查询结构对比:CTE/子查询/视图/临时表</h5>
1084<!--kg-card-begin: html-->
1085<table>
1086<thead>
1087<tr>
1088<th>类型</th>
1089<th>适用场景</th>
1090<th>注意事项</th>
1091</tr>
1092</thead>
1093<tbody>
1094<tr>
1095<td>子查询(Subquery)</td>
1096<td>几乎不用</td>
1097<td>严重降低数据管道可读性</td>
1098</tr>
1099<tr>
1100<td>临时表(Temp Table)</td>
1101<td>需重复使用某段逻辑</td>
1102<td>会被物化,能提升后续执行效率</td>
1103</tr>
1104<tr>
1105<td>视图(View)</td>
1106<td>需长期存储逻辑(超出自定义管道执行周期)</td>
1107<td>不存储数据,仅保存查询逻辑</td>
1108</tr>
1109<tr>
1110<td>CTE(公共表表达式)</td>
1111<td>其他所有场景</td>
1112<td>最提升代码可读性,优先用</td>
1113</tr>
1114</tbody>
1115</table>
1116<!--kg-card-end: html-->
1117<h5 id="%EF%BC%883%EF%BC%89%E5%88%86%E5%B8%83%E5%BC%8F%E7%8E%AF%E5%A2%83%E4%B8%AD%E7%9A%84sql%E9%80%BB%E8%BE%91">(3)分布式环境中的SQL逻辑</h5><ul><li>触发“数据混洗(Shuffle)”的关键字:JOIN、GROUP BY、ORDER BY(需注意性能)</li><li>高可扩展性关键字(Map端直接执行):SELECT、FROM、WHERE、LIMIT(效率高)</li></ul><h5 id="%EF%BC%884%EF%BC%89%E7%AA%97%E5%8F%A3%E5%87%BD%E6%95%B0%EF%BC%9A%E9%9D%A2%E8%AF%95%E9%AB%98%E9%A2%91%E8%80%83%E7%82%B9">(4)窗口函数:面试高频考点</h5><p>基础语法:</p><pre><code class="language-sql">RANK() OVER (
1118  PARTITION BY &lt;分区字段&gt;  -- 按什么分组(如部门、国家)
1119  ORDER BY &lt;排序字段&gt;      -- 组内排序规则
1120  ROWS BETWEEN &lt;前N行&gt; PRECEDING AND &lt;后N行&gt; FOLLOWING  -- 窗口范围
1121)
1122
1123</code></pre><ul><li>若不指定<code>ROWS BETWEEN</code>,默认是“从窗口起始行到当前行”(累积计算)。</li></ul><h6 id="rankdenserankrownumber%E7%9A%84%E5%8C%BA%E5%88%AB">RANK/DENSE_RANK/ROW_NUMBER的区别</h6>
1124<!--kg-card-begin: html-->
1125<style>
1126/* Pretty, responsive table (Notion-like) */
1127/* ——— theme vars ——— */
1128:root{
1129  --tbl-bg:#fff;
1130  --tbl-text:#111827;
1131  --tbl-muted:#6b7280;
1132  --tbl-border:#e5e7eb;
1133  --tbl-border-strong:#d1d5db;
1134  --tbl-radius:12px;
1135  --tbl-accent:#f59e0b; /* accent color (amber). tweak to fit your brand */
1136}
1137
1138/* wrapper (keeps layout from breaking on small screens) */
1139.xw-table-wrap{margin:1rem 0;}
1140
1141/* base table */
1142.xw-table{
1143  width:100%;
1144  border-collapse:separate;
1145  border-spacing:0;
1146  background:var(--tbl-bg);
1147  color:var(--tbl-text);
1148  font-size:14.5px;
1149  line-height:1.6;
1150  border:1px solid var(--tbl-border);
1151  border-radius:var(--tbl-radius);
1152  overflow:hidden;
1153}
1154
1155/* caption */
1156.xw-table caption{
1157  text-align:left;
1158  font-weight:600;
1159  padding:.75rem 1rem;
1160  color:var(--tbl-muted);
1161}
1162
1163/* header */
1164.xw-table thead th{
1165  background:linear-gradient(0deg,rgba(245,158,11,.08),rgba(245,158,11,.08));
1166  text-align:left;
1167  font-weight:600;
1168  padding:.75rem 1rem;
1169  border-bottom:1px solid var(--tbl-border-strong);
1170}
1171
1172/* cells */
1173.xw-table th,
1174.xw-table td{
1175  padding:.75rem 1rem;
1176  vertical-align:top;
1177}
1178
1179.xw-table tbody tr + tr td{
1180  border-top:1px solid var(--tbl-border);
1181}
1182
1183/* hover highlight */
1184.xw-table tbody tr:hover{
1185  background:rgba(245,158,11,.06);
1186}
1187
1188/* code styling for function names */
1189.xw-table code{
1190  background:rgba(0,0,0,.04);
1191  padding:.15rem .35rem;
1192  border-radius:.35rem;
1193  font-family:ui-monospace,"Noto Sans Mono",SFMono-Regular,Menlo,Consolas,monospace;
1194  font-size:.95em;
1195}
1196
1197/* compact variant (optional): add class "xw-compact" on table */
1198.xw-table.xw-compact{font-size:13.5px;}
1199.xw-table.xw-compact th,
1200.xw-table.xw-compact td{padding:.55rem .75rem;}
1201
1202/* mobile: stack rows as cards */
1203@media (max-width:640px){
1204  .xw-table thead{display:none;}
1205  .xw-table,
1206  .xw-table tbody,
1207  .xw-table tr,
1208  .xw-table td{display:block;width:100%;}
1209  .xw-table tr{
1210    border-top:1px solid var(--tbl-border);
1211    padding:.25rem 0;
1212  }
1213  .xw-table tr:first-child{border-top:0;}
1214  .xw-table td{
1215    display:grid;
1216    grid-template-columns:38% 1fr;
1217    gap:.5rem;
1218    padding:.55rem 1rem;
1219  }
1220  .xw-table td::before{
1221    content:attr(data-label);
1222    font-weight:600;
1223    color:var(--tbl-muted);
1224  }
1225}
1226</style>
1227
1228<div class="xw-table-wrap">
1229  <table class="xw-table">
1230    <caption>窗口函数排名差异对比</caption>
1231    <thead>
1232      <tr>
1233        <th scope="col">函数</th>
1234        <th scope="col">无重复值时</th>
1235        <th scope="col">有重复值时(如并列第1)</th>
1236      </tr>
1237    </thead>
1238    <tbody>
1239      <tr>
1240        <td data-label="函数"><code>RANK</code></td>
1241        <td data-label="无重复值时">结果一致</td>
1242        <td data-label="有重复值时(如并列第1)">跳过排名(并列1后下一名是3)</td>
1243      </tr>
1244      <tr>
1245        <td data-label="函数"><code>DENSE_RANK</code></td>
1246        <td data-label="无重复值时">结果一致</td>
1247        <td data-label="有重复值时(如并列第1)">不跳过排名(并列1后下一名是2)</td>
1248      </tr>
1249      <tr>
1250        <td data-label="函数"><code>ROW_NUMBER</code></td>
1251        <td data-label="无重复值时">结果一致</td>
1252        <td data-label="有重复值时(如并列第1)">强制唯一排名(按数据自然顺序分配)</td>
1253      </tr>
1254    </tbody>
1255  </table>
1256</div>
1257
1258<!--kg-card-end: html-->
1259<h6 id="%E5%AE%9E%E6%88%98%E5%9C%BA%E6%99%AF%EF%BC%9A">实战场景:</h6><p>连续登录(面试常考,下面只给出示例,其他<a href="https://xiaowantree.com/gao-pin-sqlti/" rel="noreferrer">高频sql考题</a>在这里 → <a href="https://xiaowantree.com/gao-pin-sqlti/" rel="noreferrer">高频SQL考题</a>)</p><pre><code class="language-sql">select a.user_id,
1260	a.date_rslt,
1261	count(1) as cnt
1262from(
1263	select 
1264		t.user_id, 
1265		t.login_time,
1266		date_sub(login_time, INTERVAL t.num DAY) date_rslt
1267	from 
1268		(
1269		select user_id, 
1270		login_time, 
1271		row_number() over(partition by user_id order by login_time) num 
1272		from login_log
1273		) t
1274) a
1275group by a.user_id, a.date_rslt; 
1276
1277</code></pre><ul><li>注意:需确保每个日期都有数据(即使收入为0),否则结果会错。</li></ul><h5 id="%EF%BC%885%EF%BC%89%E6%95%B0%E6%8D%AE%E5%86%99%E5%85%A5%EF%BC%9Ainsert-intoinsert-overwritemerge">(5)数据写入:INSERT INTO/INSERT OVERWRITE/MERGE</h5><ul><li>INSERT INTO:仅追加数据,<strong>极易重复</strong>,需搭配TRUNCATE/DELETE使用</li><li>INSERT OVERWRITE:覆盖目标分区数据,大厂最常用</li><li>MERGE:对比新旧数据,仅更新/删除差异行,缺点是超大规模数据下速度慢</li></ul><hr><h3 id="%E4%BA%8C%E3%80%81%E5%88%86%E5%B8%83%E5%BC%8F%E8%AE%A1%E7%AE%97%E5%8E%9F%E7%90%86"><strong>二、分布式计算原理</strong></h3><p><strong>1. 大数据生态的底层逻辑 </strong></p><ul><li><strong>HDFS:</strong>分布式文件系统,采用块存储与多副本机制保障可靠性。重点理解 NameNode / DataNode 的职责、文件写入流程、心跳与故障恢复。</li><li><strong>MapReduce:</strong>最早的分布式计算模型,由 Map(映射)和 Reduce(聚合)两阶段组成。理解 Shuffle 阶段的数据倾斜和排序代价。</li><li><strong>Spark:</strong>Hadoop 的升级演进。核心概念包括 RDD、DAG、窄依赖与宽依赖、Shuffle 机制、Catalyst 优化器和内存管理。</li><li><strong>Flink:</strong>实时流处理框架。要掌握有界/无界流、事件时间(Event Time)、Watermark、窗口计算、状态管理与 Checkpoint。</li></ul><p><strong>2. 核心性能与稳定性话题</strong></p><ul><li><strong>Shuffle 机制:</strong> 代价高但必要,优化手段包括广播小表、分桶、预聚合。</li><li><strong>数据倾斜:</strong> 使用随机盐(Salting)或分区调优;Spark 3.x 可启用 AQE 自适应执行。</li><li><strong>Exactly-once 一致性:</strong> 通过 Checkpoint + 幂等 Sink 实现。</li><li><strong>资源调度:</strong> 理解 YARN / Kubernetes 调度思想与任务并发控制。</li></ul><h4 id="%E5%B8%B8%E8%A7%81%E9%9D%A2%E8%AF%95%E9%A2%98%EF%BC%9A"><strong>常见面试题:</strong></h4><ul><li>解释 Hadoop 的 MapReduce 执行流程。</li><li>Spark 的宽依赖和窄依赖有什么区别?</li><li>Flink 的 Watermark 是如何处理延迟数据的?</li></ul><hr><h3 id="%E4%B8%89%E3%80%81%E6%95%B0%E6%8D%AE%E5%BB%BA%E6%A8%A1%E4%B8%8E%E6%95%B0%E6%8D%AE%E8%B4%A8%E9%87%8F%EF%BC%9A%E8%AE%A9%E6%95%B0%E6%8D%AE%E2%80%9C%E8%83%BD%E7%94%A8%E3%80%81%E6%95%A2%E7%94%A8%E3%80%81%E5%A5%BD%E7%94%A8%E2%80%9D"><strong>三、数据建模与数据质量:让数据“能用、敢用、好用”</strong></h3><h4 id="1-%E5%88%86%E5%B1%82%E5%BB%BA%E6%A8%A1%E6%80%9D%E6%83%B3"><strong>1. 分层建模思想</strong></h4><p>国内主流架构:<strong>ODS → DWD → DWS → ADS</strong></p>
1278<!--kg-card-begin: html-->
1279<table>
1280<thead>
1281<tr>
1282<th><strong>层级</strong></th>
1283<th><strong>主要作用</strong></th>
1284</tr>
1285</thead>
1286<tbody>
1287<tr>
1288<td>ODS</td>
1289<td>存储原始数据,轻清洗</td>
1290</tr>
1291<tr>
1292<td>DWD</td>
1293<td>事实层,保证口径统一</td>
1294</tr>
1295<tr>
1296<td>DWS</td>
1297<td>公共指标层,用于复用</td>
1298</tr>
1299<tr>
1300<td>ADS</td>
1301<td>应用层,为报表/分析提供高效查询</td>
1302</tr>
1303</tbody>
1304</table>
1305<!--kg-card-end: html-->
1306<h4 id="2-%E6%A8%A1%E5%9E%8B%E7%B1%BB%E5%9E%8B%E4%B8%8E%E5%8F%96%E8%88%8D"><strong>2. 模型类型与取舍</strong></h4><ul><li><strong>范式建模</strong>:数据一致性强,适合高写入、查询复杂的系统。</li><li><strong>维度建模</strong>(事实表+维度表):分析型任务常用。</li><li><strong>宽表(One Big Table)</strong>:查询快但冗余多,适合固定报表。</li></ul><h4 id="3-%E6%95%B0%E6%8D%AE%E8%B4%A8%E9%87%8F%E4%B8%8E%E6%B2%BB%E7%90%86"><strong>3. 数据质量与治理</strong></h4><ul><li><strong>质量维度</strong>
1306:准确性、完整性、一致性、时效性。</li><li><strong>治理方法</strong>:<ul><li>“写入-校验-发布(W-A-P)” 模式;</li><li>建立异常监控与对账机制;</li><li>PII 数据脱敏与权限分级。</li></ul></li></ul><p><strong>常见面试题:</strong></p><ul><li>如何保证离线与实时数据口径一致?</li><li>什么是 WAP 模式?在数仓开发中如何落地?</li><li>说说你理解的数据质量治理体系。</li></ul><hr><h3 id="%E5%9B%9B%E3%80%81%E4%BD%9C%E5%93%81%E9%9B%86%E9%A1%B9%E7%9B%AE%EF%BC%88%E4%B8%8D%E6%98%AF%E5%BF%85%E9%A1%BB%EF%BC%8C%E4%BD%86%E6%9C%89%E5%88%99%E6%9E%81%E5%85%B7%E5%8A%A0%E5%88%86%EF%BC%89"><strong>四、作品集项目(不是必须,但有则极具加分)</strong></h3><p>国内招聘中展示作品集的候选人较少,如果你能展示一套完整的 Demo,将会大幅脱颖而出。</p><h4 id="%E5%8F%AF%E9%80%89%E5%B1%95%E7%A4%BA%E6%96%B9%E5%90%91%EF%BC%9A"><strong>可选展示方向:</strong></h4><ul><li><strong>小型数仓项目</strong>:模拟电商日志 → Hive → DWD → DWS → 报表。</li><li><strong>批流一体示例</strong>:实现实时指标看板。</li><li><strong>数据治理案例</strong>:自动校验脚本 + 指标对账系统。</li></ul><p>建议在简历中附上<strong>架构图</strong>与<strong>核心代码链接(GitHub/Gitee)</strong>,能有效提升技术说服力。</p><hr><h3 id="%E4%BA%94%E3%80%81%E6%80%BB%E7%BB%93%EF%BC%9A2025-%E5%B9%B4%E5%85%A5%E9%97%A8%E6%95%B0%E6%8D%AE%E5%B7%A5%E7%A8%8B%EF%BC%88%E5%9B%BD%E5%86%85%EF%BC%89%E2%80%94%E2%80%94%E4%BB%A5%E7%90%86%E8%AE%BA%E4%B8%BA%E6%9C%AC%EF%BC%8C%E5%8E%9F%E7%90%86%E4%B8%BA%E7%8E%8B"><strong>五、总结:2025 年入门数据工程(国内)——以理论为本,原理为王</strong></h3><p>当下国内企业更看重你是否能:</p><ul><li>理解 Hadoop 生态体系及其原理;</li><li>设计合理的数仓分层模型;</li><li>讲清楚批流系统的协同机制;</li><li>在面试中通过原理解释问题的解决思路。</li></ul><p>真正的竞争力,不在于用哪个云,而在于你能否<strong>用清晰的逻辑讲明白一个系统为什么这么设计</strong>。</p><blockquote>如果这份路线图对你有帮助,欢迎分享给正在学习数据工程的朋友。</blockquote><blockquote>也可以发送你的问题到 <a href="/cdn-cgi/l/email-protection#4f3e28227d7d797e7c7e0f28222e2623612c2022"><span class="__cf_email__" data-cfemail="6c1d0b015e5e5a5d5f5d2c0b010d0500420f0301">[email&#160;protected]</span></a>,我会挑选常见考点整理成下一篇文章。</blockquote><hr><hr><h2 id="%E6%95%B0%E6%8D%AE%E5%B7%A5%E7%A8%8B%E5%85%A5%E9%97%A8%E8%B7%AF%E7%BA%BF%E5%9B%BE%EF%BC%88%E6%B5%B7%E5%A4%96%EF%BC%89">数据工程入门路线图(海外)</h2><h3 id="%E5%BC%95%E8%A8%80-1">引言</h3><p>想找到一份数据工程相关工作并不容易,但是也没有想象中的那么难。在网上看到很多人都说进入公司后就是做”SQL Boy“的言论后,却在学习阶段发现技术要学 ——— Spark、Flink, Hadoop、Datalake、 Lakehouse等等</p><p>要明确入门方向、掌握学习方法,你需要一些指引。本文将拆解数据工程所需的全部步骤!</p><h3 id="%E5%85%88%E8%81%8A%E7%AD%96%E7%95%A5%EF%BC%9A%E5%88%AB%E8%BF%B7%E4%BF%A1%E2%80%9C%E5%88%B7leetcode%E6%8B%BFoffer%E2%80%9D">先聊策略:别迷信“刷LeetCode拿Offer”</h3><p>很多人觉得刷够一定数量的LeetCode题目,工作就会自动找上门。但现实几乎从不是这样!</p><blockquote><a href="https://xiaowantree.com/">xiaowantree.com</a> 里面会定期更新优质文章,大家可以注册免费或者付费观看,如果需要了解产品,请点击这里,助力您的offer之路。</blockquote><h3 id="%E8%A6%81%E6%8B%BF%E5%88%B0%E6%95%B0%E6%8D%AE%E5%B7%A5%E7%A8%8Boffer%EF%BC%8C%E4%BD%A0%E9%9C%80%E8%A6%81%E8%BF%99%E4%BA%9B">要拿到数据工程Offer,你需要这些</h3><h4 id="1-%E5%8F%AF%E9%AA%8C%E8%AF%81%E7%9A%84%E6%A0%B8%E5%BF%83%E6%8A%80%E8%83%BD">1. 可验证的核心技能</h4><ul><li>SQL与Python(基础中的基础)</li><li>分布式计算能力(三选一:Snowflake/Spark/BigQuery)</li><li>任务编排知识(三选一:Airflow/Mage/Databricks Workflows)</li><li>数据建模与数据质量管控能力</li></ul><h4 id="2-%E8%83%BD%E8%AF%81%E6%98%8E%E8%83%BD%E5%8A%9B%E7%9A%84%E2%80%9C%E7%A1%AC%E9%80%9A%E8%B4%A7%E2%80%9D">2. 能证明能力的“硬通货”</h4><ul><li>1个以上可展示的作品集项目</li><li>在LinkedIn和面试中能体现的个人品牌</li></ul><hr><h3 id="%E4%B8%80%E3%80%81%E5%AD%A6%E4%B9%A0sql%EF%BC%9A%E6%95%B0%E6%8D%AE%E5%B7%A5%E7%A8%8B%E7%9A%84%E2%80%9C%E6%AF%8D%E8%AF%AD%E2%80%9D">一、学习SQL:数据工程的“母语”</h3><p>回避SQL就等于放弃数据工程工作——SQL是你必须掌握的最基础语言。</p><h4 id="1-%E6%8E%A8%E8%8D%90%E5%AD%A6%E4%B9%A0%E8%B5%84%E6%BA%90-1">1. 推荐学习资源</h4><ul><li><a href="https://leetcode.com/?ref=xiaowantree.com">Leetcode</a>: 侧重sql题目的练习</li><li><a href="https://datalemur.com/?ref=xiaowantree.com">DataLemur</a>:侧重面试级SQL题,贴近实战</li><li>
1306<a href="https://www.stratascratch.com/?ref=xiaowantree.com">StrataScratch</a>:含真实业务场景的SQL练习</li></ul><h4 id="2-%E5%BF%85%E9%A1%BB%E6%8E%8C%E6%8F%A1%E7%9A%84%E6%A0%B8%E5%BF%83%E7%9F%A5%E8%AF%86%E7%82%B9-1">2. 必须掌握的核心知识点</h4><p>(1)基础操作与聚合</p><ul><li>JOIN类型:重点掌握INNER JOIN、LEFT JOIN、FULL OUTER JOIN,<strong>几乎不用RIGHT JOIN</strong></li><li>GROUP BY聚合:明确COUNT与COUNT(DISTINCT)的区别(分布式环境中COUNT(DISTINCT)速度慢)</li></ul><p>基数缩减:将高基数维度转为低基数,例如按年龄分桶:</p><pre><code class="language-sql">SELECT
1307  CASE WHEN age &gt; 30 THEN 'old' ELSE 'young' END as age_bucket,
1308  COUNT(1)
1309FROM users
1310GROUP BY 1
1311
1312</code></pre><p>CASE WHEN+聚合:例如统计过期订单数:</p><pre><code class="language-sql">COUNT(CASE WHEN status = 'expired' THEN order_id END)
1313
1314</code></pre><p>(2)查询结构对比:CTE/子查询/视图/临时表</p>
1315<!--kg-card-begin: html-->
1316<table>
1317<thead>
1318<tr>
1319<th>类型</th>
1320<th>适用场景</th>
1321<th>注意事项</th>
1322</tr>
1323</thead>
1324<tbody>
1325<tr>
1326<td>子查询(Subquery)</td>
1327<td>几乎不用</td>
1328<td>严重降低数据管道可读性</td>
1329</tr>
1330<tr>
1331<td>临时表(Temp Table)</td>
1332<td>需重复使用某段逻辑</td>
1333<td>会被物化,能提升后续执行效率</td>
1334</tr>
1335<tr>
1336<td>视图(View)</td>
1337<td>需长期存储逻辑(超出自定义管道执行周期)</td>
1338<td>不存储数据,仅保存查询逻辑</td>
1339</tr>
1340<tr>
1341<td>CTE(公共表表达式)</td>
1342<td>其他所有场景</td>
1343<td>最提升代码可读性,优先用</td>
1344</tr>
1345</tbody>
1346</table>
1347<!--kg-card-end: html-->
1348<p>(3)分布式环境中的SQL逻辑</p><ul><li>触发“数据混洗(Shuffle)”的关键字:JOIN、GROUP BY、ORDER BY(需注意性能)</li><li>高可扩展性关键字(Map端直接执行):SELECT、FROM、WHERE、LIMIT(效率高)</li></ul><p>(4)窗口函数:面试高频考点</p><p>基础语法:</p><pre><code class="language-sql">RANK() OVER (
1349  PARTITION BY &lt;分区字段&gt;  -- 按什么分组(如部门、国家)
1350  ORDER BY &lt;排序字段&gt;      -- 组内排序规则
1351  ROWS BETWEEN &lt;前N行&gt; PRECEDING AND &lt;后N行&gt; FOLLOWING  -- 窗口范围
1352)
1353
1354</code></pre><ul><li>若不指定<code>ROWS BETWEEN</code>,默认是“从窗口起始行到当前行”(累积计算)。</li></ul><p>RANK/DENSE_RANK/ROW_NUMBER的区别</p>
1355<!--kg-card-begin: html-->
1356<table>
1357<thead>
1358<tr>
1359<th>函数</th>
1360<th>无重复值时</th>
1361<th>有重复值时(如并列第1)</th>
1362</tr>
1363</thead>
1364<tbody>
1365<tr>
1366<td>RANK</td>
1367<td>结果一致</td>
1368<td>跳过排名(并列1后下一名是3)</td>
1369</tr>
1370<tr>
1371<td>DENSE_RANK</td>
1372<td>结果一致</td>
1373<td>不跳过排名(并列1后下一名是2)</td>
1374</tr>
1375<tr>
1376<td>ROW_NUMBER</td>
1377<td>结果一致</td>
1378<td>强制唯一排名(按数据自然顺序分配)</td>
1379</tr>
1380</tbody>
1381</table>
1382<!--kg-card-end: html-->
1383<p>实战场景:滚动计算</p><p>按部门计算30天滚动收入(面试常考):</p><pre><code class="language-sql">SELECT
1384  revenue_date,
1385  SUM(revenue) OVER (
1386    PARTITION BY department
1387    ORDER BY revenue_date
1388    ROWS BETWEEN 30 PRECEDING AND CURRENT ROW
1389  ) as thirty_day_rolling_revenue
1390FROM daily_sales
1391
1392</code></pre><ul><li>注意:需确保每个日期都有数据(即使收入为0),否则结果会错。</li></ul><p>(5)数据写入:INSERT INTO/INSERT OVERWRITE/MERGE</p><ul><li>INSERT INTO:仅追加数据,<strong>极易重复</strong>,需搭配TRUNCATE/DELETE使用</li><li>INSERT OVERWRITE:覆盖目标分区数据,大厂最常用</li><li>MERGE:对比新旧数据,仅更新/删除差异行,缺点是超大规模数据下速度慢</li></ul><hr><h3 id="%E4%BA%8C%E3%80%81%E5%AD%A6%E4%B9%A0python%EF%BC%88%E6%88%96scala%EF%BC%89%EF%BC%9A%E7%AA%81%E7%A0%B4sql%E7%9A%84%E5%B1%80%E9%99%90">二、学习Python(或Scala):突破SQL的局限</h3><p>SQL虽好,但有局限性。要处理复杂逻辑(如API调用、自定义函数),需掌握Python或Scala。</p><h4 id="1-%E8%AF%AD%E8%A8%80%E9%80%89%E6%8B%A9%EF%BC%9Apython%E8%BF%98%E6%98%AFscala%EF%BC%9F">1. 语言选择:Python还是Scala?</h4><ul><li>零基础:选Python(入门快、生态全,能快速出成果)</li><li>
1392有编程基础:可试Scala(静态类型,对schema和数据质量管控更严格,适合Spark深度开发)</li></ul><h4 id="2-%E6%A0%B8%E5%BF%83%E5%AD%A6%E4%B9%A0%E5%86%85%E5%AE%B9">2. 核心学习内容</h4><p><strong>(1)基础:数据类型与结构</strong></p><ul><li>基础类型:string、integer、decimal、boolean</li><li>复杂结构:list(数组)、dictionary(字典)、stack(栈)、queue(队列)</li><li>非必需但面试可能考:heap(堆)、tree(树)、graph(图)(不用深钻,了解基础即可)</li></ul><p><strong>(2)算法基础</strong></p><ul><li>必学:循环、线性查找、二分查找</li><li>必懂:大O表示法(能分析时间/空间复杂度)</li><li>非必需但面试可能考:动态规划、贪心算法、迪杰斯特拉算法(不用深钻)</li></ul><p><strong>(3)数据工程实战场景</strong></p><ol><li>编写Airflow DAG:用Python定义数据管道流程(最常用场景)</li><li>调用REST API:用<code>requests</code>库处理GET/POST/PUT请求(获取外部数据)</li><li>编写Spark UDF:<ul><li>部分场景下需用 Python UDF 处理数据,但注意:Spark 1.x/2.x 版本中 Python UDF 性能较差,若使用旧版本,建议用 Scala 编写 UDF。</li></ul></li></ol><p><strong>(4)代码测试</strong></p><ul><li>Python:用<code>pytest</code>框架测试代码。推荐搭配<code>Chispa</code>库 —— 它专门用于测试 PySpark 任务,功能非常实用。</li><li>Scala:用<code>JUnit</code>框架</li></ul><hr><h3 id="%E4%B8%89%E3%80%81%E5%AD%A6%E4%B9%A0%E5%88%86%E5%B8%83%E5%BC%8F%E8%AE%A1%E7%AE%97%EF%BC%9A%E5%A4%84%E7%90%86%E6%B5%B7%E9%87%8F%E6%95%B0%E6%8D%AE%E7%9A%84%E6%A0%B8%E5%BF%83">三、学习分布式计算:处理海量数据的核心</h3><p>分布式计算的核心是“多台机器协同,每台处理一小部分数据”,突破单机算力限制。需三选一:Spark/BigQuery/Snowflake。</p><h4 id="1%E6%A0%B8%E5%BF%83%E6%A6%82%E5%BF%B5%EF%BC%9A%E6%95%B0%E6%8D%AE%E6%B7%B7%E6%B4%97%EF%BC%88shuffle%EF%BC%89"><strong>1.核心概念:数据混洗(Shuffle)</strong></h4><ul><li>作用:确保特定数据到特定机器(如“按用户汇总消息数”,需同一用户数据到同一机器)</li><li>触发场景:仅JOIN、GROUP BY、ORDER BY时必须Shuffle</li><li>误区:Shuffle不是“坏事”——它让分布式计算逻辑接近单机,但超大规模数据下会成性能瓶颈</li></ul><h4 id="2-%E8%B6%85%E5%A4%A7%E8%A7%84%E6%A8%A1%E5%9C%BA%E6%99%AF%EF%BC%9Ashuffle%E4%BC%98%E5%8C%96%E6%8A%80%E5%B7%A7">2. 超大规模场景:Shuffle优化技巧</h4><ul><li>广播连接(Broadcast JOIN):小表(&lt;5GB)广播到所有节点,无需Shuffle</li><li>分桶连接(Bucket JOIN):大表先分桶,后续多次JOIN可复用,减少重复Shuffle</li><li>数据集分区:按时间(如按天)分区,避免JOIN全量数据</li><li>累积表设计:计算“月活”时用“日活累积表”,不用扫30天原始数据</li></ul><h4 id="3-%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98%EF%BC%9A%E6%95%B0%E6%8D%AE%E5%80%BE%E6%96%9C%EF%BC%88skew%EF%BC%89">3. 常见问题:数据倾斜(Skew)</h4><ul><li>问题:某台机器数据量远大于其他机器,拖慢整体速度</li><li>解决方案:<ul><li>Spark 3.x+:启用“自适应执行(Adaptive Execution)”(Databricks支持最好)</li><li>Spark 3.x以下:对JOIN/GROUP BY关键字“加盐(Salting)”,用随机数平衡数据分布</li></ul></li></ul><h4 id="4-%E8%BE%93%E5%87%BA%E6%95%B0%E6%8D%AE%EF%BC%9Aparquet%E6%96%87%E4%BB%B6%EF%BC%88%E9%87%8D%E7%82%B9%EF%BC%89">4. 输出数据:Parquet文件(重点)</h4><ul><li>特性:支持“行程长度编码(Run-Length Encoding)”,压缩率极高(曾将Airbnb数据集压缩90%+)</li><li>场景:用Spark处理后,通常以Parquet格式写入S3/GCP/Azure</li><li>注:Snowflake/BigQuery用自研格式,可参考其他专项资料(如“西雅图数据Guy”的内容)</li></ul><hr><h3 id="%E5%9B%9B%E3%80%81%E6%95%B0%E6%8D%AE%E5%BB%BA%E6%A8%A1%E4%B8%8E%E6%95%B0%E6%8D%AE%E8%B4%A8%E9%87%8F%EF%BC%9A%E5%81%9A%E2%80%9C%E5%8F%AF%E7%94%A8%E7%9A%84%E5%A5%BD%E6%95%B0%E6%8D%AE%E2%80%9D">四、数据建模与数据质量:做“可用的好数据”</h3><p>数据工程师的“产品”是数据——需满足“准确、易用、合规”三个标准。</p><h4 id="1-%E6%95%B0%E6%8D%AE%E8%B4%A8%E9%87%8F%E7%9A%843%E4%B8%AA%E6%A0%B8%E5%BF%83%E6%A0%87%E5%87%86">1. 数据质量的3个核心标准</h4><ul><li>准确性:无重复、无NULL、格式正确、行数符合预期</li><li>易用性:有文档、列名规范、查询高效(避免巨额云资源成本)</li><li>合规性:不存储多余的个人身份信息(PII),保护用户隐私</li></ul><h4 id="2-%E5%A6%82%E4%BD%95%E5%AE%9E%E7%8E%B0%E9%AB%98%E8%B4%A8%E9%87%8F%E6%95%B0%E6%8D%AE%EF%BC%9F">2. 如何实现高质量数据?</h4><p>(1)确保准确性:校验+自动化</p><ul><li>首轮校验:让数据分析师参与(参考Airbnb的MIDAS流程)</li><li>自动化校验:用Great Expectations等工具,嵌入数据管道</li><li>关键模式:遵循“写入-校验-发布(Write-Audit-Publish)”,不合格数据不进生产</li></ul><p>(2)提升易用性:文档+建模</p><ul><li>文档先行:搭建管道前,先写需求文档、确认 stakeholder(避免重复开发)</li></ul><p>
1392数据建模三选一(按场景选):</p>
1393<!--kg-card-begin: html-->
1394<table>
1395<thead>
1396<tr>
1397<th>建模类型</th>
1398<th>核心逻辑</th>
1399<th>适用场景</th>
1400</tr>
1401</thead>
1402<tbody>
1403<tr>
1404<td>关系型建模</td>
1405<td>减少数据重复,查询较复杂</td>
1406<td>存储成本敏感,数据一致性要求高</td>
1407</tr>
1408<tr>
1409<td>维度建模(Kimball)</td>
1410<td>拆分为事实表+维度表,平衡存储与计算</td>
1411<td>多数业务分析场景</td>
1412</tr>
1413<tr>
1414<td>单一大表(One Big Table)</td>
1415<td>事实+维度合并,查询极快,数据重复多</td>
1416<td>计算成本敏感,查询场景固定</td>
1417</tr>
1418</tbody>
1419</table>
1420<!--kg-card-end: html-->
1421<p>(3)确保合规:隐私保护</p><ul><li>识别PII:任何可定位到具体用户的信息都算(如手机号、邮箱、精准地址)</li><li>处理方式:匿名化处理,平衡“长期分析需求”与“用户隐私”</li></ul><hr><h3 id="%E4%BA%94%E3%80%81%E6%89%93%E9%80%A0%E4%BD%9C%E5%93%81%E9%9B%86%E9%A1%B9%E7%9B%AE%EF%BC%9A%E8%AE%A9offer%E4%B8%BB%E5%8A%A8%E6%89%BE%E4%BD%A0">五、打造作品集项目:让Offer主动找你</h3><p>作品集是“能力证明”的硬通货,我之前看过一个很典型的例子,他是因为开发过一款产品,这个产品的月活达到了4w人,才被顶级科技公司录取了,所以说如果你现在有很多充足的时间,我非常推荐可以用自己的专业知识去开发一个小产品,虽然不一定会得到市场的认可,但是你可以把社交媒体变成你的名片,向面试官展示你的能力,会给你加分不少。</p><h4 id="1-%E9%A1%B9%E7%9B%AE%E9%80%89%E6%8B%A93%E4%B8%AA%E5%8E%9F%E5%88%99">1. 项目选择3个原则</h4><ul><li>选你热爱的领域:能支撑你3个月每周5-10小时的投入(不热爱很难坚持)</li><li>包含前端展示:用Tableau/Power BI做可视化,或用JavaScript搭简单网站,方便展示</li><li>难度匹配:别做“一键完成”的项目(太简单没竞争力),要包含“数据管道+质量校验+自动化运行”</li></ul><h4 id="2-%E9%A1%B9%E7%9B%AE%E5%BF%85%E9%A1%BB%E5%8C%85%E5%90%AB%E7%9A%844%E4%B8%AA%E8%A6%81%E7%B4%A0">2. 项目必须包含的4个要素</h4><ol><li>完整文档:说明输入/输出数据集、质量校验规则、运行频率</li><li>生产环境运行:用Heroku Scheduler(每月5-10美元)或自建Airflow集群,证明管道可落地</li><li>用热门技术:融入Spark/Snowflake/Iceberg/Delta Lake等,贴合行业趋势</li><li>开源到GitHub:附清晰README,方便面试官查看</li></ol><hr><h3 id="%E5%85%AD%E3%80%81%E6%89%93%E9%80%A0%E4%B8%AA%E4%BA%BAip%EF%BC%9A%E9%9D%A2%E8%AF%95%E4%B8%8E%E4%BA%BA%E8%84%89%E7%9A%84%E2%80%9C%E5%8A%A0%E5%88%86%E9%A1%B9%E2%80%9D">六、打造个人IP:面试与人脉的“加分项”</h3><p>有技能+作品集后,IP 能帮你“脱颖而出”。</p><h4 id="1-linkedin%E4%BA%BA%E8%84%89%E6%90%AD%E5%BB%BA">1. LinkedIn人脉搭建</h4><p>(1)重点连接两类人</p><ul><li>招聘经理/猎头:让他们记住你(即使当前无空缺,未来可能有机会)</li><li>同行:互相分享求职技巧,缓解求职压力</li></ul><p>(2)人脉搭建技巧</p><ul><li>先“互动”再“求内推”:评论对方内容、分享见解,建立信任后再提需求</li><li>用“提问”开场:想内推时,先问“您目前负责的业务中,数据管道主要解决什么问题?”,比直接要内推成功率高</li><li>接受“低回复率”:初期发20条私信可能仅1-2条回复,优先联系与你资历相近的人</li></ul><p>(3)创作内容:让更多人认识你</p><ul><li>分享学习笔记(如“Spark Shuffle优化实战”),我也是近一两年才开始养成了写文章的习惯,博士的学习生活也让我更加清楚什么是好的文章,也切切实实感受到了写文章带给我的好处,在这里我可以分享一个小技巧:你写完后你要回过头看看你的这篇文章,如果你是读者,你会愿意看下去吗,如果读者不是那么的专业,他能够看懂吗?想好这些问题,我相信你可以写出一个很棒的文章。</li><li>记录项目复盘(如“用Snowflake搭建电商数据管道的踩坑记录”)</li><li>我也是靠内容创作赚了一些钱,而且随着你越做你会知道的越多——内å
1421®¹åˆ›ä½œçš„力量远超想象!</li></ul><h4 id="2-%E9%9D%A2%E8%AF%95%EF%BC%9A%E5%81%9A%E2%80%9C%E7%9C%9F%E5%AE%9E%E7%9A%84%E4%BA%BA%E2%80%9D%EF%BC%8C%E5%88%AB%E5%81%9A%E2%80%9C%E7%AD%94%E9%A2%98%E6%9C%BA%E5%99%A8%E2%80%9D">2. 面试:做“真实的人”,别做“答题机器”</h4><p>(1)面试前准备</p><ul><li>研究面试官:查LinkedIn,了解其在公司的年限、负责业务</li><li>问猎头细节:明确“用什么技术”“团队规模”“文化风格”</li></ul><p>(2)面试中要展现的3个特质</p><ul><li>热情:对岗位和业务表现出兴趣</li><li>沉稳:回答问题时逻辑清晰,不慌不忙</li><li>好奇:主动问“这个数据管道当前最大的挑战是什么”,展现思考深度</li></ul><hr><h3 id="%E6%80%BB%E7%BB%93%EF%BC%9A2025%E5%B9%B4%E5%85%A5%E9%97%A8%E6%95%B0%E6%8D%AE%E5%B7%A5%E7%A8%8B%EF%BC%8C%E9%9A%BE%E4%BD%86%E5%8F%AF%E8%A1%8C%EF%BC%8Cai-%E7%9A%84%E5%8F%91%E5%B1%95%E6%9C%89%E7%82%B9%E6%8A%B9%E6%9D%80%E4%BA%86ds%E7%9A%84%E5%B8%82%E5%9C%BA%EF%BC%8C%E4%BD%86%E6%98%AFai%E4%B9%9F%E4%BB%8D%E7%84%B6%E9%9C%80%E8%A6%81%E6%95%B0%E6%8D%AE">总结:2025年入门数据工程,难但可行,AI 的发展有点抹杀了DS的市场,但是AI也仍然需要数据</h3><p>当前就业市场确实严峻,但优秀的工程师和初创公司,往往在困境中成长。跟着这份路线图走,能帮你更接近理想的 data engineering 岗位!</p><blockquote><strong>互动:你觉得还有哪些关键技能?</strong> 我是否遗漏了什么重要内容?可以发邮件 <a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="2b5a4c4619191d1a181a6b4c464a424705484446">[email&#160;protected]</a>或者在小红书私信告诉我,你的需求会被优先满足。</blockquote><p>若觉得这份路线图有用,欢迎分享给朋友!</p><hr><h2 id="%E6%B5%B7%E5%86%85%E5%A4%96%E5%AF%B9%E6%AF%94">海内外对比</h2><p>国内:</p><p>面试更偏“笔试化”和“知识覆盖型”,面试官常从底层原理切入——例如问你 Hadoop 的执行流程、Spark 的 Shuffle 为什么慢、Flink 的 Exactly-once 如何实现、数仓的 DWD 和 DWS 各代表什么含义。你可能需要白板手写 SQL、画出数据分层图或解释数据倾斜的解决方案。核心目的是判断:<strong>你是否理解整个数据体系的运作逻辑</strong>。</p><p>海外:</p><p>他们关注你在真实场景中如何设计系统、如何应对挑战、如何做权衡。例如问你:“在项目里如何保证 pipeline 的稳定性?”、“遇到数据延迟时你怎么 debug?”、“为什么选择 Snowflake 而不是 BigQuery?”甚至“如果要重构你的数据模型,你会如何做?”。核心目的是判断:<strong>你能否独立设计与落地工程系统</strong>。</p>
1422<!--kg-card-begin: html-->
1423<table>
1424<thead>
1425<tr>
1426<th><strong>面试维度</strong></th>
1427<th><strong>国内数据工程岗位</strong></th>
1428<th><strong>海外数据工程岗位</strong></th>
1429</tr>
1430</thead>
1431<tbody>
1432<tr>
1433<td>面试重点</td>
1434<td>理论原理:Hadoop、Spark、Flink、数仓建模、数据倾斜、SQL 手写</td>
1435<td>工程落地:Pipeline 设计、Airflow/dbt、云平台(AWS/GCP/Azure)实践</td>
1436</tr>
1437<tr>
1438<td>题型风格</td>
1439<td>技术笔试题+口头问答:考点密集、覆盖全面</td>
1440<td>系统设计+项目复盘:注重思维与场景分析</td>
1441</tr>
1442<tr>
1443<td>评估核心</td>
1444<td>原理理解深度、执行逻辑是否透彻</td>
1445<td>工程经验、可维护性、问题分析与沟通能力</td>
1446</tr>
1447<tr>
1448<td>回答方式</td>
1449<td>注重基础知识、技术细节与公式化表达,但是大厂的考察会把原理挖的很深,如果有实习经历的话主要还是看解决问题的思路</td>
1450<td>注重故事性、思考过程与工程权衡</td>
1451</tr>
1452<tr>
1453<td>成功关键</td>
1454<td>扎实基础、熟悉框架原理、清晰讲解数据流</td>
1455<td>能讲出决策过程、项目复盘逻辑、体现实战能力</td>
1456</tr>
1457</tbody>
1458</table>
1459<!--kg-card-end: html-->
1460</div>
1461
1462      
1463        <footer class="lesson-footer">
1464          <div>
1465            <strong data-completion-title>完成后会更新你的学习进度</strong>
1466            <span data-completion-copy>你可以随时返回学习路径,自由选择下一节。</span>
1467          </div>
1468          <button class="primary-button" type="button" data-complete-lesson="68fa2955512f5e0001d71394">标记完成 <span aria-hidden="true">→</span></button>
1469        </footer>
1470      
1471    </article>
1472  </main>
1473
1474  <aside class="right-rail article-rail" id="article-reading-rail" data-article-toc-panel aria-label="文章阅读辅助">
1475  <section class="rail-section reading-progress-section">
1476    <div class="reading-progress-heading">
1477      <span class="rail-eyebrow">▣ 阅读进度</span>
1478      <strong data-reading-progress>0%</strong>
1479    </div>
1480    <div class="horizontal-progress" role="progressbar" aria-label="文章阅读进度" aria-valuemin="0" aria-valuemax="100" aria-valuenow="0" data-reading-progressbar><i></i></div>
1481  </section>
1482  <nav class="rail-section toc-section" aria-label="本页目录" data-reading-toc>
1483    <div class="toc-section__header">
1484      <strong>本页目录</strong>
1485      <button type="button" data-toc-density-toggle aria-controls="article-toc-items" aria-expanded="true" hidden>只看大纲</button>
1486    </div>
1487    <a class="is-root is-active" href="#lesson-top" data-toc-link="lesson-top">数据工程入门路线图(必看)</a>
1488    <div id="article-toc-items" data-toc-items></div>
1489  </nav>
1490  <button class="rail-secondary-button" type="button" data-bookmark="68fa2955512f5e0001d71394">☆ 收藏这节课</button>
1491</aside>
1492  <button class="article-toc-toggle" type="button" data-article-toc-toggle aria-controls="article-reading-rail" aria-expanded="false" aria-label="打开本页目录">
1493    <i class="ph-duotone ph-list-bullets" aria-hidden="true"></i>
1494    <span class="article-toc-toggle__label">目录</span>
1495  </button>
1496</div>
1497
1498  </div>
1499  <!-- Ghost Code Injection: Site Header -->
1500<!-- DE Academy safe version: GA4 + isolated roadmap styles only. -->
1501
1502<!-- Keep this block only if GA4 is not already installed by another Ghost integration. -->
vendor: 109 bytes, lines 1502-1503
1502
1503<script data-cfasync="false" src="/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js"></script>
vendor: 63 bytes, line 1503
1503<script async src="https://www.googletagmanager.com/gtag/js?id=
1503G-MJ2YRQ6SYD
vendor: 12 bytes, line 1503
1503"></script>
1504<script>
1505  
vendor: 133 bytes, lines 1505-1508
1505window.dataLayer = window.dataLayer || [];
1506  function gtag(){dataLayer.push(arguments);}
1507  gtag('js', new Date());
1508  gtag('config', '
1508G-MJ2YRQ6SYD
vendor: 4 bytes, line 1508
1508');
1509</script>
1509
1510
1511<!-- Optional roadmap embed compatibility. All selectors are scoped to #roadmapEmbed. -->
1512<style>
1513  #roadmapEmbed {
1514    background: #fff;
1515    border: 1px solid rgba(15, 23, 42, 0.08);
1516    border-radius: 16px;
1517    box-shadow: 0 18px 50px rgba(15, 23, 42, 0.08);
1518    height: min(86vh, 980px);
1519    margin: 28px auto;
1520    max-width: 1680px;
1521    overflow: hidden;
1522    position: relative;
1523    width: min(100%, calc(100vw - 64px));
1524  }
1525
1526  #roadmapEmbed .roadmap-toolbar {
1527    display: flex;
1528    flex-wrap: wrap;
1529    gap: 10px;
1530    left: 16px;
1531    pointer-events: auto;
1532    position: absolute;
1533    top: 16px;
1534    z-index: 10;
1535  }
1536
1537  #roadmapEmbed .rm-btn {
1538    align-items: center;
1539    background: rgba(255, 255, 255, 0.94);
1540    border: 1px solid rgba(15, 23, 42, 0.14);
1541    border-radius: 10px;
1542    cursor: pointer;
1543    display: inline-flex;
1544    font: inherit;
1545    font-size: 14px;
1546    line-height: 1;
1547    padding: 10px 12px;
1548  }
1549
1550  #roadmapEmbed .rm-exit {
1551    display: none;
1552  }
1553
1554  #roadmapEmbed.is-active .rm-enter {
1555    display: none;
1556  }
1557
1558  #roadmapEmbed.is-active .rm-exit {
1559    display: inline-flex;
1560  }
1561
1562  #roadmapEmbed .roadmap-iframe {
1563    border: 0;
1564    height: 100%;
1565    inset: 0;
1566    pointer-events: none;
1567    position: absolute;
1568    width: 100%;
1569  }
1570
1571  #roadmapEmbed.is-active .roadmap-iframe {
1572    pointer-events: auto;
1573  }
1574
1575  #roadmapEmbed .rm-enter {
1576    background: #dc2626;
1577    border-color: #dc2626;
1578    color: #fff;
1579    font-weight: 600;
1580  }
1581
1582  @media (max-width: 700px) {
1583    #roadmapEmbed {
1584      border-radius: 12px;
1585      height: 76vh;
1586      margin: 20px auto;
1587      width: 100%;
1588    }
1589
1590    #roadmapEmbed .roadmap-toolbar {
1591      left: 10px;
1592      top: 10px;
1593    }
1594  }
1595</style>
1596<script type="module" src="https://static.cloudflareinsights.com/beacon.min.js/v31edd6df95cf4e85bb4c19e7a9bdbcba1788362987495" integrity="sha512-iIg7k2xntmwu6/uSb5tpc/hySgZc4eoL31yB29W6tJFo2akwjPWcEqnCEdJvGexCL0KEQwVYv5BlowfhVz26hg==" data-cf-beacon='{"version":"2024.11.0","token":"5c190cf9e68b43378c6b04174de597d5","r":1,"spa":2}' crossorigin="anonymous"></script>
1596
1597</body>
1598</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.