PageSourceSearch

https://woowabros.github.io/woowabros/2021/03/02/search-system.html

html woowabros.github.io collected 2026-10-03 10:10:48 UTC 53,976 bytes, 673 lines download raw bytes

1<!DOCTYPE html>
2<html>
3
4  <head>
5  <meta charset="utf-8">
6  <meta http-equiv="X-UA-Compatible" content="IE=edge">
7  <meta name="viewport" content="width=device-width, initial-scale=1">
8
9  <title>검색을 위한 데이터 다루기 - 우아한형제들 기술 블로그</title>
10  <meta name="description" content="  안녕하세요. 우아한형제들 검색개발팀 정철입니다.">
11  <meta http-equiv="refresh" content="0;url=https://techblog.woowahan.com/change/?key=2021-03-03-search-system.md" />
12  <link rel="canonical" href="https://techblog.woowahan.com/change/?key=2021-03-03-search-system.md" />
13
14  <meta property="fb:pages" content="1012405502198937" />
15  <meta property="fb:app_id" content="596588250503455"/>
16  <meta property="og:title" content="검색을 위한 데이터 다루기 - 우아한형제들 기술 블로그"/>
17  <meta property="og:site_name" content="우아한형제들 기술 블로그"/>
18  <meta property="og:type" content="website"/>
19  <meta property="og:url" content="https://woowabros.github.io/woowabros/2021/03/02/search-system.html"/>
20  <meta property="og:image" content=""/>
21  <meta property="og:description" content="  안녕하세요. 우아한형제들 검색개발팀 정철입니다."/>
22  <meta name="naver-site-verification" content="9b6a9f890dc78f921d5c4b87fdb6ceaaf5549a17" />
23
24  <link rel="stylesheet" href="/css/main.css">
25  <link rel="canonical" href="https://woowabros.github.io/woowabros/2021/03/02/search-system.html">
26  <link rel="alternate" type="application/rss+xml" title="우아한형제들 기술 블로그" href="https://woowabros.github.io/feed.xml">
27  <link rel="shortcut icon" href="https://www.woowahan.com/favicon/favicon.ico">
28  
28<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script>
28
29  
29<script type="text/javascript" src="//cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
29
30</head>
31
32
33  <body>
34
35    
36<div id="fb-root"></div>
37<script>(function(d, s, id) {
38  var js, fjs = d.getElementsByTagName(s)[0];
39  if (d.getElementById(id)) return;
40  js = d.createElement(s); js.id = id;
41  js.src = "//connect.facebook.net/ko_KR/sdk.js#xfbml=1&version=v2.6&appId=596588250503455";
42  fjs.parentNode.insertBefore(js, fjs);
43}(document, 'script', 'facebook-jssdk'));</script>
43
44
45
46    <header class="site-header">
47
48  <div class="wrapper">
49
50    <a class="site-title" href="/">
51      <span class="site-logo">우아한형제들</span>
52      <span class="site-title">기술 블로그</span>
53    </a>
54
55    <nav class="site-nav">
56      <a href="#" class="menu-icon">
57        <svg viewBox="0 0 18 15">
58          <path fill="#424242" d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.031C17.335,0,18,0.665,18,1.484L18,1.484z"/>
59          <path fill="#424242" d="M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0c0-0.82,0.665-1.484,1.484-1.484 h15.031C17.335,6.031,18,6.696,18,7.516L18,7.516z"/>
60          <path fill="#424242" d="M18,13.516C18,14.335,17.335,15,16.516,15H1.484C0.665,15,0,14.335,0,13.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.031C17.335,12.031,18,12.696,18,13.516L18,13.516z"/>
61        </svg>
62      </a>
63
64      <div class="trigger">
65      
66        
67        
68        
69        
70        <li>
71          <a class="page-link" href="/category/interview">
72            interview (3)
73          </a>
74        </li>
75      
76        
77        
78        
79        
80        <li>
81          <a class="page-link" href="/category/tools">
82            tools (23)
83          </a>
84        </li>
85      
86        
87        
88        
89        
90        <li>
91          <a class="page-link" href="/category/nerd">
92            nerd (1)
93          </a>
94        </li>
95      
96        
97        
98        
99        
100        <li>
101          <a class="page-link" href="/category/woowabros">
102            woowabros (38)
103          </a>
104        </li>
105      
106        
107        
108        
109        
110        <li>
111          <a class="page-link" href="/category/study">
112            study (15)
113          </a>
114        </li>
115      
116        
117        
118        
119        
120        <li>
121          <a class="page-link" href="/category/experience">
122            experience (108)
123          </a>
124        </li>
125      
126        
127        
128          
129        
130        
131        
132        
133        <li>
134          <a class="page-link" href="/category/security">
135            security (6)
136          </a>
137        </li>
138      
139        
140        
141        
142        
143        <li>
144          <a class="page-link" href="/category/swift">
145            swift (4)
146          </a>
147        </li>
148      
149        
150        
151        
152        
153        <li>
154          <a class="page-link" href="/category/rnd">
155            rnd (2)
156          </a>
157        </li>
158      
159        
160        
161        
162        
163        <li>
164          <a class="page-link" href="/category/techcourse">
165            techcourse (18)
166          </a>
167        </li>
168      
169        
170        
171        
172        
173        <li>
174          <a class="page-link" href="/category/devrel">
175            devrel (9)
176          </a>
177        </li>
178      
179        
180        
181        
182        
183        <li>
184          <a class="page-link" href="/category/sre">
185            sre (2)
186          </a>
187        </li>
188      
189        
190        
191        
192        
193        <li>
194          <a class="page-link" href="/category/growth">
195            growth (1)
196          </a>
197        </li>
198      
199        
200        
201        
202        
203        <li>
204          <a class="page-link" href="/category/culture">
205            culture (1)
206          </a>
207        </li>
208      
209        <a class="page-link" href="http://woowabros.github.com">Github</a>
210      </div>
211    </nav>
212
213  </div>
214
215</header>
216
217
218    <div class="page-content">
219      <div class="wrapper">
220        <section class="content-layout">
221	<article class="post" itemscope itemtype="https://schema.org/BlogPosting">
222	  <meta itemscope itemprop="mainEntityOfPage"  itemType="https://schema.org/WebPage" itemid="https://woowabros.github.io"/>
223
224      <div itemprop="image" itemscope itemtype="https://schema.org/ImageObject">
225        <meta itemprop="url" content="">
226        <meta itemprop="width" content="800">
227        <meta itemprop="height" content="800">
228      </div>
229
230      <div itemprop="publisher" itemscope itemtype="https://schema.org/Organization">
231        <meta itemprop="name" content="우아한형제들">
232        <div itemprop="logo" itemscope itemtype="https://schema.org/ImageObject">
233          <meta itemprop="url" content="https://woowabros.github.io/logo_220x57.png">
234          <meta itemprop="width" content="220">
235          <meta itemprop="height" content="57">
236        </div>
237      </div>
238
239	  <header class="post-header">
240	    <h1 class="post-title" itemprop="name headline">검색을 위한 데이터 다루기</h1>
241	    
242	    <p class="post-meta"><time datetime="2021-03-02T09:30:00+09:00" itemprop="datePublished dateModified">Mar 2, 2021</time> • <span itemprop="author" itemscope itemtype="https://schema.org/Person"><span itemprop="name">정철</span></span><img src="https://www.gravatar.com/avatar/400ebaf765e049eb24b119f389b85c68?s=50" class="gravatar"></p>
243	  </header>
244	
245	  <div class="post-content" itemprop="articleBody">
246	    <blockquote>
247  <p>안녕하세요. 우아한형제들 검색개발팀 정철입니다.</p>
248
249</blockquote>
250
251<p>배달의민족 검색시스템에서 검색에 사용되는 데이터를 적재하면서 경험했던 어려움과 해결했던 방법을 공유하고자 합니다.</p>
252
253<h2 id="검색시스템이란">검색시스템이란?</h2>
254<p><code class="language-plaintext highlighter-rouge">
254먹고 싶은 음식을 찾는 첫 단계</code> <br />
255배달의민족에서 검색시스템은 주문하고 싶은 가게 또는 메뉴를 빠르게 찾을 수 있도록 도와주는 여러 기능을 개발하고 있습니다.</p>
256<ul>
257  <li>검색 기능</li>
258  <li>인기검색어</li>
259  <li>1인분, 채식 등의 테마 지면 리스팅</li>
260  <li><code class="language-plaintext highlighter-rouge">배달 빠른 순, 배달팁 낮은 순</code>과 같은 정렬과 <code class="language-plaintext highlighter-rouge">최소주문금액, 쿠폰, 예약</code>등의 필터 기능</li>
261</ul>
262
263<div style="text-align: center;">
264  <img src="/img/2021-03-03/s1.jpg" /> <span style="display: block; color:#808080;">검색개발팀이 제공하는 서비스</span>
265</div>
266<p><br /></p>
267
268<p>단순하다고 생각될 수 있는 시스템이지만 실시간으로 가게에 대한 정보를 동기화하기 위해서 십여개의 외부시스템으로부터 실시간 정보를 저장해야 하며, 시간, 위치를 기반으로 복잡한 수식으로 가게의 여러정보를 계산해야하는 복잡한 시스템입니다.<br /><br /></p>
269
270<h2 id="추가되는-서비스-늘어나는-과부하">추가되는 서비스, 늘어나는 과부하</h2>
271<p>검색시스템은 위의 소개했던 것과 같은 서비스를 제공하기 위해서 광고, 메뉴, 리뷰, 통계… 등 십여 개의 데이터를 aws의 SQS를 사용하여 수신하고 이를 검색엔진 <code class="language-plaintext highlighter-rouge">Elasticseach</code>에 저장하고 있습니다.<br /></p>
272<div style="text-align: center;">
273  <img src="/img/2021-03-03/s2.jpg" /><span style="display: block; color:#808080;">데이터 수신 구조</span>
274</div>
275
276<p><br />
277초기 시스템 설계 시 들어오는 데이터의 양이 그렇게 많지 않았고 예측 가능했기 때문에 개별적으로 들어오는 이벤트를 건바이건으로 update request를 만들어서 반영하였습니다.<br /></p>
278<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// 가게 이벤트 수신</span>
279<span class="nd">@SqsListener</span><span class="o">(</span><span class="n">value</span> <span class="o">=</span> <span class="s">"shopSqs"</span><span class="o">,</span> <span class="n">deletionPolicy</span> <span class="o">=</span> <span class="no">NEVER</span><span class="o">)</span>
280<span class="kd">public</span> <span class="kt">void</span> <span class="nf">tagsUpdateListen</span><span class="o">(</span><span class="nd">@Payload</span> <span class="nc">ShopChangeEvent</span> <span class="n">shopChangeEvent</span><span class="o">,</span> <span class="nc">Acknowledgment</span> <span class="n">acknowledgment</span><span class="o">)</span> <span class="o">{</span>
281    <span class="n">shopElasticsearch</span><span class="o">.</span><span class="na">update</span><span class="o">(</span><span class="n">createUpdateRequest</span><span class="o">(</span><span class="n">shopChangeEvent</span><span class="o">));</span>
282<span class="o">}</span>
283<span class="o">...</span>
284<span class="c1">// 리뷰 이벤트 수신</span>
285<span class="nd">@SqsListener</span><span class="o">(</span><span class="n">value</span> <span class="o">=</span> <span class="s">"reviewSqs"</span><span class="o">,</span> <span class="n">deletionPolicy</span> <span class="o">=</span> <span class="no">NEVER</span><span class="o">)</span>
286<span class="kd">public</span> <span class="kt">void</span> <span class="nf">tagsUpdateListen</span><span class="o">(</span><span class="nd">@Payload</span> <span class="nc">ReviewChangeEvent</span> <span class="n">reviewChangeEvent</span><span class="o">,</span> <span class="nc">Acknowledgment</span> <span class="n">acknowledgment</span><span class="o">)</span> <span class="o">{</span>
287    <span class="n">shopElasticsearch</span><span class="o">.</span><span class="na">update</span><span class="o">(</span><span class="n">createUpdateRequest</span><span class="o">(</span><span class="n">reviewChangeEvent</span><span class="o">));</span>
288<span class="o">}</span>
289</code></pre></div></div>
290<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">POST</span><span class="w"> </span><span class="err">baemin-search/_update/</span><span class="mi">1234567</span><span class="w">
291</span><span class="p">{</span><span class="w">
292  </span><span class="nl">"doc"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
293    </span><span class="nl">"shopName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"배민의 맛집가게"</span><span class="p">,</span><span class="w">
294    </span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"송파구 방이동 44-2"</span><span class="w">
295  </span><span class="p">}</span><span class="w">
296</span><span class="p">}</span><span class="w">
297</span></code></pre></div></div>
298<p>하지만 가게가 증가함에 따라 가게가 오픈하는 시간대나 주문이 많아서 메뉴 품절 등의 상황이 많이 발생하는 경우에는 이벤트를 많이 수신하게 되었고 그에 따라 모든 부담이 Elasticsearch에 집중되면서 Elasticsearch Writer Thread Pool이 가득 차서 request가 rejected 되는 문제와 Disk I/O 문제로 cpu 수치가 기하급수적으로 상승하는 문제가 발생하였습니다.<br /></p>
299
300<p>이런 상황 속에 현재 상태를 유지하기도 어려워졌고 더불어 <code class="language-plaintext highlighter-rouge">배달 빠른 순, 배달팁 낮은 순</code>정렬/필터와 같이 사용자에게 편의를 줄 수 있는 여러 feature를 저희 시스템 문제로 개발이 지연되는 문제가 발생하였습니다.<br /><br /></p>
301
302<h2 id="원인-분석">원인 분석</h2>
303<h3 id="문제-1-잦은-변경에-의한-disk-io-이슈">문제 1. 잦은 변경에 의한 Disk I/O 이슈</h3>
304<p>Elasticsearch의 Index는 Lucene 기반의 작은 segment 파일로 구성되어 있습니다. 이 segment는 검색 요청을 빠르게 처리하기 위해서 여러 개로 생성이 되어있습니다. 하지만 이 segment는 속도, 성능, 동시성 이슈, 압축 등의 이유로 immutable 한 속성을 가지고 있기 때문에 Index에 대한 업데이트 요청이 오게 되면 기존 segment에 mark만 하고 새로운 segment를 생성하여 데이터를 기록하는 동작을 진행합니다.</p>
305<div style="text-align: center;">
306  <img src="/img/2021-03-03/s3.jpg" /><span style="display: block; color:#808080;">이미지 출처 구글</span>
307</div>
308<p><br /></p>
309
310<p>저희에 문제는 바로 여기에 있었습니다. 잦은 업데이트로 인해 다량의 segment가 생성되면서 Disk I/O가 증가하게 되었고 그 결과 cpu가 동반 상승했으며, 삭제 대상 segment를 지우고 단편화 해소를 위해 동작하는 force merge 시에도 부하가 발생하는 문제가 확인되었습니다. <br /></p>
311<div style="text-align: center;">
312  <img src="/img/2021-03-03/s4.jpg" /><span style="display: block; color:#808080;">문제의 cpu 상태</span>
313</div>
314<p><br /></p>
315
316<h3 id="문제-2-건바이건-요청으로-인한-thread-pool-초과">문제 2. 건바이건 요청으로 인한 thread pool 초과</h3>
317<p>가게에 해당하는 속성 변경 이벤트가 몰리는 시간대에는 최대 수십만 ê±´ 이상의 이벤트가 발생합니다. 이 이벤트를 건바이건으로 업데이트를 하면서 Elasticsearch의 write thread pool을 모두 점유하고 queue까지 가득 차게 되면서 일부 이벤트가 정상적으로 업데이트 되지 못하고 rejected 되는 경우가 ë¹
317ˆë²ˆí•˜ê²Œ 발생하였습니다.</p>
318<div style="text-align: center;">
319  <img src="/img/2021-03-03/s5.jpg" /><span style="display: block; color:#808080;">rejected request 개수</span>
320</div>
321<p><br /><br /></p>
322
323<h2 id="정답은-언제나-기본에-있다">정답은 언제나 기본에 있다</h2>
324<blockquote>
325  <p>To optimize insert speed, combine many small operations into a single large operation. Ideally, you make a single connection, send the data for many new rows at once, and delay all index updates and consistency checking until the very end.<br />
326출처 : https://dev.mysql.com/doc/refman/8.0/en/insert-optimization.html</p>
327
328  <p><code class="language-plaintext highlighter-rouge">insert 성능을 높이기 위해서는 작은 여러 연산을 큰 하나의 연산으로 만들어라.</code> 라는 말을 위의 mysql doc에서 볼 수 있듯이 그렇게 배웠고 그렇게 개발을 진행했었습니다.<br /></p>
329</blockquote>
330
331<p>Elasticsearch 또한 같은 개념을 가지고 있으며 단건 request 수행 보다 Bulk request의 효능이 큰 것은 여러 벤치마킹을 통해서도 입증이 되었습니다.<br /></p>
332
333<p>그럼 우리도 이 문제를 해결하기 위해 bulk로 동작을 시키는 방법을 적용해보기로 했습니다. 하지만 언제나 문제는 쉽게 해결되지 않듯이 역시 문제가 있었습니다.
334검색의 현재 구조는 모든 이벤트를 별도의 sqs를 listen 하여 받고 있기 때문에 서로 다른 sqs에서 데이터를 받아서 처리하기에는 무리가 있었습니다.</p>
335
336<p>그러다 Elasticsearch의 resthighlevelclient java api에서 <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-docs-bulk-processor.html">Bulk processor</a> 라는 기능을 사용하여 request를 모아서 업데이트하는 기능을 발견하게 되었습니다.
337Bulk Processor는 하나의 독립적인 프로세스를 실행시키고 request를 buffer에 모아서 정해진 시간, 크기 등에 맞ê²
337Œ Elasticsearch에 데이터를 Bulk request를 만들어서 flush하도록 동작하고 있습니다.</p>
338<div style="text-align: center;">
339  <img src="/img/2021-03-03/s6.jpg" /><span style="display: block; color:#808080;">Bulk processor 구조</span>
340</div>
341<p><br /></p>
342
343<p>이걸 사용해서 저희의 문제를 풀어보기로 했으며 있는 그대로 사용하지 않고 조금 더 나은 성능을 만들기 위해 튜닝도 함께 진행했습니다.<br /></p>
344
345<h3 id="튜닝-1">튜닝 1.</h3>
346<p>기존 Bulk processor는 DocWriteRequest를 담는 List를 가지고 있는 BulkRequest라는 Buffer용 객체를 사용하여 데이터를 담고 시간, 사이즈 등의 flush 타이밍이 되면 해당 버퍼를 BulkProcessorHandler를 통해 전달하고 Elasticsearch에 Bulk request를 요청하는 구조로 되어있습니다.</p>
347<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// BulkRequest (Buffer 역할을 하는 객체)</span>
348<span class="kd">public</span> <span class="kd">class</span> <span class="nc">BulkRequest</span> <span class="kd">extends</span> <span class="nc">ActionRequest</span> <span class="kd">implements</span> <span class="nc">CompositeIndicesRequest</span><span class="o">,</span> <span class="nc">WriteRequest</span><span class="o">&lt;</span><span class="nc">BulkRequest</span><span class="o">&gt;</span> <span class="o">{</span>
349
350    <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span> <span class="no">REQUEST_OVERHEAD</span> <span class="o">=</span> <span class="mi">50</span><span class="o">;</span>
351
352    <span class="cm">/**
353     * Requests that are part of this request. It is only possible to add things that are both {@link ActionRequest}s and
354     * {@link WriteRequest}s to this but java doesn't support syntax to declare that everything in the array has both types so we declare
355     * the one with the least casts.
356     */</span>
357    <span class="kd">final</span> <span class="nc">List</span><span class="o">&lt;</span><span class="nc">DocWriteRequest</span><span class="o">&lt;?&gt;&gt;</span> <span class="n">requests</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ArrayList</span><span class="o">&lt;&gt;();</span>
358    <span class="o">..</span>
359    <span class="o">..</span>
360    <span class="o">..</span>
361
362    <span class="cm">/**
363     * Adds a list of requests to be executed. Either index or delete requests.
364     */</span>
365    <span class="kd">public</span> <span class="nc">BulkRequest</span> <span class="nf">add</span><span class="o">(</span><span class="nc">DocWriteRequest</span><span class="o">&lt;?&gt;...</span> <span class="n">requests</span><span class="o">)</span> <span class="o">{</span>
366        <span class="k">for</span> <span class="o">(</span><span class="nc">DocWriteRequest</span><span class="o">&lt;?&gt;</span> <span class="n">request</span> <span class="o">:</span> <span class="n">requests</span><span class="o">)</span> <span class="o">{</span>
367            <span class="n">add</span><span class="o">(</span><span class="n">request</span><span class="o">,</span> <span class="kc">null</span><span class="o">);</span>
368        <span class="o">}</span>
369        <span class="k">return</span> <span class="k">this</span><span class="o">;</span>
370    <span class="o">}</span>
371<span class="o">}</span>
372
373<span class="c1">// BufferProcessor add메소드를 통해 BulkRequest에 request를 만들어서 담는 부분</span>
374<span class="n">bulkProcessor</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="k">new</span> <span class="nc">IndexRequest</span><span class="o">(</span><span class="s">"twitter"</span><span class="o">,</span> <span class="s">"_doc"</span><span class="o">,</span> <span class="s">"1"</span><span class="o">).</span><span class="na">source</span><span class="o">(</span><span class="cm">/* your doc here */</span><span class="o">));</span>
375<span class="n">bulkProcessor</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="k">new</span> <span class="nc">DeleteRequest</span><span class="o">(</span><span class="s">"twitter"</span><span class="o">,</span> <span class="s">"_doc"</span><span class="o">,</span> <span class="s">"2"</span><span class="o">));</span>
376
377<span class="c1">// BulkProcessor Handler</span>
378<span class="kd">public</span> <span class="kt">void</span> <span class="nf">execute</span><span class="o">(</span><span class="nc">BulkRequest</span> <span class="n">bulkRequest</span><span class="o">,</span> <span class="kt">long</span> <span class="n">executionId</span><span class="o">)</span> <span class="o">{</span>
379    <span class="nc">Runnable</span> <span class="n">toRelease</span> <span class="o">=</span> <span class="o">()</span> <span class="o">-&gt;</span> <span class="o">{};</span>
380    <span class="kt">boolean</span> <span class="n">bulkRequestSetupSuccessful</span> <span class="o">=</span> <span class="kc">
380false</span><span class="o">;</span>
381    <span class="k">try</span> <span class="o">{</span>
382        <span class="n">listener</span><span class="o">.</span><span class="na">beforeBulk</span><span class="o">(</span><span class="n">executionId</span><span class="o">,</span> <span class="n">bulkRequest</span><span class="o">);</span>
383        <span class="n">semaphore</span><span class="o">.</span><span class="na">acquire</span><span class="o">();</span>
384        <span class="n">toRelease</span> <span class="o">=</span> <span class="nl">semaphore:</span><span class="o">:</span><span class="n">release</span><span class="o">;</span>
385        <span class="nc">CountDownLatch</span> <span class="n">latch</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">CountDownLatch</span><span class="o">(</span><span class="mi">1</span><span class="o">);</span>
386        <span class="n">retry</span><span class="o">.</span><span class="na">withBackoff</span><span class="o">(</span><span class="n">consumer</span><span class="o">,</span> <span class="n">bulkRequest</span><span class="o">,</span> <span class="k">new</span> <span class="nc">ActionListener</span><span class="o">&lt;</span><span class="nc">BulkResponse</span><span class="o">&gt;()</span> <span class="o">{</span>
387            <span class="nd">@Override</span>
388            <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onResponse</span><span class="o">(</span><span class="nc">BulkResponse</span> <span class="n">response</span><span class="o">)</span> <span class="o">{</span>
389                <span class="k">try</span> <span class="o">{</span>
390                    <span class="n">listener</span><span class="o">.</span><span class="na">afterBulk</span><span class="o">(</span><span class="n">executionId</span><span class="o">,</span> <span class="n">bulkRequest</span><span class="o">,</span> <span class="n">response</span><span class="o">);</span>
391                <span class="o">}</span> <span class="k">finally</span> <span class="o">{</span>
392                    <span class="n">semaphore</span><span class="o">.</span><span class="na">release</span><span class="o">();</span>
393                    <span class="n">latch</span><span class="o">.</span><span class="na">countDown</span><span class="o">();</span>
394                <span class="o">}</span>
395            <span class="o">}</span>
396
397            <span class="nd">@Override</span>
398            <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onFailure</span><span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
399                <span class="k">try</span> <span class="o">{</span>
400                    <span class="n">listener</span><span class="o">.</span><span class="na">afterBulk</span><span class="o">(</span><span class="n">executionId</span><span class="o">,</span> <span class="n">bulkRequest</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
401                <span class="o">}</span> <span class="k">finally</span> <span class="o">{</span>
402                    <span class="n">semaphore</span><span class="o">.</span><span class="na">release</span><span class="o">();</span>
403                    <span class="n">latch</span><span class="o">.</span><span class="na">countDown</span><span class="o">();</span>
404                <span class="o">}</span>
405            <span class="o">}</span>
406        <span class="o">});</span>
407        <span class="n">bulkRequestSetupSuccessful</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
408        <span class="k">if</span> <span class="o">(</span><span class="n">concurrentRequests</span> <span class="o">==</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
409            <span class="n">latch</span><span class="o">.</span><span class="na">await</span><span class="o">();</span>
410        <span class="o">}</span>
411    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">InterruptedException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
412        <span class="nc">Thread</span><span class="o">.</span><span class="na">currentThread</span><span class="o">().</span><span class="na">interrupt</span><span class="o">();</span>
413        <span class="n">logger</span><span class="o">.</span><span class="na">info</span><span class="o">(()</span> <span class="o">-&gt;</span> <span class="k">new</span> <span class="nc">ParameterizedMessage</span><span class="o">(</span><span class="s">"Bulk request {} has been cancelled."</span><span class="o">,</span> <span class="n">executionId</span><span class="o">),</span> <span class="n">e</span><span class="o">);</span>
414        <span class="n">listener</span><span class="o">.</span><span class="na">afterBulk</span><span class="o">(</span><span class="n">executionId</span><span class="o">,</span> <span class="n">bulkRequest</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
415    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
416        <span class="n">logger</span><span class="o">.</span><span class="na">warn</span><span class="o">(()</span> <span class="o">-&gt;</span> <span class="k">new</span> <span class="nc">ParameterizedMessage</span><span class="o">(</span><span class="s">"Failed to execute bulk request {}."</span><span class="o">,</span> <span class="n">executionId</span><span class="o">),</span> <span class="n">e</span><span class="o">);</span>
417        <span class="n">listener</span><span class="o">.</span><span class="na">afterBulk</span><span class="o">(</span><span class="n">executionId</span><span class="o">,</span> <span class="n">bulkRequest</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
418    <span class="o">}</span> <span class="k">finally</span> <span class="o">{</span>
419        <span class="k">if</span> <span class="o">(</span><span class="n">bulkRequestSetupSuccessful</span> <span class="o">==</span> <span class="kc">false</span><span class="o">)</span> <span class="o">{</span>  <span class="c1">// if we fail on client.bulk() release the semaphore</span>
420            <span class="n">toRelease</span><span class="o">.</span><span class="na">run</span><span class="o">();</span>
421        <span class="o">}</span>
422    <span class="o">}</span>
423<span class="o">}</span>
424</code></pre></div></div>
425
426<p>저희는 이 부분에서 수신한 이벤트를 Bulk processor에 add 할 때 Update request를 만들어서 buffer에 밀어 넣는 부분이 번거로웠고 같은 index에 동일한 Update Request 규칙을 만들때도 모든 이벤트 수신부분에 일괄 적용하기 어려웠습니다. 그래서 buffer 역할을 하는 BulkRequest를 걷어내고 Update Request를 만들기 위한 Shop이라는 객체를 만들고 이 객체들을 담을 수 있는 Map으로 대체하였습니다<br /><br /> 
427각 이벤트 수신위치에서는 Bulk Processor에 add 할 때 각 이벤트별 변경 부분만 전달할 수 있도록 별도의 객체(ShopMenu, ShopReview..)들을 구성하였고, 이 값들을 전달받은 Bulk processor는 Shop객체로 전환하여 buffer에 저장을 하도록 구성하였습니다.(<span style="color: red; weight: bold;">이 부분 코드 및 활용은 튜닝2에서 다룰 예정</span>) 그리고 별도의 serializer 정책을 적용해서 flush 주기에 BulkProcessorHandler에 데이터를 전달하여 필요한 부분만 업데이트 될 수 있도록 BulkRequest를 생성하는 등의 작업을 진행하였습니다.<br /><br /></p>
428
429<p>Bulk processor 도입으로 인한 효과를 이론적으로 생각했던 부분이었고 테스트를 통해 검증도 해봤지만 실제 운영에서도 어느 정도 효과가 발생될지 궁금했었습니다. 기대감을 가지고 운영에서 확인해봤습니다.<br /></p>
430
431<p>운영 데이터 기준 이벤트 수신 건수는 총 <em>9,261,187개</em> 였습니다.<br /></p>
432
433<p>Bulk processor 도입전에는 모든 이벤트를 단건으로 처리하였기 때문에 총 update request는 이벤트 수신건수와 동일하게 9,261,185개 였습ë
433‹ˆë‹¤.<br />
434Bulk processor 도입후에는 update request를 모아 Bulk request를 만들면서 request건수는 233,949건 발생하였습니다.</p>
435
436<blockquote>
437  <p>도입 전 : 9,261,185건<br />
438도입 후 : 233,949건</p>
439
440</blockquote>
441
442<p>Elasticsearch에 요청하는 request 건수가 <span style="color:#d23c3c; font-weight:700"> 총 97.5% </span> 감소하였습니다.
443이로 인해 write thread pool 감소로 많은 이벤트 수신 시 발생하던 rejected가 해소되었고, bulk로 단일 요청 대비 응답 대기 시간 및 네트워크 부하 감소 효과를 받았습니다.</p>
444
445<h3 id="튜닝-2">튜닝 2.</h3>
446<p>튜닝 1에서 얻은 효과도 좋았지만 조금 더 성능을 올려보고 싶었습니다. Bulk request가 single request보다 효능은 있지만 Bulk의 사이즈가 커질수록 오히려 Elasticsearch에 부담을 줄 수 있기 때문에 Bulk에 담기는 request를 더 줄여보고 싶었습니다.</p>
447
448<p>데이터를 가만히 보니 검색에서 사용하는 모든 데이터는 shopId를 기준으로 업데이트 쿼리가 만들어졌습니다. 그럼 request를 add 하는 순간에 동일한 가게에 대한 request가 존재하면 변경되는 부분만 기존 request에 넣어주면 되겠다는 생각을 가지게 되었습니다.<br /></p>
449
450<div style="text-align: center;">
451  <img src="/img/2021-03-03/s11.jpg" /> <span style="display: block; color:#808080;">가게 중심으로 request 단일화 과정</span>
452</div>
453<p><br /><br /></p>
454
455<p>그래서 다음과 같이 Bulk processor에 buffer역할을 하는 map의 키를 shopId로 부여하고 value를 Shop 객체로 부여를 하였고 각 이벤트를 통해 들어오는 요청을 하나의 Shop으로 묶어서 Bulk request를 만들도록 진행하였습니다.</p>
456<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">#</span> <span class="n">실제</span> <span class="n">코드는</span> <span class="n">더</span> <span class="n">복잡하여</span> <span class="n">예시</span> <span class="n">코드를</span> <span class="n">기재합니다</span><span class="o">.</span>
457
458<span class="cm">/**
459 * data는 유연하게 받을 수 있도록 제네릭을 사용하였고, 실제 검색에서는 Shop객체를 받고 있다.
460 * acknowledgments는 수신된 이벤트들에 대한 정상 처리 여부에 따라 sqs에 ack 처리를 하기 위해 보관  
461*/</span>
462<span class="nd">@Getter</span>
463<span class="nd">@Setter</span>
464<span class="nd">@NoArgsConstructor</span>
465<span class="kd">public</span> <span class="kd">class</span> <span class="nc">BulkData</span><span class="o">&lt;</span><span class="no">T</span> <span class="kd">extends</span> <span class="nc">IndexDomain</span><span class="o">&gt;</span> <span class="o">{</span>
466    <span class="kd">private</span> <span class="no">T</span> <span class="n">data</span><span class="o">;</span>
467    <span class="kd">private</span> <span class="nc">Set</span><span class="o">&lt;</span><span class="nc">Acknowledgment</span><span class="o">&gt;</span> <span class="n">acknowledgments</span><span class="o">;</span>
468
469    <span class="nd">@Builder</span>
470    <span class="kd">public</span> <span class="nf">BulkData</span><span class="o">(</span><span class="no">T</span> <span class="n">data</span><span class="o">,</span> <span class="nc">Set</span><span class="o">&lt;</span><span class="nc">Acknowledgment</span><span class="o">&gt;</span> <span class="n">acknowledgments</span><span class="o">)</span> <span class="o">{</span>
471        <span class="k">this</span><span class="o">.</span><span class="na">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">;</span>
472        <span class="k">this</span><span class="o">.</span><span class="na">acknowledgments</span> <span class="o">=</span> <span class="n">acknowledgments</span><span class="o">;</span>
473    <span class="o">}</span>
474<span class="o">}</span>
475
476<span class="nd">@Getter</span>
477<span class="nd">@Setter</span>
478<span class="kd">public</span> <span class="kd">class</span> <span class="nc">Shop</span> <span class="kd">extends</span> <span class="nc">IndexDomain</span> <span class="o">{</span>
479    <span class="kd">private</span> <span class="nc">Long</span> <span class="n">shopNumber</span><span class="o">;</span>
480    <span class="kd">private</span> <span class="nc">List</span><span class="o">&lt;</span><span class="nc">Menu</span><span class="o">&gt;</span> <span class="n">menus</span><span class="o">;</span>
481    <span class="kd">private</span> <span class="nc">String</span> <span class="n">content</span><span class="o">;</span>
482    <span class="kd">private</span> <span class="nc">String</span> <span class="n">subject</span><span class="o">;</span>
483    
484    <span class="nd">@Builder</span>
485    <span class="kd">public</span> <span class="nf">Shop</span><span class="o">(</span><span class="nc">Long</span> <span class="n">shopNumber</span><span class="o">,</span> <span class="nc">List</span><span class="o">&lt;</span><span class="nc">Menu</span><span class="o">&gt;</span> <span class="n">menus</span><span class="o">,</span> <span class="nc">String</span> <span class="n">content</span><span class="o">,</span> <span class="nc">String</span> <span class="n">subject</span><span class="o">)</span> <span class="o">{</span>
486        <span class="k">this</span><span class="o">.</span><span class="na">shopNumber</span> <span class="o">=</span> <span class="n">shopNumber</span><span class="o">;</span>
487        <span class="k">this</span><span class="o">.</span><span class="na">menus</span> <span class="o">=</span> <span class="n">menus</span><span class="o">;</span>
488        <span class="k">this</span><span class="o">.</span><span class="na">content</span> <span class="o">=</span> <span class="n">content</span><span class="o">;</span>
489        <span class="k">this</span><span class="o">.</span><span class="na">subject</span> <span class="o">=</span> <span class="n">subject</span><span class="o">;</span>
490    <span class="o">}</span>
491<span class="o">}</span>
492
493<span class="c1">// 이벤트 변경정보를 가지는 객체의 공통 추상 클래스</span>
494<span class="kd">public</span> <span class="kd">abstract</span> <span class="kd">class</span> <span class="nc">ShopUpdatableDomain</span> <span class="kd">extends</span> <span class="nc">UpdatableDomain</span><span class="o">&lt;</span><span class="nc">Shop</span><span class="o">&gt;</span> <span class="o">{</span>
495
496    <span class="nd">@Override</span>
497    <span class="kd">public</span> <span class="nc">Shop</span> <span class="nf">convert</span><span class="o">()</span> <span class="o">{</span>
498        <span class="nc">Shop</span> <span class="n">shop</span> <span class="o">=</span> <span class="nc">Shop</span><span class="o">.</span><span class="na">builder</span><span class="o">().</span><span class="na">build</span><span class="o">();</span>
499        <span class="n">convert</span><span class="o">(</span><span class="n">shop</span><span class="o">);</span>
500        <span class="k">return</span> <span class="n">shop</span><span class="o">;</span>
501    <span class="o">}</span>
502<span class="o">}</span>
503
504<span class="c1">// 메뉴 변경정보를 가지고 있는 ShopMenu</span>
505<span class="nd">@Getter</span>
506<span class="nd">@Setter</span>
507<span class="nd">@NoArguments</span>
508<span class="kd">public</span> <span class="kd">class</span> <span class="nc">ShopMenu</span> <span class="kd">extends</span> <span class="nc">ShopUpdatableDomain</span> <span class="o">{</span>
509    <span class="kd">private</span> <span class="nc">Long</span> <span class="n">shopNumber</span><span class="o">;</span>
510    <span class="kd">private</span> <span class="nc">List</span><span class="o">&lt;</span><span class="nc">Menu</span><span class="o">&gt;</span> <span class="n">menus</span><span class="o">;</span>
511    
512    <span class="nd">@Builder</span>
513    <span class="kd">public</span> <span class="nf">ShopMenu</span><span class="o">(</span><span class="nc">Long</span> <span class="n">shopNumber</span><span class="o">,</span> <span class="nc">List</span><span class="o">&lt;</span><span class="nc">Menu</span><span class="o">&gt;</span> <span class="n">menus</span><span class="o">)</span> <span class="o">{</span>
514        <span class="k">this</span><span class="o">.</span><span class="na">shopNumber</span> <span class="o">=</span> <span class="n">shopNumber</span><span class="o">;</span>
515        <span class="k">this</span><span class="o">.</span><span class="na">menus</span> <span class="o">=</span> <span class="n">menus</span><span class="o">;</span>
516    <span class="o">}</span>
517
518    <span class="nd">@Override</span>
519    <span class="kd">public</span> <span class="kt">void</span> <span class="nf">convert</span><span class="o">(</span><span class="nc">Shop</span> <span class="n">shop</span><span class="o">)</span> <span class="o">{</span>
520        <span class="n">shop</span><span class="o">.</span><span class="na">setShopNumber</span><span class="o">(</span><span class="n">shopNumber</span><span class="o">);</span>
521        <span class="n">shop</span><span class="o">.</span><span class="na">setMenus</span><span class="o">(</span><span class="n">menus</span><span class="o">);</span>
522    <span class="o">}</span>
523<span class="o">}</span>
524
525<span class="c1">// 리뷰 변경정보를 가지고 있는 ShopReview</span>
526<span class="nd">@Getter</span>
527<span class="nd">@Setter</span>
528<span class="nd">@NoArguments</span>
529<span class="kd">public</span> <span class="kd">class</span> <span class="nc">ShopReview</span> <span class="kd">extends</span> <span class="nc">ShopUpdatableDomain</span> <span class="o">{</span>
530    <span class="kd">private</span> <span class="nc">Long</span> <span class="n">shopNumber</span><span class="o">;</span>
531    <span class="kd">private</span> <span class="nc">String</span> <span class="n">content</span><span class="o">;</span>
532    <span class="kd">private</span> <span class="nc">String</span> <span class="n">subject</span><span class="o">;</span>
533
534    <span class="nd">@Builder</span>
535    <span class="kd">public</span> <span class="nf">ShopReview</span><span class="o">(</span><span class="nc">Long</span> <span class="n">shopNumber</span><span class="o">,</span> <span class="nc">String</span> <span class="n">content</span><span class="o">,</span> <span class="nc">String</span> <span class="n">subject</span><span class="o">)</span> <span class="o">{</span>
536        <span class="k">this</span><span class="o">.</span><span class="na">shopNumber</span> <span class="o">=</span> <span class="n">shopNumber</span><span class="o">;</span>
537        <span class="k">this</span><span class="o">.</span><span class="na">content</span> <span class="o">=</span> <span class="n">content</span><span class="o">;</span>
538        <span class="k">this</span><span class="o">.</span><span class="na">subject</span> <span class="o">=</span> <span class="n">subject</span><span class="o">;</span>
539    <span class="o">}</span>
540
541    <span class="nd">@Override</span>
542    <span class="kd">public</span> <span class="kt">void</span> <span class="nf">convert</span><span class="o">(</span><span class="nc">Shop</span> <span class="n">shop</span><span class="o">)</span> <span class="o">{</span>
543        <span class="n">shop</span><span class="o">.</span><span class="na">setShopNumber</span><span class="o">(</span><span class="n">shopNumber</span><span class="o">);</span>
544        <span class="n">shop</span><span class="o">.</span><span class="na">setContent</span><span class="o">(</span><span class="n">content</span><span class="o">);</span>
545        <span class="n">shop</span><span class="o">.</span><span class="na">setSubject</span><span class="o">(</span><span class="n">subject</span><span class="o">);</span>
546    <span class="o">}</span>
547<span class="o">}</span>
548
549<span class="c1">// 이벤트 수신과 Bulk processor에 이벤트 내용 전달</span>
550<span class="nd">@SqsListener</span><span class="o">(</span><span class="n">value</span> <span class="o">=</span> <span class="s">"menu-sqs"</span><span class="o">,</span> <span class="n">deletionPolicy</span> <span class="o">=</span> <span class="no">NEVER</span><span class="o">)</span>
551<span class="kd">public</span> <span class="kt">void</span> <span class="nf">shopMenuV3UpdateListen</span><span class="o">(</span><span class="nd">@Payload</span> <span class="nc">ShopMenuEvent</span> <span class="n">eventDto</span><span class="o">,</span> <span class="nc">Acknowledgment</span> <span class="n">acknowledgment</span><span class="o">)</span> <span class="o">{</span>
552    <span class="n">bulkProcessor</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="nc">BulkProcessorBucket</span><span class="o">.&lt;</span><span class="nc">ShopMenu</span><span class="o">&gt;</span><span class="n">builder</span><span class="o">()</span>
553        <span class="o">.</span><span class="na">data</span><span class="o">(</span><span class="nc">ShopMenu</span><span class="o">.</span><span class="na">builder</span><span class="o">()</span>
554            <span class="o">.</span><span class="na">shopNumber</span><span class="o">(</span><span class="n">eventDto</span><span class="o">.</span><span class="na">getShopId</span><span class="o">())</span>
555            <span class="o">.</span><span class="na">menus</span><span class="o">(</span><span class="n">eventDto</span><span class="o">.</span><span class="na">getMenus</span><span class="o">())</span>
556        <span class="o">.</span><span class="na">build</span><span class="o">())</span>
557    <span class="o">.</span><span class="na">build</span><span class="o">());</span>
558<span class="o">}</span>
559<span class="o">...</span>
560<span class="nd">@SqsListener</span><span class="o">(</span><span class="n">value</span> <span class="o">=</span> <span class="s">"reviewSqs"</span><span class="o">,</span> <span class="n">deletionPolicy</span> <span class="o">=</span> <span class="no">NEVER</span><span class="o">)</span>
561<span class="kd">public</span> <span class="kt">void</span> <span class="nf">tagsUpdateListen</span><span class="o">(</span><span class="nd">@Payload</span> <span class="nc">ReviewChangeEvent</span> <span class="n">reviewChangeEvent</span><span class="o">,</span> <span class="nc">Acknowledgment</span> <span class="n">acknowledgment</span><span class="o">)</span> <span class="o">{</span>
562    <span class="n">bulkProcessor</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="nc">BulkProcessorBucket</span><span class="o">.&lt;</span><span class="nc">ShopReview</span><span class="o">&gt;</span><span class="n">builder</span><span class="o">()</span>
563        <span class="o">.</span><span class="na">data</span><span class="o">(</span><span class="nc">ShopReview</span><span class="o">.</span><span class="na">builder</span><span class="o">()</span>
564            <span class="o">.</span><span class="na">shopNumber</span><span class="o">(</span><span class="n">eventDto</span><span class="o">.</span><span class="na">getShopId</span><span class="o">())</span>
565            <span class="o">.</span><span class="na">content</span><span class="o">(</span><span class="n">eventDto</span><span class="o">.</span><span class="na">getContent</span><span class="o">())</span>
566            <span class="o">.</span><span class="na">subject</span><span class="o">(</span><span class="n">eventDto</span><span class="o">.</span><span class="na">getSubject</span><span class="o">())</span>
567        <span class="o">.</span><span class="na">build</span><span class="o">())</span>
568    <span class="o">.</span><span class="na">build</span><span class="o">());</span>
569<span class="o">}</span>
570
571
572<span class="c1">// bulk processor add 코드</span>
573<span class="nd">@Slf4j</span>
574<span class="kd">public</span> <span class="kd">class</span> <span class="nc">BulkProcessor</span><span class="o">&lt;</span><span class="no">S</span> <span class="kd">extends</span> <span class="nc">IndexDomain</span><span class="o">&gt;</span> <span class="kd">implements</span> <span class="nc">Closeable</span> <span class="o">{</span>
575
576    <span class="kd">private</span> <span class="nc">Map</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">,</span> <span class="nc">BulkData</span><span class="o">&lt;</span><span class="no">S</span><span class="o">&gt;&gt;</span> <span class="n">datas</span><span class="o">;</span>
577    <span class="o">...</span>
578    <span class="o">..</span>
579    <span class="o">...</span>
580    <span class="kd">private</span> <span class="kd">synchronized</span> <span class="kt">void</span> <span class="nf">addData</span><span class="o">(</span><span class="nc">BulkProcessorBucket</span> <span class="n">request</span><span class="o">)</span> <span class="o">{</span>
581        <span class="nc">String</span> <span class="n">id</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="na">getData</span><span class="o">().</span><span class="na">idValue</span><span class="o">();</span>
582
583        <span class="c1">// bulk buffer안에 동일한 가게가 존재하는지 조회</span>
584        <span class="k">if</span> <span class="o">(</span><span class="k">this</span><span class="o">.</span><span class="na">datas</span><span class="o">.</span><span class="na">containsKey</span><span class="o">(</span><span class="n">id</span><span class="o">))</span> <span class="o">{</span>
585            <span class="nc">BulkData</span><span class="o">&lt;</span><span class="no">S</span><span class="o">&gt;</span> <span class="n">bulkData</span> <span class="o">=</span> <span class="k">this</span><span class="o">.</span><span class="na">datas</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">id</span><span class="o">);</span>
586            
587            <span class="c1">// 들어온 이벤트 데이터를 기존 shop 변경분 정보의 매핑</span>
588            <span class="n">request</span><span class="o">.</span><span class="na">getData</span><span class="o">().</span><span class="na">convert</span><span class="o">(</span><span class="n">bulkData</span><span class="o">.</span><span class="na">getData</span><span class="o">());</span>
589            <span class="k">return</span><span class="o">;</span>
590        <span class="o">}</span>
591        <span class="k">this</span><span class="o">.</span><span class="na">datas</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">id</span><span class="o">,</span> <span class="nc">BulkData</span><span class="o">.&lt;</span><span class="no">S</span><span class="o">&gt;</span><span class="n">builder</span><span class="o">()</span>
592            <span class="o">.</span><span class="na">data</span><span class="o">((</span><span class="no">S</span><span class="o">)</span> <span class="n">request</span><span class="o">.</span><span class="na">getData</span><span class="o">().</span><span class="na">convert</span><span class="o">())</span>
593            <span class="o">.</span><span class="na">build</span><span class="o">());</span>
594    <span class="o">}</span>
595    <span class="o">...</span>
596    <span class="o">..</span>
597    <span class="o">.</span>
598<span class="o">}</span>
599</code></pre></div></div>
600<p><br />
601해당 튜닝 포인트도 효과가 있었습니다. Bulk request를 만들어서 flush하는 주기에 같은 가게 아이디로 묶이는 이벤트 종류별 건수와 횟수에 대한 정보는 아래와 같이 적지 않은 수치를 볼 수 있었으며 실제로 이에 대한 효과도 볼 수 있었습니다.</p>
602<div style="text-align: center;">
603  <img src="/img/2021-03-03/s7.jpg" /><span style="display: block; color:#c2b4b4;"> request 군집 지표</span>
604</div>
605<p><br /><br /></p>
606
607<h2 id="성능-테스트-및-실제-운영-리소스-변화">성능 테스트 및 실제 운영 리소스 변화</h2>
608<h3 id="성능테스트-결과">성능테스트 결과</h3>
609<div style="text-align: center;">
610  <img src="/img/2021-03-03/s8.jpg" /><span style="display: block; color:#c2b4b4;"> 27만건 무작위 이벤트 테스트 결과</span>
611</div>
612<p><br /></p>
613
614<p>27만 건의 데이터를 무작위로 전송했고 리소스 변화량을 체크해봤습니다.
615CPU 사용량은 15% 이상 감소되었고 request 단위가 작아지면서 write complete 수도 30배 정도 감소하는 효과를 얻었습니다.<br /></p>
616
617<h3 id="운영-결과">운영 결과</h3>
618<p>실제 운영에 적용 결과 위의 처음 문제가 발생하던 시기에 피크시간 기준 61%를 육박하던 cpu가 비교 대상 기준 이벤트가 수십만건이 더 추가되었음에도 불구하고 최대 29%정도의 안정적인 상태를 유지하고 있습니다.</p>
619<div style="text-align: center;">
620  <img src="/img/2021-03-03/s10.jpg" /> <span style="display: block; color:#c2b4b4;">안정된 시스템</span>
621</div>
622<p><br /><br /></p>
623
624<p>이 문제 해결로 배달 빠른 순, 배달팁 낮은 순과 같이 실시간성 데이터가 들어가야하는 서비스가 적용될 수 있었고 고객에게 더 좋은 가치를 드리는게 가능해졌습니다.</p>
625
626<p>2021년에는 검색창이 하단으로 옮겨지고 인기 검색어 지면 등 많은 기능들이 추가되고 있습니다. 이 글을 읽고 모든 코드를 보고 싶거나, 검색개발팀에 관심이 생겼다면 <a href="https://resume.woowahan.com/jobs/view.aspx?cc=244001&amp;jidx=105376&amp;pidx=10893&amp;bidx=166014">[배민서비스실] 검색서비스 개발자 모집</a> 에 많은 지원 부탁드립니다.</p>
627
628<p>긴 글 읽어주셔서 감사드립니다.</p>
629
630	  </div>
631	  
632    <div class="fb-comments"
633         data-href="http://woowabros.github.io/woowabros/2021/03/02/search-system.html" data-width="100%" data-numposts="10"></div>
634    
635	  <a href="/" class="btn btn-block" id="back">목록으로 돌아가기</a>
636	
637  </article>
638  <a href="https://www.woowahan.com/#/recruit/tech" class="sub-banner"></a>
639</section>
640
641<script>
642	var goback = document.getElementById("back");
643    goback.addEventListener('click', function(event) {
644        window.history.back();
645        event.stopPropagation();
646    });
647</script>
647
648      </div>
649    </div>
650
651    <footer class="site-footer">
652
653  <div class="wrapper">
654    © Woowabros. All rights reserved. Powered by GitHub Pages.
655  </div>
656
657</footer>
658
659<script>
660  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
661  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
662  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
663  })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
664  //ga('create', 'UA-77115139-1', 'auto');
665  ga('create', 'UA-167267783-1', 'auto');
666  ga('send', 'pageview');
667</script>
667
668
669
670
671  </body>
672
673</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.