https://www.rabbitmq.com/assets/js/6184fe49.2f3b56b8.js

1"use strict";(self.webpackChunkrabbitmq_website=self.webpackChunkrabbitmq_website||[]).push([["11611"],{75580(e,t,s){s.r(t),s.d(t,{assets:()=>h,contentTitle:()=>r,default:()=>c,frontMatter:()=>o,metadata:()=>n,toc:()=>d});var n=s(78326),a=s(74848),i=s(28453);let o={title:"RabbitMQ, backing stores, databases and disks",tags:["Introductory"],authors:["matthew"]},r,h={authorsImageUrls:[void 0]},d=[];function l(e){let t={a:"a",code:"code",em:"em",p:"p",...(0,i.R)(),...e.components};return(0,a.jsxs)(a.Fragment,{children:[(0,a.jsxs)(t.p,{children:["From time to time, on\nour ",(0,a.jsx)(t.a,{href:"https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss",children:"mailing\nlist"})," and elsewhere, the idea comes up of using a\ndifferent ",(0,a.jsx)(t.em,{children:"backing store"})," within RabbitMQ. The backing store is\nthe bit that's responsible for writing messages to disk (a message can\nbe written to disk for a number of reasons) and it's a fairly frequent\nsuggestion to see what RabbitMQ would look like if its own backing\nstore was replaced with another storage system."]}),"\n",(0,a.jsx)(t.p,{children:"Such a change would permit functionality that is not currently\npossible, for example out-of-band queue browsing, or distributed\nstorage, but there is a fundamental difference in the nature of data\nstorage and access patterns between a message broker such as RabbitMQ\nand a generic database. Indeed RabbitMQ deliberately does not store\nmessages in such a database."}),"\n",(0,a.jsxs)(t.p,{children:["Firstly we need to discuss what properties RabbitMQ itself expects of\nany backing store. RabbitMQ writes messages to disk in two cases:\neither the message has been published in such a way that it must be\nwritten to disk (e.g. published with ",(0,a.jsx)(t.code,{children:"delivery_mode = 2"}),")\nor memory pressure is causing RabbitMQ to start running out of RAM and\nso it is pushing messages to disk in order to free up RAM. In the\nfirst case, just because we've written the message to disk, does not\nmean that we're going to forget about it from RAM: if memory is\nabundant then there's no reason to incur the cost of a subsequent disk\nread."]}),"\n",(0,a.jsxs)(t.p,{children:['In the second case, it means that any backing store that keeps\neverything in RAM all the time is immediately not a good fit: RabbitMQ\nwrites messages to disk in order to free up RAM, thus if the "writing\nto disk" bit actually just moves the message from one area of RAM to\nanother without freeing up RAM then nothing has been gained. Using\nsuch a backing store might work and it might achieve the improvements\nin functionality desired, but such a change would have substantial\nimpact on the scalability of RabbitMQ: it would no longer be able to\nabsorb more messages than can be kept in RAM, which was one of the\n',(0,a.jsx)(t.em,{children:"raisons d'\xeatre"})," for the ",(0,a.jsx)(t.em,{children:"new persister"})," work that led to\nRabbitMQ's current default backing store."]}),"\n",(0,a.jsxs)(t.p,{children:["Some databases or key-value stores write disk contents\nby initially writing a snapshot of their entire data set, and then\nwriting deltas to that data set. After a while, either time based or\nbased on the number of deltas, or ratio of deltas to snapshot size, a\nnew snapshot is written, and then the previous snapshot and all its\ndeltas can be thrown away. This is how RabbitMQ's ",(0,a.jsx)(t.em,{children:"old persister"}),"\nworked. The problem with this is that it can repeatedly cause vast\namounts of data to be unnecessarily rewritten. Imagine you have two\nqueues, one of which is entirely static: no one is publishing messages\nto it, and no one is consuming messages from it, it's just sitting\nthere, but it contains several million messages, all of which have\nbeen written to disk. The other queue is almost always empty, but is\nmoving very quickly -- thousands of messages a second are being\npublished and consumed from it. Every message sent to that queue has\nto be written to disk, but they're all being consumed as soon as\nthey've been written to disk. Consider the effect of this scenario on\nthe backing store: the second queue will cause a rapid stream of\ndeltas to occur but whenever the snapshot is rewritten, it'll cause\nthe entire contents of the first queue to be rewritten too ",(0,a.jsx)(t.em,{children:"even\nthough there has been no change to that queue's contents"}),". So\nagain, backing stores that write messages to disk in this way are\nlikely to be a poor fit for RabbitMQ's needs."]}),"\n",(0,a.jsx)(t.p,{children:"So suitable backing stores (assuming the performance and scalability\nproperties that RabbitMQ has need to be kept: this is by no means\ncertain in all scenarios) would be able to store a volume of data\nbounded only by disk size rather than RAM, and also have a reasonably\nsophisticated means of storing data on disk such that unchanged data\nwon't be rewritten indefinitely."}),"\n",(0,a.jsx)(t.p,{children:"There are a couple of further aspects of RabbitMQ's default backing\nstore that are worth mentioning. Queues themselves decide when and\nwhether to write a message to disk. But a single message can be sent\nto multiple queues and it is obviously advantageous to make sure each\nmessage only gets written to disk once. However, there are two\ndistinct pieces of information here: firstly, the message content\nitself. This is the same in every queue that the message has been sent\nto, and should only be written to disk once, regardless of the number\nof queues it goes to; note that subsequent writes of this do not need\nto do a value comparison: if the ID of the message is known to the\nbacking store then the message body will match what is already on disk\n-- message content is never altered by the broker. The second piece of\ninformation is the existence of the message in each queue: where in\nthe queue it lies, what its neighbours are, and what its\nqueue-specific status is. This second piece of information is what\nallows RabbitMQ to 

1start up, recover messages and queues from disk and\nensure that the messages in each queue are in the same order as when\nRabbitMQ was shut down."}),"\n",(0,a.jsxs)(t.p,{children:["Thus RabbitMQ's default backing store consists of a\nnode-global ",(0,a.jsx)(t.em,{children:"message store"})," which is concerned only with writing\nmessage contents to disk; and a per queue ",(0,a.jsx)(t.em,{children:"queue index"})," which\nuses a very different format for writing per message per queue data to\ndisk. Because these two needs are very specific, there are an awful\nlot of optimisations that can be applied (and we have!)."]}),"\n",(0,a.jsxs)(t.p,{children:["Generic database benchmarks normally show that read performance vastly\nout performs write performance. If it doesn't then that normally means\nthe writes aren't actually going to disk (with ",(0,a.jsx)(t.code,{children:"fsync"}),"), or\nthere's a bug which is crippling read performance. And indeed,\ndatabases have historically been optimised for read-heavy\nworkloads. This matches their general use case: there is a slowly\nexpanding data set which must be queried in various different ways.\nDeletions tend to be quite rare: if you think about the typical\nwebsite shopping basket on top of a relational database, then unless a\ncustomer deletes their account, there are very few reasons to ever\nissue deletions -- even if a product is discontinued, you're probably\njust going to set a flag on that product row because otherwise you\nrisk stopping customers from being able to see their order history\n(assuming it's normalised)."]}),"\n",(0,a.jsxs)(t.p,{children:["So the vast volume of data in most databases is fairly static. This is\nthe exact opposite of data in message brokers: for us, ",(0,a.jsx)(t.em,{children:"reading"}),"\ndata is the rarest operation, and ",(0,a.jsx)(t.em,{children:"writing"})," and ",(0,a.jsx)(t.em,{children:"deleting"}),"\ndata are the common cases. Ideally, if RabbitMQ is running in\nplenty of memory, there will never be any reads from disk at\nall. There will only be writes for messages that are published in such\na way that they have to be written to disk, and even then, provided we\ncan get the message out to a consumer quickly enough, there are many\nways in which we can optimise out those writes. We only ever read data\nwhen memory pressure has forced us to write messages to disk and then\nforget about the message from RAM. Read performance is certainly\nimportant: we work hard to make sure RabbitMQ gets rid of data as fast\nas possible (without utilising ",(0,a.jsx)(t.code,{children:"/dev/null"}),") and being able\nto read messages from disk quickly is part of that. But avoiding the\nwrite in the first place is the goal."]}),"\n",(0,a.jsx)(t.p,{children:"In fact, as far as message brokers are concerned, it's best to think\nof RAM as a large write-back cache for the disk, and then the task is\nto optimise the management of this cache to maximise the elimination of\nwrites by delaying them for as long as possible in the hope that the\ncorresponding deletion occurs before the write has really gone to\ndisk. This is quite obviously very different from normal databases\nwhich do not try to make gains from the lifespan of data being so\nshort as it frequently is in a message broker."}),"\n",(0,a.jsxs)(t.p,{children:["None of this is meant to deter efforts to make RabbitMQ work with\nalternative backing stores, but merely to explain why we decided to do\nour own thing when writing the ",(0,a.jsx)(t.em,{children:"new persister"})," for RabbitMQ\n(which first came out with RabbitMQ version 2.0.0) rather than use an\noff-the-shelf data store. It explains why building a high performance\nmessage broker directly on top of a normal database is tricky at best,\nand why the nature of data in a message broker is very different from\nthe nature of data in a database."]})]})}function c(e={}){let{wrapper:t}={...(0,i.R)(),...e.components};return t?(0,a.jsx)(t,{...e,children:(0,a.jsx)(l,{...e})}):l(e)}},28453(e,t,s){s.d(t,{R:()=>o,x:()=>r});var n=s(96540);let a={},i=n.createContext(a);function o(e){let t=n.useContext(i);return n.useMemo(function(){return"function"==typeof e?e(t):{...t,...e}},[t,e])}function r(e){let t;return t=e.disableParentContext?"function"==typeof e.components?e.components(a):e.components||a:o(e.components),n.createElement(i.Provider,{value:t},e.children)}},78326(e){e.exports=JSON.parse('{"permalink":"/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks","editUrl":"https://github.com/rabbitmq/rabbitmq-website/tree/main/blog/2011-01-20-rabbitmq-backing-stores-databases-and-disks/index.md","source":"@site/blog/2011-01-20-rabbitmq-backing-stores-databases-and-disks/index.md","title":"RabbitMQ, backing stores, databases and disks","description":"From time to time, on","date":"2011-01-20T00:00:00.000Z","tags":[{"inline":true,"label":"Introductory","permalink":"/blog/tags/introductory"}],"readingTime":7.43,"hasTruncateMarker":true,"authors":[{"name":"Matthew Sackman","key":"matthew","page":null}],"frontMatter":{"title":"RabbitMQ, backing stores, databases and disks","tags":["Introductory"],"authors":["matthe

1w"]},"unlisted":false,"prevItem":{"title":"Who are you? Authentication and authorisation in RabbitMQ 2.3.1","permalink":"/blog/2011/02/07/who-are-you-authentication-and-authorisation-in-rabbitmq-231"},"nextItem":{"title":"Ruby AMQP 0.7 released!","permalink":"/blog/2011/01/19/ruby-amqp-0-7-released"}}')}}]);