From patchwork Mon May 7 22:15:28 2007 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Artur Skawina X-Patchwork-Id: 12458 Received: from mx10.go2.pl ([193.17.41.74] helo=poczta.o2.pl) by www.linuxtv.org with esmtp (Exim 4.50) id 1HlBUv-00083i-EW for vdr@linuxtv.org; Tue, 08 May 2007 00:16:01 +0200 Received: from poczta.o2.pl (mx10.go2.pl [127.0.0.1]) by poczta.o2.pl (Postfix) with ESMTP id C73E658016 for ; Tue, 8 May 2007 00:15:28 +0200 (CEST) Received: from dlf153.neoplus.adsl.tpnet.pl (dlf153.neoplus.adsl.tpnet.pl [83.24.35.153]) by poczta.o2.pl (Postfix) with ESMTP for ; Tue, 8 May 2007 00:15:28 +0200 (CEST) Received: (qmail 20149 invoked from network); 7 May 2007 22:15:28 -0000 Received: from unknown (HELO ?172.19.43.221?) (172.19.43.221) by 172.19.43.250 with SMTP; 7 May 2007 22:15:28 -0000 Message-ID: <463FA500.4010200@o2.pl> Date: Tue, 08 May 2007 00:15:28 +0200 From: Artur Skawina User-Agent: Thunderbird 3.0a1 (X11/20070320) MIME-Version: 1.0 To: VDR Mailing List X-Enigmail-Version: 0.95b Subject: [vdr] [PATCH] dynamically sized ringbuffers v1 X-BeenThere: vdr@linuxtv.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: VDR Mailing List List-Id: VDR Mailing List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2007 22:16:01 -0000 Status: O X-Status: X-Keywords: X-UID: 12808 vdr uses quite a lot of memory, RSS here used to be ~70M, so i decided to see if it could be reduced a bit. First suspects became the ringbuffers, cause i was seeing a lot of log lines such as "buffer stats: 96068 (1%) used". Turns out many buffers are way over sized, only a few percent of the memory are typically used, yet because of how a ringbuffer works _all_ of it is constantly accessed. I first tried to make the ringbuffer users smarter about picking the size and eg after changing the transfer buffer to start out small and use bigger sizes only when the current one isn't large enough it seemed a more reasonable size for that vdr instance was 384k instead of 2M. Similarly the TS buffer could be just a few hundred K instead of 2M. But that approach was not ideal as there isn't anything one could do after realizing the buffer is too small, it could be grown next time, after channel switch etc, but that's it. Also some of the huge buffers came from plugins, and I wasn't all that excited about fixing all of them (one example w/ the one-digit % fill was streamdev-server). It is hard to estimate what the proper size would be under load, with many simultaneous recordings. So I did attack this issue differently -- by making the ringbuffers themselves smarter. Patch below changes the interpretation of the 'Size' parm given when creating a buffer from a fixed length to a max limit. The buffers start out much smaller (currently 128k, instead of many megabytes) and grow automatically when they become almost full. This way all callers benefit, and they don't have to be modified at all. The buffers are actually always allocated using the max size, but only the necessary buffer part is used, the rest of the memory is never accessed; this is done this way so that no extra locking is necessary. I've tested this on two vdrs for a few hours and it seems to work, but it needs a lot more testing. There is some extra logging in this patch so that you can see what's going on; every time a ring buffer gets deleted it logs a line which tells you exactly how much memory was requested and how much was really needed. vdr: [19729] Deleting ring buffer "TS" size: 196608 / 2097152 (used 9.375% of requested size) vdr: [19718] Deleting ring buffer "Result" size: 262144 / 262144 (used 100% of requested size) vdr: [19718] Deleting ring buffer "Recorder" size: 442368 / 5242880 (used 8.4375% of requested size) vdr: [19718] Deleting ring buffer "Result" size: 262144 / 262144 (used 100% of requested size) vdr: [19718] Deleting ring buffer "Recorder" size: 131072 / 5242880 (used 2.5% of requested size) vdr: [19733] Deleting ring buffer "TS" size: 131072 / 2097152 (used 6.25% of requested size) vdr: [19718] Deleting ring buffer "Result" size: 262144 / 262144 (used 100% of requested size) vdr: [19718] Deleting ring buffer "Recorder" size: 442368 / 5242880 (used 8.4375% of requested size) So each of the several 'Recorder' buffers was an order of magnitude smaller than w/o this patch and several megabytes less ram was used. artur diff --git a/ringbuffer.c b/ringbuffer.c index 0633bd3..8dd1efe 100644 --- a/ringbuffer.c +++ b/ringbuffer.c @@ -151,13 +151,29 @@ void cRingBufferLinear::PrintDebugRBL(void) } #endif +// cRingBufferLinear are dynamically sized, or at least we can pretend they are ;) +// We treat 'Size' as the maximum size, but start with a small buffer, which can +// grow later as it fills up. Memory is always allocated for the full buffer, but +// the unused RAM portion remains untouched until (if at all) it is actually needed. +// Note that we can not start with a larger than requested buffer because there are +// ring buffer users that cause crashes then (eg softdevice mpegdecoder absolutely +// needs 32/64k). + +// Startup size. 64k still causes overflows before buffer starts to grow, 128k doesn't. +// The buffers grow to 200..500k anyway, so maybe increasing this a bit more would +// make sense, but let's first see if it's really needed. +#define DEFRBSIZE KILOBYTE(128) + cRingBufferLinear::cRingBufferLinear(int Size, int Margin, bool Statistics, const char *Description) -:cRingBuffer(Size, Statistics) +:cRingBuffer(min(Size,DEFRBSIZE), Statistics) { description = Description ? strdup(Description) : NULL; tail = head = margin = Margin; gotten = 0; buffer = NULL; + maxsize = Size; + growbuf = 0; + dsyslog("New ring buffer \"%s\" size: %d margin: %d", description, Size, Margin ); if (Size > 1) { // 'Size - 1' must not be 0! if (Margin <= Size / 2) { buffer = MALLOC(uchar, Size); @@ -183,6 +199,8 @@ cRingBufferLinear::~cRingBufferLinear() #ifdef DEBUGRINGBUFFERS DelDebugRBL(this); #endif + dsyslog("Deleting ring buffer \"%s\" size: %d / %d (used %g%% of requested size)", description, + size, maxsize, size/(double)maxsize*100.0 ); free(buffer); free(description); } @@ -207,6 +225,13 @@ void cRingBufferLinear::Clear(void) int cRingBufferLinear::Read(int FileHandle, int Max) { + if (Available() >= size*2/4) + growbuf = 1; + if (growbuf && head>=tail && size 0) ? diff - 1 : Size() - head; @@ -236,6 +261,8 @@ int cRingBufferLinear::Read(int FileHandle, int Max) lastHead = head; lastPut = Count; #endif + if (Available() >= size*3/4) + growbuf = 1; EnableGet(); if (free == 0) WaitForPut(); @@ -245,6 +272,13 @@ int cRingBufferLinear::Read(int FileHandle, int Max) int cRingBufferLinear::Put(const uchar *Data, int Count) { if (Count > 0) { + if (Available()+Count >= size*2/4) + growbuf = 1; + if (growbuf && head>=tail && size