| Message ID | 45A52BC6.3040203@cadsoft.de |
|---|---|
| State | New |
| Headers |
Received: from raven.cadsoft.de ([217.7.101.211]) by www.linuxtv.org with esmtp (Exim 4.50) id 1H4htj-0007jj-VQ for vdr@linuxtv.org; Wed, 10 Jan 2007 19:10:04 +0100 Received: from [192.168.100.10] (hawk.cadsoft.de [192.168.100.10]) by raven.cadsoft.de (8.13.3/8.13.3) with ESMTP id l0AI9Agk012799 for <vdr@linuxtv.org>; Wed, 10 Jan 2007 19:09:10 +0100 Message-ID: <45A52BC6.3040203@cadsoft.de> Date: Wed, 10 Jan 2007 19:09:10 +0100 From: Klaus Schmidinger <Klaus.Schmidinger@cadsoft.de> Organization: CadSoft Computer GmbH User-Agent: Thunderbird 1.5.0.9 (X11/20060911) MIME-Version: 1.0 To: vdr@linuxtv.org Subject: Re: [vdr] error on large EPG loads via SVDRP References: <45A512B3.7000605@xs4all.nl> <45A52889.8020206@gmx.de> In-Reply-To: <45A52889.8020206@gmx.de> Content-Type: multipart/mixed; boundary="------------070302070803080206070905" X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (raven.cadsoft.de [192.168.1.1]); Wed, 10 Jan 2007 19:09:11 +0100 (CET) X-BeenThere: vdr@linuxtv.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: VDR Mailing List <vdr@linuxtv.org> List-Id: VDR Mailing List <vdr.linuxtv.org> List-Unsubscribe: <http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr>, <mailto:vdr-request@linuxtv.org?subject=unsubscribe> List-Archive: <http://www.linuxtv.org/pipermail/vdr> List-Post: <mailto:vdr@linuxtv.org> List-Help: <mailto:vdr-request@linuxtv.org?subject=help> List-Subscribe: <http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr>, <mailto:vdr-request@linuxtv.org?subject=subscribe> X-List-Received-Date: Wed, 10 Jan 2007 18:10:05 -0000 Status: O X-Status: X-Keywords: X-UID: 11636 |
Commit Message
Klaus Schmidinger
Jan. 10, 2007, 6:09 p.m. UTC
Udo Richter wrote: > Pjotr Kourzanov wrote: >> I am experiencing VDR errors when loading EPG via SVDRP. If the EPG >> data to >> be loaded is larger that 3 MB then VDR silently drops the SVDR >> connection. >> >> Everything is fine if the data is truncated to fit 3 megs. > > There is an issue if the EPG download is too slow. In that case, the VDR > watchdog timer may restart VDR to bring back VDR to life. You can try to > increase the watchdog timeout value. > VDRAdmin-AM can also read the epg data directly from the /video/epg.data > file, avoiding the lengthly and slow SVDRP download. I have recently sent the attached patch to Thomas Koch <tom@linvdr.org>. It dramatically speeds up getting EPG information from VDR. Klaus
Comments
En/na Klaus Schmidinger ha escrit: > Udo Richter wrote: >> Pjotr Kourzanov wrote: >>> I am experiencing VDR errors when loading EPG via SVDRP. If the EPG >>> data to ... > > I have recently sent the attached patch to Thomas Koch <tom@linvdr.org>. > It dramatically speeds up getting EPG information from VDR. I think the OP problem is the reverse: getting EPG data into vdr, not out of it. Bye
Luca Olivetti wrote: > En/na Klaus Schmidinger ha escrit: >> Udo Richter wrote: >>> Pjotr Kourzanov wrote: >>>> I am experiencing VDR errors when loading EPG via SVDRP. If the EPG >>>> data to > ... >> >> I have recently sent the attached patch to Thomas Koch <tom@linvdr.org>. >> It dramatically speeds up getting EPG information from VDR. > > I think the OP problem is the reverse: getting EPG data into vdr, not > out of it. > > Bye Indeed, I was referring to the problem of loading the data to VDR. Tweaking the watchdog timer looks like a dirty workaround to me - the value depending on amount of EPG data to be loaded divided by the speed at which this data could be transferred to VDR (do we assume loopback access, or ethernet? maybe wireless or WAN/VPN access?). I have neither investigated the problem deeper yet (just found a way to split the EPG data into parts), nor looked in the source code. Is it so difficult to convince the watchdog that while SVDR thread is busy it is actually doing useful work? Pjotr
On Wed, 10 Jan 2007 23:48:38 +0100, Pjotr Kourzanov wrote > I have neither investigated the problem deeper yet (just found a way > to split the EPG data into parts), nor looked in the source code. Is > it so difficult to convince the watchdog that while SVDR thread is > busy it is actually doing useful work? SVDRP is not implemented as a thread. Processing the SVDRP interface is part of the VDR main loop. The problem is that if you are uploading EPG fast enough, the SVDRP code will remain in cSVDRP::Process() until you're done. So the watchdog timer will not be refreshed and the watchdog eventually fires. There are two possible workarounds: 1. increase the watchdog timeout 2. take a short break from time to time while uploading the EPG. The "while (file.Ready(false))" loop in cSVDRP::Process() has a timeout of just 10 milliseconds. With a sleep of 100ms you should be on the safe side, as you still need to take into account that network traffic is buffered on both, the client and the server. Cheers, Frank
Hi, I heard that in new versions multiple SVDR connections are/will be allowed. Does it mean that they will be then implemented via a separate thread and my problem is solved? Frank Schmirler wrote: > On Wed, 10 Jan 2007 23:48:38 +0100, Pjotr Kourzanov wrote >> I have neither investigated the problem deeper yet (just found a way >> to split the EPG data into parts), nor looked in the source code. Is >> it so difficult to convince the watchdog that while SVDR thread is >> busy it is actually doing useful work? > > SVDRP is not implemented as a thread. Processing the SVDRP interface is part > of the VDR main loop. The problem is that if you are uploading EPG fast > enough, the SVDRP code will remain in cSVDRP::Process() until you're done. So > the watchdog timer will not be refreshed and the watchdog eventually fires. > > There are two possible workarounds: > 1. increase the watchdog timeout > 2. take a short break from time to time while uploading the EPG. The "while > (file.Ready(false))" loop in cSVDRP::Process() has a timeout of just 10 > milliseconds. With a sleep of 100ms you should be on the safe side, as you > still need to take into account that network traffic is buffered on both, the > client and the server. Well, as I said earlier, these things look like dirty hacks to me. For now I will just continue using my own dirty hack by splitting the EPG into parts. > > Cheers, > Frank > > _______________________________________________ > vdr mailing list > vdr@linuxtv.org > http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr >
Pjotr Kourzanov wrote: > Hi, > > I heard that in new versions multiple SVDR connections are/will be > allowed. > Does it mean that they will be then implemented via a separate thread > and my problem is solved? This will most likely be the case. Klaus
Pjotr Kourzanov wrote: > Luca Olivetti wrote: >> En/na Klaus Schmidinger ha escrit: >>> Udo Richter wrote: >>>> Pjotr Kourzanov wrote: >>>>> I am experiencing VDR errors when loading EPG via SVDRP. If the EPG >>>>> data to >> ... >>> >>> I have recently sent the attached patch to Thomas Koch <tom@linvdr.org>. >>> It dramatically speeds up getting EPG information from VDR. >> >> I think the OP problem is the reverse: getting EPG data into vdr, not >> out of it. >> >> Bye > > Indeed, I was referring to the problem of loading the data to VDR. > Tweaking the watchdog > timer looks like a dirty workaround to me - the value depending on > amount of EPG data to > be loaded divided by the speed at which this data could be transferred > to VDR (do we assume > loopback access, or ethernet? maybe wireless or WAN/VPN access?). > > I have neither investigated the problem deeper yet (just found a way to > split the EPG data into > parts), nor looked in the source code. Is it so difficult to convince > the watchdog that while SVDR thread is busy it is actually doing useful > work? Pjotr, How did you split the information up? I have a 17MB xmltv file which I would like to upload into VDR. Is it possible to adjust xmltv2vdr.pl to split the xmltv epg into parts too? My python knowledge is limited, but I am willing to have a stab at it.
Rob Davis wrote: > Pjotr Kourzanov wrote: >> Luca Olivetti wrote: >>> En/na Klaus Schmidinger ha escrit: >>>> Udo Richter wrote: >>>>> Pjotr Kourzanov wrote: >>>>>> I am experiencing VDR errors when loading EPG via SVDRP. If the EPG >>>>>> data to >>> ... >>>> >>>> I have recently sent the attached patch to Thomas Koch >>>> <tom@linvdr.org>. >>>> It dramatically speeds up getting EPG information from VDR. >>> >>> I think the OP problem is the reverse: getting EPG data into vdr, not >>> out of it. >>> >>> Bye >> >> Indeed, I was referring to the problem of loading the data to VDR. >> Tweaking the watchdog >> timer looks like a dirty workaround to me - the value depending on >> amount of EPG data to >> be loaded divided by the speed at which this data could be transferred >> to VDR (do we assume >> loopback access, or ethernet? maybe wireless or WAN/VPN access?). >> >> I have neither investigated the problem deeper yet (just found a way >> to split the EPG data into >> parts), nor looked in the source code. Is it so difficult to convince >> the watchdog that while SVDR thread is busy it is actually doing >> useful work? > > Pjotr, > > How did you split the information up? See script attached (given stdin, creates epg.[0-9]* files of ~3megs, without overwriting). > > I have a 17MB xmltv file which I would like to upload into VDR. > > Is it possible to adjust xmltv2vdr.pl to split the xmltv epg into parts > too? > Maybe. I found xmltv2vdr just too slow - it took ~9 hours to output 17MB of VDR epg.data. The XML was btw. ~54MB. > My python knowledge is limited, but I am willing to have a stab at it. > xmltv2vdr.pl is perl:-( But if you know AWK, I've posted my version of xmltv2epg on this list that takes ~40 minutes on the same data:-) Also, it takes extra info like credits into VDR's epg... Regards, Pjotr #!/usr/bin/awk -f BEGIN { max=3*1024*1024 if (!base) base="epg." n=1 advance() size=0 } /^[CEeTSDX]/ { size+=length } { print > base n } /^c/ { if (size>max) { advance() size=0 } } func advance() { while (!system("stat " base n " >/dev/null 2>&1")) n++ }
--- vdradmind.pl 2006-12-23 13:36:30.353879013 +0100 +++ vdradmind-2.pl 2006-12-23 13:37:42.186068197 +0100 @@ -538,21 +538,22 @@ sub EPG_buildTree { $SVDRP->command("lste"); + my @DATA = $SVDRP->readresponse; + CloseSocket(); my($i, @events); my($id, $bc) = (1, 0); undef(%EPG); - while($_ = $SVDRP->readoneline) { - chomp; + while($_ = shift @DATA) { if(/^C ([^ ]+) *(.*)/) { $bc++; undef(@events); my($channel_id, $channel_name) = ($1, $2); my $vdr_id = get_vdrid_from_channelid($channel_id); - while($_ = $SVDRP->readoneline) { + while($_ = shift @DATA) { if(/^E (.*) (.*) (.*) (.*)/ || /^E (.*) (.*) (.*)/) { my($event_id, $time, $duration) = ($1, $2, $3); my($title, $subtitle, $summary); - while($_ = $SVDRP->readoneline) { + while($_ = shift @DATA) { if(/^T (.*)/) { $title = $1 } if(/^S (.*)/) { $subtitle = $1 } if(/^D (.*)/) { $summary = $1 } @@ -3157,6 +3158,20 @@ return undef; } } + +sub readresponse { + my $this = shift; + my @a = (); + if($connected) { + while (<$SOCKET>) { + chomp; + my $end = substr($_, 3, 1) ne "-"; + push(@a, substr($_, 4, length($_))); + last if ($end); + } + } + return @a; +} # #############################################################################