maximum articles to download

apokalips · Posted: Tue Jan 20, 2004 12:14 pm Post subject: maximum articles to download

Does anyone know what the maximum nr. of headers is that Newsleecher can have in the "download screen". Downloading 10 atricles goes OK. But when i try to download 170 articles, the program keeps running, but working extremely slow (using 100% processortime) and not responding anymore to any mouseclick.

Does anyone no a solution to this?

I have 10 bots divided over 5 servers.

Icefreez · Joined: 13 Nov 2003 Posts: 279

I have had 2000 files in my queue before Razz

and about 300, 50 mb files before. So I don't think the number of files or the size of the number of files determines how much CPU you use.

The more bots you use the more CPU you will end up using. I don't know if the number of servers has anything to do with the increase in CPUbut it might.
_________________
- Icefreez

Gregoor · Joined: 12 Nov 2003 Posts: 168

This is a known limitation. When for example files are unable on a server, this will take a lot of cpu power, and therefore time, when NL is checking if the parts are available on the server. When you have a lot of for example 200 part files, this will indeed take a LONG time.

Let it download this week, and queued the files when I left. Because of that, the first files were downloaded very slow, as the scanning for parts made the entire pc, and also the downloading almost come to a halt. So when I came home later and checked the filetimes, it turned out the first 2 files took just over an hour to download, where all other (equally sized files) took only about 6 minutes. So it is obvious that after about two hours, NL was done scanning for the unavailabe parts on one server.

invalidKey · Joined: 13 Nov 2003 Posts: 22 Location: Midwest, USA

This is probably exacltly the same thing Gregoor said but I had a hard time understanding him. Probably just me thinking like an american. Rolling Eyes

From what I've seen beta 10 and above scan each server in order until it finds one which has the part. This takes time. If you have a lot of files or parts cued up that are not on your primary server this can take a verrrrrry long time. If everything was on my first server it seemed to work okay. If it had to go to the second or third server to get the odd part it was bearable. If most of the files were on the third server it was time for a nap. Maybe it was just me or my setup, I used my ISP, then UNS, then Easynews with 3 bots per server. Since my ISP has retention of at most 2 days and I update headers every 3-5 days this was unbearable for me.

Fortunately beta 9 does not do this. I've had almost 33,000 files cued and it just sings along at about 12-20% CPU. Beta 12 was over 90% with less than 300 files. If you can live with the increased memory usage I would suggest beta 9 which also cures the garbage cleaning problem if your affected by it.

BTW, if you ever have over 20,000 files cued don't ever (and I do mean ever) try to use the move to top of cue feature. Over an hour later at 99% CPU it wasn't done so I went to bed and 7 hours later it still wasn't done. Confused

I know that was a lot of stuff to move around but it shouldn't take over 8 hours on a P4 3.0 w/1Gig DDR400. It wasn't swapping either as I have no swap file. I finally just killed it and restarted.
_________________
(c) 2003 No one in particular

Gregoor · Joined: 12 Nov 2003 Posts: 168

Yeah, that's what I meant, but it was late for me you know :p

And for me it already starts being downright slow when trying the first server that hasn't got the files. But might be my slow dual celeron 300 @ 414 MHz Wink

invalidKey · Joined: 13 Nov 2003 Posts: 22 Location: Midwest, USA

apokalips · Posted: Thu Jan 22, 2004 7:29 am Post subject:

invalidKey · Joined: 13 Nov 2003 Posts: 22 Location: Midwest, USA