Jump to content


RSS downloader in perl


  • Please log in to reply
5 replies to this topic

#1 kpw

kpw

    n00b

  • Members
  • 2 posts
  • 0 thanks
  • Gender:Male

Posted 11 February 2009 - 12:50 AM

Hi,
Recently I've discovered that utorrent has nice feature that allows you to download files directly from rss, and since every show on tvu has individual feed, why not use it ? This is my first script in perl so I'm not so sure about it, but it works so far. It's designed to work with mldonkey but if you are using amule you can just execute amulecmd instead of telnet, and windows users can execute emule.exe with ed2klink. Configuration is pretty easy, fill up connection info(mldonkey), which season you wish to keep eye on and which files you already have. Variables are now set to 7th season of '24' hdtv (350mb) version, without the last episode. Paste it to file, chmod +x and execute in screen or dtach :)

#!/usr/bin/perl -w

use strict;
use HTML::Element;
use HTML::TreeBuilder;
use XML::RSS;
use LWP::Simple;
use Net::Telnet;
use Time::Local;

sub if_not_in_tab {
 my ($needle,@haystack) = @_;
 my $count=@haystack;
 my $i=0;
 while($i<$count){
  if($haystack[$i]==$needle){
   return 0;
  }
  $i++;
 }
 return 1;
}

my $rss = new XML::RSS;
#mld host user and pass
my $host="";
my $user="";
my $pass="";
#season id
my $season = '24566';
#fill array with file_ids that you already have
my @ihave = (270587, 371711, 371715, 371827, 371832, 373105, 374096, 375322);

while(1){

my $data = get('http://tvunderground.org.ru/rss.php?se_id='.$season);
if($data){

 if($rss -> parse($data)){

  my @tab = ();

  foreach my $item (@{$rss->{'items'}}){
   next unless defined($item->{'guid'});
   $_=$item->{'guid'};
   if( /%5B(.*)%5D/ ) {
	if(if_not_in_tab($1,@ihave)){
	 unshift(@tab,$1);
	}
   }
  }

  my $howmanyfound=@tab;
  
  if($howmanyfound>0){

   my $i=0;
   print "Found ".$howmanyfound." new files ".localtime(time())."\n";
   while($i<$howmanyfound){

   my $url='http://tvunderground.org.ru/index.php?show=ed2k&season='.$season.'&sid['.$tab[$i].']=1';

   my $tree = HTML::TreeBuilder->new();
   $data=get($url);
   if($data){
	$tree->utf8_mode(1);
	if($tree->parse($data)){

	 my $ed2k = $tree->find_by_attribute('name','ed2klink');

	 print "Found: ".$ed2k->attr('href')."\n";
	 my $telnet = new Net::Telnet (Timeout=>10,Errmode=>'die',Port=>4000);
	 if($telnet){
	  $telnet->open($host);
	  $telnet->waitfor('/> $/');
	  sleep(1);
	  $telnet->print('auth '.$user.' '.$pass);
	  sleep(1);
	  $telnet->print('dllink '.$ed2k->attr('href'));
	  $telnet->print('q');
	  unshift(@ihave,$tab[$i]);
	 }
	 $i++;
   }
  }
  $tree=$tree->delete;
 }
}
}
}
 sleep(1800);#sleep for 30 min
}

This script shows what files are already available (paste output to ihave array)

#!/usr/bin/perl -w

use strict;
use XML::RSS;
use LWP::Simple;

my $rss = new XML::RSS;

#season id
my $season = '24566';

my $data=get('http://tvunderground.org.ru/rss.php?se_id='.$season);

$rss->parse($data);

my @tab = ();

foreach my $item (@{$rss->{'items'}}){
 next unless defined($item->{'guid'});
 $_=$item->{'guid'};
 if( /%5B(.*)%5D/ ) {
  unshift(@tab,$1);
 }
}

my $i=0;
my $hm=@tab;
while($i<$hm){
 print "$tab[$i]";
 $i++;
 if($i!=$hm){print ",";}
}

print "\n";


#2 imognimma

imognimma

    TVU Star

  • VIP
  • PipPipPip
  • 409 posts
  • 1 thanks
  • Gender:Male
  • Location:North Korea

Posted 01 March 2009 - 02:53 AM

wow ! real guys use mldonkey and program in perl, I always knew that ! ;)

will take a look at your script since that's what I was looking for.
I wrote myself a perl script called "donkey feeder": Imagine you want to download several hundred files, but don't want to enter them into mldonkey all at once (i.e. to keep a low profile). instead you feed the list of edk urls into the donkey feeder, which will start downloading the next file from the list as soon as a previous file has finished.

will post the script here if someone is interested.
Posted ImagePosted Image
Posted ImagePosted Image
Posted ImagePosted Image
---

#3 kpw

kpw

    n00b

  • Members
  • 2 posts
  • 0 thanks
  • Gender:Male

Posted 02 March 2009 - 11:31 PM

But isn't it easier to just paste all links to donkey, then modify user with `userdls name_of_user 1`, so only one file will be downloading ?

#4 Che Guevara

Che Guevara

    Ghetto Coder

  • Root Admin
  • PipPipPipPipPip
  • 1,471 posts
  • 4 thanks
  • Gender:Male

Posted 29 March 2009 - 07:36 PM

does mldonkey have encryption yet for ed2k protocol
"For God so loved the world that he gave his only beogtten Son, that whoever believes in him shall not perish but have eternal life" John 3:16

#5 imognimma

imognimma

    TVU Star

  • VIP
  • PipPipPip
  • 409 posts
  • 1 thanks
  • Gender:Male
  • Location:North Korea

Posted 03 April 2009 - 11:56 PM

View PostChe Guevara, on Mar 29 2009, 03:36 PM, said:

does mldonkey have encryption yet for ed2k protocol

not that I know. but I'm really looking forward to seeing this feature implemented.
Posted ImagePosted Image
Posted ImagePosted Image
Posted ImagePosted Image
---

#6 imognimma

imognimma

    TVU Star

  • VIP
  • PipPipPip
  • 409 posts
  • 1 thanks
  • Gender:Male
  • Location:North Korea

Posted 04 April 2009 - 12:02 AM

View Postkpw, on Mar 2 2009, 07:31 PM, said:

But isn't it easier to just paste all links to donkey, then modify user with `userdls name_of_user 1`, so only one file will be downloading ?

I'd like to keep the total number of files in the download queue low for obvious reasons. The script also supports categories ("classes"), allowing parallel downloads of (1 or more) files per class.
Posted ImagePosted Image
Posted ImagePosted Image
Posted ImagePosted Image
---




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users