Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I'd like to compare it (speed) to my version using python+mechanize..


I'm intrigued as to why you think there would be a speed difference. If your language of choice can't iterate around a loop faster than your network card can push out a couple of network packets then I think you have some issues.


Factor in RTT/latency and concurrency. There's no reason multiple requests can't be in-flight at once, up to the point where you're overloading your uplink or the server.


like flatline said below, I was wondering if you did any concurrency. I recently scraped another site and I found a "sweet spot" of about 10 different sessions each trying different chunks... maxed out at about 200kb/s from the site.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: