1. Can you modify your system to return results based on a subset of their followers? If this is still of value to the user, it looks like the way to go, providing results based on a subset followed by another set of "final" results based on the full follower graph.
2. If you have enough celebrity-scale accounts using your service, are you able to share their followers details and cut down on time used to pull them?
3. On the business side, the service sounds dangerously like something that would be killed by Twitter if it becomes successful, by some definition of successful because you are pulling out their follower graphs. Look at Tumblr and Instagram. While it is good to milk it while you can and build it with ambition, it is also wise to look further and prepare for the day if it gets shut down and revenue goes to 0. If you have nothing to lose, go ahead, but be aware.
Great, great points. Here are my responses:
1) Yes, that seems to be the only way out right now. But my core features (most popular followers / most valuable followers) depend on a full set and will not be accurate. Growth data works fine though. The main reason why my service got traction were the most popular followers / most valuable followers metrics though. So that sucks. But seems to be my only resort right now.
2) I can get IDs faster, and then theoretically use my own DB to see if I need details for that ID (i.e. follower details) from Twitter or have them stored. Problem is: I am duplicating Twitter's database, also, I had this before and the database grew to immense size and kept crashing all the time despite efforts to avoid it. Also: Celebrities have a large distribution of followers, so unlikely to save much time by seeing who's details I already got elsewhere.
3. Yes, although Twitter announced their quadrants recently, of services that will be supported by them, one of them is social analytics. This is what I do. So that should be good. But you have a point, I can't touch any money for up to 1 year in case Twitter shuts it down, I want to pay back the remaining unused time to my users. So it's frozen money.
Focusing on 1): if "most popular follower" means "follower with the most followers", you could try one of three strategies.
First, just check which of the the top 100 (1000) most-followed users follow the celebrity - this will likely eliminate the heaviest hitters immediately.
For accounts with normal amount of followers, do as you already do.
For the in-betweeners,
Pick your celebrity.
For a random 1% (.1%) of his/her followers:
Retrieve list of people followed by this follower
For each user followed by a follower of the celebrity:
Increase number of people following this user by 1
For each user in the top 1% (10%, .1%) of the above table of users:
Retrieve number of followers
Report user with highest number of followers from the above
This is, of course, based on the idea that often-followed users who follow a celebrity will also have many followers among the followers of the celebrity. Results will become more accurate as you poll more followers, of course.
(I don't use Twitter or their API, and the above may be completely wrong.)
see, this is what I came for. I believe there's room for optimization within the limits I have to adhere to right now. your approach seems like a stab at it and I'll think this through again in a bit if it makes sense. but you're hitting the nail on the head in terms of where my problem is, i can't change what twitter does, i can display partial results but it's suboptimal in many ways, so how can i make these partial results the best possible quality? this is where you answer seems to make sense - thanks. i'll think about this one for sure.
Digging further in 3), if your app is proven to be in the 'good' quadrant, have you ever considered contacting Twitter directly and requesting a higher API limit rate.
I can imagine they are ok with that if your app provides value to their ecosystem.
well i constantly mention the issue through their bug reports and discussions, they acknowledge it and promised to talk to the team about re-evaluating the way things are handled now, but I'd assume this means it's not a priority.
i tried signing up through one of their partner thing forms but as expected, no response.
With the right infrastructure and tech, the "mirror" DB shouldn't crash. And also keep in mind how many people follow a lot of celebrities… common point!
1. Can you modify your system to return results based on a subset of their followers? If this is still of value to the user, it looks like the way to go, providing results based on a subset followed by another set of "final" results based on the full follower graph.
2. If you have enough celebrity-scale accounts using your service, are you able to share their followers details and cut down on time used to pull them?
3. On the business side, the service sounds dangerously like something that would be killed by Twitter if it becomes successful, by some definition of successful because you are pulling out their follower graphs. Look at Tumblr and Instagram. While it is good to milk it while you can and build it with ambition, it is also wise to look further and prepare for the day if it gets shut down and revenue goes to 0. If you have nothing to lose, go ahead, but be aware.