Context:
I am currently developing a quite complex social media network. (Not for any company, just for the heck of it) Now one of the main priveleges of having a profile on this social media platform, is having a very wide spectrum of "modules" with different functions that can be placed on your own profile as you please. The system for that is not much of a challenge, but I am intending to make the users able to integrate pieces of profiles they have on other platforms as well. Oh and by the way, I am using node.js with the express framework. But I am pretty much willing to adapt however needed for this purpose.
What I am looking for:
I am trying to find a resolution to properly access the public data in real time without being restricted by the amount of traffic or making it easy for the other social networks to deny the information.
Methods I have considered, wich do not work or are not suited for this job:
Quote:
After looking at the twitter api, I quickly realized how limitted it is in the amount of requests one can make. Additionally pretty much every api seems to use authentication, wich makes it really easy for them to shut down my service. Just because they feel like it that is. I would prefer not to beg for other social networks mercy to please allow users to integrate pieces of their own personal accounts on my network. (Wich, judging from a companies perspective would be a somewhat stupid thing to allow. Making it very likely for them to deny it somwaht quickly imo)
Quote:
|
Request the pages normally using your server!
|
Honestly, this is semi doable. Until enough users use my service and the requests become spammy, getting caught in detection and ultimatively just receiving 404. (also very easy to detect if they wanted to shut it down)
Quote:
|
Same as above, but use proxies!
|
That's something I am actually seriously considering. But I am quite afraid of the immense bulk of proxies and ip bans I could run into. This could easily become quite expensive very quickly and the only way I could see making them almost impossible to detect, is assigning every user a private proxy. And... I don't have that kind of money.
Is there something like ajax that could work for this purpose?
I clearly understand that the same origin policy does not allow an ajax call in this case. But the general concept of having the user fetch the information would make this so much easier. Is there maybe like a function that does the same as an ajax call, but sends a version of the wesbite requested, without using cookies etc? I mean, that should be save for the user, as sensitive information would not be accessable. But at the same time they could not kick my ass by completly blocking the functionality of those modules as they please or by me getting caught in automated systems.
Alternatively I am also completly fine with any server side resolution, as long as it is not bottlenecked in terms of maximum requests, does not get caught in automated protective systems (spammy requests) and can not be easily detected as coming from our network. (Therefore a clientside request being more efficient, since their network would easily block us from accessing their pages, but not the users accessing their own data)
