Curated Twitter timelines and the tech stack

2014-09-08 1 min read

    Apparently Twitter is considering curation user’s timelines. A perspective people haven’t really discussed is the impact on the tech side. Right now each user has a unique timeline that needs to be presented in near-real time in case they need to see it. This results in a massive storage operation using Redis where these timelines are continuously generated and cached. By moving to a model where every user can be categorized into a group that sees a particular set of tweets Twitter can drastically reduce the amount of data they need to store per user. I’m sure Twitter already has a way of categorizing users in order to support the ad product and this approach would extend it to the “stream” product. In a way it’s akin to how compression works - find repeated patterns and replace every occurrence with something shorter. Then when you want to uncompress you just reverse the process.

    I doubt this is the primary driver of the curation discussion and there are clearly more important issues at stake but this may be the proverbial “cherry on top” that will get Twitter to move to the curated model.