Thanks. I wrote 500B, but I did do the calculation with 500M; I used much larger estimates (as is always recommended in back-of-the-envelope calculation) - namely, 256 bytes per tweet - which, in addition to 140 bytes of text[0], leaves e.g. 8 bytes for user id (2 billion uids for every living person on earth), 8 bytes for microsecond timestamp, and 80 more bytes for pointers/caching/stats/whatever. (Those 16 bytes, incidentally, are enough to have a unique identifier for every tweet)
I assume the average tweet is closer to 80 bytes, and taking advantage of that can let you store twice as much tweets in memory.
[0] Does twitter count utf-8 bytes, code points, graphemes, or something else towards the 140-"char" limit? Regardless, the computation would not yield materially different results.
Ok. I was assuming the average tweet is < 144 chars, and that there would/could be compression applied.
In fact I would guess that the effective average information content in a tweet is very low :) (~10-20 bytes?)
I assume the average tweet is closer to 80 bytes, and taking advantage of that can let you store twice as much tweets in memory.
[0] Does twitter count utf-8 bytes, code points, graphemes, or something else towards the 140-"char" limit? Regardless, the computation would not yield materially different results.