Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Snowden releases information about Australian Intelligence gathering programs (theage.com.au)
245 points by SomeoneWeird on July 8, 2013 | hide | past | favorite | 98 comments


This is now getting so entirely overwhelming. It's like you're on the beach getting repeatedly smashed by waves that get bigger and bigger. There's nothing you can do to fight it, stop it, or move, and it just keeps coming. I feel so helpless with every new leak.


There's actually nothing new in this leak. Pine Gap [1] and HMAS Harman [2] were discussed on here a couple of weeks ago. Really what we're seeing here is the current generation of Echelon [3].

I'm not saying it's OK, far from it. It's just that there's nothing new in this story.

[1] https://news.ycombinator.com/item?id=5884858

[2] https://news.ycombinator.com/item?id=5887539

[3] https://en.wikipedia.org/wiki/ECHELON


Nothing new? Did you even read the article?

"Tempora is the first 'I save everything' approach ('full take') in the intelligence world. It sucks in all data, no matter what it is, and which rights are violated by it"

The other systems have always been given some degree of plausible deniability by government doing some hand-waving and suggesting that they have controlled methods by which wiretaps are only placed on specific targets, for specific reasons, etc. etc.

When you start saving everything, especially stuff that you know you are not supposed to, you are violating UK law (Human Rights Act of 1988) as well as international law (https://en.wikipedia.org/wiki/Data_Protection_Directive)

Trying to pass this off as nothing to be concerned about means that you are not paying attention, or don't care.


Yes of course I read the article.

Tempora isn't new news either [1].

> Trying to pass this off as nothing to be concerned about means that you are not paying attention, or don't care.

Did you even read my comment? Where did I say nothing to be concerned about?

[1] https://www.hnsearch.com/search#request/all&q=Tempora


> When you start saving everything

I'm pretty sure that's media sensationalism - were they get quite ahead of the facts and details, and spin the headlines. Or worse, leave out the logic of it all and replace it with something else - the narative the reader wants to believe (ex: they are gathering data on me to be used against me).

Not only does it take time to process the data, but an operational window (of that data) needs to be maintained for the next couple of days - were that data can be cross-referenced with the new feed and possibly be mind and connected to new events as they are unfolding. As otherwise it's useless.

The data is filtered for specific targets, or on specific triggers, or graph data, which is then kept. The rest (like 99.9999995% or more) is just noise that has no use to intelligence gathering - and I don't see why anyone would store it - or would want to store it.


Was the "full take" system in the UK mentioned previously? That's the scariest thing, to me. They're literally saving every bit of data that enters or leaves the UK, and they can store three days' worth already.


{EDIT: I meant to upvote you, but I might have accidentally downvoted you. Sorry.}

Yes, it has been mentioned previously.

UK media is under a notice from the Department of Defence - a D notice - which is an informal arrangement where the DoD asks media not to report things which they think are strategically important, and the media agrees.

The Official Secrets Act is the legal framework for secrets. People "Sign the Official Secrets Act", but all that means is that they've been made aware that the act exists. You can be prosecuted for revealing secrets even if you haven't signed the act. (I seem to remember some protesters being prosecuted for publishing photographs of missile transport in the 80s?)

So, while the D notice system is informal and is not a prohibition I guess it's a warning about what is considered really important. There's interesting discussion about why a free media pays any attention at all to D Notices.

I think the D Notice for the full take system is an abuse of the D Notice system. (Location of troops during war, for example.)


exactly - secret keeping is a privilege that is granted to the state in order for it to win fights against an "enemy". What is this enemy that the state is currently fighting that warrants these secrets? i don't believe "terrorists" is the answer.


trade enemies, preservation of self-interest


Is that a blanket term for everyone that isn't us, or do we actually have an official list of countries we consider trade enemies?


So "the rest of the world" counts as "trade enemies" now?


Heh, that's just buffering.....

(BTW that was a bad joke, for those who take tings seriously)

Tell you what this does mean though, our parliament has been lied too, which is supposed to be a big deal. But I guess all the opposition MP's are too worried that GCHQ know all the porn sites they visit. Lets face it, we all have something to hide, and chances are now that the gov knows about it, or can do with the right Db query.


Does this not mean that GCHQ are the greatest child pornographers on the planet, by their own standards? They've successfully prosecuted and imprisoned private individuals who've ended up with child porn on their computers due to botnets etc., so if they're wilfully collecting this data, they surely have a criminal case to answer?



We can stop it, and redirect it.

Once the people start creating new laws, and getting them passed, we'll win. Currently, politicians hand us laws that are drafted by corporations & lobbyists, then we simply react.

It's all about us being more assertive.


We will win.

We already have the bill of rights. Creating new laws won't help us. It's too late for that. Maybe repealing some laws and executive orders might help... but what we're seeing is the symptom of something bigger than governments.

We need to see the cage and step out of it. Educating ourselves and honing our common sense will help us. Becoming self sufficient will help us.


There's still plenty of time to pass laws. Why would there be a time limit?


Because nobody takes the law seriously anymore. Not the cops, not the bankers, not the president, not congressmen, not the underprivileged public, not the entrepreneurs. The judges can't even agree on what is constitutional. The illegitimate wars drag on, since before 9/11.

Because the way the bill of rights is treated now is farcical.


@jaekwon

You sound frustrated, as a lot of people are these days. Is it helping to be reactive to all of the things you mentioned? What if you instead were proactive, and defined a new law & helped promote it?

For instance, you mentioned illegal wars. What if we created a new law, that required a popular vote, yearly, to continue any war? That would be awesome, right? :)

Let's write some good laws, and get them passed.


>> You sound frustrated, as a lot of people are these days

You're still in the denial stage. I've been there. You've got a long ways to go. Keep reading about recent history. Next you'll get depressed...

I'm actually quite happy nowadays. Every day is a day full of purpose. I'm not reactive, I'm proactive. I see the patterns and I know what's coming. I know what needs to be done.

I agree with Lessig on one point and one point alone. Code is law.

Not saying we shouldn't try the traditional approach, though. Here's some good news:

- http://www.reddit.com/r/restorethefourth/comments/1htaqf/the...


>> You're still in the denial stage. I've been there. You've got a long ways to go. Keep reading about recent history. Next you'll get depressed...

I'm sorry to hear you are dealing with depression.

>> I'm actually quite happy nowadays. Every day is a day full of purpose. I'm not reactive, I'm proactive.

Oh, well that's good.


And watch the government and any one with enough money to sway a court roll over them and do what they want anyway?

Now that would be frustrating.


By my layman's point of view, we already have existing laws that are being flouted, among the the 4th Amendment which requires specific warrants for specific investigations, and the various laws authorizing and limiting surveillance. They do whatever they want despite these existing laws, by a) just doing it, and b) getting the administration through the Attorney General to reinterpret laws so that what was illegal is now justified by the very laws that previously made things illegal, and c) a captive, secret court that only hears one side of any query by the government and rubber stamps all NSA actions.

Given that, what good would a new law do? They already have everything in place to subvert any law they want, and their chief tool is mere will.

What might change things is public outrage and muscular congressional hearings, a la Watergate. I can't see that ever happening again.


> Once the people start creating new laws, and getting them passed, we'll win.

There's an important step you forgot to include: Getting the new laws enforced. When most of the people in charge of the enforcement are used to (and may well have liked) the old laws, that will not be simple.


Could you provide an example? Most police & military have good intentions, in my experience.


Cash money trumps good intentions when it comes to systemic corruption.

A bit off-topic but related to law enforcement: http://www.salon.com/2013/07/07/“why_did_you_shoot_me_i_was_...


you forget that a lot of human institutions are very hierarchical. Sure, most people in most institutions have good intentions, but it is usually the top that makes the rules/policy decisions, and it's more likely than not that those at the top don't have good intentions...


Why is that more likely, if you choose those at the top appropriately?


Laws: rules for the weak, guidance for the powerful.


You're far, far from alone.

Relax and join the movement... it's going to be a long ride.

--> http://goo.gl/lc7Fz


Except:

DON'T relax. These feelings of frustration are absolutely NORMAL and they must be channeled into action, not relaxation.


Except:

DON'T panic! And don't forget your towel.


well if it's a tsunami situation we quite literally need to run for the hills. The question is, where are "the hills" and how do we get there in this situation?

Or maybe we need to democratize the analogue of a submarine.


Tsunami situation verified. Relax, nobody on our side wants a violent revolution. The only hills you will find are the global enlightened community.

You will initially get depressed as you see the injustice around you.

You can deny what you see and go back to browsing Facebook.

Or, you can embrace the movement. I've been traveling the rabbit hole and I assure you that the thing is manageable if you put your time into it.

Here are some guidelines:

- You already have local communities (hacker/creative spaces) that have members who are on top of the news and working to escape the system. Find them.

- Find the commonality between libertarianism, anarcho-capitalism and anarcho-socialism. None of these ideologies are sufficient, but the conversations and debates are converging.

- Learn history and social/economic sciences. The history of debt money, the collapse of rome, today's corporate power structures, etc.

- Understand how the dollar works and how it's controlled and enforced. Support alternative currencies.

- Support open hardware. The EFF is your friend. Learn more about encryption and understand the vulnerabilities online and offline.

I could go on, share links, etc.


> find the commonality between libertarianism, anarcho-capitalism, and anarcho-socialism.

There is no commonality. At it's core, anarchism wishes to abolish capitalism, and libertarians/ancaps wish to keep it. It's a fundamental disagreement that starts the conversation off. There's no reconciliation here.


Well, both camps agree that the State as we know it today is mostly a Bad Thing, so there is some common ground. That said, you're right to suggest that it's hard for anarcho-socialist types and an-cap types to cooperate. Strangely enough, however, I find that it's usually the anarcho-socialist types who want nothing to do with an-caps, even when our goals do overlap. I guess they don't care much for the "the enemy of my enemy is my friend" thinking, even in a limited scenario.

Personally, I think if we could accomplish a RADICAL reduction in the size, scope and power of the State, that that would be a great first step, and we could then start figuring out what to do about economic systems and the property debate. But that's just me...


Saying "their goals overlap" is a fundamental misunderstanding of the anarchist position. Ancaps are not "the enemy of my enemy," they are my (in my anarchist days...) enemy: capitalists.

If you radically reduce the state without protecting the poor, you end up doing more harm to the poor. That's why anarchists are, for example, anti-austerity, even though it could be construed as "making the state smaller." Anarchists are for _the liberation of the working class_, (and all people...) and happen to believe that the state is one of the largest obstacles in achieving that goal. But removing the state without protections is just going to do far more harm than good.

You can also see this in their histories: anarchism comes from the First International, and is primarily about a means dispute with Marxists. ancap/voluntaryism comes from Rothbard, (who incidentally said that ancaps aren't anarchists...) who is... just a bit different than Marx. To understate it.

Ancaps and anarchists have almost nothing in common.


... besides the common goal of achieving peace and liberty by abolishing the tyranny of government. So yes, they do have a lot in common.

Anarcho-socialists (what steveklabnik calls "anarchists") have a reactionary attitude towards "capitalism" and see capitalism as the source of government tyranny. You'll see them claiming the word "anarchism" for themselves, and rejecting any form of capitalism.

Yet you'll find that in denying the freedom of voluntary trade, this particular group fail to propose an alternative that doesn't just turn into another tyrannical state.

I say, find an anarchist who can understand the Bitcoin protocol, and you'll find yourself an anarcho-capitalist.

The problem with anarcho-capitalism is that they believe the free market will solve everything, and they don't believe in environmental limitations, and they have no proposed mechanism of protecting the liberties of the poor. (the poor can't buy freedom, damnit).

In practice, most people admit that there should be personal property (such as your home and your body), and that there should be limitations to property (there are more empty houses in san francisco than homeless people). It's just common sense. Between anarcho-socialists and anarcho-capitalists, there is a way to achieve a good balance.


Yet you'll find that in denying the freedom of voluntary trade, this particular group fail to propose an alternative that doesn't just turn into another tyrannical state.

Yeah, that's always been one of my issues with anarcho-socialists - in the obsessive quest to eliminate all hierarchy, it seems that they've created a system that could only be enforced through... a hierarchy. Unless you can get every human being in a group larger than about 2 to agree on something, that is. And I'm not betting any money on that happening.

In practice, most people admit that there should be personal property (such as your home and your body), and that there should be limitations to property (there are more empty houses in san francisco than homeless people). It's just common sense. Between anarcho-socialists and anarcho-capitalists, there is a way to achieve a good balance.

Part of the problem with that is this: When you start having those kinds of discussions, what you refer to as "common sense" often winds up being a form of Utilitarianism, or based on Utilitarian arguments. And while Utilitarianism isn't necessarily Bad in any inherent way, it's an ideology like any other, and taken to it's further-most logical extremes, also leads to Bad Things being supported. See: the whole swathes of novels written about the evils of medical utilitarianism[1] gone too far.

For this reason, among others, finding a common ground between (left) anarchism and anarcho-capitalism isn't necessarily as easy as it might seem.

[1]: http://en.wikipedia.org/wiki/Utilitarian_bioethics


Saying "their goals overlap" is a fundamental misunderstanding of the anarchist position

Right, but I didn't say that. Not as a universal statement, anyway. I'm saying that there are certain, specific situations / circumstances where they do overlap.

Ancaps and anarchists have almost nothing in common.

Agreed, but the word I'm focusing on is the "almost". Where there is something in common, we could all probably accomplish more by working together a bit. But, in my experience, an-caps are often open to collaboration with anarchist-socialists / left-anarchists / whatever-term-you-prefer, but those guys usually don't want anything to do with us. I guess I'd say it's that we don't see "traditional" anarchists as our enemy, but they see us as theirs. But what can ya do? shrug


> Right, but I didn't say that.

Fair enough. Frankly, I probably just should have not even replied, as my hostility stems from being sick of having this conversation over and over and over. See your siblings implying that anarchists are against 'free trade', or that they don't understand bitcoin...

The rhetoric when you understand both sides of this is immensely tiring.


Well go on, make your point.

I never tire of correcting someone, even correcting myself or allowing others to correct me. So go ahead and convince me of something.


As I said above, I'm not really in the mood, but since I made a snipe-y comment at you, I guess I at least owe you this:

1. No anarchist wants to stop you from trading things with anyone. Capitalism != trade. The issue is ownership of the means of production, not movables.

2. I personally understand Bitcoin quite well, which is one of the reasons I don't like it. however, I've seen quite a few of my fellow anarchists support it thoroughly, to the point of saying "If you don't support Bitcoin, you're not an anarchist." I _will_ say that I'm warming up to it slightly as a certain tactic for certain situations.

As I'm already irritated, you'll have to forgive me if I don't say more than that. Maybe another time.


> I could go on, share links, etc.

Please do.


We're understanding our reality better. Whistleblower leaks and more...

- http://www.youtube.com/watch?v=vSSKpL87_Rs

- http://www.boilingfrogspost.com/2013/06/19/podcast-show-112-...

- http://m.democracynow.org/web_exclusives/1781

- http://www.dailykos.com/story/2013/06/23/1218189/-HBGary-Pal...

Books illuminate how power works

- http://www.amazon.com/The-Master-Switch-Information-Empires/...

- http://en.wikipedia.org/wiki/Debt:_The_First_5000_Years

- http://en.wikipedia.org/wiki/A_People's_History_of_the_Unite... (I haven't read it yet, I've heard good and bad things about the book)

- http://daviddfriedman.com/The_Machinery_of_Freedom_.pdf (I've only skimmed a few sections, but I suspect this is good to understand anarcho-capitalism)

To combat low level surveillance, we need open hardware.

- http://en.wikipedia.org/wiki/List_of_open-source_hardware_pr...

- http://www.bunniestudios.com/blog/?p=2686

Our own mesh network for community penetration. Mesh networks sort of died off a decade ago, but there is a resurgence.

- http://karmanebula.com/chicagomeshnet/

- https://www.wepay.com/donations/oakland-community-mesh-netwo...

Bitcoin is still growing, and more people are understanding the mechanism behind it. Many altcoins are just Bitcoin ripoffs, but we'll see interesting developments going forward.

- https://cryptocointalk.com

- There are many technologies and protocols being built for distributed exchanges, etc. Discussions under bitcointalk.org etc.

Other links

- http://liberationtechnology.stanford.edu/

- https://mailman.stanford.edu/pipermail/liberationtech/2013-J...

- http://en.wikipedia.org/wiki/Libertarianism

- http://www.democracynow.org/

- http://www.dailykos.com/

Those have just been on my mind recently. Reddit is actually a good source of reading material once you have an account, unfollow stupid subreddits and follow the right ones like politics, worldnews, restorethefourth, libertarian, anarcho_* etc.


I'm sure something will eventually happen. Maybe not tomorrow, maybe only in 5 years.

The thing is, Edward Snowden was the 1st one to show us he's ready to die for this cause.

I'm sure the number of people willing to go further will only increase. And that will ultimately lead to actions.

EDIT: The movement will "only" have to make sure not be sabotaged and killed off by the surveillance state, along the way.


EDIT II: The thing is, life under these circumstances (and the logical consequences which will appear, sooner or later) is only worth so much.


To me, much more concerning is Snowden's revelation that the British intelligence program 'Tempora' "saves everything":

"Tempora is the first 'I save everything' approach ('full take') in the intelligence world. It sucks in all data, no matter what it is, and which rights are violated by it. ... Right now, the system is capable of saving three days' worth of traffic [...]. 'Full take' means that the system saves everything. If you send a data packet and if makes its way through the UK, we will get it."


To me there's something of the "if a tree falls in a forest" koan here.

"If an agency caches the data but doesn't grep the data for anything related to me has my privacy been violated?"

I would be a lot more worried if it was a US and not a UK agency.

I would also be a lot more worried if it wasn't GCHQ, but was a police force.

There are other databases in the UK that are a lot more concerning to me. All the stuff that supermarkets collect on people with loyalty cards[1], all the medical databases, the misuse of criminal records bureau checking. (These last two have real world examples of real world harm being done to individuals. But perhaps GCHQ's doesn't because it's secret and the information is shared with weird routes.)

[1] Many years ago some friends and I had a loyalty card for "Ivor Trots". He only ever bought loo roll.


If an agency caches the data but doesn't grep the data for anything related to me has my privacy been violated?

If it's stored indefinitely (as they have the ambition to do, and will soon have the ability), it doesn't matter if it is not used today, or in the past. What if at some point in your life you run for parliament, or your business strikes a prominent deal with a foreign power? At that point your records can be taken out and scrutinised by whoever has access to this data at that point in time, and you won't even know, because it's secret. Information has a habit of spreading beyond the intended boundaries, and if it is collected in secret, we have no way of controlling that spread or controlling its use.

I would be a lot more worried if it was a US and not a UK agency.

You should be worried then, because this data is routinely shared with the NSA, seems to be collected under the remit of a joint program, and is probably stored indefinitely by several GCHQ partners where someone is judged to be of interest or enemy of the state/agency, the definition of which will of course vary with the agency.

I would also be a lot more worried if it wasn't GCHQ, but was a police force.

I'd be surprised if access is not given the police on some level, given their participation in counter-terrorism activities, and there's nothing to stop a secret policy change giving more access, you wouldn't even know about it. So you should be worried.

There are other databases in the UK that are a lot more concerning to me.

Given that those databases exist online, and are probably transmitted regularly as backups etc as well as accessed via the internet, you can assume that all that data is also accessible by GCHQ partners and clients. The important difference though is that, as you point out, these databases are public, acknowledged to exist, and controlled by our existing law. If you want to you can challenge the use or collection of the data in a court, sometimes you can stop using these services, or ask to see all information collected on you, but no such options are available for information gathered and shared in the name of security.

What I find troubling here is that the only limit on the indefinite storage of all this material by GCHQ is technical, not legal or moral, and that limit will soon be overcome.


> You should be worried then, because this data is routinely shared with the NSA, seems to be collected under the remit of a joint program

Exactly. Snowden says "If you send a data packet and if [sic] makes its way through the UK, we will get it."

Given that Snowden is an NSA whisteblower, who's talking here about the "Five Eyes" intelligence alliance (U.S., U.K., Canada, Australia and New Zealand), this suggests pretty strongly that the "we" is the NSA.


In any other situation I can think of, the important part is copying the data and not accessing it. If I copied someone's personal files, or files containing a company's trade secrets, or top secret files from the government, but promised to never look at them I'm still breaking the law. This might be a simplistic view, but why should it be any different for the government?


Why would the UK worry you less than the US? Just a matter of scale, or something structural?


Because of my personal, irrational biases.

I tell myself that the UK hasn't demonstrated contempt for the rule of law to the same degree as the US has.

I accept that this is probably wrong, and had I been more aware of my bias I wouldn't have used that wording in the post you replied to.

It's good that I keep challenging myself on this stuff, so thanks for pointing it out.


even more concerning when coupled with the fact that no UK news outlet is covering Tempora due to the D-Notice.

http://www.guardian.co.uk/world/2013/jun/17/defence-d-bbc-me...


Worse than that, the BBC is running puff pieces like this one today telling us how GCHQ kept the lights on at the Olympic ceremony:

http://www.bbc.co.uk/news/uk-23195283

Telling us how useful GCHQ is in the fight against terror without mentioning Tempora.


I find the drip-feed of information a really interesting choice, though I'm curious of its efficacy versus the wikileaks style dump.

It keeps the news alive for a longer period of time, sure, but it also means fewer eyes are looking over it and connecting the dots between various different pieces of the revealed information.

Wouldn't it be better if everything was revealed at once, and the various investigative reporters out there all got to go through it independently and write up a cohesive summation of the results, instead of this two-three page exposé per day?


No, I think this is incredibly clever. Imagine how this release would have looked if the Australian government had thrown a fit and tried to make an example of the USs bad behavior[0].

If you're dealing with a criminal, having a slight upper hand in information while playing it as though you know everything is a proven tool. Like a police interview where an accused gets presented with a crime so you can watch him or her come up with an excuse, which, in turn, might reveal information that you didn't even know before.

In fact, one minister in Germany made a pretty grave statement about how the US behavior was "monstrous". So now I'm kind of waiting for the Snowden story to get to the part where things about Germany are revealed.

[0]It seems like they did the opposite, as they usually do, doubling down in 'Merica adoration. Which is still similarly funny.


Would Breaking Bad be as good if every episode were released at once?


this isn't for the same reason, its because they learned from wikileaks that the publics ability to absorb information is limited. a lot of important issues were burried in the wikileaks' releases.


Maybe it's done in view of personal security? Maybe he doesn't want to make all his enemies at once.


He's also in the middle of finding out who his enemies already are. I'll be _very_ surprised to see Bolivia, Venezuela, or Nicaragua show up on the leak schedule.


If he leaves out that data specifically then he'd being doing a great injustice to those countries.


A more favorable interpretation might be that the countries who are offering asylum quite likely have nothing to hide.

Russia's conditions are kind of telling.


News seems to last 5 seconds these days so I'm all for this choice.


I'm curious of its efficacy versus the wikileaks style dump.

How do you propose to measure such a thing?


The same O Globo article reported a list of priority targets of the NSA and they are basically the BRIC ( interestingly the population here did not gave a shit about knowing US spies on them) also Iran andsome other powerful countries. North Korea,.Cuba, Venezuela or Afghanistan are not on the priority list, making clear that the "terrorism" is just a misdirection, and that PRISM true purpose still is.intelligence for potential.enemies.in a.conventional war.


This is funny, if you actually google the keyword X-Keyscore, you'll find job opportunities that matches the criteria of intelligence gathering. And if you look closely, you'll find out it's a company called Raytheon that's awarded the contract to execute these works. And they have main offices in these crucial locations, eg; Fort Meade, Australia etc. And that they do all these kind of intelligence works.

And ironically, Raytheon's scientist was the dude who invented microwave.

Edit: Found this post from Feb 27th. About X-Keyscore, http://www.nowtheendbegins.com/blog/?tag=xkeyscore and the interesting snippet.

What happens next looks like a 21st-century data assembly line. At the NSA’s headquarters in Fort Meade, Maryland, a program called Xkeyscore processes all intercepted electronic signals before sending them to different “production lines” that deal with specific issues. Here, we find another array of code names.

Pinwale is the main NSA database for recorded signals intercepts, the authors report. Within it, there are various keyword compartments, which the NSA calls “selectors.” Metadata (things like the “To” and “From” field on an e-mail) is stored in a database called Marina. It generally stays there for five years. In a database called Maui there is “finished reporting,” the transcripts and analysis of calls. (Metadata never goes here, the authors found.)

As all this is happening, there are dozens of other NSA signals activity lines, called SIGADS, processing data. There’s Anchory, an all-source database for communications intelligence; Homebase, which lets NSA analysts coordinate their searches based on priorities set by the Director of National Intelligence; Airgap, which deals with missions that are a priority for the Department of Defense; Wrangler, an electronic intelligence line; Tinman, which handles air warning and surveillance; and more.

Lest you get confused by this swirl of code names and acronyms, keep this image in mind of the NSA as a data-analysis factory. Based on my own reporting, the agency is collecting so much information every day that without a regimented, factory-like system, analysts would never have the chance to look at it all. Indeed, they don’t analyze much of it. Computers handle a chunk, but a lot of information remains stored for future analysis.


I struggle to believe that GCHQ has the ability to store 3 or even 30 days worth of internet traffic flowing through the UK borders and also make reasonable sense of it all. It's such a mind boggling amount of data. I looked at some bandwidth graphs from places like LINX, which is just one of presumably many, and they are moving around a terrabyte a second - how can you possibly store, let alone sift through that kind of volume? 30 days at a terrabyte a second is nearly 3,000 petabytes of data. And that's just in one exchange. Is this kind of thing feasible? Facebook allegedly deals with 200PB of photo data - 3000PB is a lot more than that.


Three seconds of data at 1TB/s, will fit on the largest consumer hard drive (3TB). There are 3 x 86400 seconds in a day = 259200 seconds. Therefore three days' data requires 86400 drives. Cost of said drives is about $100 each, meaning the cost of disks to store 3 days of data is $8.64 million, which is negligible to the government. Even with multiple links at 1TB/s the problem is tractable.


Storing it really is the trivial and cheap part. There are other really hard problems to solve:

(1) Where do you place collection points so you get a full take of not only international traffic (moving across your borders) and domestic traffic (all traffic within your borders but that doesn't leave your borders) (2) Since the number of collection points is limited, that means there is a lot of data that has to be recorded at select points. How do you record that data to disk in real time? (3) How do you avoid the duplication of packets that travel through multiple collection points. (4) Lastly, the most difficult problem is figuring out how to query all that data and not end up with a haystack. When you have millions and millions of pieces of communication from people with no involvement in the criminal activity, then all that communication becomes noise.


1) If it's your own country's traffic, at the choke point in the network, which you (ie. govt) have full control over, since they are domestic.

2) Since you already have 100,000 disks, run 5000 of them in parallel. Each disk interface does 200MB/s, for an aggregate of 1TB/s. If necessary, use fibre to transport data from the collection point to the storage point.

3) Don't. Record the lot.

4) Spend a billion on a super computer?

I think it comes down to the bandwidth of computers now vastly exceeding the bandwidth of human thought. It is now possible to record an person's entire life, with plenty of headroom to spare, to account for any attempt to overwhelm the recording system.


I know, i did the numbers too - but it's still an almost unthinkable amount of data to try and make sense of. If we conservatively say that there are 5 others like linx, that's nearly half a million 3TB drives. And that's only for 3 days! edit: clarity


> but it's still an almost unthinkable amount of data to try and make sense of.

It's only money, and compared with keeping a division supplied in Afghanistan or putting an enormous reconnaissance satellite in space, it's an obvious winner in the minds of the people running the security apparatus.


I thought Snowden had already released everything and it was up to the media to decide what will be released when?


Isn't that nitpicking? Snowden released some information through the Guardian.


It is only nitpicking if you consider the Guardian to have zero editorial control over what they publish.

There is a supposed distinguishable feature between a user/self publishing website, and a publisher with editorial control. One is considered as free from responsibility regarding the content and get legal protection to said effect (with exceptions like DMCA), and the other is supposed to be responsible for the content being published. News papers has always been responsible when they publish secret information, but are protected under the freedom of the press. Self/users-publishing sites like youtube, wikileaks, facebook, twitter, flickr and so on is not responsible for the content available on their websites (again, with some exceptions). Content on those sites are published by users, and the user is responsible.

One could of course argue that Snowden is ultimate responsible for providing the Guradian the ability in publishing the information, through that would be a slight lie. "Ultimate" would mean that there aren't anyone with even more responsibility, which the term whistleblower tend to disagree with. As such, there is plenty of blame to throw around if one wishes to build a blame tree.


Russia said it would grant him immunity under the condition there were no more leaks. So who is responsible for this leak is certainly relevant.

That said I could imagine this is enough for Russia to make their case either way.


I'm pretty sure Snowden has already refused asylum from Russia, he wouldn't be stuck in an airport otherwise.


I'm a bit confused too.

From the article:

"Classified US National Security Agency maps leaked by Mr Snowden and published by US journalist Glenn Greenwald in the Brazilian O Globo newspaper reveal the locations of dozens of US and allied signals intelligence collection sites that contribute to interception of telecommunications and internet traffic worldwide."

It sounds like the Guardian released this information, but I'm not really sure.


The Australian journalists just "decoded" the references to Australia from the slides first time presented in Brazil's O Globo by Glenn Greenwald some days ago.


iirc he did store the files is various places on the internet, encrypted.


[deleted]


Time is a tool that Snowden is using well.

By dumping everything at once he creates work for everybody else to scour through for the interesting bits.

Instead by releasing bits at a time he lets us digest it all before moving on to the next info.

As for enemies, yes, he may never leave the Moscow airport alive but I'm pretty sure he understands this.

He's doing what he feel is right, not what is easiest on him.


It's the info about what Wikipedia knows as

http://en.wikipedia.org/wiki/ECHELON

As you can see in Wikipedia, a lot of it was known for many years (decades?) The new thing is that these topics got a bit more exposure mostly because of the thriller-like situation Snowden is in.

The Australian journalists actually just "decoded" the references to Australia from the slides first time presented in Brazil some days ago.


This is... WOW..


I thought we were clear of this shit.


-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Really?

My immediate thoughts when Snowden's first revelations came to light were "I wonder just how much our local Australian government has rolled over and played lap-dog to every US intelligence request, no matter _how_ illegal or immoral?"

I also have very little faith in the Australian civil rights movement - the US has some powerful, smart, and noisy civil rights activism - here I see very little that appears in any way effectual.

Encrypt all the things… -----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAlHabm0ACgkQniy+nkMiY+13aQCgzAN24gQyZpyKZvw38qao2QN2 ryUAn0tLNaZWRMuExrQGHUMzgjB5ourZ =Sd2U -----END PGP SIGNATURE-----


Well if your local job centre front desk can access your metadata with the nod of your supervisor upstairs, or if the upcoming referendum on local councils coming under federal auspices succeeds, and would allow as a consequence all levels of government to probe citizens metadata, I'm sure the NSA can.

The furore over the NSA data hoovering and the fourth amendment has no parallel in Australia. The general citizenry is completely uninformed, complacent and lazily uninterested in these matters.

http://m.heraldsun.com.au/news/law-order/victorians-private-...

http://powerhouse.theglobalmail.org/the-australian-governmen...


Non-Americans are going to have to come to terms with the likelihood that their governments cooperate with the US not because of blackmail or coercion or fear, but because they agree with the justification for and procedures surrounding this kind of surveillance, and very much appreciate the US's willingness to throw billions of dollars into the pot to help it happen.


You thought wrong, and apparently there is more to come, so put on your seatbelt.


Not for a long time, I suspect.


I wonder what would cause you to say something so stupid.

When you say "this shit" do you mean the negative image the USA is gaining from this? Why on earth would you prefer to remain in ignorant bliss about these happenings?


I read that not as "stupid", but (perhaps incorrectly) as a fellow Australian finally admitting to themselves that we're fooling ourselves if we think we can point an laugh at USA's treatment of their citizens while hiding behind some expectation that our local government is treating us with any more respect.

The sad thing is, our government is no better, out intelligence service is just as amoral and unaccountable, and our civil liberty movement is ineffectual to the point of being invisible. Perhaps "ignorant bliss" was preferable…


Actually it is likely worse than the US, the US government spying would generally stay US government only. Whereas ours in Australia is likely accessible by US agencies.


Indeed, and given the NSA's unique interpretations of some fairly commonly understood english words (like for example "collect" and "no"), I have no doubt that they're capable of creative legal interpretations where they can look their elected representatives and (in theory) overseers in the eye and say "we're not spying on US citizens" - claiming later if caught that it's "the least untruthful answer" - when what they really meant was "we asked the Australians to spy on US citizens for us, and hand over everything they found. _We_ didn't engage in any spying."

On the plus side, while I have niggling doubts about whether GPG and encfs/OpenSSL/AES are really secure against the NSA - I'm reasonably sure that even if they've got practical attacks against them, they aren't likely to be sharing even the existence of them with ASIO. I'm as close to 100% certain as makes no difference that GPG/encfs are secure against even the most powerful Australian government agencies (which is to say, only as secure as anything that'd break easily with rubber hose cryptography…).


No, I thought that Australia was independent enough that we wouldn't be ferrying our data to the United States. It's a reasonably enough assumption.


People on HN have been mentioning ECHELON and five eyes. ECHELON is a system from the 90s, and involved Australia, New Zealand, Canada, UK, and the US.


Five Eyes itself dates back literally to the last World War as well.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: