Also, this isn't a hypothetical attack. Basically the same setup is used for client-side bitcoin wallets, and there have been reports of thefts (stolen keys).