The one continuing weakness in security is the human element. You would have to have physical security on all of those agents. Unless you were making a small "internet" you'd have to have other people in the mix to help you maintain it.

Once you bring in other people you have to consider the level of trust in all of them. I don't mean that people are necessarily evil but you'd also have to be able to trust them to do their job as completely as you would, negligence can bring down a network as quick as malevolence.