Swarm Intelligence

Reading through my blog feeds I came across something Hoff wrote in response to Reuven Cohen’s “Elastic Vapor: Life In the Cloud Blog, in particular I wanted to respond to the the following comment (here)

This basically means that we should distribute the sampling, detection and prevention functions across the entire networked ecosystem, not just to dedicated security appliances; each of the end nodes should communicate using a standard signaling and telemetry protocol so that common threat, vulnerability and effective disposition can be communicated up and downstream to one another and one or more management facilities.

I also wrote about this concept in a series of post on swarm intelligence…

Evolving Information Security Part 1: The Herd Collective vs. Swarm Intelligence (here)

The only viable option for collective intelligence in the future is through the use of intelligent agents, which can perform some base level of analysis against internal and environmental variables and communicate that information to the collective without the need for centralized processing and distribution. Essentially the intelligent agents would support cognition, cooperation, and coordination among themselves built on a foundation of dynamic policy instantiation. Without the use of distributed computing, parallel processing and intelligent agents there is little hope for moving beyond the brittle and highly ineffective defenses currently deployed.

Evolving Information Security Part 2: Developing Collective Intelligence (here)

Once the agent is fully aware of the state of devices it resides on, physical or virtual, it will need to expand its knowledge of the environment it resides in and it’s relative positioning to others. Knowledge of self, combined with knowledge of the environment expands the context in which agents could effect change. In communication with other agents the response to threats or other problems would be more efficiently identified, regardless of location.

As knowledge of self moves to communication with others there is the foundation for inter-device cooperation. Communication and cooperation between seemingly disparate devices, or device clusters, creates collective intelligence. This simple model creates an extremely powerful precedent for dealing with a wide range of information technology and security problems.

Driving the intelligent agents would be a lightweight and adaptable policy language that would be easily interpreted by the agent’s policy engine. New polices would be created and shared between the agents and the system would move from simply responding to changes and begin to adapt on its own. The collective and the infrastructure will learn. This would enable a base-level of cognition where seemingly benign events or state changes coupled with similarly insignificant data could be used to lessen the impact of disruptions or incidents, sometimes before they even occur.

The concept of distributed intelligence and self-healing infrastructure will have a major impact on a highly mobile world of distributed computing devices, it will also form the foundation for how we deal with the loss of visibility and control of the “in the cloud” virtual storage and data centers that service them.

Web threats are up 1564% since 2005, vulnerabilities continue to number in the thousands annually, malware infections have skyrocketed to over 8 million in November of 2007 alone, SPAM accounts for up to 90% of all email traffic, there is an estimated 3 million plus bot-compromised machines connected to the internet at any given moment, high-impact regional threats and targeted attacks have increased dramatically year over year since 2005, and there is a breach a day in what has become an orgy of disclosure, punctuated by a tsunami of useless loss statistics. This is all against a backdrop of new vectors of attack introduced by mobile computers, virtualization, SaaS, and other disruptive technologies. Clearly the current reactive, ad-hoc, threat enumeration, information security model is broken and given the economics of malware and cybercrime it will only get worse…

Sample data from research on the underground digital economy in 2007 from Trend Annual Threat Report 2007 (here)

Pay-out for each unique adware installation – $.30 in the US

Malware package, basic version $1,000 – $2,000

Malware package with add-on services – $20 starting price

Undetected copy of an information stealing Trojan – $80, may vary

10,000 compromised PCs – $1,000

Stolen bank account credentials – $50 starting price

1 million freshly-harvested emails – $8 up, depending on quality

Recently I posted some thoughts on evolving information security to move towards distributed, collective intelligence or swarm intelligence, (here) and (here), and came across a project at the University of Washington called Phalanx – (here) via /.

Their system, called Phalanx, uses its own large network of computers to shield the protected server. Instead of the server being accessed directly, all information must pass through the swarm of “mailbox” computers.

The many mailboxes do not simply relay information to the server like a funnel – they only pass on information when the server requests it. That allows the server to work at its own pace, without being swamped.

“Hosts use these mailboxes in a random order,” the researchers explain. “Even an attacker with a multimillion-node botnet can cause only a fraction of a given flow to be lost,” the researchers say.

Phalanx also requires computers wishing to start communicating with the protected server to solve a computational puzzle. This takes only a small amount of time for a normal web user accessing a site. But a zombie computer sending repeated requests would be significantly slowed down.

This is a very interesting way to deal with the problem of DDoS attacks, it isn’t difficult to imagine how one could use a swarm of intelligent agents to cooperate and shield, or even work to identify patterns of behavior that are representative of malicious or nefarious actions and counter an attack in progress or impending attack before it has a chance to impact the environment.