The integration of security into virtual machine (VM) technology is an exciting and encouraging prospect that is likely to provide an additional line of defense in the never-ending fight against malware and malicious activity.

A VM typically refers to running multiple operating systems simultaneously on a single physical computer. This has long been the territory for high-end servers or test labs, but new software and hardware is bringing virtualization to the mainstream. Aside from letting users support a variety of applications on more than one operating system on the same physical computer, this also offers new possibilities for computer security.

A major historical challenge for security vendors is that security products are almost always running in a potentially infected or compromised environment. The operating system and security software can be tricked or disabled altogether by malicious code. In the past, the fix has been to create an “emergency disk” with a bootable OS image or file-system reader, reboot the machine from a write-protected device, and then scan the computer.

But this is not a workable option not only because of the size and complexity of security products, but also because proprietary operating systems have not provided inexpensive or practical access to the file system without using the entire OS (or its license). Furthermore, this solution forces someone to physically visit the machine to provide a security check and the approach does not provide real-time protection or detection.

A virtual machine can do better. There are numerous opportunities for future security products to protect applications and services from isolated environments. One that has garnered some attention is running security software alongside the operating system it is protecting. We call that “the sentinel.”

In this approach, the security software itself resides in its own virtual machine outside and parallel to the system it is meant to protect, which could be another virtual machine running an operating system such as Windows. This enables the security technology to look omnisciently into the subject OS and its operation and take appropriate action when malware or anomalous behavior is detected. The security software would run in an uncompromised environment monitoring in real-time, and could avoid being disabled, detected or deceived (or make the bad guys work a lot harder.)

This kind of security is not necessarily a one-to-one relationship between sentinel and OSs. One physical machine can run several virtual machines, so one virtual sentinel could watch and service many virtual machines.

The effectiveness of security products in a VM is based on the premise of trust barriers. The VM or hypervisor code itself is not designed to be a general purpose OS. Ideally very little should be allowed to run on the VM. And very little functionality should be exposed.

Weaknesses in a VM or the technology that supports it could pose a threat. This threat must be considered possible when designing security for these systems.

Code signing, while impractical to secure a general purpose OS such as Windows, could be used to enhance the security of virtual machines. Code signing uses digital signatures to make sure executables and scripts have not been tampered with. The process is too cumbersome and expensive for a general OS, but a virtual machine is only meant to run specific code and therefore only few things need to be signed.

Virtualization is a changing field and the potential for security to be tightly integrated into this technology is an exciting prospect. Look for more posts here soon concerning the use of virtualization providing security.