The secret life of a rootkit

Still a rarity but likely to become less so.

Much as in the movie The Matrix, where the world presented is not a true presentation of what is really going on, a rootkit is a program that can be used to hide files, registry settings, network connections, processes and other information from computer users.

For example, a rootkit can make processes that run but are hidden from Windows Task Manager, registry keys that can't be seen with Regedit, and network connections that are not viewable by Netstat.
Rootkit technology allows malicious software (malware) to be stealthier, and that in general makes it more effective.

This is not about just evading detection by a casual user; rootkit technology allows malware to evade many antivirus and antispyware programs. There are few legitimate uses for this kind of technology, although some companies do market "hidden folders" that enable users to hide sensitive or embarrassing information from other users of the same computer.

All rootkits rely on the ability to manipulate the results of the function calls made by programs. For example, in order for the Task Manager program in Windows to show a list of running processes, it calls a Windows API function (EnumProcesses) that returns a list of identifiers (or process IDs), which are obtained from a data structure in the kernel. A rootkit works by intercepting the call and filtering out the processes that it is trying to hide. They can be implemented either in user space or in the kernel, with the kernel rootkits being the most dangerous.

Files, registry entries and network connections can all be hidden in analogous ways by altering the results of the appropriate function calls. Because most antivirus and antispyware programs rely on these calls (for example, to find files to scan), files hidden by rootkits are invisible to an antivirus program. The machine could be infected, but an antivirus program would be unable to detect it.

Kernel-mode rootkits require some code to be loaded into the kernel (normally a device driver or .sys file). They can do this by following the legitimate route that low-level device drivers use (using the service control manager services .exe), or there are a few undocumented ways to insert code into the kernel. Once inside, the code can modify the results of functions calls made into the kernel or modify kernel structures.

How to spot a rootkit
There are two main ways to detect the presence of a rootkit on an infected machine: scanning and event monitoring. The scanning technique involves comparing a view of the system using user-space tools and a view from inside the kernel. If anything is hidden, it should be visible in the kernel, but not in user space. Recently, a variety of programs have been released that do these scans.

This technique is good in principle - rootkits hide resources, so the best way to detect them is to look for things that are hidden. There are, however, a couple of weaknesses with this approach. The first is that if the kernel itself has been compromised, then the scan from kernel space may be tainted by the rootkit.

Whether this happens or not depends on the details of exactly how the scan and rootkit are implemented. Since the Windows kernel is largely undocumented, it is hard to be sure that the scan is giving the correct results. Also, rootkits can evade detection by hiding from all processes except the rootkit detector.

The alternative approach is to use an event-based system that monitors continuously to catch the rootkit in the act of installation. These programs are often called intrusion-prevention systems (IPS). It is important that the program behaviour is monitored from the kernel. IPS systems that monitor in user space are just as vulnerable to rootkits as any other user-space program.

These systems can detect and block the loading of kernel modules. However, blocking all kernel modules is impractical - many legitimate programs install kernel modules. For example, some antivirus programs use kernel modules to perform on-demand scanning.

It is possible to make a better decision regarding whether the loading of the module is malicious by looking at other properties of the installer and other associated programs. While a rootkit and an antivirus program might have actions in common (e.g., installing a kernel module), there are many other characteristics that they would be less likely to share.

For example, a rootkit might try to be stealthy by not having a visible window, while the antivirus program will likely advertise its presence to reassure the user. The rootkit program may also install a keylogger, which one would not expect a well-behaved antivirus program to do. By combining various behavioral characteristics (carefully chosen so that they capture the common behaviors associated with malicious code), it is possible to reliably detect rootkit programs. In fact, this general approach, termed "behavioral heuristics," can be applied more broadly, to detect other classes of malicious code such as Trojans, bots and spyware.

Being based on heuristics, this type of system can make mistakes (classifying normal programs as malicious). The normal technique used to deal with this is to have exclusion lists for the common errors, which would then have to be maintained.

Undoubtedly, rootkits are serious and increasingly problematic for computer users. They enable malicious code to remain undetected by most security software. They work by manipulating the results of function calls, which they gain access to by a variety of mechanisms.

The good news is that they are detectable, with two mechanisms currently available - a scanning technology that can detect them after they are installed, and an event-based system that can catch them in the act of installation (by observing their behavior) - and thus prevent compromise in the first place.

Matthew Williamson is senior research scientist at San Mateo, Calif.-based Sana Security


What are your views on this subject? Use the form below to post a comment on this article up to 500 characters.


Characters remaining: 500

Related Security news

Hacker attacks on US military jump sharply in 2009

China source of most attacks, says report

Microsoft denies building security 'backdoor' in Windows 7

Privacy organisations shouldn't read too much into NSA involvement it says

Pentagon expands exclusive deal with McAfee

Department of Defense uses McAfee products

Police arrest pair over global banking web scam

Man and woman arrested in Manchester for using notorious Zeus Trojan



Email this article to a friend or colleague:


PLEASE NOTE: Your name is used only to let the recipient know who sent the story, and in case of transmission error. Both your name and the recipient's name and address will not be used for any other purpose.

Techworld White Papers

Database security: Preventing enterprise data leaks at the source

IDC discusses the growing internal threats to business information, the impact of government regulations on the protection of data, and how enterprises must adopt database security best practices...

Download Whitepaper

Service-oriented security

SOA has become an integral part of enterprise software by providing a framework to efficiently develop software as services that is easily sharable, reusable, and integrated. No where is the need more apparent than in the Identity Management space. Welcome to the age of Service-Oriented Security (SOS).

Download Whitepaper

Data protection prospective vendor checklist

Organisations need a way to map business needs against all these challenges in procuring a technical solution. To help, SANS has developed the following Prospective Vendor Checklist.

Download Whitepaper

Unlock the power of the mainframe

This whitepaper presents the notion of CICS as an integration hub based on a component-based, service-oriented architecture supporting Web services. Highlights will review the challenges and contrasted support for Web services natively in CICS.

Download Whitepaper

Techworld UK - Technology - Business

COLT White Paper

Are all VoIP services the same?

Questions to ask your service provider to ensure you get the VoIP service you need
With careful choice of partner, your business can have all the advantages of VoIP access - reduced costs, flexibility and simplicity - without the drawbacks.
This white paper is your guide to ensure you get right the VoIP service and details the pitfalls which businesses would do well to avoid.

Download white paper
BMC

Ride the express lane in the journey to speed ITIL adoption

Explore the challenges in making the journey to ITIL and the criteria for selecting consulting services
By following ITIL practices, your IT organisation will become more closely integrated with the business. We recommend making the journey to ITIL in a sequence of six incremental steps, the phases of which are driven through execution of a strategic transformational roadmap.

Download white paper

Webcast: IT Financial Management: Cost Optimisation for Efficiency and Agility.
On Demand Webcast
Join this webcast to learn about the techniques and technologies that can help you prove the value of IT to the business by understanding the true cost of today's IT services and those that will be necessary to deliver future success.

Register Today

Site Map

IDG Network

* *