Questions dog Cisco routers
Cisco vs Michael Lynn has a way to run.
By Ellen Messmer and Phil Hochmuth, Network World | Network World US | Published: 13:00, 15 August 2005
Heavy fallout continues on several fronts from a security researcher's recent disclosure that unpatched Cisco routers can be subverted by buffer-overflow attacks and shell-code exploits.
Among the developments of recent weeks: Cisco continually revised its security bulletin, adding details as to how versions of unpatched IOS software could be undermined by a "specifically crafted IPv6 packet." Sources at Cisco say testing will continue indefinitely and could include findings related to more than simply IPv6-related exploits.
The researcher who touched off the uproar, Michael Lynn, says he is now the subject of inquiries by FBI agents, and he continues to defend the propriety of his actions.
The episode rekindled debate about "responsible disclosure," the notion that information about major security problems should be made public in a way that brings minimal risk to customers.
According to Lynn and other experts, what Lynn described and demonstrated at the Black Hat Conference on July 27 could potentially lead to manipulation of Cisco router tables, denial-of-service attacks and access to confidential data.
Through a security advisory, Cisco has indicated that the way some unpatched IOS routers handle IPv6, which has seen little adoption in North America outside of research labs, is a conduit for the type of buffer-overflow exploit revealed by Lynn. But this week, a Cisco spokesman acknowledged the exploit may be possible in other ways. "There's ongoing information gathering and more testing," says Cisco spokesman John Noh.
Cisco this week also released a new patch for Cisco IOS-XR, its new carrier-focused router operating system, which was introduced last year for its CRS-1 Internet core router, and ported to the 12000 series of carrier routers this year.
Experts and users say the hole in IOS appears not to be an immediate concern based on what is public knowledge at the moment, since patches are available. But what concerns some is that Lynn's exploit techniques take router hacking to a new level, which eventually could have security implications for Cisco customers.
"Strategically, this is a very serious issue for Cisco," says David Lawson, vice president and director of global security practice at Greenwich Technology Partners, a New York integration and consulting firm that specializes in Cisco technology. "It proves something we've been saying in the security field for a long time, that a router is breakable."
Many IOS exploits in the past would simply cause a router to crash or reload itself, he adds.
"The big key to what [Lynn] did was to demonstrate a way to fool [the router] into thinking it was already crashing, so that it didn't initiate the shutdown sequence. If you can do that, that opens up the ability to open up other exploits. Now you can actually get code running that does god-only-knows what."
As for the question of responsible disclosure and whether Lynn represented that ideal or not, opinions continue to differ.
"I personally wouldn't have done it the way he did it," says Justin Bingham, CTO at security vendor Intrusic, referring to Lynn's action in defying Cisco and Internet Security Systems (ISS) - his employer until he quit just hours before giving his demonstration. "I like my career being a security researcher and a lot of that is based on trust with your customers and other companies."
Lynn, who has acknowledged breaking non-disclosure agreements in speaking out about the router exploit, says he took the step out of concern that withholding the knowledge would help would-be attackers and even posed a national security concern.
"The vulnerability which I demonstrated - but didn't give any information about - was properly disclosed to Cisco months in advance," Lynn says. "They had patches publicly available for months before I went on stage.
"That said, the disclosure debate is one that needs to happen. The idea of full disclosure is just about as dangerous as no disclosure at all. As with most things, we have to find the proper balance."
While Lynn has settled one lawsuit with Cisco and ISS, agreeing not to disclose anything he knows about the exploit, his problems don't seem to be over. The FBI is investigating him and interviewing friends and roommates, he says.
ISS, which declined to discuss the Lynn matter this week, has sought to stop the spread of the electronic version of the presentation slides that Lynn showed at Black Hat -- many of which are labeled with the ISS logo -- by threatening legal action against Web sites posting them.
ISS has benefited from its research by including preemptive protections for the vulnerabilities in its Proventia IPS product line and Internet Scanner products. ISS had been planning to make a big splash at Black Hat by unveiling the Cisco router flaw, but backed down when Cisco balked. But Lynn, after quitting his job at ISS, spoke out anyway.
Customers want more info.
Cisco customers say they would like to know about these types of security problems as soon as possible.
"I'd like to be the first one to find out," says Bob Lescaleet, MIS department manager at Pace Suburban Bus Service, a government agency in Arlington Heights, Ill., serving a six-county region. "I'm not sure Cisco should have kept this quiet as long as they have."
John Monaghan, vice president of IT for Marnell Corrao Associates, a Las Vegas construction and architectural firm that uses Cisco routers and firewalls in its corporate and field offices, says he was troubled that Cisco was working with ISS on how to present the shell-code exploit at a hacker conference, but not telling customers about the potential threat.
"We are concerned that a vulnerability has existed, and that Cisco didn't come clean and let us know about it," Monaghan says. "As far as getting information from Cisco, it's more of a pull from our end than a push from their end. You had to dig through an awful lot of rhetoric to find out that this vulnerability only has to do with IPv6."
"As a user, you worry if there's stuff out there already in the wild," says Dennis Schwind, network specialist at Miami University in Oxford, Ohio. "Cisco is not telling us anything about" the shell-code exploit, he says. "You're just left saying, I sure as hell hope this isn't big. That's really what you're left [with], because there isn't any real detail on what the real impact would be if this is exploited other than the 'execution of arbitrary code,'" he says, referring to language used in Cisco's security notice issued this week.
Microsoft weighs in
Microsoft this week offered its view on responsible disclosure, saying it entails seeking to ensure there's a fix in place before publicly identifying a flaw -- but that there should a time frame for this, says Stephen Toulouse, Microsoft's security program manager in the Microsoft security response center.
In general, Microsoft supports the "Guidelines for Security Vulnerability Reporting and Response" published under the aegis of the Organization for Internet Safety.
These guidelines, while declaring there's "no single universally appropriate time frame for investigating and remedying security vulnerabilities," does state that 30 days is a "good starting point."
The guidelines also suggest a 30-day "grace period" during which the remedy and information about the security problem is shared only with people and organizations "that play a critical role in advancing the security of users, critical infrastructures and the Internet." However, Toulouse says if a security vulnerability is highly critical, he would consider releasing information within a day.
Symantec, which has IPS products but doesn't do the type of security research ISS does, didn't have the advance knowledge about the exploit that ISS did, says Alfred Huger, senior director of engineering at Symantec Security Response. Nonetheless, he noted that sometimes researchers do share information about exploits across vendor boundaries, usually based on personal relationships.
Huger says Symantec would probably have treated the situation differently than ISS and Cisco did based on its own corporate guidelines for responsible disclosure, which give an IT vendor 30 days to correct an identified problem before going public.
McAfee President Gene Hodges said his company's policy is "to share as much information as you need to share and nothing more." The Cisco router flaw is "a very important vulnerability, probably one that's had the biggest impact of anything we've seen all year."
Among the questions surrounding the Cisco router exploit is whether a researcher's attempt to use reverse engineering and disassemble code to discover flaws is illegal - a charge raised against Lynn by Cisco and ISS in legal filings.
"In the anti-virus business, that's exactly what we do," Hodges says. "You put it in the de-compiler and try to figure out how it operates."
Mark Rasch, chief security counsel at security firm Solutionary in Omaha, Neb., says, "Reverse engineering is not clearly illegal."
Lynn maintains that he was simply following orders from his then-employer.
"It seems to me there is a license agreement dispute over that now, but the license was with ISS, not me," Lynn says.