Top three indicators of compromised web servers

Web servers are a popular target for attackers, and the number of servers, frameworks, and web apps can make it difficult to recognize where the threats are. Here are some common indicators. 

By Lonnie Benavides

You slowly push open your unusually unlocked door only to find that your home is ransacked. A broken window, missing cash, all signs that someone has broken in and you have been robbed.

In the physical world it is very easy to understand what an indicator of compromise would mean for a robbery. It would simply be all the things that clue you in to the event's occurrence. In the digital world however, things are another story.

My area of expertise is breaking into web applications. I’ve spent many years as a penetration tester attempting to gain access to internal networks through web applications connected to the Internet. I developed this expertise because of the prevalence of exploitable vulnerabilities that made it simple to achieve my goal.  In a world of phishing and drive-by downloads, the web layer is often a complicated, over-looked, compromise domain.

A perimeter web server is a gem of a host to control for any would-be attacker. It often enjoys full Internet connectivity with minimal downtime while also providing an internal connection to the target network.  These servers are routinely expected to experience attacks, heavy user traffic, bad login attempts, and many other characteristics that allow a real compromise to blend in with "normal" behavior.  The nature of many web applications running on these servers are such that encoding, obfuscation, file write operations, and even interaction with the underlying operating system are all natively supported, providing much of the functionality an attacker needs to do their bidding.  Perimeter web servers can also be used after a compromise has occurred elsewhere in the network to retain remote access so that pesky two-factor VPN's can be avoided.  

With all the reasons an attacker has to go after a web server, it's a wonder that there isn't a wealth of information available for detecting a server compromise by way of the application layer.  Perhaps the sheer number of web servers, application frameworks, components, and web applications culminate in a difficult situation for any analyst to approach with a common set of indicators.  While this is certainly no easy task, there are a few common areas that can be evaluated to detect a compromise with a high degree of success.

#1 Web shells

Often the product of vulnerable image uploaders and other poorly controlled file write operations, a web shell is simply a file that has been written to the web server's file system for the purpose of executing commands. Web shells are most commonly text files with the appropriate extension to allow execution by the underlying application framework, an obvious example being commandshell.php or cmd.aspx.  Viewing the text file generally reveals code that allows an attacker to interact with the underlying operating system via built-in calls such as the ProcessStartInfo() constructor in .net or the system() call in php.  The presence of a web shell on any web server is a clear indicator of compromise in virtually every situation.

Web Shell IOC's (Indicators of Compromise)

• Scan all files in web root for operating system calls, given the installed application frameworks
• Check for the existence of executable files or web application code in upload directories or non-standard locations
• Parse web server logs to detect commands being passed as GET requests or successive POST requests to suspicious web scripts
• Flag new processes created by the web server process because when should it ever really launch cmd.exe

#2 Administrative interfaces

Many web application frameworks and custom web applications have some form of administrative interface. These interfaces often suffer from password issues and other vulnerabilities that allow an attacker to gain access to this component. Once inside, an attacker can utilize all of the built-in functionality to further compromise the host or it's users. While each application will have its own unique logging and available functionality, there are some common IOC's that should be investigated.

Admin interface IOC's

• Unplanned deployment events such as pushing out a .war file in a Java based application
• Modification of user accounts
• Creation or editing of scheduled tasks or maintenance events
• Unplanned configuration updates or backup operations
• Failed or non-standard login events

#3 General attack activity

The typical web hacker will not fire up their favorite commercial security scanner to try and find ways into your web application as they tend to prefer a more manual approach. The ability for an attacker to quietly test your web application for exploitable vulnerabilities makes this a high reward, low risk activity.  During this investigation the intruder will focus on the exploits that lead them to their goal of obtaining access. A keen eye can detect some of this activity and isolate it to a source.

General attack IOC's

• Scan web server logs for (500) errors or errors handled within the application itself.  Database errors for SQL injection, path errors for file write or read operations, and permission errors are some prime candidates to indicate an issue
• Known sensitive file access via web server process.  Investigate if web configuration files like WEB-INF/web.xml, sensitive operating system files like /etc/passwd, or static location operating system files like C:\WINDOWS\system.ini have been accessed via the web server process.
• Advanced search engine operators in referrer headers.  It is not common for a web visitor to access your site directly from an inurl:foo ext:bar Google search
• Large quantities of 404 page not found errors with suspicious file names may indicate an attempt to access un-linked areas of an application

Web application IOC's still suffer from the same issues as their more traditional counterparts in that the behavior of an attacker must be highly predictable to detect their activity.  If we're honest with ourselves, an attacker's ability to avoid detection is only limited by their creativity and skill set.  An advanced attacker could easily avoid creating most, if not all, of the indicators in this article. That said, many attackers are not as advanced as the media makes them out to be; even better, some of them are just plain lazy. Armed with the web-specific IOC's above, the next time you walk up to the unlocked front door of your ransacked web server, you might actually get to see who has their hand in your cookie jar.

Lonnie Benavides is a Senior Technical Director of Information Security at DocuSign with 16 years of industry experience.  In 2007, he was part of a hand picked Air Force team that took over The White House network in two hours. Lonnie led the Application Security Team at Washington Mutual and created the Red Team at Boeing. Lonnie is an expert in web application security and enterprise assessment strategy.