There are a variety of indicators that a system has been compromised, ranging from the obvious to the very subtle.

Less obvious indications of a compromise include increased bandwidth, subtle IDS alerts (such as those indicating anomalous behavior) and mysterious configuration changes on systems. The questions that are typically asked include "How did they get in?" and "What did they do?" Tenable's Passive Vulnerability Scanner (PVS) provides useful information for answering these questions. Following are some of the alerts PVS may generate during an intrusion:

It's 4:55 PM on a Friday and you are looking forward to an enjoyable dinner with your family. Your Blackberry starts buzzing from across your desk while your inbox starts filling up with alerts from your SecurityCenter along with frantic emails from Human Resources. It seems a disgruntled employee named Jack Black quit today and nobody remembered to tell the IT group to disable his accounts until after important files started disappearing. Suddenly, you are stuck in Incident Response mode, gathering data on the user's activities. Do you cancel your reservations?

Fortunately, you have deployed Tenable Network Security's Unified Security Monitoring products, and have a wide array of resources[1] at hand to streamline the response process. These resources include SecurityCenter, the Passive Vulnerability Scanner (PVS) and Log Correlation Engine (LCE). At a high level, what can these resources do for you?

SecurityCenter

SecurityCenter provides a unified view of both vulnerability and event data along with the alerting, ticketing and reporting required for thorough user forensics.

Passive Vulnerability Scanner

PVS not only tracks vulnerabilities, but logs user and network activities detected in real-time on the wire. These activities include:

If you have a SQL server on your network, you know how important it is to monitor transactions to identify suspicious activity.

The Passive Vulnerability Scanner (PVS) can sniff SQL traffic in real-time and then send SYSLOG data to the Log Correlation Engine (LCE) that looks like this:

<36> Jun 21 08:34:05 pvs: 149.X.X.X:0|149.X.X.X:0|7019|Database command logging|version: 1.19 PVS has observed the following command from a database client to the database server (206.X.X.X): SELECT COUNT(CASE WHEN e.EmailType = ‘User’ OR t.ProcessType = ‘Borrowing’ THEN 1 ELSE null end) as Borrowing,|INFO

This blog entry describes a basic worm detection that triggers multiple types of correlation rules. All detections were done passively using the Passive Vulnerability Scanner (PVS) and by observing network session traffic using the Log Correlation Engine (LCE). The principals in this blog entry and others in our ‘Event Analysis Training’ blog series can be used with a variety of NBAD, IDS and SIM solutions.

Previously, we’ve blogged about the various advantages and disadvantages of using reputation based analysis of NetFlow, firewall and network sessions for event analysis. The basic concept is to use an external source of “badguy” IP addresses from commercial providers or free providers such as the SANS Internet Storm Center and see if any of your network IP addresses communicate with them.

I recently worked with a customer who asked for advice on the following “virus” events:

They were seeing “virus” traffic more or less continually. If you run a network IDS, and operate a busy email server, you will likely sniff virus traffic contained in inbound email messages.

I recently observed a SSH worm in progress at one of the research sites running our suite of products. I was looking into a spike of SSH events that had been alerted on by the Log Correlation Engine’s stats daemon. Filtering on the remote IP address (that came from the 240.0.0.0/8 Class A address space) that was causing the anomalies, displayed this screen:

Passwords are just so easy to abuse...

It was interesting to see that the top scorer in the game (who went by the handle of "ftp", and coincidentally had 21 scores in the first day of game play!) did not use fancy new exploits, 0day attacks or a wide range of open-source or even commercial tools. He was able to gain access to systems because the teams left default or easily guessable passwords set on some of the Linux servers. He used SSH to login to the systems, then SCP to upload some Python code, that was used to update the scoring engine. From there he was able to maintain access, not by rootkit technology or anything sophisticated, but just hiding in plain sight. The Python script makes a TCP connection to the scoring server and sends a message. It was moved into a file called "/dev/vfat", to make it look like a system file. Next, a shell script was written to call the python script every ten minutes. This file was called "getty" and ran in the background, and was also inserted into the startup scripts to ensure it kept running. The teams never found these processes running and "ftp" won the game, no exploits required.

Hacker "ftp" at work, winning the game using built-in tools such as bash, python and SSH.

Cyber Exercise

Over this past weekend I attended Cyberdawn, a cyber exercise that was hosted by Battlefield High School in Haymarket, Virginia.

At one of the research sites that we monitor, an analyst noted that a few servers were consistently making a large number of IRC connections. These connections occurred in a periodic manner and appeared to be automated. This blog entry describes the various steps taken in analyzing the connections and historical data. We used Tenable’s log analysis, network monitoring and passive profiling solutions to perform this analysis, but the principals could be applied to various SIMs, NBADs and analytical tools.

On Friday April 10, the Conficker worm was supposed to wake up and start network scanning. I grabbed the following screen shot from one of Tenable’s research sites:

I was recently working with a Log Correlation Engine customer who had gone through a typical deployment. Tenable advises customers to carefully consider which system and application logs they want their LCE Clients to send to the LCE server, in addition to a variety of Snort, Firewall and network activity logs. In this case, the customer had recently configured their system to send SSH logs to the LCE whereas before, they were only getting "network" events. When I had the opportunity to chat with them, they were very concerned about various worms or potential intruders performing a network scan such as those shown below:

Recently, I saw a spike in “compliance” violations in the logs of one of the large research deployments of Tenable’s network and log monitoring products. At first glance, a web server appeared to be sharing content that matched our passive adult media data rules. On further analysis, we discovered that this was actually a worm infection. This blog demonstrates many of the techniques to use network analysis to identify the worm’s activity.

Initial Detection on port 25109

The Passive Vulnerability Scanner (PVS) focuses on finding information about hosts on the network by identifying their vulnerabilities. As it sees new traffic, its database of known systems and their vulnerabilities is updated and sent to the Security Center.

Some of the PVS’s rules are set in “real-time” mode and will generate a syslog entry when the PVS encounters events such as new vulnerabilities or in this case, compliance violations. In this particular case, the logs were sent to the Log Correlation Engine (LCE) and from February 16th through the 19th, had this type of consistent activity:

This particular research site has many university students and occasionally a student server will get compromised and be used to host movies, music and other types of media that has been infected with viruses.

Looking at the actual passively sniffed vulnerability (shown below) it could be seen that the web server was hosted on port 25109.

Hosting web servers on ports other than 80 or 443 is a common way to share content that might not normally be blocked by a firewall or monitored by security tools.

Activity Profile

Now that we know a host is sharing some sort of pornographic content on port 25109, the next step is to profile the activity. This particular environment has the Tenable Network Monitor which is an LCE agent that sniffs packets on a span port and logs network session information.  Snort is also running with the Emerging Threats signature database.

For the last few days, the overall activity profile for the host in question and port 25109 looked like this (click for a high-res image):

The blue area is the past weekend activity for Saturday and Sunday. I’ve indicated where some backdoor events (these are normalized logs from Snort) as well as the PVS’s indication of a new open port occurrence.

We could conclude that sometime on Monday, a compromise took place and then port 25109 was opened as a new service. Once the service is open, many different hosts on the Internet started to connect to it and, at the time of the screen shot, the activity still seemed to be ongoing.

All of the above traffic was related to port 25109. There could be a lot more activity occurring on other ports. Below is the screen shot of the same host’s time period but without the port 25109 filter (click for a full screen shot):

There was a little bit of activity before the initial compromise, but most of the continuous activity and constant communications occurred from Monday onward. There were a few more “never before seen” events on ports other than 25109 as well. These events indicated that the IP address in question had never been attacked like this in the past.

Traffic Analysis

How can we pull apart all of these logs and get a sense of what is occurring? Let’s start by comparing inbound versus outbound traffic.

The Security Center has a tool that can graph all matching events over time based on their direction. In the case of our worm outbreak, the following graph was shown:

There is slightly more inbound traffic than outbound traffic. The traffic loads are also linked. If you look at the spikes and valleys, they correspond almost identically. This means that whenever this host increased its inbound traffic, it also increased its outbound traffic.

This could mean that the compromised server was part of a network of nodes that communicated with each other. This is unexpected since typically, when you have a worm or server controlled by a remote “command and control” node, there is much more scanning and probing traffic than actual command traffic. Based on other worms I had looked at under the Security Center and the Log Correlation Engine, I expected to see a huge amount of outbound connections and only a few inbound connections.

What ports are being used here? The Security Center also has a tool to quickly sort this information and perform a port summary. Below is an example screen shot:

That was my second surprise in this analysis. I was expecting to see a lot of outbound port 445 sessions. Port 445 is a port used by the Conficker worm to infect Windows servers that are still vulnerable to MS08-67.

However, in this case we saw traffic from port 444. Port 444 is commonly known as the Simple Network Paging Protocol (SNPP) service. Sniffing port 444 traffic showed it to be nothing like the paging protocol. Instead, when we graphed out the port 444 activity over the same time period for this infected host, we saw the following:

Reading this from left to right, we can see that the PVS identified new Internet activity for this host on port 444. This was the first time the IP address in question had browsed on port 444. After that, there was a wide variety of almost continuous network sessions of various lengths and bandwidth. The largest percentage of logged connections was “TNM-TCP_Session_Timedout” events. These events indicate TCP sessions that started but didn’t complete. These logs show that our host was involved with almost 60,000 attempted connections over a two day period that did not complete.

Port 25 (SMTP) was also on the list of ports (shown below). This had a similar profile of activity starting shortly after being compromised on Monday. Sniffing this traffic showed that the host was repeatedly sending rejected email and was likely being used to send SPAM.

Although this is conjecture, you can see an initial spike of TCP sessions that taper off within a day. It’s very possible that the content of the SPAM message was fingerprinted by anti-spam vendors and the email messaging attempts were immediately denied.

I thought it would be interesting to compare the amount of traffic and destination for both port 25 and port 25109. The Security Center has a tool that will use the source and destination Class A IP address of each matching event and generate a graph. This is an easy way to see where and how much traffic is occurring. Below are two screen shots for port 25 and port 25091:

Port 25	Port 25109

The X and Y axes are not shown (which would give away our general location). What can we conclude from these graphs? The large circles indicate that there have been more events on port 25109 than on port 25. In addition, port 25 traffic seems to be limited to just a few class A address spaces and not as many as the ones used by port 25109. The large grey circle in the port 25 graph could indicate a Class A address space that was targeted or was particularly successful at sending spam email.

The Compromise

What do the compromise IDS events look like? Consider these sanitized logs:

<158>snort[11035]: [1:2001099:6] ET EXPLOIT Attempt to execute VBScript code [Classification: Misc Attack] [Priority: 2]: {TCP} xxx.yyy.135.66:80 -> aaa.bbb.72.123:1822

<158>snort[11035]: [1:2008803:1] ET CURRENT_EVENTS Possible Downaup/Conficker-A Infection Checking Geographical Location [Classification: A Network Trojan was detected] [Priority: 1]: {TCP} aaa.bbb.72.123:4477 -> xxx.yyy.69.70:80

<158>snort[11035]: [1:2008335:3] ET TROJAN Beizhu/Womble/Vipdataend Controller Keepalive [Classification: A Network Trojan was detected] [Priority: 1]: {TCP} xxx.yyy.121.146:15648 -> aaa.bbb.72.123:25109

<158>snort[11035[1:2001689:6] ET WORM Potential MySQL bot scanning for SQL server [Classification: A Network Trojan was detected] [Priority: 1]: {TCP} aaa.bbb.72.123:1151 -> xxx.yyy.59.186:3306

What we have here is a set of exploit steps. VB Script is downloaded, a potential geo-location attempt is launched by a worm and then we have some potential Trojan and scanning activity. The first two alerts occurred within a minute or so of each other. The other scanning alerts started about an hour later.

End Game

Although fun to watch, the worm had to go. After making the appropriate contacts, the host was turned off as shown below (click for hi-res):

It is easy to see the complete drop off in activity across all log sources. The two spikes of lone “SnortET-Trojan_Activity” events were UDP based worm probes from external hosts.

For More Information

If this type of analysis is of interest to you, you may want to review all of the blog entries in our “Event Analysis Training” category. There are many example articles including working with blacklisted IP addresses and an example Windows NT 4 system being infected. There are also many example demo videos of using Tenable solutions to look at anomalies, network traffic and intrusion detection logs. If you are interested in learning more about the Security Center or Log Correlation Engine, please contact us at sales@tenablesecurity.com.

Previous blog entries have described how blacklist analysis for logs and network traffic can lead to interesting evidence of compromises and network probes. In this blog entry, we will examine a situation where the continuous blacklist activity led to multiple compromises.  This particular network has the Security Center, Passive Vulnerability Scanner and two Log Correlation Engines (LCEs). One LCE is receiving netflow data and logs from Snort while the other LCE is gathering network session data and syslog messages from routers, servers and access control devices on the network.

Initial Detection

Tenable products are deployed at several different live research and testing sites. For blacklist analysis, we typically see occasional hits during the course of a day such as shown in the following screen shot:

At one of the university locations we monitor, we saw the following sequence of events:

Notice that the “Blacklist-Emerging Threats Compromised Host” event is almost continuous. The default query is for 24 hours. Since the event seems continuous, we zoomed out for a five day summary of events: 

Clearly, we can see that the bulk of the Blacklist events occurred over the past 2 days, but there were no events in large numbers prior to that time.

Who is causing us problems?

The next step is to determine the IP address or addresses involved. Clicking on the “78909” events instruct the Security Center to summarize IP addresses, as shown in the following screen capture (actual addresses have been sanitized):

All of the IP addresses involved were from our network except one in the 123.151.0.0/16 class B network. To trace the owner of the address, we first searched for it in the ARIN “whois” database, which can be found at this link: http://www.arin.net/index.shtml. Entering the IP address in the search box labeled “Whois” showed that the address was registered with the Asia Pacific Network Information Center (APNET).  Querying the “Whois” search on the the APNET site showed that the address was registered from a location in China. Tracing one IP address is much easier than tracing multiple simultaneous attackers.

Analyzing the Attacker

An important point with blacklists is that an IP address could be scanning you today, yesterday or even a few weeks ago and it might never be placed on a public blacklist. Even when it is, you should always attempt to go back in time and see if there was any detected activity from the IP before it was placed on the blacklist.

In this case, we took the IP address and had the Security Center summarize all events from it for the past 48 hours:

This single display tells a very interesting story and is the primary focus for this blog post.

There is no activity prior to the blacklist events. We should always be suspicious that there may have been activity that was not logged or sniffed, but in this case, we are pulling netflow data from inside the network and are performing direct network sniffing on the university perimeter.

On January 15 around 9:00 AM (the first vertical blue line is 12:00 noon on the 15th) are two “Never Before Seen Correlated” events.  This means that two hosts on the network have had a correlated event occur from the LCE, which has never happened before. The correlated event that has never been seen before is the “Long Term Intrusion Activity” as indicated by this sanitized log entry:

Never_Before_Seen-correlated event, destination IP address 128.xxx.xxx.xxx has never seen event Long_Term_Intrusion_Activity in the past. This event originated from 123.151.xxx.xxx. If this event is unusual for this IP address, please investigate this event further.

Other machines on our network may have indeed had correlated events and need to be investigated, but the IP addresses involved in this attack could be our first priority.

The bulk of the network traffic is short network sessions, consistent with probing. These are indicated by both the TNM-TCP_Session_Short and the TFM-TCP_Session_Partial. The TNM events indicate network sessions that were sniffed with the Tenable Network Monitor agent and the TFM events indicate netflow events logged with the Tenable NetFlow Monitor.  A majority of the events are short events that last less than 5 minutes. However, in both the netflow data and sniffed network session logs, there are several sessions that have lasted 15, 30, 60 minutes or longer. Towards the end of this graph can be seen some network sessions that have transferred hundreds of megabytes of data and several that have transferred more than a gigabyte.

Performing a port summary of just the network events provides this distribution:

Clearly, this is on port 22 and is likely a Secure Shell probe of some sort. With this in mind, the long network sessions and sessions that had large bandwidth in them seem more ominous.

Many SSH worms that have been active in January of 2009 perform brute force password guessing. These sessions are relatively long, but utilize low bandwidth.  Picking a TNM-Long_TCP_Session_60_Minutes event at random, we can analyze its sanitized log:

Fri Jan 16 13:49:15 - TNM-Long_TCP_Session_60_Minutes[2102]:123.151.xxx.xxx:13861 -> 128.xxx.xxx.xxx:22 [u:521 d:1581]|1232128074|1232131755|3681

The “u:521 d:1581” entries indicate that the client uploaded 521 bytes and also downloaded 1581 bytes of data. Moving slightly more than 2000 bytes in an hour is not a large file transfer. It is very likely that the network connection was used to attempt several login and password guesses.

Perhaps more concerning is that later on in the graph, there are several TNM-TCP_Session_Whole events that had 100MB to 1GB of data transferred. This could indicate that a server was compromised and was being used for a more specific purpose. Each of these sessions also involved port 22. Large file transfers on this port are very alarming. Keep in mind that the filter on this graph only showed events that involved our suspicious entry from China. If that IP had connected to a host on a target network and then connected out to other targets across the Internet, it would not be shown on this graph. Finding all of the IP addresses that these target hosts had connected to would be a very good next step.

Lastly, there are two “Suspicious Proxy” events. These events are the result of an LCE correlation that examines terminating TCP sessions that are longer then five minutes and attempts to find “leap frog” connections. The behavior we are trying to find is when one computer connects to a second computer and then uses that second computer to make a connection to a third computer. Every time the LCE sees a terminating network session, it checks lists of other previously terminated network sessions to see if there could be a link. If so, it logs a Suspicious Proxy event with a log entry that looks like this:

Suspicious_Proxy - host: 192.168.20.91 just completed a network session lasting from client 208.73.181.192 to port 1361 and during that time completed the following TCP sessions (destination IP:destination port:duration): 208.73.181.192:5223:

Although simple and elegant, an issue with this technique is that since we are purely watching the network traffic, we have no way to correlate system activity that could be generating normal outbound connections. There could be other processes or even other users on the system that are making legitimate connections outbound.  Even in the case of our blacklisted IP address, the fact that the IP address has connected with us over SSH  is no reason to conclude a different user on the system was not performing some sort of network activity.

Having said all this about the two Suspicious Proxy events, knowing that the blacklisted IP address in this case had made numerous connections to the network, and eventually was seen using large amounts of bandwidth from the network, these events should be analyzed much further. With the Security Center and Passive Vulnerability Scanner, we could learn the IP addresses involved in these Suspicious Proxy events and then obtain vulnerability and profile information about the targets.

What are we not seeing?

One of the interesting perspectives about running Tenable’s products at different test locations is that in almost every case, we get to deal with a different level of vulnerability and event information. In some places, we get to scan with credentials and in others, we can only sniff. Some test sites provide us full logs from their network servers and even user desktops while others only provide us netflow data and sniffed TCP sessions. With that in mind, what types of logs and access would be of most useful to us in this case?

• System logs for these targeted Linux servers would be most useful. It would be very interesting to see if an account was guessed and which account it was. Was there any software installed via RPMs? Were any network devices placed into promiscuous mode?  Were any user accounts added? This sort of information is readily processed by the LCE and can quickly build a full picture of what an attacker is capable of.
• Full system audit access would allow Nessus to perform more detailed patch analysis, and also determine which ports had which processes listening on them. If an unsophisticated backdoor were added to the server, it is possible a new Nessus process scan could identify the listening process.

This sort of analysis is local to our network. It is also very interesting to see if the remote attackers are active on other lists or monitored sites. In this case, the IP address was added to an Emerging Threats Snort rule. After gathering the information for this blog, we went and queried several of our other test sites and saw logs like this:

Fri Jan 16 03:53:04 – TNM-ICMP_Activity:123.151.xxx.xxx:11 -> 149.xxx.xxx.xxx:0

This is a log from the Tenable Network Monitor that captured some ICMP probes also from the remote blacklisted IP address.

Previous Event Analysis Training Blogs

This blog entry is the fifth is a series of entries that offers in-depth example analysis of detecting a wide variety of scanning, probing and compromise issues. To view all of them, click on this link. The specific blog entries in this series are as follows:

• Run NT and Pay The Price – shows how a Windows NT 4.0 system is compromised and participating in a command and control network.
• Advanced Blacklist Analysis – shows examples of how IP blacklists can be used.
• Working with “Blacklists” – introduction to working with publicly available sources for IP address reputations.
• Working with Emerging Threats Events – introduction to work with IDS alerts generated from Snort running the Emerging Threats signature rules

Although we focus on using Tenable products to view and analyze a variety of security events, readers can expect to learn a wide variety of monitoring strategy from reading these blog entries. If you would like more information about Tenable products, please contact as at sales@tenablesecurity.com or consider viewing the following product videos.

Most large enterprise networks have a few legacy systems around – either because they were “forgotten” or because they support an old application that was never ported to a newer release. Such legacy systems can be the Achilles  heel of network security.

The following sanitized screen shot comes from one of Tenable’s research sites:

What we are looking at (click on it to see a high resolution image) is a Windows NT 4.0 system that has been “forgotten” and is also being controlled as part of a botnet. This blog entry will discuss how the above Security Center screen shot can be analyzed to arrive at this conclusion.

Analyzing IDS Events

The Security Center can process network IDS events from a wide variety of sources including application specific detections from the Passive Vulnerability Scanner (PVS). The primary focus of the PVS is to passively identify all client and server applications in network traffic and generate an alert when there are specific vulnerabilities associated with them. In the case of botnet traffic, a certain percentage of HTTP, IRC and other client detection rules implemented by Tenable for the PVS will often highlight compromised systems.

In the above screen shot, the Security Center user has listed IDS events and focused on a system that had multiple “Generic BOTNET Client Detection” events. Regardless of what type of network monitoring solution you use (Snort, TippingPoint, etc.), I tend to find the signatures that look for “known” botnet and command and control events to be very reliable with a low false positive rate.

Another example of this was detected by the Snort sensor also running in this environment, with the Emerging Threats signature set. One of the interesting rules in that signature set is to look for IRC traffic on ports not normally associated with IRC, such as port 80:

The above screen was generated by analyzing logs gathered with the Log Correlation Engine (LCE).

How do we know it is an NT 4 System?

Nessus has a very sophisticated operating system identification system designed for IT audits. It uses the most reliable forms of system fingerprinting available.

In this case, the remote system was detected as being Windows NT 4.0 through interaction with the MS RPC service as shown below:

This same network also had the PVS running on it, and it also fingerprinted it as NT 4.0:

Looking at the Windows User Management set of plugins (shown below) we can see a few that have been protected. In particular plugin 10907 checks if the Guest account belongs to a group, which was very common on NT 4.0 default installations.

How do we know it is forgotten?

Determining that a computer has been “forgotten” is more a matter of opinion than something that can be detected with a plugin. Let’s ask some questions:

Why would a Windows NT 4.0 system still be around?  Perhaps the server is part of an embedded device such as a medical system, printer or other legacy service. In this case however, there were no unknown services on this host and no other “odd” devices on the network this computer was on.

Why would it not be in DNS? If you notice on the popup “System Information Summary” screen, the DNS name for this IP address is unknown. If a system on a large corporate network is not part of DNS or the AD, there is a good chance that the IT group does not know about this system.

How Could this System Have Been Exploited?

The LCE was running on this network for some time, but the botnet activity was only recently discovered. There were no inbound network connections that resulted in “attack” events or other types of correlated activity. One day, the “guest” account simply was part of a group.There are many “LAN” aware worms such as SirCam and Nimba variants that look for local network shares in an attempt to exploit more computers. It is quick likely that this NT 4.0 system was victim to a worm-style attack.

For More Information

Nessus is very effective at identifying a wide variety of “Unsupported” operating systems. These include Windows NT 4.0, Windows 95/98/ME and a variety of unsupported Linux and UNIX systems.

Microsoft runs a web page of their products which have been “end of lifed” (EOL) here:

http://support.microsoft.com/gp/lifesupsps

In particular, Windows NT has been EOL’ed since December 31st 2004. It contains many vulnerabilities that Microsoft has not offered patches for.

There may be many good reasons for having an older computer system on your network. A great example would be to run older software application that was never ported to a newer OS platform. You might not have a choice either. It is possible that Windows NT comes as part of your printer, phone system, security camera or other type of embedded applications.

Regardless, being able to identify these types of systems on an ongoing basis is very useful. If the system is needed, you can develop a risk mitigating strategy to compensate for any vulnerabilities associated with these unsupported operating systems. If the system is not needed it be removed from the network, thus closing the security exposure.

DNS blacklists are publicly accessible lists of IP addresses that have been identified as associated with undesirable Internet behavior, mostly involving the sending of spam e-mails.

Using lists of IP addresses on a blacklist can be an effective way to look for potential security issues in your logs. We've previously blogged about using blacklists with the Log Correlation Engine. In this post, we will examine more of the basic assumptions and principals for analyzing traffic associated with blacklisted IPs.

A Quick Blacklist Review

There are many different free and commercially available lists of "blacklisted" IP addresses. These lists attempt to identify IP addresses that have a reputation for scanning, sending spam email, hosting malware and many other types of undesirable behavior.

The lists differ in how often they are updated,  how many IP addresses are published, the source of the IP addresses and the nature of the activity from the IP. For example, Arbor's Atlas list comes directly from their deployed sensors whereas the SANS Internet Storm Center derives its lists from voluntary user submissions.

Here Today -- Blacklisted Tomorrow

Using blacklists to filter your logs for evidence of connectivity with potentially hostile servers is an effective way to identify potential security issues on your network. However, if a system interacts with an IP address on a blacklist, don't forget to analyze what the IP did before it was blacklisted.

Consider this screen shot from the Log Correlation Engine deployed at a large university:

This is a 5-day graph of any log matching an IP address that the security administrator identified  after seeing this (sanitized) alert:

Outbound_Blacklist_Communication src – X.X.32.99 dst – X.X.182.175 , following is known about 217.175.182.175 , message - ISC-BlockList , dns_lookup – xxxx.xxxxl.fr

To determine the activities associated with this IP address for all systems (not just the “blacklist” events), the security administrator  summarized all normalized event types for the past 5 days. If you view the graph from left to right you can see that there was a large spike in firewall and connection events. This particular network has Stonegate firewalls and logs not only firewall “deny” events, but also permitted connections. There is also a variety of statistical and never-before-seen events from this IP address.

It is interesting to note that about 3 days in the past (around the middle of the graph of events) is when our first normalized blacklist event occurs. Sometime between 5 days ago and 3 days ago, the IP in question was involved in activities that caused it to be placed on a blacklist. Only after this occurred did we get blacklist correlation events -  prior to that the connections were logged and correlated purely on their own merit.

The point of this type of analysis is to always consider what the blacklisted IP addresses may have been doing prior to being placed on a blacklist.

Analyzing Blacklist Connection Directionality

The most basic form of blacklist correlation is to get a list of IP addresses and then use them as a filter to see if any logs match with the addresses on the list.

If you are analyzing logs from public facing servers and devices, the blacklist correlation will likely tell you that you are being scanned by "known" IP addresses that scan the Internet. Although interesting, this type of activity is not alarming and should be expected.

What is more interesting is to consider the direction of the traffic  in the logs that include blacklisted connections. It may be expected that we would be scanned from a blacklisted IP, but what we would be really concerned about is if one of our servers connected to the blacklisted IP.

Why might one of our systems connect to a known blacklisted IP address? Possibly, it has been compromised and is connecting back for a payload or is part of a command and control botnet.

On the other hand, "connections" can mean many things and some care should be made when looking at these types of correlations. Consider some of these scenarios where the "outbound" connection is a legitimate connection, but that would generate a false alarm:

• The attacking/scanning blacklisted IP address is actually a compromised DNS server. When this IP makes connections to your mail, web and other types of services that perform DNS lookups, they may in fact send DNS queries back to the blacklisted IP address because it is the DNS server. This might look like the host has been attacked and then compromised.
• If the blacklisted IP address has been placed on a list because it is a communication channel for a botnet, other legitimate users of this system will look like they are communicating with a hostile IP address. This often happens when a public IP address of an IRC system is blacklisted. When an innocent IRC user connects to this address, a correlation occurs, although there has been no compromise.
• In some SIMs I've encountered, the normalization of NIDS events and even system logs such as login failures swap the source and destination IP address. This can cause a correlation engine to conclude that an outbound connection has occurred. For example, consider a Snort rule that logged a VNC login failure event. This event has a source IP of the target host and a destination IP of the attacker's system. If a correlation engine were not smart enough to consider the direction of network traffic of this event, it could incorrectly conclude that a system had  made a new connection when in fact it had not.

Considering the network traffic direction of connections which terminate or originate with a blacklisted IP address is exactly what the Log Correlation Engine blacklist.tasl script performs. It considers any network connection event which was logged by a system, application, firewall, netflow or other type of network monitor and then alerts if the IP addresses involved correspond to a blacklisted IP address. Other correlation scripts attempt to see if “blacklist” connections were preceded by an intrusion detection event, or if a system repeatedly attempts to make connections to a blacklisted IP address.

For More Information

The topic of blacklist analysis as well as generic event analysis has been discussed on this blog several times:

• Working With Blacklists
• Passive SPAM Traffic Analysis
• Finding Low Frequency Events
• Working with Emerging Threats events
• Looking  for “Never Before Seen” Events
• ShmooCon - Network Monitoring Notes

Many SIM, NIDS and NBAD solutions have some sort of "blacklist" functionality which highlights when systems on your network interact with IP addresses that have been identified as being associated with scanning, virus propogation, SPAM originators and so on. These solutions typically take one or more lists of potentially "bad" IP addresses and then scour your netflow, IDS, firewall or other types of logs and to see if there are any correlations. This blog entry will focus on how these types of events can be analyzed.

Why Do Blacklist Correlation?

On any given day, you may be able to count all of the individual network sessions in and out of your network in the millions. Auditing these can be very difficult. If you know ahead of time where users are supposed to be going, you can use a firewall to filter this access. If you don't you can try to use a network IDS/IPS to look into these sessions and find evidence of foul play and policy abuse.

However with a blacklist, someone else has done the work for you. All you need to do is sit back and wait for your network to be connected to by one of these "bad guys" on the black list, or worse, wait for one of your systems to connect to these IP addresses.

Tenable runs our technology at a variety of locations and we often have access to our systems during customer evaluations. For a given network, if we aggregate network IDS/IPS events and there are perhaps 10,000 to 1,000,000 events per day, a heavy day of blacklist correlation could mean 10 to 30 hits. The effort required to analyze a blacklist interaction is very minimal and often of very useful value. And if you have a solution like the Log Correlation Engine, you can do this type of correlation on any log source you have including netflow, sniffed network traffic, firewall accept events and so on.

Where do Blacklists come from?

There are many sources of lists of "bad" IP addresses. At Tenable, we've enabled the Log Correlation Engine to work with published black lists from the Internet Storm Center, Arbor Networks, the Emerging Threats project and a few other. These sites have a variety of different techniques they use to aggregate information about potential:

• large scale scanners
• botnet command and control nodes
• malware payload distributors
• SPAM sending sources
• phishing sites

Some of these blacklists rely on input from sensors from all over the world. For example, if someone sends their logs to the Internet Storm Center, the sources of their activity are correlated with other submissions and the top sources of abuse are published. Sometimes, when a specific worm or trojan is analyzed, the IP addresses it communicates with are published and these are added to these lists as well.

Accuracy

Being limited to an IP address, there are often false positives where there is more than one function at that node. For example, a botnet that is receiving commands over IRC may result in having an IRC server IP address being placed onto a blacklist. Anyone that used IRC and connected to this server would then be seen communicating with a blacklisted IP address, even though the botnet is likely limited to a few specific IRC channels.

Similarly a web server may be identified as distributing malware or be a known phishing site, but if the IP address actually hosts many virtual web servers, then traffic going to those other web sites could be flagged as suspicious, even though they all travel to the same IP address.

The quality of being on a list or off a list is also something that changes over time. A hostile host could be scanning a site for several days or weeks before there is enough activity for it to be listed on someone's blacklist. For example, in the screen shot below, the last 5 days of all normalized events from a smaller university are shown:

Reading from left to right, there has been some sort of Snort based detection of malware (the SnortET-Malware_Activity events), but within the last day, there was a variety of logged network traffic with one hit of a connection to a known scanning source IP address as tracked by Arbor Networks. This would be the Blacklist_Arbornetworks_Top_Attack_Source event, as well as the Outbound_Blacklist_Communication event.

Compromise Detection

When looking for a compromise, any type of outbound connectivity from your network (the "good guys") to a blacklisted IP address should be investigated. A wide variety of trojans, malware and other types of backdoors will reach out to one or more hosts to receive instructions.

This outbound connection to a blacklisted IP address should be treated much more seriously than being scanned by a known blacklisted IP address. Imagine you are in charge of security for a large network and a blacklisted IP address scans you. Your SIM or NBAD may see 1000s of connections and lots of scanning events, but they will all be "inbound" to your network. However, if your systems are compromised, you may see an outbound connection. Consider the below example screen shot from a Log Correlation Engine monitoring netflow:

In this image (which was over a 48 hour period), there were a variety of blacklist events in the past (reading left to right), but the last few events also included 4 Outbound_Blacklist_Communication events. All of the previous events included connections "from" the bad guys to our network. They may be hostile and should be investigated, but at least the good guys didn't connect back. However, these last four events were "outbound" in nature. For some reason, hosts on the monitored network reached out to these "blacklisted" sites.

Since there are only 4 events, analyzing them can be accomplished in short order.When this occurs, you should attempt to see if this connection is a false positive or if the system is indeed compromised. If you are collecting logs from other sources (NIDS, firewall, anti-virus, .etc) you may be able to correlate the outbound blacklist connection with other types of activity. When trying to discriminate between a harmless false positive alerts and a compromise, keep the following in mind:

• If you have collected web proxy logs, these should contain information about the visited domain name and web URIs involved.
• If the system in question is not a desktop or user workstation, yet it is sending mail, browsing the web and doing chat, this is a good indication that you have a compromised server. One of the techniques our Security Center customers use is to create asset lists of various servers and then use these as a report or filter to analyze blacklist connections quickly.
• Even if the system is a desktop, even power users don't surf the net 24x7. Analyzing how much traffic is normally sent can show "off hours" activity from a host.
• Keep in mind the protocol you are dealing with. If you run FTP servers on your network and configure them such that the server connects to the client to transfer data, you may get a valid outbound connection on some high port. Although highly interesting (why is a blacklisted IP address talking with your FTP server) its likely not a compromise.

If you don't have access to any other corroborating logs, you can still do some interesting analysis with the original netflow or network session data. In this redacted screen shot below, we list each network session from hosts on a network that have made a connection to a known blacklist source:

Host 208.87.149.250 was listed by Arbor Networks as an attack source. At one of our monitoring sites, we saw several hosts connect to this IP address. Reading through the logs (which came from the Log Correlation Engine's network monitor agent), the first session transfered 1363 bytes, but less than a day later, a different host (perhaps the same host with a new DHCP lease which LCE could have figured if it was collecting DHCP logs) attempts two connections, both of which time out. This could indicate that the blacklisted host at the target IP has been taken offline and is no longer available. If this is the case, then that first host that was able to transfer 1363 bytes could have been infected or have received some sort of hostile communication.

Lastly, if no additional log sources are available, bringing in vulnerability, asset and configuration data can help. If the source system looks like a desktop and is not running the latest web browser patch, it may be vulnerable to a variety of client-side web exploits. If the system is a server, and it has made an outbound connection, perhaps it has been compromised. Servers generally don't  "surf" the network and if it has made such a connection either this is "normal" or it just started. If it just started, then trying to figure out when it happened and maybe why it occurred could be interesting.

Scan And Generic Activity Detection

Besides looking for compromised systems, tracking blacklist connections can be useful on many levels:

Follow On Traffic - Some SIMs and NBAD solutions can correlate blacklist connection events with various types of NIDS events. For example, the Log Correlation Engine can look for hosts that have been attacked, and then interact with a blacklisted IP address which produce logs such as:

Compromised_System_Blacklist_Connection - source host 24.3.2.1 has connected to host 77.3.43.112 on port 80 with a blacklist event of Blacklist-BleedingSnort_Botnet_Source. Within the last five minutes, the source host was targeted with 24 critical IDS attacks. This could indicate that the system has been compromised and is participating in a botnet. Time of activity was 10/03/2007 09:11:45

SPAM - If you receive a correlation with connections from a known "spammer", you should check to make sure that each of the target mail servers are indeed authorized mail servers AND they have anti-SPAM filtering in place. On the other hand, if your computers are making connections to a known SPAM source, this could indicate that you have one or more systems on your network that have been compromised and are being used to send SPAM. This could also mean that one of your users has happened to send mail to a server that was blacklisted for sending SPAM.

Large Network Scans - Although your SIM, NBAD or NIDS may detect port scans and scanning activity, correlating these events with blacklist source IP addresses can easily identify probes and potential worm infection events. If you can recognize these types of scans, including the target ports, you may be able to further analyze this and compare what the worm was probing for with known vulnerabilities on your network.

Phishing - Some blacklist sites track IP addresses that have been associated with phishing scams. If you see many users visit one of these IP addresses, it could indicate a scam that has targeted your network.

Botnet Command and Control - Some blacklist sites (such as Emerging Threats) identify IP addresses that are used to manage various types of botnets.

For More Information

This blog entry is second in a series of "Event Analysis Training" entries. The previous blog entry and several other event analysis links are listed below:

• Event Analysis Training -- Working With Emerging Threats Alerts
• Passive SPAM Traffic Analysis
• Zombies and Botnets -- Detecting Crowd Surges

In the next few weeks, I will be posting a series of blog entries which provide examples of analyzing logs and events in large enterprise networks. We will be using the Security Center, Nessus, Log Correlation Engine and the Passive Vulnerability Scanner in these examples, but the principals can be applied to most SIMs, NBAD and network IDS Consoles. Today's blog considers working backwards from a few interesting alerts from a Snort sensor running the Emerging Threats signature set.

Emerging Threats

Tenable recently began supporting logs from Snort sensors running rules from the Emerging Threats project in both Security Center and the Log Correlation Engine. The Emerging Threats project classifies Snort events with a variety of names such as "ET EXPLOIT", "ET POLICY" and "ET SCAN". At one of our test sites (a large university) a typical "day" of logs looks as follows:

There are many ways to analyze this data such as summarizing by port, time, business asset and so on. Visualizing certain types of discrete activity is also useful. Below is a graph of all "Class A" address space that has had at least one Emerging Threats "P2P" event:

You can see that this university has a wide variety of P2P activity that communicates with a wide variety of address space.

Attacks and Exploit Attempts

A quick filter that can be used in the Security Center is to match all event names based on a unique string or text. To see all Emerging Threat Exploits, we'd type in "ET E*" into the query tool. All Emerging Threat exploit rules start with the name "ET EXPLOIT" such as "ET EXPLOIT WinProxy Host port buffer overflow". Similarly, we'd type in "ET A*" to match all attacks. Below are two screen shots
of the events which occurred under these filters:

Attack Events	Exploit Events

Under the attack filter, the events detected were IRC events that might indicate that a host has been compromised and is now receiving command and control instructions over an IRC channel. We will talk more about analyzing potential botnets in a future blog entry. However, in this case, the events were found to be false positives because:

• there was only one host of interest
• it never attacked nor scanned anyone
• there was a history of IRC usage as well as typical email and web browsing
• the IP address was found to be part of the student network

I say that these were false positives because the intent of the signature was to look for a compromised system logging into IRC in a suspicious manner. The signatures themselves looked fine, it is just that the packets involved that matched did not indicate abuse.

Analyzing The Events

In the Exploit screen, there were several denial of service events, a client side embedded GIF attack, a surge of PHP exploit attempts and a Solaris Telnet attack.

DOS Event Analysis

The DOS events were directed against one MS SQL server for which we had been receiving event logs from the underlying system. These logs indicated no interruption in service around the time of the attack. Vulnerability data from previous Nessus scans and continuous updates from the Passive Vulnerability Scanner did not indicate any major vulnerabilties or potential denial of service issues
either.

The actual packets detected in this case may have been real attacks. If we wanted to do deeper analysis, we could consider the sources of the attacks and see if this indicates some sort of  longer term activity. I generally don't like any type of unencrypted SQL access across a perimeter of a university.

Analyzing a Client Side Attack

The client side embedded GIF attack referenced MS05-036. This targets Internet Explorer browsers. In this case, Nessus was not being used to perform a client side audit and the Passive Vulnerability Scanner does not have a plugin to find this vulnerability since there isn't enough data on the wire to determine if it is present. However, we can do a few things.

First, we can look for evidence of a browser being used other than IE. In this case, it was Firefox. The PVS will report what browser is being used through passive analysis.

Second, we can see if this computer is receiving updates from Microsoft. PVS plugins #1925 and #4433 detect if a host is performing updates from Microsoft. And third, we were performing analysis of this host several days after these events occurred. After this IDS event happened, we did not observe any statistical anomalies, any types of scanning, outbound attacks or correlated events which could indicate a compromised system with Malware or a back door recently installed.

Analyzing the Telnet Attacks

The last attack that I felt was interesting were the Telnet attacks against a Solaris system. First, I thought it would be interesting to see how much Telnet traffic was occurring and to where:

Compared to the P2P traffic in the screen shot above, there is much less port 23 traffic. I am surprised at how much port 23 traffic is occurring and originating from multiple places on the Internet. There are likely scanners looking for open Telnet ports to exploit them with vulnerabilities or attempt brute force password guessing.

Analyzing the specific attack further, I found that the source of the attack originated from outside the United States and had not sent previous attacks. Wanting to see if there was other data on this host besides the IDS events, I turned to the Log Correlation Engine we had deployed there. This included logs from the Tenable Network Monitor to collect all network session data as well as system logs from a few servers and applications. Performing the query for our IP of interest we see this following data:

In this graph, we see that there were 70 normalized Emerging Threats Snort events, and slightly more TCP sessions. However, there was only one network session that completed. All of the other sessions "Timed Out" which is very indicative of a network scan. For the one session that was observed to complete (a full, SYN, data and good close with a FIN), it would be very interesting to see what the target and port were and this is shown below:

In this case, the target was an email server. SPAM logs were not being sent to the LCE, but delivered email was. This could allow us to conclude that the 34k bytes in the session on port 25 was likely some sort of spam email. Correlating this with the fact that the same IP address also tried a Solaris Telnet attack means that the remote host is most likely a SPAM source that also probes for vulnerable services.

The Security Center automatically correlates IDS events to discovered vulnerabilities and would have generated such an alert if the target host was vulnerable to the Telnet attack. To manually make sure, looking at the Nessus and Passive Vulnerability Scanner data that has been detected does not show any issues on port 23:

None of the vulnerabilities listed here seem serious enough to be concerned about.

Conclusions

This is the first in a series of several articles on different examples of event analysis. We've covered some related topics previously in this blog and they are listed below here:

• ShmooCon -- Network Monitoring Notes
• Finding Low Frequency Events
• Good and Bad Uses of Vulnerabiltiy Data for IDS Event Correlation
• Tracking Users Through Logs and Network Activity
  
• Blacklist Event Correlation
• Looking for "Never Before Seen" Events
• Example NBAD with the Log Correlation Engine