Assessing the security of SCADA devices has always been a challenging task. SCADA devices are used in several critical infrastructure industries, including power plants, manufacturing, chemical processing, and nuclear reactors. Thus, the high availability and security of these devices are of the utmost importance. The challenge lies in assessing the security of SCADA devices without causing any adverse effects. The special purpose-built systems often operate within a limited scope and use protocols that are specific to the tasks being performed, such as Modbus, OPC, and DNP3.

In 2006, Tenable Network Security released the first Nessus® vulnerability scanner and Tenable Passive Vulnerability Scanner (PVS) SCADA plugins (you can read the original release notes for PVS in a post titled "SCADA Network Monitoring" and the original release for Nessus titled "SCADA Checks For Nessus 3"). In April 2011, a new round of SCADA plugins were released for Nessus (covering devices from Movicon, 7-Technologies, and more).

Tenable is now pleased to announce the availability of additional SCADA plugins for Nessus ProfessionalFeed, Tenable SecurityCenter, and PVS users. Tenable's research team worked alongside SCADA experts from Digital Bond to test and identify a wide variety of common SCADA devices. The plugins were announced at Digital Bond’s S4 Conference on SCADA security held on January 19, 2012. Note: Digital Bond’s Dale Peterson joined us on the Tenable Network Security podcast episode 110 and spoke about the new plugins and SCADA security.

Below is a sample of some of the new SCADA plugins:

Burying Stumps

Recently I've been planning and executing a plan to fix some of the landscaping around my house (as a side note, try not to plan this to happen in the middle of July when it’s 90 degrees). In talking with people who have experience with landscaping projects we seem to always hit the topic of digging up and burying stumps, and whether this is a good idea or a bad idea. For the short term, it seems like a good idea. The stumps take up space in the ground so you need less fill (which saves money), burying is cheaper than grinding them down or having them hauled away, and you don't have to look at an ugly stump. The downside is that 7-10 years down the road, the stumps begin to rot and you are left with sinkholes in your yard.

There are a variety of indicators that a system has been compromised, ranging from the obvious to the very subtle.

Less obvious indications of a compromise include increased bandwidth, subtle IDS alerts (such as those indicating anomalous behavior) and mysterious configuration changes on systems. The questions that are typically asked include "How did they get in?" and "What did they do?" Tenable's Passive Vulnerability Scanner (PVS) provides useful information for answering these questions. Following are some of the alerts PVS may generate during an intrusion:

It's 4:55 PM on a Friday and you are looking forward to an enjoyable dinner with your family. Your Blackberry starts buzzing from across your desk while your inbox starts filling up with alerts from your SecurityCenter along with frantic emails from Human Resources. It seems a disgruntled employee named Jack Black quit today and nobody remembered to tell the IT group to disable his accounts until after important files started disappearing. Suddenly, you are stuck in Incident Response mode, gathering data on the user's activities. Do you cancel your reservations?

Fortunately, you have deployed Tenable Network Security's Unified Security Monitoring products, and have a wide array of resources[1] at hand to streamline the response process. These resources include SecurityCenter, the Passive Vulnerability Scanner (PVS) and Log Correlation Engine (LCE). At a high level, what can these resources do for you?

SecurityCenter

SecurityCenter provides a unified view of both vulnerability and event data along with the alerting, ticketing and reporting required for thorough user forensics.

Passive Vulnerability Scanner

PVS not only tracks vulnerabilities, but logs user and network activities detected in real-time on the wire. These activities include:

Tenable Network Security has released version 3.6 of the Log Correlation Engine. This new version includes many performance enhancements as well as its own web-based user interface. This blog entry describes the new user interface, the increased performance and the new features of LCE 3.6.

I was interviewed for episode #173 of the Risky Business information security podcast.

The previous Risky Business episode that discussed the recent release of the open source El Jefe project by Immunity Inc, focused on how process execution tracking for Windows can be a great source of security data - especially compared to raw network traces.  

During my interview with Patrick Gray, we covered how many SIEMs already have this sort of capability, but most SIEM users don't enable these features because they are complex. I also covered how Tenable's Log Correlation Engine can collect logs from both Unix and Windows computers that reflect process execution traces and how they can organized for attack detection, change detection, forensics, alerting, reporting and anomaly detection.  

Awful, awful, awful.....Magic!

It was my wife’s turn to choose a movie the other night, which means there were no kung fu fight scenes, sword fights or car chases. Instead, there was a scene that depicted a father-to-be talking to a father of three children. The father with three children was explaining to the father-to-be what parenthood was really like and stated: "Parenthood is awful... awful… awful... but then there is this magical moment that makes it all worth it… then awful... awful... awful and repeat". Parents reading this, especially ones with small children, are probably laughing. However, I thought that the "awful, awful, awful, magic!" analogy also very accurately described penetration testing.

Passwords are just so easy to abuse...

It was interesting to see that the top scorer in the game (who went by the handle of "ftp", and coincidentally had 21 scores in the first day of game play!) did not use fancy new exploits, 0day attacks or a wide range of open-source or even commercial tools. He was able to gain access to systems because the teams left default or easily guessable passwords set on some of the Linux servers. He used SSH to login to the systems, then SCP to upload some Python code, that was used to update the scoring engine. From there he was able to maintain access, not by rootkit technology or anything sophisticated, but just hiding in plain sight. The Python script makes a TCP connection to the scoring server and sends a message. It was moved into a file called "/dev/vfat", to make it look like a system file. Next, a shell script was written to call the python script every ten minutes. This file was called "getty" and ran in the background, and was also inserted into the startup scripts to ensure it kept running. The teams never found these processes running and "ftp" won the game, no exploits required.

Hacker "ftp" at work, winning the game using built-in tools such as bash, python and SSH.

Setting up the Log Correlation Engine & Splunk

Tenable has recently released a new Log Correlation Engine (LCE) client that allows you to collect log data from Splunk installations to send to LCE, Tenable’s solution for log storage, normalization and correlation. If you have instances of Splunk in your environment, it’s a simple process to configure the integration. Below is an overview of the traffic flow:

With the recent news of more than 500,000 web sites becoming compromised, Tenable's research team added support into Nessus and the Passive Vulnerabiltiy Scanner to look for evidence of recently installed Javascript that may be indicative of a mass compromise.

With Nessus, the webmirror.nasl and webserver_infected.nasl plugins enumerate the web pages of a scanned web server and look for evidence of a compromise. With the PVS, plugin #4487 watches for unencrypted web traffic which contains evidence of these compromises.

Previously, Tenable has blogged about this type of active and passive detection for a different mass compromise event. Also, last week we blogged about auditing Internet facing web servers. The techniques outlined there should be utilized when auditing web servers that may have been infected with malicious content.

News References

• http://blogs.iis.net
• http://blog.washingtonpost.com

Tenable has recently released version 2.0.3 of the Log Correlation Engine (LCE). This blog entry will highlight the new features as well as recent enhancements to the log parsing rule sets and the event correlation algorithms.

Daemon and Agent Enhancements

The main log processing daemon has enhanced performance. Several optimizations were added which drastically increase the overall events per second throughput. LCE customers should notice substantially lower CPU utilization as well.

Additionally, the stability of the remote LCE clients (such as the tail agents, netflow, network sniffing or OPSEC agents) has also been enhanced.

We are encouraging all customers to upgrade their daemon and clients to 2.0.3.

Log Parsing Rule Enhancements

The entire signature library of log parsing rules has also been analyzed and rewritten for higher performance and accuracy. This new library is roughly ten times more efficient than before which also leads to much higher events per second rates and lower CPU utilization.

Tenable has also added more unique event 'types' which enhances analysis and reporting. The current list of supported event types, with new event types indicated with an asterisk, is as follows:

• application - Logs from generic applications and daemons.
• backdoor (*) - Primarily normalized network IDS events indicating a backdoor or covert channel. Events from the blacklist.tasl correlation script which correlate with Arbor, SANS, Bleeding Threats and other types of blacklisted IP addresses.
• compliance - Events which violate PCI, SOX and other types of compliance issues.
• compromise (*) - Primarily network IDS events which are critical in nature or indicate a successful attack.
• connection (*) - Firewall 'accept', 'allow' and 'permit' events which indicate a network connection.
• correlated (*) - A generic type for a variety of correlated events.
• detected-change - Indicates a change detected on the network, at a host, with a user or with an application.
• dhcp (*) - Covers all DHCP logs such as IP address leases.
• dos - Primarily network IDS events that indicate some sort of denial of service attack.
• dns - Logs from DNS servers such as Bind.
• error (*) - A generic type to catch error log messages from a wide variety of applications and network devices.
• firewall - All network deny firewall events as well as system changes.
• ftp - Logs from a variety of file transfer protocol network daemons.
• hids - Logs from host based IDS programs.
• honeypot - Logs from a variety of network and system honeypots.
• intrusion - Generic normalization of non-critical IDS events.
• lce - Status and connection logs from the Log Correlation Engine.
• login (*) - Generic type for successful logins from a wide variety of OSes, network devices and applications.
• login-failure (*) - Generic type for failed logins from a wide variety of OSes, network devices and application.
• logout (*) - Generic type for logouts from a wide variety of OSes, network devices and application.
• mail - Logs from Sendmail, Postgres and other mail daemons.
• mysql (*) - Logs from MySql.
• nbad - Logs from network based anomaly systems such as Stealthwatch.
• nessus - Logs from the Windows or UNIX Nessus 3 daemons.
• network - Logs from the Tenable Network Monitor or Tenable NetFlow Monitor agents.
• p2p-activity (*) - Generic network IDS or firewall logs which indicate P2P activity.
• pup-activity (*) - Generic network IDS or firewall logs which indicate some sort of spyware or undesired software.
• radius - Logs from  various radius authentication devices.
• reboot (*) - Logs indicating network devices, OSes or applications which have been shutdown or restarted.
• router (*) - System logs from network routers.
• scanning (*) - Logs from firewalls, network IDSes and other sources that indicate a host is performing port scanning.
• spam - Logs from mail applications and network monitors that indicate spam activity.
• stats - Logs from the LCE's statistical event daemon (the stats daemon).
• switch - Logs from a wide variety of network switches.
• system - Generic type for all system events such as time changes, new hardware discovery and software installation or removal.
• user-activity - Events for new users, changes to user privileges and many other user related activities.
• virus (*) - Logs from network IDSes, firewalls and host-based programs which indicate detection of a virus infection.
• vmware (*) - Logs from VMware systems including startup and shutdown of virtual machines as well as addition of new virtual machines.
• vpn (*) - Events such as VPN to VPN connections, successful remote connections of users and modifications of VPN configuration.
• vulnerability (*) - Normalizes real-time logs from the Passive Vulnerability Scanner.
• web (*) - Logs from Apache, IIS and many different web proxy devices.
• wireless (*) - Logs from a variety of wireless access points.

These new event types are also made use of heavily by the updated TASL correlation scripts which is covered in the next section.

If you have not upgraded to version 2.0.3 yet, these plugins are available by doing a manual plugins update. If you do upgrade to version 2.0.3, we recommend doing an additional plugin update to get the very latest available rules.

Updated TASL Correlation Scripts

Also with this release of LCE 2.0.3, Tenable has enhanced the accuracy, performance and ease of use of the existing library of TASL correlation scripts.

Many of the TASLs which perform similar algorithms on different events have been combined. Some TASLs which performed analysis on specific event sources (like just the Dragon IDS) have been made more generic to work on any source. This generalization occurred through the use of the new 'types' in the log parsing rules.

In addition, a performance analysis of each TASL was also accomplished and many optimizations were made.

And finally, with the new 'type' tagging in the underlying log parsers, several new classes of TASLs were written to generically look for a wide variety of interesting compromise and suspicious activity.

There are currently 35 TASLs available. The main TASL download site has also been re-categorized with new sections for easier comprehension of what they do.

New and updated TASLs that I would like to point out include the following:

• attack_and_connect_to_blacklist.tasl - Any system which is attacked as detected by a critical IDS event and then connects to a blacklisted IP address will have an alert generated. This finds systems which have been compromised and are part of or being controlled by a botnet.
• blacklist.tasl - This script has an external helper Perl script which downloads publicly available blacklisted IP address from sources like Arbor, SANS and Bleeding Threats. For any firewall connection events or sniffed or netflow sessions, it evaluates in real-time, any connections to or from a blacklisted IP address.
• crowd_surge.tasl - Detects when a large number of local systems reach out at the same time to a remote system. This can indicate spyware, rootkit and botnet activity. The script now supports firewall 'connection' type events as well as Tenable netflow and network monitor logs.
• new_network_user.tasl and new_system_user.tasl - These scripts subscribe to any login events and automatically learn the current network users as well as current system users. Previously the new_network_user.tasl was known as thew new_nw_usr.tasl script and it has been renamed to be more clear.
  
• nids_compromise_detection.tasl - Looks for any host which has been attacked with a critical NIDS event and then attacks a different system. This can indicate a compromised system.
• nids_compromised_server.tasl - This script automatically learns (through Passive Vulnerability Scanner real-time events) where your servers are and if any of them attack another system, an alert is generated.
• long_term_scanning.tasl - Detects several conditions such as systems performing continuous scanning for at least three hours, systems that have been attacked which have started to scan and systems that have been scanned which are now scanning others as in a worm outbreak.

Many of the older TASLs also have new types of functionality. If you are running additional TASLs on your LCE, we strongly recommend checking to see if it has been updated and if the new functionality is relevant for your environment.

Below is a screen shot of an alert generated by the attack_and_connect_to_blacklist.tasl script:

This script alerts when a host is attacked, and then the host reaches out or is connected with an IP address on one or more suspected "black lists" of IP addresses. In this particular case, a host was attacked and this event was detected by an IntruSheild IPS and then an outbound network connection was detected with a Tenable Network Monitor. This connection was found to terminate with an IP address tracked by the SANS Internet Storm Center.

Many of the TASL correlation scripts perform this level of analysis in real time.

For More Information

Installation and upgrade instructions are available on the Tenable Support Portal. After upgrading to version 2.0.3, users should perform a plugin update and then manually audit their TASLs to see if they want to remove or replace any of them with the new ones which are now available. Tenble LCE customers should contact Tenable Support if they have any questions regarding the upgrade to 2.0.3

Very often when I speak with Tenable customers about performing IDS or Event analysis, I ask them if they use the Time Distribution tool under the Security Center. This tool is used to identify any combination of low frequency events for any query or time period it works with raw IDS events under the Security Center as well as normalized log or network events under the Log Correlation Engine. Regardless if you are analyzing the last million events which occurred in the last hour, or the entire last 90 days of events, this tool can quickly let you find what is unique and "interesting".

Why Find Low Frequency Events?

Many activities (checking mail, surfing the web, performing backups, .etc) occur at similar times over and over. These result in network and system logs which also occur over and over. Similarly, repetitive activities also generate repetitive false positives in your network IDS.

These events may be very interesting but it is much more likely that they are very boring. Since they occur over and over, an interesting filter would be to remove them and see what is left behind. Another way to look at this is to assume that your network isn't compromised or severely attacked each day. This can be a dangerous assumption in some cases, but as a filter which can be invoked as an analysis tool, can be very effective and useful. 

Basic Algorithm

The Security Center is used to configure any query you want. Maybe you are looking at the default "last 24 hours" view of events. Maybe you want to see all port 21 traffic for the last 5 days or all "User Activity" type events for the last month.

Regardless of your filter, the Time Distribution tool computes the oldest event time and the newest event time and then breaks this time period up into 20 parts. Then, for each unique event or log that has occurred, it counts the total that have occurred in each part. If an event has occurred in at least twelve  of the buckets, it is considered "high frequency" and is suppressed.

Example Output

Below is an image of all logs and events in a 24 hour period involving port 21, 22, 53 and 80.

There are several thousands events each hour in this trace. However, analyzing this data with the Time Distribution tool gives us this view:

In this view, we can see that even though there are thousands of events, the only really "low frequency" or very unique ones occurred at specific times. Clicking on the specific times would allow all events to be analyzed for that specific time period. 

Obtaining This Tool

This feature has been available in the Security Center and Log Correlation Engine for several years and is available while analyzing raw IDS events as well as normalized IDS, netflow, firewall, windows events and other types of logs.

For More Information

For a true low frequency event, Log Correlation Engine customers should consider using the "Never Before Seen" TASL script. This script remembers when a certain type of event first occurs on a host and alerts if a new event (such as an SSH login failure) has occurred for the first time.

Readers interested in learning more about event correlation should consider the existing TASL scripts for the Log Correlation Engine and also consider the "stats" log anomaly engine.

Tenable Network Security also offers several webinars and white papers online:

• Correlating IDS Events with Vulnerabilities (webinar)
• Network and Behavioral Anomaly Detection (webinar)
• Security Event management (white paper)

Several new PRM libraries and one TASL script have been updated and are available for download and use with the Log Correlation Engine. The list below shows what has changed. Each PRM or TASL links to the URL for downloading.

• detect_change tasl Support for Windows and FreeBSD system time change events.
• os_freebsd.prm Support for FreeBSD system time change, disk errors and more types of user login events.
• os_linux.prm Support for Linux named logs.
• os_win2k_app.prm Uses the Windows server name as the 'sensor' name.
• os_win2k_sec.prm Uses the Windows server name as the 'sensor' name.
• os_win2k_sys.prm Uses the Windows server name as the 'sensor' name.
• web_squid.prm Support for more Squid logging formats.
• virus_symantec.prm Support for Symantec anti-virus 'virus removed' messages.

To install these files, simply download them and place them in the /usr/thunder/daemons/plugins directory and then restart the thunderd process.

Customers are encouraged to periodically monitor their notmatched.txt file, which contains a list of all logs that were collected, but did not match a known pattern. Please contact Tenable if one of the supported applications or products is missing logs in your environment.