ACCC IDS Process Description

Details of the complete Intrusion Detection System at ACCC.

The IDS system has pieces located on several systems: acccscan6, tigger, ead-ids-{1..5} and radius-1.

ACCC's IDS capabilities now consist of two pieces: snort and our flow IDS.

Snort is deployed on ead-ids-{1..5}. This machine is connected via gig-ethernet to a switch port that has been configured to mirror the data traversing UIC's internet border. The snort sensor is configured with the latest VRT snort rules as well as some locally written rules.

See Document 82930 is unavailable at this time.

ACCC analyzes the snort logs to determine which rules are most effective in determining when a machine on campus has been compromised. When such a rule is found, it is configured in such a way that when triggered, it will cause a script to be executed by swatch. This script introduces the data into the autofilter queue for automatic filtering as described below.

The flow IDS system is based off of the Cisco Flow Logs. Although ideally, an IDS would be able to view the contents of individual packets, the flow logs only provide a summary of a connection. This summary of what happened during a connection is what Cisco refers to as a flow. Each flow log entry provides the following pieces of information (as well as a few more that we don't use):
  • Start/Stop times of the connection
  • Source/Destination IP addresses
  • Source/Destination ports
  • Protocol ( 1=ICMP, 6=TCP, 17=UDP)
  • TCP Flags set during the connection
  • Number of packets sent during the connection
  • Number of octets (bytes) sent during the connection

Flow log line for example **

The IDS is based upon this information in these flow records.

Theory

When a machine is scanning, by definition it will generate a flow record for each IP/port that it connects to. There are two basic types of scans: sequential and non-sequential. The sequential scan is the easiest to detect as it simply requires that traffic from a single on-campus machine that connects to a N consecutive IP addresses either on-campus or off-campus. For our purposes, N is currently set to the value of 30 and it seems to work fairly well.

The nice thing about sequential scans is that they can be detected even if they represent some never before seen type of infection. This is true as it will not matter what port or protocol is being contacted to detect the scan, simply the fact that the machine is talking to consecutive IP addresses.

Non-sequential scans are more difficult to detect. The main problem with detecting this type of traffic is that it is imperative that valid traffic not be mis-interpreted as a virus/worm/attack. To detect non-sequential attacks, a rules database is used. It is possible using this rules database to successfully detect traffic of specific types and to classify that traffic as a worm. In reality, due to the generalized nature of the data from the flowlogs, it is sometimes impossible to state with certainty what worm or virus is represented by the traffic as more than one worm may have the same flow signature. Nevertheless, the classification can be modified to indicate that a given signature may be one of a small group of worms.

Practice

The Cisco flowlogs are collected from the routers by the networking group. After they receive the flow log, they copy the log onto an mstore filesystem for our use. This mstore filesystem is mounted on acccscan6 at /var/flows. The flows for the current day are located directly in /var/flows. A process runs on acccscan6 shortly after midnight every day and archives the logs from the previous day to a directory under /var/flows/logs/mm-dd where mm-dd is (obviously) the month and day that the logs were for. Currently, there isn't any process to remove the files. In time, a process will be created and cronned to erase the logs after a certain period of time (I'm currently thinking of retaining 3 months of logs).

Processes

The main IDS program is located under /usr/local/flowtools on the acccscan-6. The ids script is called 'fscan' (flow scanner) and is run every five minutes. When the IDS starts up, it checks to see which flowlog files have been updated in the last five minutes that haven't yet been processed. These files are read in by fscan (using the Ohio State flow-print utility) and analyzed. As previously mentioned, the fscan program looks for sequential scanners and also for those that match the entries in the rules file.

It should be noted that when looking for scanners, we are really looking for machines that are misbehaving. For this reason, all of the rules become associated with SOURCE IP addresses when the conditions are met.

The rules file is located at /mnt/global/security/configs/ids.rules on the farm. The format of the rules file is somewhat strict and a mistake in the rules file may keep the IDS from starting. A sample rule may look like:
id=blaster;prot=17;dp=135;pkts=1;oct=43;thr=100&Blaster_Worm

In this example, the signature id is "blaster". This id is used internally by the ids to keep track of items that match. NOTE that the ID tag MUST be unique. The ids will not start if there are multiple rules with the same ID tag present. The other parts of the rule are somewhat self-explanatory. In words, this rule associates the id blaster with flow logs lines that are for UDP (prot=17) going to port 135 (dp=135) that have 1 packet (pkts=1) of 43 bytes (oct=43). The description for this rule is "Blaster_Worm". Note that for all rules, the first entry must be the 'id' tag and the last entry must be the description. The other tags can be in any order and can be present or not. Note that there is also a "thr" (threshold) tag. This means that the rule is only matched if, in this case, 100 flows from a given source IP are seen.

Note that some other rule options that haven't been discussed: sp (source port) and req (requisites). The source port option is obvious. For the other, look at the example below:
id=WormA1;prot=17;dp=135;pkts=1;oct=43;req=WormA2,WormA3;thr=100&Unknown_WormA
id=WormA2;prot=1;dp=2048;pkts=1;oct=42;thr=100;req=WormA1,WormA3&Unknown_WormA
id=WormA3;prot=6;dp=445;pkts=3;oct=198;thr=50;req=WormA1,WormA2&Unknown_WormA

In order for this ruleset to match, all of the individual rules must match. This is indicated by the separate req rules that require each other. Note also that the thresholds can be different for the rules in a ruleset. In this example, the worm will try to ping all addresses but will only attempt to infect those that actually respond, so we have set the threshold for the ICMP packets to be higher.

Once fscan determines that a given ruleset matches for a UIC source IP address, it tags that IP address with that description. Due to the way the flow records are produced and the fact that many worms and viruses have different variants, it is possible that any worm may have more than one signature. Regardless of how many different rulesets for the same description match, the IDS will only tag that source IP with the description once.

Once a UIC IP address has been detected by fscan as either scanning or attempting to spread a worm, a file is created in /mnt/global/flowlogs/uic. The file contains fifty records from the flowlogs that show some of the traffic that was observed for the IP address. Originally, all of the traffic for that five minute interval was written to the file, but the files were occasionally very large when a machine was very talkative during that five minute interval. If more than fifty records are needed to determine what the problem was, the flow information can be retrieved from the archived flow logs. This will be discussed later. The name of the file that is created is in the format:
aaa.bbb.ccc.ddd.hostname.uic.edu.__Filter_reasons

where aaa.bbb.ccc.ddd is the IP address of the machine. Note that these files are later renamed to include the MAC address associated with that IP address by the autofilter program.

Note that when a dialin user is detected as being infected with a worm or scanning, the file is created in a different directory, specifically /mnt/global/security/dialin_sus_queues/sus_request. This is done so that a separate process running on the radius server can check this queue (directory) and determine the user that was using that IP address at the time of the scanning. Once the user has been identified, the file is moved to the normal "uic" directory mentioned above and renamed to include "[netid]_" at the beginning of the filename. Obviously, "netid" is the real netid of the person responsible for the traffic. If the radius server script was unable to determine who the user was (if it was via a dialin server that uses tacacs instead of radius), then the username will be "Unknown".

A couple of recent additions to the rules:

  • log="filename" causes flow data that matches that rule to be logged to a file (useful for watching traffic on a certain port (think irc/botnets)

  • scan=1 on a rule causes the machine to be put into the scan queue rather than filtered when the rule matches.

Autofilter

Autofilter (actually autofilter.pl) is the program that handles the automatic filtering of machines that have been identified as scanning, etc. Autofilter and it's associated files currently reside in ~edzsu/production_filter. Files of importance are:

  • autofilter.pl (The script itself)

  • autofilter.config ( Controls which viruses etc. are filtered for and which message files to use)

  • autofilter.exclude ( Exclude high priority IPs from the possibility of being autofiltered)

  • syncfiles.pl ( Creates all variants of message files from a single msg file)

  • message files (ending in .msg, this info is sent when a machine is filtered for the corresponding reason)

autofilter.pl is run every five minutes on tigger. It ignores any files that have the word "autofilter" in them as these files were already processed. For dialin users, the filename is parsed and the netid is suspended (unless of couse, it is "Unknown). The suspend command provides the boilerplate id that should be passed to csomenu. This is a tailored version of the msg file for that reason that says that the infected user was on the dialins. Note that tailored versions of the boilerplates also exist for tut users and wireless.

For regular IP addresses that have been discovered to have a worm that autofilter is configured for, the IP address is filtered and the filename is changed by adding the MAC address of the machine and by appending the word autofilter followed by the process id. This guarantees (almost) a unique filename. The MAC addresses are found by first loading the contents of the arpdata file when autofilter starts. It assumes that the last MAC found for a given IP is the correct one and this assumption is almost always correct.



Keywords:snort flow ids   Doc ID:82936
Owner:Jason R.Group:University of Illinois at Chicago ACCC
Created:2018-06-14 14:11 CDTUpdated:2018-07-13 12:24 CDT
Sites:University of Illinois at Chicago ACCC, University of Illinois at Chicago Sandbox KB
CleanURL:https://answers.uillinois.edu/accc-ids-process-description
Feedback:  0   0