The IT Pro index is a Splunk index available to all University of Illinois IT staff which contains BPDU and interface change log messages collected from campus networking equipment. 

What is the IT Pro index?

The IT Pro index provides IT staff the ability to find log messages related to campus network equipment serving their responsible departments. This can be used as a troubleshooting tool when attempting to identify network related issues. Additionally, IT Pros can create custom alerts that notify them when particular conditions are met. Some possible scenarios where this index could help include:

• Creating alerts for switchports that provide connectivity to important infrastructure (such as servers, important lab equipment, etc.)
• Identifying the reason that a port has entered an "Error" state in Iris. The IT Pro index will show ports that enter a disabled state due to receiving BPDU frames. If you would like to learn more about BPDU frames, see Networking, Iris Scenarios: Ports disabled by Spanning Tree BPDUs
• Troubleshooting layer 1 (physical) link problems between an endpoint and campus network switch.

How do I search the index?

To search the IT Pro index, start by navigating to the TechSvc Splunk cloud instance. The landing page will allow you to enter searches, change the time range for your search, and see your search history (see Fig. 1). 

(Fig. 1 - Tech Services Splunk landing page)

There are numerous ways to customize your search to narrow down a query. 

• To search the entire index, enter the IT Pro index name into the search box: index=network-itpros-illinois_techsvc (see Fig. 2).

(Fig 2. - IT Pro index listing log messages from equipment across campus)

• To search for log messages reported by a specific network device, enter the IT Pro index followed by the network device in quotes: index=network-itpros-illinois_techsvc "sw-hab5" (see Fig. 3).
  • You can use the asterisk (*) character to display messages from all devices that match the name prior to the asterisk. For example, if you are wanting to display log messages for all switches located in the Henry Administration Building, you would enter: index=network-itpros-illinois_techsvc "sw-hab*" (notice that the asterisk replaces the switch number). 

(Fig 3. - IT Pro index listing log messages recorded from network equipment with the sw-hab* prefix)

• To search for log messages reported by a specific device and switchport, enter the IT Pro index followed by the network device and switchport both in separate quotes: index=network-itpros-illinois_techsvc "sw-hab5" "B3" (see Fig. 4).

(Fig. 4 - IT Pro index listing log messages from sw-hab5 port B3)

• Audio/Video equipment, wireless access points, and network switches are the most common devices on campus that send BPDU frames in an attempt to establish a Spanning Tree topology. If you have one of these devices connected to a network jack and are not getting layer 1 (physical) link, the switchport that it is connected to may be in an error state. To search for BPDU related log messages, enter the IT Pro index followed by BPDU in quotes: index=network-itpros-illinois_techsvc "BPDU" (see Fig. 5).
  • If you know the switch name/port that the device is connected to, you can also add that into the query. 

(Fig. 5 - IT Pro index listing log messages for ports that have been disabled due to receiving BPDU frames)

How do I create custom alerts?

To create custom alerts, create a search string that matches the criteria that you want to monitor. For example, if you wanted to be notified when port f5 on sw-gregory3 changes its interface state, you would enter: index=network-itpros-illinois_techsvc "sw-gregory3" "f5". Or, if you specifically wanted to know when port f5 on sw-gregory3 changes its interface state to off-line, you would enter: index=network-itpros-illinois_techsvc "sw-gregory3" "f5" "off-line". 

Once you identify the specific string that you want to be notified about, and search for that string, click Save As and then Alert in the top right corner above the search box (see Fig. 6).

(Fig. 6 - Navigating to the alert option in Splunk)

When saving as an alert, you will be prompted for information such as:

• Alert Title
• Alert Permissions 
  • It is recommended that you select "Private" since selecting "Shared in App" will let everyone capable of accessing the index to see your alert. 
• Alert Frequency
  • Alerts can be run on Cron Schedule, or by the hour/day/week/month.
• Trigger Conditions
  • Determines how the alert triggers. Identify the number of results before a notification is sent.
  • Alerts can trigger once for many results or once per result.
• Trigger Actions
  • Clicking the +Add Actions drop down will display a list of potential actions when the alert conditions are met. For example, if you want to receive an email notification when an alert is triggered, you will select Send email (see Fig. 6). 

(Fig. 7 - Splunk alert settings)

• • Email notifications can be sent to multiple recipients with a message priority, subject, and message (see Fig. 8). Additionally, the notification can include useful information such as:
    • Link to the alert
    • Link to the alert results
    • Trigger condition
    • Trigger time
    • & more

(Fig. 8 - Splunk trigger actions)

Summary and contact

This article has covered searching the IT Pro index in Splunk and how to create custom alerts which will notify recipients. Please direct any questions, comments, or recommendations to Technology Services Networking.