Networking, Lens API Quick Start Guide
Required access credentials
Lens API clients must use an IP address that is registered in DNS with a hostname ending in uiuc.edu, illinois.edu or uillinois.edu. Access to the Lens API requires a current campus Active Directory user account whose name is known as a network contact in Technology Services Contacts Database. The Lens API provides users with data pertaining to the networks for which they are listed contacts.
Not just for NetIDs anymore!
The Lens API authentication mechanism recognizes user accounts anywhere in the campus Active Directory and can service these user accounts if they are also known to Contacts Database with network Iris permissions. This allows you to grant privileges to service users in your OU for use by departmental scripts and applications. See Access by service user section below for details.
Access by NetID
The root URL for browser-based person NetID access is https://lens-api.techservices.illinois.edu/web
Your NetID is represented by an [AD.UILLINOIS.EDU] Active Directory user account which identifies you personally. As a network contact, you can access the Lens API using your NetID and AD password. This is appropriate for your direct, interactive use of the API because you can supply your name and password as necessary. For example, you can use your NetID with command-line utilities you create and for direct browser-based access.
Other applications require Lens API access when no user is involved. For these we do not recommend using your personal NetID and AD password when such use requires you to place them in persistent storage (e.g. a disk file).
Storing your password on disk risks more than unauthorized Lens API access. At potential risk are any other accessible University resources with which you are entrusted.
See the following section for using the Lens API with unattended applications.
Access by service user
The root URL for unattended application access is https://lens-api.techservices.illinois.edu/api
Some applications use the Lens API when no user is involved who can provide access credentials (e.g. scheduled jobs). For these you can use a campus AD user account you create within your own organizational unit and dedicate to this purpose. This service user account represents your application and neither you nor any other individual person. You can store the service user password for your unattended application without risk to personal and unrelated resources.
We strongly recommend you choose a username that begins with your unit or group followed by the hyphen character in order to identify the responsible unit and sidestep potential conflicts with NetIDs. For example, service users for the Physics or Computer Science departments might be named "physics-lens-user" or "compsci-lens-user".
You must authenticate using your full Active Directory userPrincipalName (typically serviceuser@ad.uillinois.edu, but other variations are also possible).
Steps to get service user account setup
Steps to create and authorize a non-person service user
- Create an Active Directory user object under your own organizational unit (OU) with a strong, randomly generated password. We suggest choosing a name longer than 8 characters that begins with your unit or group followed by a dash (e.g. “physics-lens-user”) for easy identification and to avoid conflicting with potential NetIDs; see also Active Directory Naming Conventions.
- Lens API will authenticate service users in Entra ID (formerly known as Azure AD). To sync the user object created in step 1 with Entra ID, please populate extensionAttribute2 with value of O365. Additional details at Entra ID documentation
- Use Contacts Database to grant Iris read-only permissions to your service user (either directly or via an AD group).
- Non-person service users must be individually exempted from MFA for Lens API. Please contact the net-trouble@illinois.edu with the Active Directory userPrincipalName of your service user to request this
Where in the Active Directory is Carmen Sandiego?
Some units operate private Active Directory trees which may or may not be somehow linked to the UOFI campus AD. Although these trees may "trust" the Campus AD and thereby recognize its users and objects, the reverse is not generally true. You need to define your pseudouser account in the UOFI Active Directory. Your unit may already own space in the UOFI AD. Contact ADSupport@illinois.edu for assistance with the UOFI AD.
As for any NetID, the Lens API imparts privileges to your service user account matching its Technology Services Contacts Database network Iris permission. Unlike the NetIDs, however, Contacts Database does not automatically know that your service user exists. You need to contact net-trouble@illinois.edu to request a Contacts Database identity made with the same name as your non-NetID service user account. Afterward, you or an authorized network contact in your unit can assign your service user to be read-only contacts for desired networks.
We strongly recommend you assign read-only privileges! Although today the Lens API just provides read-only service, the service user is also known to and carries privileges within Iris! Furthermore, the Lens API may not always remain read-only.
Through service user accounts, you can automate your applications with little need for contact beyond your unit. Although they mitigate risk, they nonetheless access University resources and can be exploited to compromise privacy. When you create and empower service user accounts as agents on your behalf or your unit's, you retain responsibility for its proper use and protection. In short: you made it, you own it, you authorized it, and you are responsible for it.
General query format
The Lens API defines several types of objects which may be returned by HTTP GET queries. (The POST method is not supported at this time. See the Lens API FAQ.)
All objects of like type provide the same set of attributes. For example, device objects all have a device_name attribute. These attributes may be used in queries to retrieve objects, and they are returned by queries as a part of the objects. Furthermore, objects are related to each other. For example, device objects are related to interface objects (because network devices have interfaces). The goal of every Lens API query is to fetch a list of some particular type of object based on the values of attributes and their interrelationships. The list may be "dressed up" with additional information to build a consistent and informative data set in one query.
All Lens API queries have a format described by the following template:
https://lens-api.techservices.illinois.edu/web/<subject type>? <param1>=<expression1>&<param2>=<expression2>&...&dressings=<dressing1>,<dressing2>,...
The subject type is the type of the objects you want to list (e.g. device). Each parameter can be the name of an attribute that is found in the subject type (e.g. param1 may be device_name). An expression describes the attribute values you're looking to match in choosing objects for your list. Alternatively, a parameter can describe how the subjects in your list must relate to other objects. The list of objects reflects the application of all these qualifications as described below. Finally, the result is elaborated with additional data as requested by the given set of "dressing" tokens (e.g. dressing1).
Example: device
To retrieve Lens device object data, point your browser to
https://lens-api.techservices.illinois.edu/web/device
You will be prompted for your netID and AD password in ad-dc-t1.ad the first time you submit a request. The URL above retrieves all devices stored in the Lens database that include vlans of your privileged network. The device attributes returned will be the "essential" attributes listed in the device specification.
Supported object types
Here is a list of all currently supported object types, linking to their specifications. Consult each specification to understand what the type represents, the attributes it supports, and how its objects relate to others.
Query mechanics
Object attribute data types
Objects such as devices can also be retrieved using any of their attributes as search criteria. The attribute names are listed in the specification, as are their basic data types. An attribute's data type defines the formats for its query expressions and its values as returned by queries. The data types currently supported are:
Datatype | Value Range | Note |
---|---|---|
Text | 256 character string | treated as a case-insensitive exact match. If the string begins with the tilde "~" character, the text is treated as a regular expression |
Integer Decimal |
38 significant digit number, or "MIN" or "MAX" token | "MIN" and "MAX" represent negative and positive infinity |
Timestamp Date |
UTC (GMT) time, or "MIN" or "MAX" token | All dates in and out of webservice are in UTC. Before you query, you must translate local times to UTC in order to get the results you expect. Likewise, you must translate time values returned by queries back to local time. Symbols "MIN" and "MAX" represent the earliest past and farthest future times recognized (1/1/1970 and 1/2/2037) |
Boolean | 1 or 0 | Boolean values are represented as the integer 1 for TRUE and the integer 0 for FALSE |
IP address | dotted-decimal IP address or subnet in CIDR notation | If a single IP is given, an exact match will be performed. If a CIDR notation is given (e.g. 192.168.100.0/24), it will perform a fully inclusive range match. Note also that a CIDR expression will be converted into the appropriate "power-of-2" subnet for the range scan, so both "192.168.100.55/24" and "192.168.100.0/24" will result in the same scan of the range 192.168.100.0 through 192.168.100.255. |
MAC address | 12 hexadecimal characters | Any delimiter is allowed, although data is always returned without any delimiter. MAC fields ONLY perform exact matches. |
Regular Expressions
The webservices supports POSIX Extended Regular Expressions for all text data types. To invoke a regular expression match, the string must begin with the tilde "~" character. The API forces all regular expression matching to be case-insensitive. The POSIX Extended Regular Expressions or ERE flavor is a standard similar to the one used by the UNIX egrep command. For a more complete description of POSIX ERE regular expressions, see POSIX Extended Regular Expressions. For a table comparing various regular expression flavors, see Regular-Expressions.info.
Range queries
The webservices recognizes the comma "," as a range separator for all numeric, date, or scalar IP data types. If a comma is used, the webservice will perform a fully-inclusive range match. For example, using "vlan_numer=100" will result in an exact match for vlan_number 100, while using "vlan_number=100,200" will result in a range scan for all vlan_numbers between 100 and 200, inclusive. IP datatypes allow single IP range scans, such as "192.168.100.0,192.180.100.255", but do not allow CIDR-notation mixed with the comma range separator ("192.168.100.0/24,192.169.101.255") as the desired outcome is unclear.
Semantics of joining multiple parameter queries
In all cases, different parameters will be "AND"ed, while multiples of the same parameter will be "OR"d. So "device_name=sw-dcl1&type=switch" translates to "device_name=sw-dcl1 AND type=switch" while "device_name=sw-dcl1&type=switch&device_name=sw-dcl5" translates to "(device_name=sw-dcl1 OR device_name=sw-dcl5) AND type=switch".
Relationship-based queries
All object types have relationships defined in their specification. Each relationship has a name and represents a real-world semantic link between an object and others. For example, a device object relates to several interface objects via its interfaces relationship. A relationship has only one "related type". Objects can be queried using relationships and the attributes of related types by the following syntax:
<{{subject_type}}>?<{{relationship_name}}>:<{{related_object_attribute}}>=<expression>
A query by relationship lists the objects (of subject_type) which relate (via relationship relationship_name) to a selection of objects (of the related type) matching the attribute query expression. Note that you may build your queries with simultaneous qualifications by attributes and by relationship, subject to the multiple-parameter semantics described in the previous section.
For example, to find all devices the have an interface that is named "A1", you would use the following query:
device?interfaces:ifname=A1
device is the base object, interfaces is a device relationship that references interface objects, and ifname is an attribute of an interface.
Note that relationships function similarily to attributes in the syntax above, making it possible to follow a "chain" of relationships to a distantly related object. For example, one could extend the previous example to find all devices that have an interface that is configured for vlan 32 untagged by using this query:
device?untagged_interfaces:vlans:vlan_number=32
The device object's "untagged_interfaces" relationship points to interface objects, which themselves have a relationship called "vlans" which points to interface_vlan objects that have the attribute "vlan_number." This query will return all device objects that include in their set of "untagged_interfaces" an interface that has within its set of "vlans" an interface_vlan object with vlan_number=32.
Object dressings
A basic query will only return a list of objects of the subject type that satisfies the query parameters. In order to get related objects returned in the same query, you must apply "dressings" to the result. Each Relationship defined for an object in the specification provides a "Dressing Parameter" to invoke the dressing. Dressings are requested by including the "dressings" parameter followed by a comma-separated list of dressings in the order you wish them applied to your result. For example, to request devices with device_name=sw-dcl5 and all of that device's interfaces:
device?device_name=sw-dcl5&dressings=dev_interfaces
The objects returned in a dressing call can also be dressed. To request the same device with all its interfaces and those interfaces' vlans:
device?device_name=sw-dcl5&dressings=dev_interfaces,ifc_vlans
Note that the ordering of the dressings listed in the request is important. When the "dev_interfaces" dressing is applied, interface objects are added to the result set. The "ifc_vlans" dressing is then applied to those interface objects. If "ifc_vlans" is requested first, there would be no interface objects in the result set to modify, so that dressing would be empty. Dressings are always applied to every object of the dressing's target type in the result set, no matter where in the list that object was applied.
The difference between Query-by-Relationship and Dressing
Objects' relationships provide the bases both for query-by-relationship and for dressing results. However, these are distinct operations which do not otherwise interact. Query-by-relationship does not automatically dress the list of subjects with the matching related objects. For example, if you build a list of devices that carry a particular VLAN, you still have to request the list be appropriately dressed if you want the VLAN information returned with the devices. Likewise, the dressing does not reflect the query-by-relationship. If you ask for VLAN information with the devices, you get information for all VLANs per device. (However, your desired result is in there.) In other words, query qualifications simply function to build the subject list and do not imply nor interact with potential dressings.
How the API returns data
Result structure
See the following link to learn about the data structures returned by the Lens API. Note: This page is taken from early design stages. While the result structure is accurately defined, please note that the examples do not reflect the true names of object types and attributes. You'll still get the idea.
Serialized data encodings
In order to package an interrelated and potentially large data set, the Lens API returns a complex data structure. The data structure is serialized in one of the available encodings:
Encoding | MIME type | Description |
---|---|---|
XML | text/xml | The result is encoded in an XML document with a very simple nested-element format. It may be trivial to decode into native structures with a basic SAX parser implementation. Note: There is no DTD nor XML Schema definition available. |
JSON | application/json | The result is encoded in a JSON program which can be deserialized into native structures using a Javascript interpreter or dedicated JSON parser. |
YAML | text/x-yaml | The result is encoded in a human-readable language-independent YAML document which can be deserialized into native structures with a standard YAML parser. NOTE: YAML is not as widely known as XML and JSON. Given the availability of the latter, YAML is not considered a supported technology. However, the API developers intend to preserve the YAML interface for some technical advantages and its human usability. |
Selecting an encoding
You can select an encoding by the following means:
- Supply a content-type URL query parameter with a value indicating the MIME type of the desired encoding. For example:
device?device_name=~sw.*&content-type=application/json
More examples
The root url for all examples is: https://lens-api.techservices.illinois.edu/web
device?type=switch&opsys=hp&opsys=os
The above query returns devices whose type="switch" and (opsys="hp" or opsys="os").
device?type=~^Swi&opsys=~^hp&opsys=os
The above query returns devices whose type satisfies the regular expression /^Swi/ and (opsys satisfies /^hp/ or opsys="os"), all case insensitive. Regular expressions can be used for string columns such as name, type, created_by, model, opsys, by prefixing the string with the tilde "~".
device?ip=172.20.14.47
The above query returns devices with an IP Address of 172.20.14.47
device?ip=172.20.14.0/23
The above query returns devices with an IP Address in the range of 172.20.14.0 through 172.20.15.255
mac_port?mac=001122334455&mac=aabb1122ccff
The above query returns a list of current ip_mac observations where the MAC address matches either 001122334455 or aabb1122ccff
ip_mac?ip=172.20.14.47&ip=172.20.14.48
The above query returns a list of ip_mac objects whose IPs match either 172.20.14.47 or 172.20.14.48.
Scripting
In a scripting language, such as perl, you can specify the return mime type as well as your credentials to allow httpd basic authentication over SSL.
#!/usr/bin/perl -w use strict; my $url='https://lens-api.techservices.illinois.edu/api/device?device_name=sw-dcl5&content-type=application/json'; my @results = get($url); foreach my $result (@results) { print $result."\n"; } sub get { use LWP::UserAgent; my $addr = shift; my $req = HTTP::Request->new(GET => $addr, HTTP::Headers->new); #$req->header(Accept => 'text/xml'); # substitute your favorite mime type here my $auth = "Basic ".ncode("<my-service-user>@ad.uillinois.edu:mysupersecretpassword"); # service user and ad password $req->header(Authorization => $auth); my $res = LWP::UserAgent->new->request($req); die $res->status_line unless $res->is_success; return $res->content; } sub ncode { use MIME::Base64; my $in = shift;
my $ret = encode_base64($in);
### remove any newline inserted by encode_base64
$ret =~ s/\n//g;
return $ret; } 1;
The result displays in the standard output. Currently, text/xml, text/x-data-dumper, text/x-json, text/x-yaml mime types are supported.