Welcome to NI&S AAA!

This documentation is intended to assist new admins of the AAA (authentication, authorization, accounting) system with quickly achieving an operational competency of its components. Here you will find a description of the component software and hardware, a description of the services backed by the system, guides for establishing system access, guides for maintaining the config, and some discussion of the design of the system itself.

This documentation is not intended to provide low-level understanding of the component software, or to be a development guide. Rather it focuses on presenting the basic procedures, and context necessary to maintain and support the existing system and config.

For an in-depth guide on the use of any software component, it is strongly suggested to read the documentation provided by that software's developers. Links to that documentation will be provided under Additional Resources.

In addition to reading the software developers' documentation, the best way to gain knowledge, is to start experimenting with the config on the -dev tier servers, or in your own sandbox. Go build something and see how it breaks!

This documentation is divided into the following sections:

• Introduction: Provides an overview of the AAA system's software components, and the services they support.
• Infrastructure: Details the hardware (physical and virtual) serving the AAA system's software components, and also details the licensing for those components.
• Service Details: Provices an explanation of each of the services backed by the AAA system, and instructions for managing their respective config.
• Renewal Quick Reference: Provides a quick reference for any significant dates related to those AAA System components which must be renewed by AAA System admins directly.
• Getting Started: Provides instructions for new admins to aquire access to the system components, and details on the management of sensitive data such as system passwords and certificates.
• Deployment: Provides instructions on how to deploy the various components.
• Management: Provides instructions on how to manage the config for the various components.
• Troubleshooting: Provides tips on how to troubleshoot some of the most common issues had by service customers.
• Design Considerations: Briefly describes the intent behind some of the design choices for the AAA system.
• Enhancements: Details some of the desired improvements to the AAA system, and new features that could/should be implemented in the future.
• Known Issues: A description of any known issues with the AAA system.
• Additional Resources: Links to various other sources of information, guides, and tools which may be of use in administering the system.

What is NI&S AAA?

NI&S's AAA system provides authentication and authorization for a number of different services, ranging from guest internet access to administrator access to the network infrastructure itself. The three main software components are FreeRADIUS, Clearpass, and OpenLDAP.

FreeRADIUS

FreeRADIUS is a free, open-source RADIUS server. Its modular design has successfully encouraged significant community development, and it supports all commonly used authentication protocols and databases. FreeRADIUS provides the AAA system with a performant, scalable RADIUS solution.

Infrastructure

Developer's Documentation

ClearPass

Aruba Clearpass is a proprietary suite of tools centered around network access control, and designed to be used with Aruba hardware. The suite includes RADIUS and TACACS servers, databases, monitoring and alerting systems, and various admin and customer tools all presented via a web front-end.

Infrastructure

Developer's Documentation

OpenLDAP

OpenLDAP is a free, open-source LDAP directory server. NI&S's instance of the directory contains entries with authentication and authorization data used by both FreeRADIUS and Clearpass.

Infrastructure

Developer's Documentation

Infrastructure

The AAA software components run a number of different systems, both physical and virtual, and potentially divided into separate tiers (dev, pprd, and prod). The following documentation provides an inventory of those systems and related data.

• FreeRADIUS Infrastructure
• Clearpass Infrastructure
• OpenLDAP Infrastructure

FreeRADIUS Infrastructure

• App Admin/Contact: NIS Wireless Team (nis-wifi-g@vt.edu)
• Sys Admin/Contact: Laurie Zirkle (lat@vt.edu)

Clearpass Infrastructure

• App Admin/Contact: NIS Wireless Team (nis-wifi-g@vt.edu)
• Sys Admin/Contact (prod): NIS Wireless Team (nis-wifi-g@vt.edu)
• Sys Admin/Contact (pprd/dev): NI&S Systems Operations

PROD

VIPs

PPRD

VIPs

DEV

OpenLDAP Infrastructure

Contact Information

• App Admin/Contact: SIS Middleware (middleware-neoldap-g@vt.edu)
• Sys Admin/Contact: NI&S Systems Operations (nis-systems-g@vt.edu)

OpenLDAP Servers

JSON table that looks great in GitLab, but not in S4

{
  "fields": [
    {"key": "name", "label": "Name", "sortable": true}, 
    {"key": "ipv4", "label": "IPv4", "sortable": true}, 
    {"key": "ipv6", "label": "IPv6", "sortable": true}, 
    {"key": "os", "label": "OS", "sortable": true}, 
    {"key": "role", "label": "Role", "sortable": true}, 
    {"key": "tier", "label": "Tier", "sortable": true }, 
    {"key": "hardware", "label": "Hardware", "sortable": true} 
],
  "items": [
    { "name": "bee.nis.vt.edu", "ipv4": "198.82.169.87", "ipv6": "2001:468:c80:210f:0:154:f38d:8e73", "os": "Alma Linux 9", "role": "consumer", "tier": "prod", "hardware": "virtual" },
    { "name": "cicada.nis.vt.edu", "ipv4": "198.82.169.19", "ipv6": "2001:468:c80:210f:0:124:a9be:cdce", "os": "Alma Linux 9", "role": "consumer", "tier": "pprd", "hardware": "virtual" },
    { "name": "conehead.nis.vt.edu", "ipv4": "198.82.169.90", "ipv6": "2001:468:c80:210f:0:165:9b7d:7dcb", "os": "Alma Linux 9", "role": "consumer", "tier": "prod", "hardware": "physical" },
    { "name": "cricket.nis.vt.edu", "ipv4": "198.82.169.55", "ipv6": "2001:468:c80:210f:0:10a:4153:c230", "os": "Alma Linux 9", "role": "provider", "tier": "prod", "hardware": "virtual" },
    { "name": "froghopper.nis.vt.edu", "ipv4": "198.82.169.90", "ipv6": "2001:468:c80:210f:0:18a:bb:8ce2", "os": "Alma Linux 9", "role": "consumer", "tier": "prod", "hardware": "virtual" },
    { "name": "gnat.nis.vt.edu", "ipv4": "198.82.169.56", "ipv6": "2001:468:c80:210f:0:10c:3728:4514", "os": "Alma Linux 9", "role": "consumer", "tier": "dev", "hardware": "virtual" },
    { "name": "grub.nis.vt.edu", "ipv4": "198.82.247.107", "ipv6": "2001:468:c80:6101:0:19b:215c:e6df", "os": "Alma Linux 9", "role": "consumer", "tier": "prod", "hardware": "physical" },
    { "name": "hornet.nis.vt.edu", "ipv4": "198.82.169.67", "ipv6": "2001:468:c80:210f:0:1e4:a709:3216", "os": "Alma Linux 9", "role": "provider", "tier": "pprd", "hardware": "virtual" },
    { "name": "midge.nis.vt.edu", "ipv4": "198.82.169.15", "ipv6": "2001:468:c80:210f:0:1f4:a62c:991c", "os": "Alma Linux 9", "role": "consumer", "tier": "prod", "hardware": "virtual" },
    { "name": "owlfly.nis.vt.edu", "ipv4": "198.82.169.47", "ipv6": "2001:468:c80:210f:0:1ad:4e6c:98b9", "os": "Alma Linux 9", "role": "provider", "tier": "dev", "hardware": "virtual" },
    { "name": "cranefly.nis.vt.edu", "ipv4": "198.82.247.75", "ipv6": "2001:468:c80:4101:0:1de:e360:f57e", "os": "Alma Linux 9", "role": "consumer", "tier": "prod", "hardware": "physical" }
  ],
  "filter": true
}

HTML table for S4

External Dependecies

NI&S's AAA system relies on a few services not presented by it's own infrastructure, and are supplied by other entites of VT Division of IT.

VMware/vCenter

The first of these depencies is vCenter, which manages the virtual machines that host many of the AAA System servers. The NI&S Systems Operations (NISSO) team is responsible for managing the virtualization config in vCenter.

Minimal access may be available to AAA System admins here: vcenter.nis.vt.edu.

However, any changes to the virtualization must be requested of NISSO, ideally with a ticket in Service Now. Tickets can be assigned to NIS-Systems.

Enterprise Directory

Virginia Tech's Secure Identity Services manages several services crucial to the function of the AAA System. Primary among these is the Enterprise Directory (ED), which presents identification data of VT-affiliates.

ED stores a pid password which is used by affiliates to access the vast majority of their networked services. For customer convenience, both Clearpass and several FreeRADIUS instances use ED as the authentication backing their own customers' access.

ED is also the authoritative source for the network password which VT-affiliates use to access eduroam and vpn services. This network password is replicated to the AAA System directories, and any changes to it must be made in ED and then allowed to propigate downstream to the AAA System.

Clearpass also leverages some of the more sensitive user data stored in ED to support role-based access for users. An ED-ID Service is maintained (along with an associated certificate) allowing the secure retrieval of that data. This ED-ID Service must be renewed yearly.

Requests for support should be directed to Identity Management Customer Support, and Service Now tickets can be assigned to IMCS.

The replication of network passwords from ED to the AAA System directory is handled with a software client designed and maintained by NI&S's own Software Development team. That team's Senior Director can be contacted for assistance, and tickets should be assigned to NIS-Software Development.

ATLAS

The NI&S Software Development team also maintains a database indirectly used by the AAA System, called ATLAS. This database stores authorization data related to VT-affiliates subscriptions to wireless and vpn services.

Similarly to Enteprise Directory, this data is replicated from ATLAS to the AAA System directory, via a software client managed by the Software Development team.

That team's Senior Director can be contacted for assistance, and tickets should be assigned to NIS-Software Development.

Password/Cert Repository

NI&S Network Operations use a combination of Pass, GPG, and GitLab to maintain two repositories for their passwords and certificates. The AAA System manages all of its admin passwords and certificates in these two stores.

• Password Store
• Certificate Store

For access to the stores, or assistance with their management, contact the Wireless Networking Team.

Licensing

FreeRADIUS

FreeRADIUS is a free, open-source product, but NI&S maintains an annual support contract with its developers, NetworkFreeRADIUS. Specifically, we pay for support of Conehead and Grub, the two physical hardware devices that provide out-of-band authentication to network infrastructure.

Since our FreeRADIUS enviroment has been incredibly stable, we've have little need for emergency support. The contract has been more useful to present the NetworkRADIUS team with design and implementation questions, or config/code review.

Our contact for contract renewal with NetworkRadius is Marc Rozier (marc.rozier@networkradius.com).

Clearpass

Clearpass clusters require a bundle of licenses and subscriptions to function. Each of our Clearpass servers includes a Platform license to activate the product, and an Access license that enables the system to authenticate, daily, some losely enforced number of unique endpoints. Licenses are also available for other Clearpass applications such as Onboard and Onguard.

Aruba has permanent, subscription, and evaluation type licenses available. NI&S's appliances have a mix of these, across the cluster tiers.

• Production

  • 4x Platform (permanent)
  • 4x Access (25k endpoints, permanent)
  • 1x Access (2.5k endpoints, permanent)
  • 1x Access (25k endpoints, subscription 1yr)

• Pre-production

  • 2x Platform (subscription 1yr)
  • 1x Access (25 endpoints, permanent)

• Development

  • 2x Platform (permanent)
  • 1x Access (25 endpoints, permanent)
  • 1x Onboard (25 endpoints, permanent)
  • 1x OnGuard (25 endpoints, permanent)

Licenses are viewable on the Clearpass publisher under Administration » Server Manager » Licensing.

Licenses are also viewable in Aruba's Customer Support Portal.

Service Details

Authentication for eduroam

eduroam is an internationally federated, wireless access service for those in the academic community. The intent is for users from any one federation member to receive free wireless network access at the physical location of any other federation member.

Functionally, each federation member presents the eduroam SSID, backed by some 802.1x authentication. Users authenticate to that network with a username of the form userId@realm, where the userId is some identifier assigned to that user, and the realm is some identifier unique to the federation member itself. Typically a member's internet domain is used, so the realm for Virginia Tech's users is vt.edu; e.g. player1@vt.edu.

Once a user attempts to connect to the eduroam network, a local authentication server analyzes the realm in the provided username. If the user is on their home campus then the realm is also local, and the server authenticates the user itself. If the realm represents some other federation member, the user is considered to be roaming, and their authentication must be proxied to their own home server.

The request is proxied to the eduroam federation which determines which of the federated members the realm is associated with, and again proxies the request to them. That remote server performs the actual user authentication, returns the result to the eduroam server, which finally returns it to the server where the user originated the network access request.

Upstream Customers

Authentication for VPN

Authentication for the F5 Load-Balancers

Overview

This page describes how an F5 systems administrator (a customer with load-balanced application servers) gets a level of access (defined by a role) to the device that is limited to their partition. This ability allows them to manage their services.

! Important Note !
By design, on the F5 load balancers, a systems administrator can have access to only one partition with only one role. Please refer chapters 8, 9, and 10 in BIG-IP TMOS Concepts.pdf for more information.

• Role: the access level a systems administrator has for the set of F5 objects used by their service.
• Partition: a space on the device that the systems administrator has access to. Once inside a partition, only the F5 objects pertinent to the services deployed by that systems administrator can be viewed/modified.
• Access Policy: refer to Access Policy for Customers-Systems Administrators for more information.

The idea is to leverage the existing AAA (Authentication, Authorization, Accounting) architecture by tweaking the FreeRADIUS and OpenLDAP servers to provide this capability.

• When F5 receives a login request from a customer, it sends an Access-Request to FreeRADIUS, which consults NI&S OpenLDAP for authorization and ED OpenLDAP for authentication.
• Assuming the authentication is successful, an Access-Accept packet is sent to the F5 with reply attributes that must match one of the remote-role groups defined on the F5.
• Once there is a match, the customer gets the appropriate role and partition access.

Remote-Role configuration on F5

auth remote-role {
    role-info {
        Admin_Group {
            attribute F5-LTM-User-Info-1=Admin
            console %F5-LTM-User-Shell
            role %F5-LTM-User-Role
            user-partition %F5-LTM-User-Partition
        }
        AppEditor_Group {
            attribute F5-LTM-User-Info-1=AppEditor
            console %F5-LTM-User-Shell
            role %F5-LTM-User-Role
            user-partition %F5-LTM-User-Partition
        }
    }

• A customer who needs access to the device must have a user object that is a member of one of the group objects in the NI&S OpenLDAP.
• The radiusAttribute attribute in the group objects determines the role a user has (0 in the example below is equal to Administrator. Refer to the F5 dictionary in FreeRADIUS), the partition to which s/he belongs, and whether they have tmsh (traffic management shell) access or not.
• The customer's user object must have a radiusprofile objectClass and a radiusProfileDn attribute that points to the group they belong to.

Group Object in OpenLDAP

# Admin, Groups, F5, Configuration, NIS, vt
dn: cn=Admin,ou=Groups,ou=F5,ou=Configuration,ou=NIS,o=vt
cn: Admin
description: Entries for the Admin group user accounts
member: nuid=1777725,ou=People,ou=NIS,o=vt
radiusAttribute: F5-LTM-User-Info-1+=Admin
radiusAttribute: F5-LTM-User-Partition+=All
radiusAttribute: F5-LTM-User-Role+=0
radiusAttribute: F5-LTM-User-Shell+=tmsh
objectClass: groupOfNames
objectClass: radiusprofile


# AppEditor, Groups, F5, Configuration, NIS, vt
dn: cn=AppEditor,ou=Groups,ou=F5,ou=Configuration,ou=NIS,o=vt
cn: AppEditor
description: Entries for the Application Editor group user accounts
member: nuid=1143470,ou=People,ou=NIS,o=vt
radiusAttribute: F5-LTM-User-Info-1+=AppEditor
radiusAttribute: F5-LTM-User-Partition+=Systems
radiusAttribute: F5-LTM-User-Role+=300
radiusAttribute: F5-LTM-User-Shell+=tmsh
objectClass: groupOfNames
objectClass: radiusprofile

User Objects in OpenLDAP

# 1777725, People, NIS, vt
dn: nuid=1777725,ou=People,ou=NIS,o=vt
nuid: 1777725
uid: afotedar
sn: Fotedar
cn: CN - afotedar
prohibited: FALSE
radiusProfileDn: cn=Admin,ou=Groups,ou=F5,ou=Configuration,ou=NIS,o=vt
objectClass: radiusprofile
objectClass: nisUserAccount
objectClass: inetOrgPerson

# 1143470, People, NIS, vt
dn: nuid=1143470,ou=People,ou=NIS,o=vt
nuid: 1143470
uid: stlee
sn: Lee
cn: CN - stlee
prohibited: FALSE
radiusProfileDn: cn=AppEditor,ou=Groups,ou=F5,ou=Configuration,ou=NIS,o=vt
objectClass: radiusprofile
objectClass: nisUserAccount
objectClass: inetOrgPerson

• FreeRADIUS's load-balancing instance is setup for role-based access. The Access-Accept packet must have the reply attributes below.
• For more information on how FreeRADIUS should be set up, see Using F5 vendor-specific attributes with RADIUS authentication.

Radtest from Cricket's FreeRADIUS server

radius@cricket(load-balancing):~
$ radtest afotedar <my pid password> 198.82.169.53:1820 234234 <shared secret
in clients.conf>
Sending Access-Request of id 81 to 198.82.169.53:1820
	User-Name = "afotedar"
	User-Password = "<my pid password>"
	NAS-IP-Address = 198.82.169.53
	NAS-Port = 234234
	Message-Authenticator = 0x00
rad_recv: Access-Accept packet from host 198.82.169.53:1820, id=81, length=68
	F5-LTM-User-Info-1 = 'Admin'
	F5-LTM-User-Partition = 'All'
	F5-LTM-User-Role = Administrator
	F5-LTM-User-Shell = 'tmsh'

Authentication for Network Admins

Overview

In addition to their wireless and VPN accounts in ou=People,ou=NIS,o=vt, NIS network administrators have an account in ou=Administrators,ou=NIS,o=vt that is used to authenticate to network infrastructure devices. The passwords on these accounts are managed independently of the wireless/VPN accounts, and administrators can change their passwords using the ldappasswd command.

This command is built-in on current versions of MacOS, and can be easily installed on Linux with the appropriate package manager.

For Windows 10/11, simply enable WSL and then follow the instructions below in your WSL Linux environment.

Ubuntu/Debian

sudo apt install ldap-utils

Red Hat/AlmaLinux

sudo yum install openldap-clients

I don't really want to install anything on my computer

No problem! These utilities are already installed on conehead and grub, if you can SSH to those hosts already. Just make sure your environment has all the right variables set.

source /apps/etc/openldap/profile

Don't want to remember to do that every time? Just add that line to the end of your .bashrc file like so:

echo "source /apps/etc/openldap/profile" >> ~/.bashrc

How to change your network administrator password

Lookup your network administrator nuid if you don't already know it

ldapsearch -LLL -H ldap://cricket.nis.vt.edu:11389/ -x -b ou=Administrators,ou=NIS,o=vt uid=your_vt_username_aka_pid nuid

Change your network administrator nuid password

Enter your old password when prompted to Enter LDAP Password:

ldappasswd -H ldap://cricket.nis.vt.edu:11389/ -x -ZZ -W -S -D nuid=your_nuid,ou=Administrators,ou=NIS,o=vt

Use manager authorization to change the password of another network administrator to a temporary value

ldappasswd \!authzid=dn:cn=Manager,o=vt -H ldap://cricket.nis.vt.edu:11389/ -x -ZZ -W -S -D nuid=your_nuid,ou=Administrators,ou=NIS,o=vt nuid=other_administrator_nuid,ou=Administrators,ou=NIS,o=vt

Enter the temporary password for the other administrator when prompted for New password: and Re-enter new password:

Enter your password when prompted to Enter LDAP Password:

For PPRD or DEV environments, respectively substitute hornet or owlfly for cricket in the commands above.

Authentication for Guest Wireless

Authentication for Device Wireless

Authentication for Hotel Wireless

Authentication for Bluefield College

NI&S Directory

Basic Information

Software: OpenLDAP Database Size: 212MB Workload: 20 - 22 million search requests daily (less on weekends)

Middleware manages the OpenLDAP servers that provide directory services to the NEO environment, and NI&S manages the services such as FreeRADIUS and ClearPass that use that directory to authenticate and authorize users.

Directory Structure

Internal Configuration

cn=config
        cn=module{0},cn=config
        cn=schema,cn=config
                cn={0}core,cn=schema,cn=config
                cn={1}cosine,cn=schema,cn=config
                cn={2}inetorgperson,cn=schema,cn=config
                cn={3}radius,cn=schema,cn=config
                cn={4}radiusClient,cn=schema,cn=config
                cn={5}vtnis,cn=schema,cn=config
        olcDatabase={-1}frontend,cn=config
        olcDatabase={0}config,cn=config
        olcDatabase={1}mdb,cn=config
                olcOverlay={0}syncprov

Under the config tree the radius, radiusClient, vtnis entries are the most notable, each containing attributes useful for the FreeRadius servers which query the directory. Radius and RadiusClient define attributes, standard to the RADIUS protocol, which may be used internally by the radiusd process. VTNis defines custom attributes which are useful for authorizing VT affiliates network- and application-access.

Virginia Tech Data

o=vt
    ou=NIS,o=vt
        ou=People,ou=NIS,o=vt
        ou=Entitlements,ou=NIS,o=vt
        ou=Administrators,ou=NIS,o=vt
        ou=Local,ou=NIS,o=vt
            ou=Updaters,ou=Local,ou=NIS,o=vt
        ou=Configuration,ou=NIS,o=vt
            ou=F5,ou=Configuration,ou=NIS,o=vt
                ou=Groups,ou=F5,ou=Configuration,ou=NIS,o=vt
            ou=FreeRADIUS,ou=Configuration,ou=NIS,o=vt
                ou=Clients,ou=FreeRADIUS,ou=Configuration,ou=NIS,o=vt

Under the VT tree are the actual records used for authentication and authorization for NEO’s customers. The People and Entitlement subtrees are the most often queried, and are important for authorizing wireless network access and access to various VPN’s provided by NIS. Records in the Administrators subtree contain accounts used for direct access to networking equipment or to various applications used to configure said equipment. A few of these accounts are used by applications to access networking equipment, rather than actual people.

The Local subtree contains a handful of accounts used to bind and query the directory. The FreeRadius and Clearpass software packages each have such an account, as do Cerberus and Orchestra. Additionally, the account used for syncrepl is in this subtree.

The Configuration subtree itself contains two important subtrees: one for F5 member groups and another for defining FreeRadius clients. The F5 member groups bundle the permissions a user might have on the F5 application. Radius clients are used to define which network application systems are allowed to send access requests to the FreeRadius servers.

Logging & Monitoring

Renewal Quick Reference

Getting Started

System Access

Password & Certificate Management

Deployment

Deploying FreeRADIUS w/ Ansible

Deploying OpenLDAP w/ Ansible

Requirements

• Physical or VPN connection to the VT network
• Local installation of Ansible 2.7 or newer
• Local installation of HashiCorp Vault v1.12 or newer
• Local installation of Git 2.13 or newer
• Local installation of OpenPGP (gpg)
• Local installation of OpenSSH client (ssh)
• VT Username (PID) with Duo MFA
• Membership in one or more of these ED groups
  • middleware.vault.neo-ldap.prod
  • middleware.vault.neo-ldap.prod.admin
• Designation as a recipient with your PGP public key signing the files in the certificates repository
• An account with the ability to sudo su - openldap on each LDAP server to be managed.

Overview

The OpenLDAP software is deployed by the CNS/ansibile-admin playbook using the Middleware/neo-ldap and Middleware/neo-ldap-group_vars roles as submodules at tasks/roles/openldap and group_vars/openldap, respectively. A separate certificates submodule at certificates contains the PKI certificates used by OpenLDAP and other software on the servers. The ansible-admin playbook needs to be passed options to use several vault encrypted variables, using the vault-id openldap.

ansible-playbook -i ansible_hosts --vault-id openldap@contrib/vault/vault-client.sh tasks/main.yml --tags openldap,certs [--limit hostname[,hostname]]

The contrib/vault/vault-client.sh helper script in the command above obtains the Ansible Vault password from a HashiCorp Vault secret. If the --limit option is not passed, careful attention should be paid to the inventory file to be sure that only the desired hosts are listed in the openldap role.

Some advice about tags

This Ansible playbook is flexible enough to address multiple deployment and maintenance scenarios through different combinations of tags, which also means it is possible to produce undesired results through incorrect use of tags. Here is a summary of the available tags, and some recommended combinations for common scenarios.

• openldap
• certs
• dump
• load
• tests
• start
• stop
• purge_db

Tag usage examples

Fresh install of OpenLDAP on a new machine

--tags openldap,certs --limit newmachine.vt.edu

Upgrade OpenLDAP on an existing machine

--tags dump,load,openldap,certs --limit existingmachine.vt.edu

Purge all o=vt data from an OpenLDAP consumer node and syncrepl the entire o=vt DIT from the provider

--tags stop,purge_db,start --limit consumernode.vt.edu

Using pass and vault to avoid password prompts from ansible-playbook

The vault command can be used to obtain a valid vault token before running the ansible-playbook command. We can also provide a gpg passphrase for the certificates decryption to ansible-playbook with its --extra-vars option. Consider the following example, which assumes you have stored your VT password and gpg passphrase in pass as vt and gpg, respectively.

export VAULT_ADDR=https://vault.es.cloud.vt.edu && \
vault login -method=ldap username=your_vt_username password=$(pass vt) && \
pass gpg | ansible-playbook --extra-vars "gpg_passphrase=@/dev/stdin" -i ansible_hosts --vault-id openldap@contrib/vault/vault-client.sh --tags openldap,certs --limit LDAP_server_hostname tasks/main.yml 

While this method of executing the playbook makes the entry of passphrases much more convenient, it still requires two separate Duo authentications when it runs, so be prepared to answer the multifactor notifications.

Upgrading OpenLDAP

Upgrade a host by updating the openldap_active_version varable in the host_vars/hostname file and run the playbook with the proper tags. The dump and load tags are used to export and import the directory data during OpenLDAP version upgrades, and can also be used independently for ad-hoc logical backup and restore operations if desired.

ansible-playbook -i ansible_hosts --vault-id openldap@contrib/vault/vault-client.sh tasks/main.yml --tags dump,openldap,certs,load --limit LDAP_server_hostname

Rotating secrets

HashiCorp Vault

The Ansible vault password is stored in HashiCorp Vault in a key/value (kv) secrets engine at dit.middleware/neo-ldap/prod with a key of openldap. If you do not already have a valid token from Vault, set the VAULT_ADDR=https://vault.es.cloud.vt.edu environment variable and login with your VT username and password.

export VAULT_ADDR=https://vault.es.cloud.vt.edu && vault login -method=ldap username=your_vt_username

Ansible vault

The playbook uses variables stored in encrypted form in the group_vars/openldap/vault file. These variables have corresponding entries in the group_vars/openldap/vars file that point to those in the group_vars/openldap/vault file so as to keep encrypted variables visible to playbook editors and users. We must rekey the Ansible vault with the secret in the HashiCorp Vault whenever it is updated (see above).

ansible-vault rekey --vault-id openldap@contrib/vault/vault-client.sh group_vars/openldap/vault

Whenever a variable needs updating in the Ansible vault, we must edit the group_vars/openldap/vault file by supplying the vault password

ansible-vault edit --vault-id openldap@contrib/vault/vault-client.sh group_vars/openldap/vault

Stripping sensitive data from production data exports

When setting up new dev and pprd instances, passwords and secrets should be redacted from LDIF backups of the production directory:

sed -i "s/userPassword:: .\+/userPassword:: `echo somegarbagepasswordthatdoesnotwork | base64 -`/g" backup/o_vt.ldif
sed -i "s/radiusClientSecret: .\+/radiusClientSecret: somegarbagesecretthatdoesnotwork/g" backup/o_vt.ldif

or simply replaced inline when exporting data for that specific purpose:

slapcat -b o=vt -F /apps/openldap/openldap/etc/openldap/slapd.d \
| sed "s/userPassword:: .\+/userPassword:: `echo somegarbagepasswordthatdoesnotwork | base64 -`/g" \
| sed "s/radiusClientSecret: .\+/radiusClientSecret: somegarbagesecretthatdoesnotwork/g" > backup/o_vt_exported.ldif

Deploying Clearpass

Management

Managing the RADIUS Instances

Using manager authorization to update the freeradius password in the OpenLDAP directory

ldappasswd -e \!authzid=dn:cn=Manager,o=vt -H ldap://cricket.nis.vt.edu:11389/ -x -ZZ -W -S -D nuid=your_nuid,ou=Administrators,ou=NIS,o=vt uid=freeradius,ou=Local,ou=NIS,o=vt

Enter the new password for the freeradius user when prompted for New password: and Re-enter new password:

Enter your password when prompted to Enter LDAP Password:

For PPRD or DEV environments, respectively substitute hornet or owlfly for cricket in the command above. If you are not a user with manager authorization, the above command will fail.

Managing the LDAP Directory

Connecting

With the exceptions of conehead and grub, all of the LDAP servers require a physical or VPN connection to the Virginia Tech network in order to be accessed for management activities. LDAP directories can be anonymously searched for public information on port 11389 from the Virginia Tech network, but service and management accounts must bind to the directory to access protected attributes of directory entries. Directory managers can bind to the directory using their uid=*,ou=Administrators,ou=NIS,o=vt accounts and obtain manager authorization as detailed in the section below.

Directory administrators that need full access to the OpenLDAP software and the root of the directory tree must connect to the server via SSH with their VT usernames and escalate their privileges with the sudo su - openldap command. After becoming the openldap user, administrative operations can be performed through the LDAP IPC socket by passing -H ldapi:/// -Y external options to the ldap client utilities: ldapadd, ldapcompare, ldapdelete, ldapmodify, ldapmodrdn, ldappasswd, ldapsearch.

Obtaining Manager Authorization

Manager authorization is granted through the SASL Proxy Authorization feature of OpenLDAP using the -e option to the ldap utilities mentioned above. Users who are authorized to change data in the o=vt subtree have authzFrom attributes set for them in the cn=Manager,o=vt entry, which is the RootDN for o=vt. They can be listed with ldapsearch -b cn=Manager,o=vt -s base authzFrom as the openldap operating system user.

Helpful Aliases for Directory Managers

It can be a little cumbersome to always have to pass your authorization credentials with every ldap command, so these aliases can make your life easier. Just put them in your ~/.bash_aliases file, and then use mgrldapsearch instead of ldapsearch to view entries that your user doesn't have access to view, or mgrldapmodify instead of ldapmodify when you need to make an update.

~/.bash_aliases

alias mgrldapadd="ldapadd -e \!authzid=dn:cn=Manager,o=vt $*"
alias mgrldapdelete="ldapdelete -e \!authzid=dn:cn=Manager,o=vt $*"
alias mgrldapmodify="ldapmodify -e \!authzid=dn:cn=Manager,o=vt $*"
alias mgrldapsearch="ldapsearch -e \!authzid=dn:cn=Manager,o=vt $*"
alias mgrldapwhoami="ldapwhoami -e \!authzid=dn:cn=Manager,o=vt $*"

The Online Configuration Database

Configuration settings for the running LDAP directory can be modified with the ldapmodify command without a restart, yet will persist after a restart.

About LDIF Files

While updates to the LDAP directory can be done inline, the more common approach is to store the operations in an LDIF file and use the -f something.ldif option of the ldap client utilities that change data. It is important to store these files securely and delete them promptly when using this technique to alter sensitive data.

About the FreeRADIUS Data

In order for any network system to send access requests to the FreeRadius servers, that system must be in a list of ‘clients’ that FreeRadius is aware of. The majority of those clients are the wireless network controllers and the routers and switches that make up the wired network infrastructure -- of which there are over two-thousand. Rather than maintain that list on every individual FreeRadius server, each server queries the directory for the known clients when its own radiusd process is started. These clients are recorded in the Clients subtree.

Among the identifying information recorded is a cleartext radiusClientSecret which the client must provide and match whenever it sends an access request to the FreeRadius server.

example

dn: radiusClientNUID=48581922809,ou=Clients,ou=FreeRADIUS,ou=Configuration,ou=NIS,o=vt
radiusClientNUID: 48581922809
radiusClientIpAddr: 172.16.246.52
radiusClientSecret: REDACTREDACTREDACTREDACTREDACTREDACTREDACTREDACTREDACTREDACT
radiusClientShortname: VTC-BC-AA-01
radiusClientIdentifier: VTC-BC-AA-01.cns.vt.edu
radiusClientLastUpdated: 1574349635
objectClass: radiusClient

The FreeRadius servers query the directory when authenticating users for the eduroam network, basic vpn, or rlan-vpn. The user’s uid (pid) is leveraged to find both a network account record and entitlement record, under the People and Entitlement subtrees respectively.

The network account contains the nt hashed password for authentication, and a prohibited attribute to denote administratively locked accounts.

example

dn: nuid=1815024,ou=People,ou=NIS,o=vt
nuid: 1815024
uid: markhw
sn: Williams
cn: Mark Williams
userPassword: {nt}REDACTREDACTREDACTREDACTREDACTED
prohibited: FALSE
objectClass: nisUserAccount
objectClass: inetOrgPerson
objectClass: radiusprofile

Each network account may be associated with up to three service subscriptions: wireless, basic vpn, or rlan-vpn access. For each subscription there will be a matching entitlement record. Existence of the record is equivalent to having a subscription.

In addition to the pid, both the network account and the entitlement reference the ‘uid’ value associated with that same pid in Enterprise Directory. The entitled attribute of each entitlement references the dn of the associated network account.

example

dn: nuid=9056724418,ou=Entitlements,ou=NIS,o=vt
nuid: 9056724418
entitled: nuid=1815024,ou=People,ou=NIS,o=vt
entitledUID: markhw
entitlement: cns.service.network.vpn
objectClass: nisEntitlement

dn: nuid=4512003484,ou=Entitlements,ou=NIS,o=vt
nuid: 4512003484
entitled: nuid=1815024,ou=People,ou=NIS,o=vt
entitledUID: markhw
entitlement: cns.service.network.wireless
objectClass: nisEntitlement

dn: nuid=0101010101,ou=Entitlements,ou=NIS,o=vt
nuid: 0101010101
entitled: nuid=010101,ou=People,ou=NIS,o=vt
entitledUID: player1
entitlement: cns.service.network.rlanvpn
objectClass: nisEntitlement

The FreeRadius servers query the directory when authenticating users and applications for access to network equipment and network configuration software. After successfully matching the salted, SHA1 hashed password, the FreeRadius server returns ALL values stored in the radiusAttribute fields to the authenticating client.

example

dn: nuid=81412786070,ou=Administrators,ou=NIS,o=vt
nuid: 81412786070
uid: markhw
cn: NetAdmin - markhw
radiusAttribute: Juniper-Local-User-Name=network-administrator
radiusAttribute: Cisco-AVPair+="shell:priv-lvl=15"
radiusAttribute: Aruba-CPPM-Role=Super-Admin
radiusAttribute: PaloAlto-Admin-Role=ITSO-Admin
mail: markhw@vt.edu
userPassword: {ssha}REDACTREDACTREDACTREDACTREDACTREDACT1234
objectClass: radiusprofile
objectClass: nisUserAccount
objectClass: radiusObjectProfile
objectClass: extensibleObject

The FreeRadius servers run a few different queries against the directory when authenticating users trying to access the F5 load-balancers. Radius filters for a uid matching the username provided in the access request, and looks for an additional radiusProfileDN attribute.

example

dn: nuid=1815024,ou=People,ou=NIS,o=vt
nuid: 1815024
uid: markhw
sn: Williams
cn: Mark Williams
userPassword: {nt}REDACTREDACTREDACTREDACTREDACTED
prohibited: FALSE
radiusProfileDN: cn=AppEditor,ou=Groups,ou=F5,ou=Configuration,ou=NIS,o=vt
objectClass: nisUserAccount
objectClass: inetOrgPerson
objectClass: radiusprofile

That value informs another query against the directory for a groupOfNames, which contain the authorization attributes to be handed back to the load-balancer in the access reply.

example

dn: cn=AppEditor,ou=Groups,ou=F5,ou=Configuration,ou=NIS,o=vt
cn: AppEditor
description: Entries for the Application Editor group user accounts
member: nuid=88888888,ou=People,ou=NIS,o=vt
member: nuid=77777777,ou=People,ou=NIS,o=vt
member: nuid=66666666,ou=People,ou=NIS,o=vt
objectClass: groupOfNames
objectClass: radiusprofile
radiusAttribute: F5-LTM-User-Info-1+=AppEditor
radiusAttribute: F5-LTM-User-Partition+=All
radiusAttribute: F5-LTM-User-Shell+=tmsh
radiusAttribute: F5-LTM-User-Role+=300

Useful NetAdmin LDAP queries

These examples require the LDAP client binaries on your local machine. For PPRD or DEV environments, respectively substitute hornet or owlfly for cricket below.

Lookup your nuid using your uid (replace xxxxxxxx with your uid)

ldapsearch -H ldap://cricket.nis.vt.edu:11389 -LLL -x ou=Administrators,ou=NIS,o=vt -s one uid=xxxxxxxx nuid

Retrieve the RADIUS client secret for a given IP address (replace xxxxxxxx with your nuid and 0.0.0.0 with IP address)

ldapsearch -H ldap://cricket.nis.vt.edu:11389 -ZZ -x -D nuid=xxxxxxxx,ou=Administrators,ou=NIS,o=vt -W -b ou=clients,ou=FreeRADIUS,ou=Configuration,ou=NIS,o=vt radiusClientIpAddr=0.0.0.0 radiusClientSecret

Retrieve the RADIUS client secret for a given shortname (replace xxxxxxxx with your nuid and yyyyyyyy with shortname)

ldapsearch -H ldap://cricket.nis.vt.edu:11389 -ZZ -x -D nuid=xxxxxxxx,ou=Administrators,ou=NIS,o=vt -W -b ou=clients,ou=FreeRADIUS,ou=Configuration,ou=NIS,o=vt radiusClientShortname=77777777 radiusClientSecret

These examples are executed on the LDAP provider (cricket, hornet, or owlfly)

Retrieve a network administrator account and save to an LDIF file

source /apps/etc/openldap/profile
ldapsearch -e \!authzid=dn:cn=Manager,o=vt -x -ZZ -H ldap://:11389 -LLL -W -D nuid=your_nuid,ou=Administrators,ou=NIS,o=vt -b ou=Administrators,ou=NIS,o=vt uid=uid_of_network_admin > ~/filename.ldif

Update a network administrator account from an LDIF file

source /apps/etc/openldap/profile
ldapmodify -e \!authzid=dn:cn=Manager,o=vt -x -ZZ -H ldap://:11389 -W -D nuid=your_nuid,ou=Administrators,ou=NIS,o=vt -f ~/filename.ldif

Network administrators should consider defining aliases for themselves on cricket, hornet, and owlfly to simplify the command syntax in the above two examples.

Managing Clearpass

(!! IMPORTANT) Conehead and Grub

Troubleshooting

Common Problems in OpenLDAP

Incorrect PIDs in Entitlements

A common problem for NEO-LDAP occurs when a user changes his/her PID in ED-LDAP, but the entitlements in NEO-LDAP do not get updated with the new PID. To resolve this problem, it's important to know three things:

1. The PID is stored in the uupid attribute in ED-LDAP, but in the uid attribute in NEO-LDAP.
2. The entitlements in ou=Entitlements,ou=NIS,o=vt are granted to entries in ou=People,ou=NIS,o=vt by setting the entitled attribute of the former to the dn of the latter, AND the entitledUID of the latter to the uupid from ED-LDAP.
3. The uid attribute of the ou=People,ou=NIS,o=vt entry is what is used by the user to authenticate to the network. Let's consider an example to see how we might resolve this kind of problem.

Example Problem:

1. Hokie Bird uses Account Manager to change his PID from hbird to hokieb.
2. The network account web service encounters an error and therefore:
   1. Fails to update the entitledUID attribute of nuid=1010101010,ou=Entitlements,ou=NIS,o=vt
   2. Fails to update the uid attribute of nuid=7777777,ou=People,ou=NIS,o=vt
   3. In rare cases, does only one of the above

Now the ED-LDAP entry looks like this:

dn: uid=12345678,ou=People,dc=vt,dc=edu
givenName: Hokie
sn: Bird
uid: 11111111
uidNumber: 11111111
virginiaTechID: 999999999
uupid: hokieb
mail: hokieb@vt.edu
mailPreferredAddress: hokieb@vt.edu
...

and the NEO-LDAP entries might look like this:

dn: nuid=1010101010,ou=Entitlements,ou=NIS,o=vt
nuid: 1010101010
entitled: nuid=7777777,ou=People,ou=NIS,o=vt
entitledUID: hbird
entitlement: cns.service.network.wireless
objectClass: nisEntitlement

and this:

dn: nuid=7777777,ou=People,ou=NIS,o=vt
nuid: 7777777
objectClass: nisUserAccount
objectClass: inetOrgPerson
prohibited: FALSE
userPassword:: eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
uid: hbird
sn: Bird
cn: Hokie Bird

or in rare situations like this:

dn: nuid=1010101010,ou=Entitlements,ou=NIS,o=vt
nuid: 1010101010
entitled: nuid=7777777,ou=People,ou=NIS,o=vt
entitledUID: hbird
entitlement: cns.service.network.wireless
objectClass: nisEntitlement

and this:

dn: nuid=7777777,ou=People,ou=NIS,o=vt
nuid: 7777777
objectClass: nisUserAccount
objectClass: inetOrgPerson
prohibited: FALSE
userPassword:: eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
uid: hokieb
sn: Bird
cn: Hokie Bird

There could also be corresponding cns.service.network.vpn entitlement entries in the two cases above, and these will have their own distinct nuid attributes. In either of the above cases, we can restore the intended entitlements to the user with the ldapmodify command. Network administrators can define and use a mgrldapmodify alias instead of ldapmodify below.

First case:

ldapmodify << EOF
dn: nuid=7777777,ou=People,ou=NIS,o=vt
changetype: modify
replace: uid
uid: hokieb

dn: nuid=1010101010,ou=People,ou=NIS,o=vt
changetype: modify
replace: entitledUID
entitledUID: hokieb
EOF

Second case:

ldapmodify << EOF
dn: nuid=1010101010,ou=People,ou=NIS,o=vt
changetype: modify
replace: entitledUID
entitledUID: hokieb
EOF

Design Considerations

FreeRADIUS w/ Local OpenLDAP

Each FreeRADIUS server connects to a local instance of OpenLDAP configured as a syncrepl consumer of the appropriate OpenLDAP provider for its tier. This allows FreeRADIUS to continue to authenticate and authorize users, even if the OpenLDAP provider node is unavailable. FreeRadius binds to the local directory using a local LDAP account in the Local organizational unit with a salted SHA-1 password.

dn: uid=freeradius,ou=Local,ou=NIS,o=vt
description: An account for freeradius to use to search the dir
cn: FreeRADIUS
uid: freeradius
userPassword: {ssha}REDACTREDACTREDACTREDACTREDACTREDACT1234
objectClass: radiusObjectProfile

Enhancements

Known Issues

Additional Resources