The NTC Mag

Python Objects

2020-02-25T00:00:00+00:00

“Everything in Python is an object” has been a common refrain among the Python community. What is an object? A simple definition is: An object is an instance of a class. For example, strings are instances of the str class.

# Creating a string variable
>>> device_type = "switch"
# device_type is an instance of the str class
>>> type(device_type)
<class 'str'>
>>>

Classes themselves are instances of type.

# Creating a class object normally
>>> class Cisco:
...     vendor = "cisco"
...
>>> Cisco
<class '__main__.Cisco'>
>>> Cisco.vendor
'cisco'
>>> isinstance(Cisco, type)
True
# Creating a class object explicitly from type
>>> Arista = type("Arista", (), {"vendor": "arista"})
>>> Arista
<class '__main__.Arista'>
>>> Arista.vendor
'arista'
>>>

In addition to being an instance of a class, objects consist of attributes defining state and behavior. It is common to refer to state attributes as data attributes or variables, and behaviors are called methods. The Python builtin classes generally have many methods, but do not have data attributes. The str class provides methods for:

Manipulating the string (capitalize, format, etc.)
Finding information about the string (startswith, isnumeric, etc.)
Segmenting the string (partition, split, etc.)

>>> example = "this is a string"
# Capitalize the first letter of each word 
>>> example.title()
'This Is A String'
# Count the number of times the letter i appears
>>> example.count("i")
3
# Convert the string into a list of words
>>> example.split()
["this", "is", "a", "string"]

It is more common for custom classes to have data attributes that store information about the objects they imitate. For an object representing a switch, some example data attributes might be: modules, uptime, and reachability. Some example methods might be: reboot, and ping.

>>> switch = Arista(hostname="nyc-hq-rt1")
# Example of a data attribute
>>> switch.uptime
'4 years, 3 days'
# Example of a method attribute
>>> ping_stats = switch.ping("10.1.1.1", 5)
'pinging 10.1.1.1 times 5'
>>> ping_stats
{
    'attempts': 5,
    'success': 5,
    'fail': 0,
    'messages': ['pinged 10.1.1.1 in 0.21ms', 'pinged 10.1.1.1 in 0.14ms', ...]
}
>>>

Data Attributes

Data attributes are distinguished between class variables and instance variables. Class variables assign values that are the same for all objects of the class. Instance variables assign values for an instance of the class. Class variables are generally used for metadata fields, such as vendor in the first example. Instance variables contain more interesting information that the object methods act upon. Using the previous example, the reachability attribute might be used in a report to identify all switches that are unreachable. The Cisco class above is redefined to create instance variables upon initialization for hostname, connection status, and reachability information.

class Cisco:
    # Class variable
    vendor = "cisco"

    def __init__(self, hostname):
        # Instance variables
        self.hostname = hostname
        self.connected = False
        ping_reachability = self.ping(hostname, 2)
        if ping_reachability["success"]:
            self.connect()
            if self.connected:
                self.reachability = "authorized"
            else:
                self.reachability = "unauthorized"
        else:
            self.reachability = "unreachable"

    def connect(self):
        ...
        self.connected = True if connected else False

    def ping(self, destination, count):
        ...
        return {
            "attempts": count,
            "success": success_count,
            "fail": fail_count,
            "messages": [message for message in ping_results],
        }

Python uses dictionaries to store class and instance variables. Class variables share the same dictionary across all instances of the class, and instance variables are stored in a unique dictionary per instance. The class dict is stored in <class>.__dict__, and this is referenced on an instance with <obj>.__class__.__dict__. The instance dict is stored in <obj>.__dict__. Here are some examples showing how these two dictionaries behave.

# View the class dict
>>> Cisco.__dict__
mappingproxy({'vendor': 'cisco', ...})
# Create two objects of the Cisco class
>>> rt1 = Cisco("nyc-hq-rt1")
>>> rt2 = Cisco("nyc-dc-rt1")
# Viewing the instance dict
>>> rt1.__dict__
{'hostname': 'nyc-hq-rt1', 'connected': True, 'reachability': 'authorized'}
# Reachability is an instance variable and can have different values across objects
>>> rt1.reachability
'authorized'
>>> rt2.reachability
'unreachable'
# The class dict is referenced within instances
>>> Cisco.__dict__ == rt1.__class__.__dict__
True
# Changing the vendor on the class is reflected on all instances
>>> Cisco.vendor = "CSCO"
>>> rt1.vendor
'CSCO'
>>> rt2.vendor
'CSCO'
>>>

Attribute Precedence

This distinction between class and instance variables is important for understanding Python’s attribute access behavior. The standard attribute lookup uses the following precedence until the attribute is found, or an Error is raised:

Attributes defined on the instance
Attributes defined on the class
Attributes defined on inherited classes

Practically, this means that overriding a class attribute on an instance will only affect the single instance and not other instances.

>>> rt1 = Cisco("nyc-hq-rt1")
>>> rt2 = Cisco("nyc-dc-rt1")
# rt1 and rt2 both have a value of "cisco" for the vendor attribute
>>> rt1.vendor == rt2.vendor == "cisco"
True
# rt1's instance dict does not have an attribute for `vendor`
>>> rt1.__dict__
{'hostname': 'nyc-hq-rt1', 'connected': True, 'reachability': 'authorized'}
>>> rt1.vendor = "Cisco"
>>> rt1.vendor
'Cisco'
# rt1's instance dict was updated with a `vendor` attribute
>>> rt1.__dict__
{'hostname': 'nyc-hq-rt1', 'connected': True, 'reachability': 'authorized', 'vendor': 'Cisco'}
# rt2's vendor attribute remains unchanged
>>> rt2.vendor
'cisco'

Mutable Attributes

The above demonstrates the built-in safety provided when naming instance variables collides with class variables. However, this behavior might have surprising results when class variables point to mutable objects (such as lists). The Arista class is rewritten below to have a class variable defining the modules, with the value using a list for modular chassis.

class Arista:
    vendor = "arista"
    modules = ["7500E-48S", "DCS-7500E-SUP"]

>>> rt1 = Arista("nyc-hq-rt1")
>>> rt2 = Arista("nyc-dc-rt1")
>>> rt1.modules
["7500E-48S", "DCS-7500E-SUP"]
# Appending to rt2's `mdoules` attribute affects rt1
>>> rt2.modules.append("7500E-72S")
>>> rt1.modules
["7500E-48S", "DCS-7500E-SUP", "7500E-72S"]
# Using the `+=` operator on rt2's `modules` attributed affects rt1
>>> rt2.modules += ["7500E-36Q"]
>>> rt1.modules
["7500E-48S", "DCS-7500E-SUP", "7500E-72S", "7500E-36Q"]
# Both modifications to rt2's `modules` attribute affect the class attribute
>>> Arista.modules
["7500E-48S", "DCS-7500E-SUP", "7500E-72S", "7500E-36Q"]
# All references to `modules` are the same shared object
>>> Arista.modules is rt1.modulues is rt2.modules
True
>>>

Conclusion

Understanding how objects work is critical to working with Python. Fundamental to working with objects is managing and using data attributes. Data attributes can be created on the class, or on each instance of the class. Instance level attributes have a higher precedence than class level attributes. Finally, when creating attributes on a class, it is important to consider how they will be used, and that mutable attributes can lead to unexpected behavior.

-Jacob

Backing up device configurations using Nornir and Ansible

2020-02-20T00:00:00+00:00

In this blog, I’m going to go over how to run a configuration backup on a Cisco IOS device using Ansible and Nornir. I’ve heard many people say, “well which tool should I use to run against my infrastructure?” That’s a great question but it depends :). There are many factors that could affect the decision of using a specific tool i.e skill level, time, access to tools, team preference…the list goes on.

Hopefully, the examples shown here can help you determine which tool best fits the need. I’m not going deep into the details on the different technologies or explaining all the components of each of the files, but I’ll provide some links to learn more about them if needed.

Installation

If you want to follow along with the blog, Nornir and Ansible need to be installed in the machine locally or you will need to use tools that isolate dependencies like Docker and virtual environments. I ended up using Docker so if you have Docker installed in your machine and know how to use it, then feel free to use the Dockerfile below:

FROM python:3.7-stretch

WORKDIR /ntc

RUN apt-get update \
&& apt-get install tree
RUN pip install black \
ansible \
nornir

After adding and saving the commands to the Dockerfile run the following commands to build the container image and start running the container.

docker build -t nornir .
docker run -it --name nornir_ansible -v ${PWD}:/ntc nornir_ansible:latest /bin/bash

Backing up with Nornir

Nornir is an automation tool used to interact with networking devices. This tool is a bit different compared to other tools in the way that you write your own Python code to control the automation. For example, Ansible is written in Python but uses its own DSL which you use to describe what you want to have done. Nornir does not need a DSL instead, you can write everything in Python.

Configuration File

Now that everything has been installed, the first file to create is the Nornir configuration file. This file is similar to the ansible.cfg file which helps set all the system defaults, but in this case it’s going to be used to find the inventory file. Click on the following links to learn more about Nornir and Ansible configuration files:

Create a file called nornir_config.yml and inside the file store the following content:

---
inventory:
    plugin: nornir.plugins.inventory.ansible.AnsibleInventory
    options:
        hostsfile: "./inventory"

Note: Since this container has access to the local machine, any tool or editors can be used with the same files the container is able to see. I’ll be using Visual Studio Code in my local machine.

Inside the container run the cat command to make sure changes took place in the file.

root@c2e446b364a8:/ntc# cat nornir_config.yml
---
inventory:
    plugin: nornir.plugins.inventory.ansible.AnsibleInventory
    options:
        hostsfile: "./inventory"

Note: The plugin used here AnsibleInventory allows us to source an inventory file that would normally be consumed by Ansible. There are other plugins that can be used like simple, NSOT and NetBox.

Inventory File

The inventory file built here will be used for both Ansible and Nornir. The main differences here will be the hostvars names. For now Nornir will be using hostname, platform, username and password. Later, when the Ansible section comes, up a few other variables will be added. Create a file called inventory and place it in the same directory path as the nornir_config.yml file. Save it and run the cat command to make sure it saved properly and it’s seen by the container.

root@c2e446b364a8:/ntc# cat inventory
[all]
csr1 hostname=csr1 platform=ios username=ntc_user password=pass123

Click on the following links to learn more about Nornir and Ansible inventory:

Backup Python Script

Now that the configuration and inventory file has been created it’s time to build the script that will backup the device configuration. The script will consist of functions that will build the necessary directory, backup files, and two different functions using the netmiko and napalm modules to backup the device configurations.

The first step to building the script is to import all the needed libraries. In this case, the libraries needed are:

os: This library is going to be used to help interact with the local environment and build the backup directory and file names for the configurations.
nornir: Nornir is an automation framework written in Python to be used with Python.
- InitNornir: Object used to provide the configuration file containing system defaults. In this case, it’s being used to provide the inventory file.
- netmiko_send_command: It’s a netmiko function used to send arbitrary commands to networking devices.
- napalm_get: Function that comes from the NAPALM library to get facts from the device. In this case, it will be used to retrieve the device configuration.

import os
import logging
from nornir import InitNornir
from nornir.plugins.tasks.networking import netmiko_send_command, napalm_get

Two global variables need to be created. The first one, BACKUP_DIR, is the directory name used to store the backed up files and the second,nr, is used to carry the data stored in the nornir_config.yml file. In this case, it will contain the inventory file data.

BACKUP_DIR = "backups/"

nr = InitNornir(config_file="./nornir_config.yml")

The create_backups_dir is used to create a local directory called backups and it’s using the os library to create a directory with the name stored in the BACKUP_DIR variable.

def create_backups_dir():
    if not os.path.exists(BACKUP_DIR):
        os.mkdir(BACKUP_DIR)

The save_config_to_file function is used to create the backed up device configuration files inside the backups directory. Notice the parameters used will take as input method, hostname and config, these parameters will get their data from the get_netmiko_backups() and get_napalm_backups() functions that will be created in the next few steps.

def save_config_to_file(method, hostname, config):
    filename =  f"{hostname}-{method}.cfg"
    with open(os.path.join(BACKUP_DIR, filename), "w") as f:
        f.write(config)

The get_netmiko_backups function will run the netmiko_send_command module to send a “show run” command to the network device stored in the nr variable and it will be stored in the backups_results variable.

def get_netmiko_backups():
    backup_results = nr.run(
        task=netmiko_send_command, 
        command_string="show run"
        )

    for hostname in backup_results:
        save_config_to_file(
            method="netmiko",
            hostname=hostname,
            config=backup_results[hostname][0].result,
        )

The backups_results variable will end up with a value that stores the hostname and configuration as a result looking something like this:

>>> print(backup_results)
AggregatedResult (netmiko_send_command): {'csr1': MultiResult: [Result: "netmiko_send_command"]}
>>>
>>>

Note: AggregatedResult is a dict-like object that aggregates the results for all devices. You can access each individual result by doing my_aggr_result["hostname_of_device"]

The data inside backups_results can be extracted by using a for loop and stored in hostname. The data iterated through with the for loop will return the hostname information and backup configuration.

The get_napalm_backups function will be built the same way, except this time it will be using the napalm_get module to extract the device configuration.

def get_napalm_backups():
    backup_results = nr.run(task=napalm_get, getters=["config"])

    for hostname in backup_results:
        config = backup_results[hostname][0].result["config"]["startup"]
        save_config_to_file(method="napalm", hostname=hostname, config=config)

The last function main() will execute all the functions and the code entry point is added to start the script when executed.

def main():
    create_backups_dir()
    get_netmiko_backups()
    get_napalm_backups()

if __name__ == "__main__":
    main()

Once the script has been built, execute the command black to clean up and make sure all formatting is correct and use the command cat to make sure everything has been saved correctly. Click on the following link to learn more about black.

root@c2e446b364a8:/ntc# black backups.py
All done! ✨ 🍰 ✨
1 file left unchanged.

root@6b91330f9674:/ntc# cat backups.py
import os
from nornir import InitNornir
from nornir.plugins.tasks.networking import netmiko_send_command, napalm_get

BACKUP_DIR = "backups/"

nr = InitNornir(config_file="./nornir_config.yml")


def create_backups_dir():
    if not os.path.exists(BACKUP_DIR):
        os.mkdir(BACKUP_DIR)


def save_config_to_file(method, hostname, config):
    filename =  f"{hostname}-{method}.cfg"
    with open(os.path.join(BACKUP_DIR, hostname), "w") as f:
        f.write(config)


def get_netmiko_backups():
    backup_results = nr.run(
        task=netmiko_send_command, 
        command_string="show run"
        )

    for hostname in backup_results:
        save_config_to_file(
            method="netmiko",
            hostname=hostname,
            config=backup_results[hostname][0].result,
        )


def get_napalm_backups():
    backup_results = nr.run(task=napalm_get, getters=["config"])

    for hostname in backup_results:
        config = backup_results[hostname][0].result["config"]["startup"]
        save_config_to_file(method="napalm", hostname=hostname, config=config)


def main():
    create_backups_dir()
    get_netmiko_backups()
    get_napalm_backups()


if __name__ == "__main__":
    main()

Running the Python Script

Now that the script is complete, it’s time to execute it by using the Python command and file name right next to it python backups.py.

root@c2e446b364a8:/ntc# python backups.py
root@c2e446b364a8:/ntc#

The tree command can be used to view the configuration files stored in the directory backups configuration files stored inside of it with the specified file names built in the save_config_to_file function.

root@c2e446b364a8:/ntc# tree
.
├── Dockerfile
├── backups
│   ├── csr1-napalm.cfg
│   └── csr1-netmiko.cfg
├── backups.py
├── inventory
├── nornir.log
└── nornir_config.yml

1 directory, 7 files

Note: Take a look at the backed up configuration files using the cat command or an editor of choice. They are left off from this as they take up a lot of screen space.

By default, Nornir automatically configures logging when InitNornir is called and a file is created locally that can be viewed what functions where ran that belong to the Nornir library.

root@c2e446b364a8:/ntc# cat nornir.log
2020-01-02 21:33:26,519 -  nornir.core -     INFO -        run() - Running task 'netmiko_send_command' with args {'command_string': 'show run'} on 1 hosts
2020-01-02 21:33:32,134 -  nornir.core -     INFO -        run() - Running task 'napalm_get' with args {'getters': ['config']} on 1 hosts

Backing up with Ansible

Ansible is a configuration management tool originally created to interact with Linux servers, eventually it started gaining traction in the networking industry to manage networking device configurations. Ansible was built using Python but it uses YAML to configure all the automation tasks. The idea is to simplify the work that it takes to start automating.

Configuration file

The ansible.cfg configuration file is used to set some of the system default values. Create the ansible.cfg file in the root of the directory next to all the other files created previously. Then use the cat command to make sure everything was stored correctly.

inventory: Sets the location of the inventory file, by default it will look for it in etc/ansible/hosts or while running the playbook the -i flag can be used to point to the location of the file. In this case, it is pointed to the inventory file we will create in the next step within the same directory.
host_key_checking: This has been disabled to prevent host key checking and prevent issues when running the playbook.
interpreter_python: Has been set in the ansible.cfg file to prevent a new warning added to the new version of Ansible and specify what interpreter to use.
gathering: Is set to smart to gather facts by default, but don’t regather if already gathered.

root@c2e446b364a8:/ntc# cat ansible.cfg
[defaults]
inventory      = ./inventory
host_key_checking = False
interpreter_python = /usr/bin/python
gathering = smart

Inventory

Edit the same inventory file that was used for Nornir, except this time add the following variables for Ansible.

ansible_user=ntc_user
ansible_password=pass123
ansible_network_os=ios

Again use the cat command to make sure the changes took effect.

root@c2e446b364a8:/ntc# cat inventory
[all]
csr1 hostname=csr1 platform=ios username=ntc_user password=pass123 ansible_user=ntc_user ansible_password=pass123 ansible_network_os=ios

Playbook

The first part to the Playbook is the play definition. The play definition will contain a list of keys such as:

name: Any arbitrary name for the Playbook.
hosts: The devices that will be targeted in the scope of the play.
connection: Type of connection. In this case network_cli which is basically SSH.
tasks: The key that contains all the automation steps that belong to the play.
gather_facts: Before Ansible version 2.9, this key was normally disabled when interacting with networking devices. Now, it has the ability to gather device facts at the play definition level rather than at the task level.

The second part of the Playbook nested under the tasks key will have a list of key:value pair tasks.

name: Optional but recommended key that can have any arbitrary name for the task.
ios_config: This is the Python module used to run configuration commands against ios devices.
backup: Module parameter used to backup the device configuration. By default, if the backup directory is not created then it will create one in the same location as the Ansible playbook and store the configuration files in the backup directory.

Note: A recent parameter was added as backup_options with sub-options to change the directory path rather than the default which is backup and the ability to change the filename of the device configuration. In this case, the default value is set with this format <hostname>_config.<current-date>@<current-time>

Create the Ansible Playbook and call it backup_playbook.yml. Use the cat command to make sure the content is stored correctly:

root@c2e446b364a8:/ntc# cat backup_playbook.yml
---
    - name: BACKUP PLAYBOOK
      hosts: csr1
      connection: network_cli
      gather_facts: yes
      tasks:

        - name: BACKUP USING IOS_CONFIG
          ios_config:
            backup: yes

Running the Playbook

To run the Ansible Playbook use the command ansible-playbook backup_playbook.yml. The output is shown below:

Note: There is an extra task that says Gathering Facts which is a new feature added to Ansible that now gathers facts from networking devices at the play definition level. Take a look at this blog Ansible gather_facts for Networking Devices to learn a little more about it.

root@c2e446b364a8:/ntc# ansible-playbook backup_playbook.yml

PLAY [BACKUP PLAYBOOK] **********************************************************************************

TASK [Gathering Facts] **********************************************************************************
[WARNING]: Ignoring timeout(10) for ios_facts

ok: [csr1]

TASK [BACKUP USING IOS_CONFIG] ***************************************************************************
changed: [csr1]

PLAY RECAP ***********************************************************************************************
csr1                 : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Use the tree command to view the content inside and notice how the device configuration is stored in the backup directory:

root@c2e446b364a8:/ntc# tree
.
├── Dockerfile
├── ansible.cfg
├── backup
│   └── csr1_config.2020-01-03@15:43:16
├── backup_playbook.yml
├── backups
│   ├── csr1-napalm.cfg
│   └── csr1-netmiko.cfg
├── backups.py
├── inventory
├── nornir.log
└── nornir_config.yml

2 directories, 10 files

Backing up with ios_facts

There’s another way of backing up the device configurations by using ios_facts, except this time rather than creating a new task to gather facts we can leverage the gathering of facts that takes place at the play definition level and add a task that will create a new directory to store the device configurations and another task to copy the content from ansible_facts as a configuration file.

---
    - name: BACKUP PLAYBOOK
      hosts: csr1
      connection: network_cli
      gather_facts: yes
      tasks:

        - name: BACKUP USING IOS_CONFIG
          ios_config:
            backup: yes
            
        - name: CREATE BACKUP DIRECTORY
          file:
            path: ./ansible_backup
            state: directory
       
        - name: BACKUP USING IOS_FACTS
          copy:
            content: "{{ ansible_net_config }}"
            dest: ./ansible_backup/{{ inventory_hostname }}_config.{{ now() }}.cfg

In the playbook above, the copy module is used to copy content stored in the ansible_net_config key that is found inside ansible_facts. Any net_* key under ansible_facts can be accessed directly by using the ansible_net_* key rather than accessing the nested data structure. Now the new tasks have been added the playbook can be run again.

root@c2e446b364a8:/ntc# ansible-playbook backup_playbook.yml

PLAY [BACKUP PLAYBOOK] **********************************************************************************

TASK [Gathering Facts] **********************************************************************************
[WARNING]: Ignoring timeout(10) for ios_facts

ok: [csr1]

TASK [BACKUP USING IOS_CONFIG] ***************************************************************************
changed: [csr1]

TASK [CREATE BACKUP DIRECTORY] ****************************************************************************
changed: [csr1]

TASK [BACKUP USING IOS_FACTS] *****************************************************************************
changed: [csr1]

PLAY RECAP *************************************************************************************************
csr1            : ok=4    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Notice how every time the first task is run a new backup configuration is created since it’s been run with a different time stamp. The same can be done with the ios_facts backup method.

root@c2e446b364a8:/ntc# tree
.
├── Dockerfile
├── ansible.cfg
├── ansible_backup
│   └── csr1_config.2020-01-03 19:01:43.277008.cfg
├── backup
│   ├── csr1_config.2020-01-03@15:43:16
│   └── csr1_config.2020-01-03@19:01:42
├── backup_playbook.yml
├── backups
│   ├── csr1-napalm.cfg
│   └── csr1-netmiko.cfg
├── backups.py
├── csr1_config.2020-01-03@15:44:07
├── inventory
├── nornir.log
└── nornir_config.yml

3 directories, 13 files

Summary

There are many automation tools and modules that can help automate the backup of device configurations. The ones shown in this blog are some of the basic ways to do it, which ever method you choose will do the job. There can be different modifications that can be done to the Nornir script or even the Ansible playbook to better fit your application and hopefully what I have shown here can give you a start on what tool to use.

-Hector

My first REST API work with Python

2020-02-11T00:00:00+00:00

I come from a network engineering background, so when I started my network automation journey, it felt like I was banging my head against new technology with very little help. When I first started using APIs, our network team had just started with this brand new ACI “thing,” and I couldn’t find a well written “What is an API” blog with code I would call “appropriate.” In this blog, I will attempt to provide that for those of you that are just getting started with APIs. If you have been using APIs for a while, this may not be as useful.

One of the hardest parts of writing this was choosing an API to discuss. Some of the things I really wanted from an API were:

A Swagger page: I always thought of this as a webpage that has the API laid out nicely and gives the opportunity to run the API calls directly on it. This is more technically known as an Open API spec.
A platform that was common in the networking world that people would be likely to run across.
A platform that was easily available and didn’t require you to build a new system.

Sadly, this combo doesn’t seem to exist. In the end, I decided to go over 2 APIs:

NetBox
- “An open source web application designed to help manage and document computer networks.”
- Documentation that is easy for a beginner with its Swagger page.
- https://netbox.readthedocs.io/en/stable/
ACI
- A Network policy-based automation model for data centers.
- Cisco DevNet has an always on sandbox where you can access a working ACI environment at any time.
- When I was working on this API about a year ago, I found the documentation hard to navigate. I can see how it would work well for someone with more experience, but it didn’t work well for me at the time, and I haven’t kept up to date to know if their documentation has gotten more friendly.
- https://learningnetwork.cisco.com/docs/DOC-32331

REST API Core Concepts

While I’m not going to go to in depth, I want to walk through a few examples to give you the general idea of how a REST API works.

There are two main methods of authentication. One where the API is provided a username and password and the system gives back a temporary token. This process is often used in systems where you are running a system such as ACI. In the other method, a token you generate from a UI serves as your authentication. These are often used for web applications such as Slack, Twitter, Facebook, and fortunately NetBox, so we can see how to do that.

The general idea behind the REST API is you are going to an URL rather than typing in commands. Let’s look at this like a Cisco CLI. For example, your command might be show run interface gi1. The equivalent URL might be something like device.com/api/conf/interface/gi1 so to work with gi2 the URL would just change device.com/api/conf/interface/gi2.

The main types of calls are below.

GET
- Pulling data.
- You can’t really cause an issue doing this as long as you don’t pull too much at one time, e.g. pulling 1,000,000 records may bog down the system.
- Equivalent of a show command.
PUT/POST/PATCH
- Changing or creating data.
- Equivalent of configuration changes.
DELETE
- Delete data.

With the same examples from above, we will do a GET on device.com/api/conf/interface/gi1. It returns the dictionary below:

{
    description: "Disconnected"
    status: "shut"
    mode: "access"
    vlans: 110
}

We now know the description on the interface, the VLAN, and that it is shut down. Let’s say we wanted to make a change, just update it as below and PUT it to the same URL.

{
    description: "VM Server Bob"
    status: "no shut"
    mode: "trunk"
    vlans: "10-100"
}

The port is now up, with a new description, as a trunk allowing VLANs 10-100 though. While not quite as easy to understand the url above this URL https://device.com/restconf/data/Cisco-IOS-XE-native:native/interface=GigabitEthernet/1 would work for the RESTCONF API for IOS-XE devices and has an easy to read URL.

NetBox

I am running an instance of NetBox in my lab, the IP is 192.168.0.159, so that’s what all my examples will be of. It’s not hardened or using https, so all of my examples are using http on port 8000. Your box may vary, adjust accordingly.

First, let’s build and grab a token from http://192.168.0.159:8000/user/api-tokens/.

Next, we’ll look at the swagger documentation page at http://192.168.0.159:8000/api/docs/. Login to the session

and find an action to do, (e.g. get a device from DCIM,) select that, then select “Try it out.”

We can pull all devices if we were to look at another option, but the API call gives us lots of options. A screenshot would be overly busy, so I am just going to pull one device in this example. Please note the required tag.

The API call gives us all the data for that we asked for. Please pay special note to the cURL script. That is the call it made to the NetBox API. It sent a GET request to http://192.168.0.159:8000/api/dcim/devices/5/ please note the initial line we chose was /dcim/devices/{id}/

The cURL script is running an HTTP request to the URL listed, and you could run the same request on nearly all systems. For example, right now you can run “curl http://www.google.com” and make the HTTP call to Google. One of the really nice things about that cURL script is we can go to https://curl.trillworks.com, paste in the cURL script, and it will give us the Python equivalent. NOTE that if the request has the token in it you should be careful. Swap out the token before posting it to the website unless you like giving admin credentials to strangers.

Honestly there’s not much to be scared of here. Just working though Swagger and putting a few functions together from what you get from the website. Super easy… Now let’s look at ACI.

ACI

In this example, the Cisco DevNet ACI emulator will be used. Set the username, password, and the base URL. This URL will be in every call made. To compare to the NetBox example, the base_url would have been http://192.168.0.159:8000/api.

username = "admin"
password = "ciscopsdt"
base_url = "https://sandboxapicdc.cisco.com/api/"

The login script is honestly one of the most difficult parts of the whole process, as documentation is often sparse and there is no industry standard for field names across REST APIs. The process involves simply logging in, getting a token, and returning it. This is how many APIs work, and there are generally small differences for every API. Most of this was from sample code obtained from DevNet a few years ago.

def get_token(username, password, base_url):
    # Disable warning messages that come with using self signed certs
    requests.packages.urllib3.disable_warnings()
    # create credentials structure
    name_pwd = {"aaaUser": {"attributes": {"name": username, "pwd": password}}}
    json_credentials = json.dumps(name_pwd)
    # Create the full URL we need in order to login
    # You'll find we do this a lot
    login_url = base_url + "aaaLogin.json"
    # log in to API
    # requests is a library that can do HTTP transactions, in this case we POSTing up (Telling it) our
    # credentials to get the token back
    # Verify=False ignores self signed certs, but would kick off errors if we hadn't disabled them up above
    post_response = requests.post(login_url, data=json_credentials, verify=False)
    # getting the token from login response structure
    auth = json.loads(post_response.text)
    login_attributes = auth["imdata"][0]["aaaLogin"]["attributes"]
    auth_token = login_attributes["token"]
    # create token dictionary that we will use for authentication from here on out for auth cookie
    cookies = {}
    cookies["APIC-Cookie"] = auth_token
    return cookies

Let’s review the documentation for the ACI API. Once again, this is how it worked back in late 2018, the documentation may have changed since. Access the API inspector,

then proceed to the interesting area in the GUI, paying close attention to your last click. The example will return with the switch information, so I went to Fabric (top tap) -> Inventory -> POD1 (expanded). Sadly, the URL alone doesn't always give you exactly what you need. In this example I had to cut off the &subscription=yes` section. Often, determinng the URL can be the hardest part.

You can also see that the URL is asking for JSON in the pod-1.json? section. For ACI, it will return either XML or JSON. Personally, I find JSON easier to deal with in Python. Using the JSON library it’s not hard to convert JSON to a dictionary; https://www.w3schools.com/python/python_json.asp does a better job at explaining the details. Let’s take a look at this process.

import json
import requests
from pprint import pprint

def pull_pod1(cookies, base_url):
    sensor_url = (
        base_url + '/node/mo/topology/pod-1.json?query-target=children&target-subtree-class=fabricNode&query-target-filter=and(not(wcard(fabricNode.dn,%22__ui_%22)),and(ne(fabricNode.role,"controller")))'
    )
    get_response = requests.get(sensor_url, cookies=cookies, verify=False)
    return get_response.json()


username = "admin"
password = "ciscopsdt"
base_url = "https://sandboxapicdc.cisco.com/api/"
## Calling the authentication function above to get the auth cookie
cookies = get_token(username, password, base_url)
pprint(pull_pod1(cookies, base_url))

This gives us all the devices in POD1. I’ll just be putting in one device for the sake of space

{
  "imdata": [
    {
      "fabricNode": {
        "attributes": {
          "adSt": "on",
          "address": "10.0.112.64",
          "annotation": "",
          "apicType": "apic",
          "childAction": "",
          "delayedHeartbeat": "no",
          "dn": "topology/pod-1/node-101",
          "extMngdBy": "",
          "fabricSt": "active",
          "id": "101",
          "lastStateModTs": "2019-12-12T14:22:14.238+00:00",
          "lcOwn": "local",
          "modTs": "2019-12-12T14:23:02.315+00:00",
          "model": "N9K-C9396PX",
          "monPolDn": "uni/fabric/monfab-default",
          "name": "leaf-1",
          "nameAlias": "",
          "nodeType": "unspecified",
          "role": "leaf",
          "serial": "TEP-1-101",
          "status": "",
          "uid": "0",
          "vendor": "Cisco Systems, Inc",
          "version": ""
        }
      }
    }
  ]
}

Note: Output omitted and modified for the sake of brevity and clarity.

You can observe several key pieces of information about this switch. The DN info, ID, model number, serial number, etc. With ACI, I often had to pull one piece of data so that I could understand how to pull another piece of data. Finding the DN was almost always an important step in my programs.

There is an easier, albeit less stable way (based on my observations) to find the URLs we want. Right-click on an interesting object and select “Open in Object Store Browser.” That will direct to a login screen. After login, the UI brings up a page where it should display the query and give the output.

To see the URL it used we click on “Show URL and response of last query.”

You can navigate the API structure and find what you are looking for. It may take some practice to develop a full understanding of this method.

I hope this has been helpful. I know it can be overwhelming, but the benefits of working with an API vs clicking though a web-page are wonderful. For example, before I left my last job I wrote a Python program that could be run on demand. It pulled what VLANS went to what VM Servers and looked for errors on interfaces. We got a call that there were lots of outages going on. The other network engineers were manually logging into everything looking for issues, but I just ran my program and was able to find a missing VLAN on a trunk to a VM server. The VM failed because it could no longer communicate out-side of the VM server. It took approximately five minutes to find and fix the issue. With dozens of VM servers and very little info on what the actual issue was, I believe that outage would have been much longer if people were looking for the issue by hand.

-Daniel

Using NTC Templates in Ansible

2020-02-04T00:00:00+00:00

Network to Code has a repository created for maintaining templates that help to convert unstructured output from network devices into structured data. Google created TextFSM that provides a custom DSL to take command line output and return as structured data. Examples range from getting an output from a show lldp neighbor command and returning the list of neighbors to getting a list of addresses in an ARP table. This post is going to dive deeper into how to use these to get the data with Ansible.

You can find more about the ntc-templates on our Github page. These are templates that have been created by both Network to Code and by the community. It is an open source project that anyone is able to contribute a template to if there is not a template already created. Take a look at the readme for how you can help contribute to NTC Templates!

Other Methods

Ansible is not the only method for using TextFSM to get structured data returned from network device output. Several Python libraries such as Netmiko and Nornir are able to use TextFSM to return structured data by setting a flag to use TextFSM as well.

Using NTC-Templates with Ansible

There are two primary methods for sending data through a TextFSM parser in Ansible. You will get to see examples for both of these. The first method available is to use the TextFSM Filter. The second is in conjunction with the Ansible Galaxy role Network Engine.

Getting Started

The first step towards leveraging these templates is getting the templates on to a compute device that is running Ansible. For this demo, we’ll be using a local machine and Git to clone the repository to a local directory.

git clone git@github.com:networktocode/ntc-templates.git

Next you will need to install TextFSM to the same Python interpreter that your Ansible installation is installed in. This demo will leverage Python3. For the installation you will also use the --user flag to install as the local user and --upgrade to ensure that the latest version of TextFSM is installed.

pip3 install textfsm --user --upgrade

For the second method of using Ansible Galaxy’s Network Engine you will also need to install the role as follows:

ansible-galaxy install ansible-network.network-engine

Now that the environment is setup it’s time to take a look at the Playbook and the execution.

Playbook Setup and Execution

This lab is setup as a 3 router traingle. RTR-1 will connect to one interface on RTR-2 and RTR-3. RTR-2 will have one connection to RTR-1 and RTR-3. RTR-3 will have the corresponding connections to RTR-1 and RTR-2.

TextFSM CLI Parser - Ansible Built In

---
- name: "PLAY 1: DEMO OF TEXTFSM"
  hosts: routers
  connection: network_cli
  gather_facts: no
  tasks:
    - name: "TASK 1: GET COMMAND OUTPUT"
      ios_command:
        commands:
          - show lldp neighbors
      register: lldp_output

    - name: "TASK 2: REGISTER OUTPUT TO DEVICE_NEIGHBORS VARIABLE"
      set_fact:
        device_neighbors: "{{ lldp_output.stdout[0] | parse_cli_textfsm('~/ntc-templates/templates/cisco_ios_show_lldp_neighbors.textfsm') }}"

    - name: "TASK 3: PRINT OUTPUT"
      debug:
        msg: "{{ device_neighbors }}"

    - name: "TASK 4: PRINT NEIGHBORS"
      debug:
        msg: "{{ item['LOCAL_INTERFACE'] }}: {{ item['NEIGHBOR'] }}"
      loop: "{{ device_neighbors }}"
      loop_control:
        label: "{{ item['LOCAL_INTERFACE'] }}"

Walking through this first play, in Task 1 you are connecting to the device and running the command show lldp neighbors. This is saved to a variable named lldp_output that will be used later.

In Task 2 the output is being sent through the parse_cli_textfsm filter. The filter is being provided the library file to be used in the parsing. This is explicitely called out. The output from going through the TextFSM parser is then getting assigned to the variable device_neighbors, with each host in the execution having their own local instance of this variable.

In Task 3 you are getting to see the output of the variable. This shows the structured data. With this particular template you get the keys of CAPABILITIES, LOCAL_INTERFACES, NEIGHBOR, and NEIGHBOR_INTERFACE back. These are all defined within the TextFSM template.

In Task 4 you get to see a practical output to a screen if you want to audit and understand the neighbor relationships seen by LLDP on the network devices.

In this example you need to set a fact to be able to access the data later or continue to send the output through the parse_cli_textfsm filter every time you want to get at the structured data, such as a line number 15 in TASK 2.

TextFSM CLI Parser Output

Here is the output that corresponds with the play above.

ansible-playbook demo_ansible_textfsm.yml

PLAY [PLAY 1: DEMO OF TEXTFSM WITH CLI PARSER] *************************************************************************

TASK [TASK 1: GET COMMAND OUTPUT] **************************************************************************************
ok: [rtr-2]
ok: [rtr-3]
ok: [rtr-1]

TASK [TASK 2: REGISTER OUTPUT TO DEVICE_NEIGHBORS VARIABLE] ************************************************************
ok: [rtr-1]
ok: [rtr-2]
ok: [rtr-3]

TASK [TASK 3: PRINT OUTPUT] ********************************************************************************************
ok: [rtr-2] => {
    "msg": [
        {
            "CAPABILITIES": "R",
            "LOCAL_INTERFACE": "Gi0/0",
            "NEIGHBOR": "rtr-3",
            "NEIGHBOR_INTERFACE": "Gi0/0"
        },
        {
            "CAPABILITIES": "R",
            "LOCAL_INTERFACE": "Gi0/1",
            "NEIGHBOR": "rtr-1",
            "NEIGHBOR_INTERFACE": "Gi0/1"
        }
    ]
}
ok: [rtr-1] => {
    "msg": [
        {
            "CAPABILITIES": "R",
            "LOCAL_INTERFACE": "Gi0/2",
            "NEIGHBOR": "rtr-3",
            "NEIGHBOR_INTERFACE": "Gi0/1"
        },
        {
            "CAPABILITIES": "R",
            "LOCAL_INTERFACE": "Gi0/1",
            "NEIGHBOR": "rtr-2",
            "NEIGHBOR_INTERFACE": "Gi0/1"
        }
    ]
}
ok: [rtr-3] => {
    "msg": [
        {
            "CAPABILITIES": "R",
            "LOCAL_INTERFACE": "Gi0/1",
            "NEIGHBOR": "rtr-1",
            "NEIGHBOR_INTERFACE": "Gi0/2"
        },
        {
            "CAPABILITIES": "R",
            "LOCAL_INTERFACE": "Gi0/0",
            "NEIGHBOR": "rtr-2",
            "NEIGHBOR_INTERFACE": "Gi0/0"
        }
    ]
}

TASK [TASK 4: PRINT NEIGHBORS FROM CLI PARSER] *************************************************************************
ok: [rtr-1] => (item=Gi0/2) => {
    "msg": "Gi0/2: rtr-3"
}
ok: [rtr-1] => (item=Gi0/1) => {
    "msg": "Gi0/1: rtr-2"
}
ok: [rtr-2] => (item=Gi0/0) => {
    "msg": "Gi0/0: rtr-3"
}
ok: [rtr-2] => (item=Gi0/1) => {
    "msg": "Gi0/1: rtr-1"
}
ok: [rtr-3] => (item=Gi0/1) => {
    "msg": "Gi0/1: rtr-1"
}
ok: [rtr-3] => (item=Gi0/0) => {
    "msg": "Gi0/0: rtr-2"
}

PLAY RECAP *************************************************************************************************************
rtr-1                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
rtr-2                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
rtr-3                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Ansible Role - Network Engine

The Ansible Role - Network Engine is a role that extracts information about network devices into Ansible Facts. You can use either TextFSM syntax or YAML (command_parser option) to extract information about a network device from the command output. You can read more about using the role at the network-engine Github page.

Here you will see a second method to get the same output with Network Engine instead of the CLI Parser. The biggest difference is that in this method, the Network Engine role will register the returned data to the ansible_facts instead of requiring you to set it to your own variable.

---
- name: "PLAY 2: DEMO OF TEXTFSM WITH NETWORK ENGINE"
  hosts: routers
  connection: network_cli
  gather_facts: no
  roles:
    - ansible-network.network-engine
  tasks:
    - name: "TASK 1: GET COMMAND OUTPUT"
      ios_command:
        commands:
          - show lldp neighbors
      register: lldp_output

    - name: "TASK 2: RUN THROUGH THE PARSER"
      textfsm_parser:
        file: "/Users/ntcblog/ntc-templates/templates/cisco_ios_show_lldp_neighbors.textfsm"
        content: "{{ lldp_output.stdout[0] }}"
        name: lldp_output

    - name: "TASK 3: SHOW ANSIBLE FACTS OUTPUT"
      debug:
        msg: "{{ ansible_facts }}"

    - name: "TASK 4: PRINT NEIGHBORS FROM ANSIBLE NETWORK ENGINE"
      debug:
        msg: "{{ item['LOCAL_INTERFACE'] }}: {{ item['NEIGHBOR'] }}"
      loop: "{{ ansible_facts['lldp_output'] }}"
      loop_control:
        label: "{{ item['LOCAL_INTERFACE'] }}"

There is a new key that we needed to add to import the role that was installed with the ansible-galaxy command earlier. This is the roles: key that you see under gather_facts and contains a list of roles to import into the play. Here you see the import of the ansible-network.network-engine role.

In Task 1 you once again have the same command to gather the LLDP neighbors from the device.

In Task 2 instead of registering to a fact and sending through a filter, you now use the module textfsm_parser that takes file as a parameter that is the file of the TextFSM template. This is the same template referenced in the filter on the first play. You also pass content of what output you want to send through the parser. The last parameter is the name of the fact that you are registering to ansible_facts.

In Task 3, you once again get to see the output for ansible_facts. This will show that there are more details available about the device.

In Task 4 you get to see the same output as Play 1 where you get the neighbors printed out.

Network Engine Output

ansible-playbook demo_ansible_textfsm.yml

PLAY [PLAY 2: DEMO OF TEXTFSM WITH NETWORK ENGINE] *********************************************************************

TASK [TASK 1: GET COMMAND OUTPUT] **************************************************************************************
ok: [rtr-2]
ok: [rtr-3]
ok: [rtr-1]

TASK [TASK 2: RUN THROUGH THE PARSER] **********************************************************************************
ok: [rtr-1]
ok: [rtr-2]
ok: [rtr-3]

TASK [TASK 3: SHOW ANSIBLE FACTS OUTPUT] *******************************************************************************
ok: [rtr-1] => {
    "msg": {
        "discovered_interpreter_python": "/usr/bin/python",
        "lldp_output": [
            {
                "CAPABILITIES": "R",
                "LOCAL_INTERFACE": "Gi0/2",
                "NEIGHBOR": "rtr-3",
                "NEIGHBOR_INTERFACE": "Gi0/1"
            },
            {
                "CAPABILITIES": "R",
                "LOCAL_INTERFACE": "Gi0/1",
                "NEIGHBOR": "rtr-2",
                "NEIGHBOR_INTERFACE": "Gi0/1"
            }
        ]
    }
}
ok: [rtr-2] => {
    "msg": {
        "discovered_interpreter_python": "/usr/bin/python",
        "lldp_output": [
            {
                "CAPABILITIES": "R",
                "LOCAL_INTERFACE": "Gi0/0",
                "NEIGHBOR": "rtr-3",
                "NEIGHBOR_INTERFACE": "Gi0/0"
            },
            {
                "CAPABILITIES": "R",
                "LOCAL_INTERFACE": "Gi0/1",
                "NEIGHBOR": "rtr-1",
                "NEIGHBOR_INTERFACE": "Gi0/1"
            }
        ]
    }
}
ok: [rtr-3] => {
    "msg": {
        "discovered_interpreter_python": "/usr/bin/python",
        "lldp_output": [
            {
                "CAPABILITIES": "R",
                "LOCAL_INTERFACE": "Gi0/1",
                "NEIGHBOR": "rtr-1",
                "NEIGHBOR_INTERFACE": "Gi0/2"
            },
            {
                "CAPABILITIES": "R",
                "LOCAL_INTERFACE": "Gi0/0",
                "NEIGHBOR": "rtr-2",
                "NEIGHBOR_INTERFACE": "Gi0/0"
            }
        ]
    }
}

TASK [TASK 4: PRINT NEIGHBORS FROM ANSIBLE NETWORK ENGINE] *************************************************************
ok: [rtr-1] => (item=Gi0/2) => {
    "msg": "Gi0/2: rtr-3"
}
ok: [rtr-1] => (item=Gi0/1) => {
    "msg": "Gi0/1: rtr-2"
}
ok: [rtr-2] => (item=Gi0/0) => {
    "msg": "Gi0/0: rtr-3"
}
ok: [rtr-3] => (item=Gi0/1) => {
    "msg": "Gi0/1: rtr-1"
}
ok: [rtr-2] => (item=Gi0/1) => {
    "msg": "Gi0/1: rtr-1"
}
ok: [rtr-3] => (item=Gi0/0) => {
    "msg": "Gi0/0: rtr-2"
}

PLAY RECAP *************************************************************************************************************
rtr-1                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
rtr-2                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
rtr-3                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Summary

This shows that there are multiple ways of pairing TextFSM templates with Ansible. Getting structured data out of unstructured output is extremely valuable when it comes to automating a network environment. There are over 300 different templates currently in the NTC-Templates repository to help get structured data out of your unstructured data.

-Josh

Versionable Database

2020-01-28T00:00:00+00:00

One of the most important components of a network automation strategy is how to manage your data and/or Source of Truth (SoT). In networking, the data elements are often massive. As an example, most Fortune 500 type companies will have to manage a few hundred thousand to potentially millions of switchports, each with several data points (VLAN, MTU, IP, etc) per interface. The scale and criticality of this data present several challenges. The intention of this blog is describe the need for a versionable database in network automation and serve as an primer to a versionable database.

Problem Space

When managing and storing your Source of Truth data, there are two primary mechanisms, each with it’s own pros and cons.

Source Control (Git)

Provides the ability to version data in such a way the exact state of the data at any point in the past is known.
Provides the ability to know who is the owner of the data.
Provides the ability to populate data in a staging area, without modifying the production data.
Tooling integration with things such as CI Systems.
Decentralized storage of the data.

Database

Provides ACID-based transactions of the data.
Has a native querying language to obtain, filter, and all around work with the data.
Provides schema enforcement of the data.
The ability to scale to large datasets.

There is a clear dichotomy between these two choices. On one hand there is great integrations with all standard DevOps tooling, on the other hand there is an enterprise-grade manager of the data.

History

For several years, a few of us have been searching for solutions that combine these two concepts. If one could build a database with Git constructs, this would allow versionable management of your data, with the ability to query, effectively store, and manage at scale. Through various searches it would seem that this has not been solved in any meaningful way. The closest I have come across was a project called NOMS, but it has limitations.

Recently a startup company called liquidata was founded and is creating a solution–a library called Dolt. Dolt is is written on top of the NOMS project, in Go. The intention is to support all Git semantics (such as branch, add, commit, etc..), as well as all MySQL semantics (such as insert, create, update, etc…).

By supporting all MySQL semantics, applications that use a MySQL server should be able to simply use a Dolt-SQL server with no disruption. Whether you are using an ODBC driver or raw SQL, in theory it should still work. You would naturally have to build in the capability for your application to take advantage of the Git capabilities or use traditional “Git-like” workflows.

By supporting all Git semantics, data should be able to be managed in a decentralized fashion using Git workflows to branch, add, diff, and merge the data. In a similar vein to GitHub, they have developed DoltHub, to provide that level of tooling expected in a standard Git user interface, such as forking, API’s, webhooks, and CI integrations.

Primer

In this brief introduction to the technology, we will create a Dolt data repository, called “simple-inventory” and run it on a dolt-sql server. If you care to follow along the only requirements are to have Docker and internet access.

Setup

Start the Docker container.

docker run -it --rm --name dolt golang:1.12.14

Install Dolt by running curl -L https://github.com/liquidata-inc/dolt/releases/download/v0.12.0/install.sh | bash

root@2a29f8a34dd3:/go# curl -L https://github.com/liquidata-inc/dolt/releases/download/v0.12.0/install.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   601    0   601    0     0   2647      0 --:--:-- --:--:-- --:--:--  2647
100  3038  100  3038    0     0   8392      0 --:--:-- --:--:-- --:--:--  8392
Downloading: https://github.com/liquidata-inc/dolt/releases/download/v0.12.0/dolt-linux-amd64.tar.gz
Installing dolt, git-dolt and git-dolt-smudge to /usr/local/bin.
root@2a29f8a34dd3:/go#

You can create a Dolt data repository by creating a folder, and then initializing the repository. Just like Git, in Dolt, you need to add yourself to the Dolt config. If you are familiar with Git, these commands will be familiar, only changing the command from git to dolt, but keeping all of the other options the same.

root@2a29f8a34dd3:/go# mkdir simple-inventory
root@2a29f8a34dd3:/go# cd simple-inventory
root@2a29f8a34dd3:/go/simple-inventory# dolt config --global --add user.email "ken@celenza.org"
Config successfully updated.
root@2a29f8a34dd3:/go/simple-inventory# dolt config --global --add user.name "itdependsnetworks"
Config successfully updated.
root@2a29f8a34dd3:/go/simple-inventory# dolt init
Successfully initialized dolt data repository.
root@2a29f8a34dd3:/go/simple-inventory# dolt status
On branch master
nothing to commit, working tree clean
root@2a29f8a34dd3:/go/simple-inventory# dolt log
commit hc8gq0434ckkjeo36rccn55mo14gk87q
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:33 +0000 2020

	Initialize data repository

root@2a29f8a34dd3:/go/simple-inventory#

Creating Tables

Ideally, if you are familiar with MySQL the only difference should be using Dolt engine instead of a MySQL server, everything should be the same. To enter the Dolt SQL server, simply use the command dolt sql.

root@2a29f8a34dd3:/go/simple-inventory# dolt sql
# Welcome to the DoltSQL shell.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit.
doltsql>

Create tables as you normally would, in this example, we will create two tables. The table device_inventory will have the inventory of the devices, with columns for hostname and IP address. The table vlan will have the hostname (with foreign key relationaship to the device_inventory table) column, VLAN , and name of the VLAN.

Note: Though the foreign key relationaship syntax is defined, it is not a currently supported Dolt feature.

doltsql> CREATE TABLE device_inventory (
      ->     hostname varchar(32) NOT NULL,
      ->     ip_address varchar(15) NOT NULL,
      ->     primary key (`hostname`)
      -> );
doltsql> CREATE TABLE vlan (
      ->     hostname varchar(32),
      ->     vlan int NOT NULL,
      ->     name varchar(32) NOT NULL,
      ->     PRIMARY KEY (hostname, vlan),
      ->     FOREIGN KEY (hostname) REFERENCES device_inventory(hostname)
      -> );
doltsql> exit

Here is where it starts to get interesting–we can combine these two different concepts and see how they interact. So far, the data has not been committed into master, only staged into our local environment. We can view this by issuing standard Git-like commands when we return back to the command line from the dolt-sql server.

Run the dolt status, dolt diff, and dolt diff -q commands.

root@2a29f8a34dd3:/go/simple-inventory# dolt status
On branch master
Untracked files:
  (use "dolt add <table>" to include in what will be committed)
	new table:      device_inventory
	new table:      vlan
root@2a29f8a34dd3:/go/simple-inventory# dolt diff
diff --dolt a/device_inventory b/device_inventory
added table
diff --dolt a/vlan b/vlan
added table
root@2a29f8a34dd3:/go/simple-inventory# dolt diff -q
CREATE TABLE `device_inventory` (
  `hostname` LONGTEXT NOT NULL COMMENT 'tag:0',
  `ip_address` LONGTEXT NOT NULL COMMENT 'tag:1',
  PRIMARY KEY (`hostname`)
);
CREATE TABLE `vlan` (
  `hostname` LONGTEXT NOT NULL COMMENT 'tag:0',
  `vlan` BIGINT NOT NULL COMMENT 'tag:1',
  `name` LONGTEXT NOT NULL COMMENT 'tag:2',
  PRIMARY KEY (`hostname`,`vlan`)
);
root@2a29f8a34dd3:/go/simple-inventory#

The dolt status command shows that the schema has not “taken effect” by being merged into master. The dolt diff with optional -q flag, shows that a new table has been created. Note the additional tag parameters, this is logic to track column names, which may change over time and cause name conflicts. The new schema can be committed into master.

Issue the commands dolt add -a, dolt commit -m 'initial schema', and dolt log.

root@2a29f8a34dd3:/go/simple-inventory# dolt add -a
root@2a29f8a34dd3:/go/simple-inventory# dolt commit -m 'initial schema'
commit 4n29eaqb1v9pmnpnda2r43fc91p1vp9l
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:41 +0000 2020

	initial schema

root@2a29f8a34dd3:/go/simple-inventory# dolt log
commit 4n29eaqb1v9pmnpnda2r43fc91p1vp9l
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:41 +0000 2020

	initial schema

commit hc8gq0434ckkjeo36rccn55mo14gk87q
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:33 +0000 2020

	Initialize data repository

root@2a29f8a34dd3:/go/simple-inventory#

Awesome! The first piece of information of the database was committed.

Adding data

Defining schema is not all the valuable without the data. Again, using standard SQL constructs we can commit the data. Let’s use proper branching strategies this time.

Create a branch and Enter dolt-sql

root@2a29f8a34dd3:/go/simple-inventory# dolt checkout -b intial_data
Switched to branch 'intial_data'
root@2a29f8a34dd3:/go/simple-inventory# dolt sql
# Welcome to the DoltSQL shell.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit.
doltsql>

Add data

doltsql> INSERT INTO device_inventory (hostname, ip_address) VALUES ("nyc-sw01", "10.1.1.1");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO device_inventory (hostname, ip_address) VALUES ("nyc-sw02", "10.1.1.2");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO vlan (hostname, vlan, name) VALUES ("nyc-sw01", 10, "user");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO vlan (hostname, vlan, name) VALUES ("nyc-sw01", 20, "printer");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO vlan (hostname, vlan, name) VALUES ("nyc-sw01", 30, "wap");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO vlan (hostname, vlan, name) VALUES ("nyc-sw02", 10, "user");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO vlan (hostname, vlan, name) VALUES ("nyc-sw02", 20, "printer");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO vlan (hostname, vlan, name) VALUES ("nyc-sw02", 30, "wap");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> exit
Bye

Now that we have staged the data, we can view the date in two different ways. The first is via raw sql entries, and the other is more akin to a unix diff.

View the diff. In the console, diffs are actually color coded green and red for add and remove respectively.

root@2a29f8a34dd3:/go/simple-inventory# dolt diff
diff --dolt a/device_inventory b/device_inventory
--- a/device_inventory @ tktob1spsoos6isfdj4o9benp9c57iic
+++ b/device_inventory @ b37haqr0n9j1e5bckj35sqt4kntojlp9
+-----+----------+------------+
|     | hostname | ip_address |
+-----+----------+------------+
|  +  | nyc-sw01 | 10.1.1.1   |
|  +  | nyc-sw02 | 10.1.1.2   |
+-----+----------+------------+
diff --dolt a/vlan b/vlan
--- a/vlan @ 7f8lqlv2k9cpdth68kob9hl8atuejrpn
+++ b/vlan @ e2kovelv2sfjuo584o0v6hp76207k720
+-----+----------+------+---------+
|     | hostname | vlan | name    |
+-----+----------+------+---------+
|  +  | nyc-sw01 | 10   | user    |
|  +  | nyc-sw01 | 20   | printer |
|  +  | nyc-sw01 | 30   | wap     |
|  +  | nyc-sw02 | 10   | user    |
|  +  | nyc-sw02 | 20   | printer |
|  +  | nyc-sw02 | 30   | wap     |
+-----+----------+------+---------+
root@2a29f8a34dd3:/go/simple-inventory# dolt diff -q
INSERT INTO `device_inventory` (`hostname`,`ip_address`) VALUES ("nyc-sw01","10.1.1.1");
INSERT INTO `device_inventory` (`hostname`,`ip_address`) VALUES ("nyc-sw02","10.1.1.2");
INSERT INTO `vlan` (`hostname`,`vlan`,`name`) VALUES ("nyc-sw01",10,"user");
INSERT INTO `vlan` (`hostname`,`vlan`,`name`) VALUES ("nyc-sw01",20,"printer");
INSERT INTO `vlan` (`hostname`,`vlan`,`name`) VALUES ("nyc-sw01",30,"wap");
INSERT INTO `vlan` (`hostname`,`vlan`,`name`) VALUES ("nyc-sw02",10,"user");
INSERT INTO `vlan` (`hostname`,`vlan`,`name`) VALUES ("nyc-sw02",20,"printer");
INSERT INTO `vlan` (`hostname`,`vlan`,`name`) VALUES ("nyc-sw02",30,"wap");
root@2a29f8a34dd3:/go/simple-inventory#

Personally, I’m pretty impressed with what is happening here, but there is still more to do to complete this workflow. The data needs to be committed, and merged from the “feature” branch into master branch.

Commit to the feature branch.

root@2a29f8a34dd3:/go/simple-inventory# dolt add -a
root@2a29f8a34dd3:/go/simple-inventory# dolt commit -m 'initial data'
commit gbt6r21mtbd0cvs5iahog912essjhn0n
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:50 +0000 2020

	initial data

root@2a29f8a34dd3:/go/simple-inventory#

Merge into the master branch.

root@2a29f8a34dd3:/go/simple-inventory# dolt checkout master
Switched to branch 'master'
root@2a29f8a34dd3:/go/simple-inventory# dolt branch
  intial_data
* master
root@2a29f8a34dd3:/go/simple-inventory# dolt merge intial_data
Updating 4n29eaqb1v9pmnpnda2r43fc91p1vp9l..gbt6r21mtbd0cvs5iahog912essjhn0n
Fast-forward
root@2a29f8a34dd3:/go/simple-inventory# dolt log
commit gbt6r21mtbd0cvs5iahog912essjhn0n
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:50 +0000 2020

	initial data

commit 4n29eaqb1v9pmnpnda2r43fc91p1vp9l
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:41 +0000 2020

	initial schema

commit hc8gq0434ckkjeo36rccn55mo14gk87q
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:33 +0000 2020

	Initialize data repository

root@2a29f8a34dd3:/go/simple-inventory#

So far this is showing the “Create” part of standard CRUD operations is working, meaning we can add data to the repository.

Update Data

Naturally, as soon as data is entered, you will want to modify it. The process is the same to modify: branch, sql statements, add, commit, and merge.

Checkout a new branch, and enter Dolt SQL.

root@2a29f8a34dd3:/go/simple-inventory# dolt checkout -b change_vlan
Switched to branch 'change_vlan'
root@2a29f8a34dd3:/go/simple-inventory# dolt sql
# Welcome to the DoltSQL shell.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit.
doltsql>

Update the data.

doltsql> update vlan set name = "prnt" where name = "printer";
+---------+---------+
| matched | updated |
+---------+---------+
| 2       | 2       |
+---------+---------+
doltsql> exit
Bye

View the diff.

root@2a29f8a34dd3:/go/simple-inventory# dolt diff
diff --dolt a/vlan b/vlan
--- a/vlan @ bjg5ou111lmu03ciu6aisphf7m6jbk1r
+++ b/vlan @ 7f8lqlv2k9cpdth68kob9hl8atuejrpn
+-----+----------+------+---------+
|     | hostname | vlan | name    |
+-----+----------+------+---------+
|  <  | nyc-sw01 | 20   | printer |
|  >  | nyc-sw01 | 20   | prnt    |
|  <  | nyc-sw02 | 20   | printer |
|  >  | nyc-sw02 | 20   | prnt    |
+-----+----------+------+---------+
root@2a29f8a34dd3:/go/simple-inventory# dolt diff -q
UPDATE `vlan` SET `name`="prnt" WHERE (`hostname`="nyc-sw01" AND `vlan`=20);
UPDATE `vlan` SET `name`="prnt" WHERE (`hostname`="nyc-sw02" AND `vlan`=20);
root@2a29f8a34dd3:/go/simple-inventory#

A keen eye will note the captured update statements shown in the diff are not the same as the update statement inputted in the SQL server. This conversion makes sense, since the data could be merged on a different set of data that had more or less data in it. It also illustrates the complexity of building this technology.

Commit to a feature branch and merge to master.

root@2a29f8a34dd3:/go/simple-inventory# dolt add -a
root@2a29f8a34dd3:/go/simple-inventory# dolt commit -m 'update data'
commit jqe3vb84fhfunn7uapksdt5thb4rr7r7
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:23:03 +0000 2020

	update data

root@2a29f8a34dd3:/go/simple-inventory# dolt checkout master
Switched to branch 'master'
root@2a29f8a34dd3:/go/simple-inventory# dolt merge change_vlan
Updating gbt6r21mtbd0cvs5iahog912essjhn0n..jqe3vb84fhfunn7uapksdt5thb4rr7r7
Fast-forward
root@2a29f8a34dd3:/go/simple-inventory# dolt sql
# Welcome to the DoltSQL shell.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit.
doltsql>

View the data from dolt sql.

doltsql> select * from vlan;
+----------+------+------+
| hostname | vlan | name |
+----------+------+------+
| nyc-sw01 | 10   | user |
| nyc-sw01 | 20   | prnt |
| nyc-sw01 | 30   | wap  |
| nyc-sw02 | 10   | user |
| nyc-sw02 | 20   | prnt |
| nyc-sw02 | 30   | wap  |
+----------+------+------+
doltsql>

You can also view who is the owner of the data by tracking to the commit, which includes the auther, time, and commit hash metadata as well, using the dolt blame <table> command.

root@2a29f8a34dd3:/go/simple-inventory# dolt blame vlan
+----------+------+--------------+-------------------+------------------------------+-------------------+
| HOSTNAME | VLAN | COMMIT MSG   | AUTHOR            | TIME                         | COMMIT            |
+----------+------+--------------+-------------------+------------------------------+-------------------+
| nyc-sw01 | 30   | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i |
| nyc-sw02 | 10   | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i |
| nyc-sw02 | 20   | update data  | itdependsnetworks | Tue Jan 28 04:23:03 UTC 2020 | jqe3vb84fhfunn7ua |
| nyc-sw02 | 30   | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i |
| nyc-sw01 | 10   | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i |
| nyc-sw01 | 20   | update data  | itdependsnetworks | Tue Jan 28 04:23:03 UTC 2020 | jqe3vb84fhfunn7ua |
+----------+------+--------------+-------------------+------------------------------+-------------------+
root@2a29f8a34dd3:/go/simple-inventory#

Summary

This is just a primer, and there is still a lot of work to be done to truly have feature parity with both MySQL and Git, but this is a great step in the right direction. I plan to continue to monitor and follow up with examples, use cases, and library updates over time. Specifically, next time, I want to extend the workflow to include DoltHub to review those capabilities as well.

-Ken

Ansible gather_facts for Networking Devices

2020-01-24T00:00:00+00:00

The purpose of this blog is to show some differences in how Ansible handled gathering facts from networking devices prior to version 2.9 and how they can be gathered in version 2.9.2. A main point to show here is how previously the gather_facts key was disabled when working with networking devices and now it can be enabled to gather device facts.

Gathering *_facts the “Old” way

Ansible core comes included with vendor specific *_facts modules. So normally to collect facts from devices the playbook was built using the vendor specific facts module and the needed key was used for any other particular task like asserting device versions, building reports, etc. The example below is a playbook used to gather facts from a Cisco device.

---
 - name: GATHER FACTS FOR IOS
   hosts: csr1
   connection: network_cli
   gather_facts: no    #<---Gather Facts Disabled

   tasks:

     - name: GATHER FACTS FOR IOS IN A TASK
       ios_facts:

     - name: VIEW ALL ANSIBLE FACT KEYS
       debug:
         var: ansible_facts.keys()

     - name: VIEW HOSTNAME
       debug:
         var: ansible_net_hostname
     
     - name: VIEW OS VERSION
       debug:
         var: ansible_net_version

Note: The first task is using the ios_facts module to gather all the data, the other three tasks are used to display the data returned from the facts module and keys inside ansible_facts can be accessed directly.

Notice how in the play definition the key gather_facts is set to no. Originally, Ansible was designed to work with Linux servers. Ansible focuses on collecting as much data as possible from the remote Linux servers and copying Python modules to the servers for automation tasks so they could be executed on the remote machines.

When working with networking devices Ansible does not copy Python modules to the remote devices to run an automation task. The Python module is executed in the local Ansible workstation and things like sending show commands or configuration changes are wrapped in the Python module and sent over through SSH. The gather_facts key was not designed to know the difference between a Linux server and a networking device, so by enabling the feature would end up gathering facts from the local Ansible workstation and not the remote networking device.

Running the Playbook with gather_facts disabled

The output of the current playbook would look like this:

Note: The current Ansible version is 2.7.10

ntc@jump-host:~$ ansible-playbook -i inventory network_facts.yml

PLAY [GATHER FACTS FOR IOS] ********************************************************************************

TASK [GATHER FACTS FOR IOS] ********************************************************************************
ok: [csr1]

TASK [VIEW ALL ANSIBLE FACT KEYS] **************************************************************************
ok: [csr1] => {
    "ansible_facts.keys()": "dict_keys(['net_serialnum', 'net_all_ipv4_addresses', 
'net_model', 'net_hostname', 'net_gather_subset', 'net_filesystems_info', 'net_interfaces', 
'net_version', 'net_neighbors', 'net_all_ipv6_add resses', 'net_memtotal_mb', 'net_filesystems', 
'net_image', 'net_memfree_mb'])"
}

TASK [VIEW HOSTNAME] ***************************************************************************************
ok: [csr1] => {
    "ansible_net_hostname": "csr1"
}

TASK [VIEW OS VERSION] *************************************************************************************
ok: [csr1] => {
    "ansible_net_version": "16.08.01a"
}

PLAY RECAP *************************************************************************************************
csr1                        : ok=4    changed=0    unreachable=0    failed=0

The current output only displays facts that belong to the remote device. The keys seen in the output are the ones available from the ios_facts module.

Running the Playbook with gather_facts enabled

When the gather_facts key is enabled in the play definition (gather_facts: yes) the output will return not only data from the remote device but also from the Ansible workstation as seen below:

ntc@jump-host:~$ ansible-playbook -i inventory network_facts.yml

PLAY [GATHER FACTS FOR IOS] ********************************************************************************

TASK [Gathering Facts] *************************************************************************************
ok: [csr1]

TASK [GATHER FACTS FOR IOS] ********************************************************************************
ok: [csr1]

TASK [VIEW ALL ANSIBLE FACT KEYS] **************************************************************************
ok: [csr1] => {
    "ansible_facts.keys()": "dict_keys(['module_setup', 'distribution_version', 'distribution_file_variety', 'env',
    'userspace_bits', 'architecture', 'default_ipv4', 'swapfree_mb', 'default_ipv6', 'cmdline', 'selinux',
    'userspace_architecture', 'product_uuid', 'pkg_mgr', 'distribution', 'iscsi_iqn', 'all_ipv6_addresses', 
    'uptime_seconds', 'kernel', 'system_capabilities_enforced', 'python', 'is_chroot', 'user_shell', 'product_serial'
    'form_factor', 'distribution_file_parsed', 'fips', 'user_id', 'selinux_python_present', 'ansible_local',
    'processor_vcpus', 'processor', 'ssh_host_key_ecdsa_public', 'mounts', 'system_vendor', 'swaptotal_mb',
    'distribution_major_version', 'real_group_id', 'lsb', 'machine', 'ssh_host_key_rsa_public', 'user_gecos',
    'processor_threads_per_core', 'eth1', 'product_name', 'all_ipv4_addresses', 'python_version',
    'product_version', 'service_mgr', 'memory_mb', 'user_dir', 'gather_subset', 'real_user_id', 
    'virtualization_role', 'tunl0', 'dns', 'effective_group_id', 'lo', 'memtotal_mb', 'device_links', 
    'apparmor', 'memfree_mb', 'processor_count', 'hostname', 'interfaces', 'machine_id', 'fqdn', 'user_gid',
    'nodename', 'domain', 'distribution_file_path', 'virtualization_type', 'ssh_host_key_ed25519_public', 
    'processor_cores', 'bios_version', 'date_time', 'distribution_release', 'os_family', 'effective_user_id',
    'sit0', 'system', 'devices', 'user_uid', 'ssh_host_key_dsa_public', 'bios_date', 'system_capabilities',
    'ens3', 'net_serialnum', 'net_all_ipv4_addresses', 'net_model', 'net_hostname', 'net_gather_subset',
    'net_filesystems_info', 'net_interfaces', 'net_version', 'net_neighbors', 'net_all_ipv6_addresses',
    'net_memtotal_mb', 'net_filesystems', 'net_image', 'net_memfree_mb'])"
}

TASK [VIEW HOSTNAME] ***************************************************************************************
ok: [csr1] => {
    "ansible_net_hostname": "csr1"
}

TASK [VIEW OS VERSION] *************************************************************************************
ok: [csr1] => {
    "ansible_net_version": "16.08.01a"
}

PLAY RECAP *************************************************************************************************
csr1                        : ok=5    changed=0    unreachable=0    failed=0

Gathering Facts from different platforms

Another thing to point out is that when using the core *_facts modules if another vendor or platform needs to be added to the playbook another play would need to be built to gather facts from that vendor or platform. Without this addition, the module will fail because the ios_facts module is vendor specific and it will try to gather facts from any devices targeted in the scope of that play. The playbook would look something like this:

---
 - name: GATHER FACTS FOR IOS
   hosts: csr1
   connection: network_cli
   gather_facts: no

   tasks:

     - name: GATHER FACTS FOR IOS
       ios_facts:

     - name: VIEW ALL ANSIBLE FACT KEYS
       debug:
         var: ansible_facts.keys()

     - name: VIEW HOSTNAME
       debug:
         var: ansible_net_hostname
     
     - name: VIEW OS VERSION
       debug:
         var: ansible_net_version
  
 - name: GATHER FACTS FOR NXOS
   hosts: nxos
   connection: network_cli
   gather_facts: no

   tasks:

     - name: GATHER FACTS FOR NXOS
       nxos_facts:

     - name: VIEW ALL ANSIBLE FACT KEYS
       debug:
         var: ansible_facts.keys()

     - name: VIEW HOSTNAME
       debug:
         var: ansible_net_hostname
    
     - name: VIEW OS VERSION
       debug:
         var: ansible_net_version

Gather Facts the “New” way

There have been a lot of changes with Ansible focused on improving how to manage networking devices. More and more the platform has been used in multi-vendor environments, meaning that the tool needs to be more robust and vendor neutral. Recent features and improvements are being built to be more “vendor neutral” - such as cli_config and cli_command (added in Ansilble version 2.7).

In Ansible version 2.9, gathering facts from devices from the play definition is now possible without having to build it as a task, like shown in the previous playbook. This time the key gather_facts can be enabled and not gather facts from the local workstation but from the devices targeted in the hosts key.

Note: The modules ios_facts and nxos_fact are still being used but are now being executed in the play definition.

---

 - name: GATHER FACTS
   hosts: csr1,nxos
   connection: network_cli
   gather_facts: yes  #<----Gather Facts Enabled 

   tasks:
   
    #NO VENDOR SPECIFIC MODULE LIKE ios_facts OR nxos_facts ANYMORE

    - name: VIEW ALL ANSIBLE FACT KEYS
      debug:
       var: ansible_facts.keys()

    - name: VIEW HOSTNAME
      debug:
        var: ansible_net_hostname
    
    - name: VIEW OS VERSION
      debug:
        var: ansible_net_version

Take a look at the current playbook. The hosts key has two different operating systems which normally would require another play when trying to gather data from different platforms because of the vendor specific modules defined in the tasks. This time gathering facts is being done at the play definition level and at the tasks level only the debug modules are being used to display the available keys and variables to see hostname and version information.

Running the Playbook with gather_facts enabled the “New Way”

ntc@jump-host:~$ ansible-playbook -i inventory network_facts.yml

PLAY [GATHER FACTS] ***************************************************************************************

TASK [Gathering Facts] ************************************************************************************


ok: [csr1]
ok: [nxos]

TASK [VIEW ALL ANSIBLE FACT KEYS] *************************************************************************
ok: [csr1] => {
    "ansible_facts.keys()": "dict_keys(['network_resources', 'net_gather_network_resources', 
    'net_gather_subset', 'net_system', 'net_model', 'net_image', 'net_version', 'net_hostname', 
    'net_api', 'net_python_version', 'net_iostype', 'net_serialnum', 'net_filesystems', 
    'net_filesystems_info', 'net_memtotal_mb', 'net_memfree_mb', 'net_config', 'net_all_ipv4_addresses',
    'net_all_ipv6_addresses', 'net_neighbors', 'net_interfaces', '_facts_gathered'])"
}
ok: [nxos] => {
    "ansible_facts.keys()": "dict_keys(['network_resources', 'net_gather_network_resources', 
    'net_gather_subset', 'net__os', 'net__platform', 'net__hostname', 'net_interfaces_list', 
    'net_vlan_list', 'net_module', 'net_fan_info', 'net_power_supply_info', 'net_filesystems', 
    'net_memtotal_mb', 'net_memfree_mb', 'net_features_enabled', 'net_all_ipv4_addresses', 
    'net_all_ipv6_addresses', 'net_neighbors', 'net_interfaces', 'net_serialnum', 'net_license_hostid',
    'net_system', 'net_model', 'net_image', 'net_version', 'net_platform', 'net_hostname', 'net_api', 
    'net_python_version', 'net_config', '_facts_gathered'])"
}

TASK [VIEW HOSTNAME] ***************************************************************************************
ok: [csr1] => {
    "ansible_net_hostname": "csr1"
}
ok: [nxos] => {
    "ansible_net_hostname": "nxos-spine1"
}

TASK [VIEW OS VERSION] **************************************************************************************
ok: [csr1] => {
    "ansible_net_version": "16.08.01a"
}
ok: [nxos] => {
    "ansible_net_version": "7.0(3)I7(4)"
}

PLAY RECAP *************************************************************************************************
csr1                        : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
nxos                       : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The output above shows the results of the playbook ran in the new version. This time only the device facts are gathered in a single play for two different platforms, rather than running vendor specific modules that have to run in two different plays.

Note: Now that gather_facts can be used with networking devices there are some configurations that can be changed in the ansible.cfg file that will affect how the gathering could behave.

Changing the settings below in the ansible.cfg file will affect how gathering facts from devices will be gathered.

# smart - gather by default, but don't regather if already gathered
# implicit - gather by default, turn off with gather_facts: False
# explicit - do not gather by default, must say gather_facts: True
# gathering = implicit

For more information on what other new changes exist in Ansible 2.9 check out the link below:

Ansible 2.9 Porting Guide

Summary

Before this new update in Ansible 2.9.2, when building playbooks for networking devices the gather_facts key was always disabled and it almost seemed like an extra key that needed to be there for no good reason unless facts from the local machine needed to be added. In this new version of Ansible gathering facts from networking devices really makes things easier and more flexible in a structured way. I’m looking forward to test this out with the new rersource modules and showing this feature to more of my students from now on.

-Hector

Workflow and Productivity Management

2020-01-14T00:00:00+00:00

Full disclosure before you read any further: I am one of “those” people …the productivity obsessed. I am constantly trying to improve my output and overall workflow - both professionally and personally. I am by no means the most productive person in the world, but I’m constantly trying to work toward some unatainable level of expert efficiency. I’ve read 7 Habits of Highly Effective People and Atomic Habits (which I HIGHLY recommend). I’ve read every Lifehacker “How I Work” post. I’ve gone from vim to emacs back to vim - currently with highly-customized vimrc file. And, I’m currently in the process of trying to reset my sleep schedule to wake up at 4:00am, to reach the pinnacle of productivity!

Take everything below with a grain of salt - there are probably much more scientific and accurate explanations for the information I provide herein. But, to quote the prophet Adam Sandler in his critically acclaimed philosophical sonnet “Wedding Singer”:

I am the one with the Microphone!

As network engineers, developers, devops practictioners, and overall technologists, we have become accustomed to working in teams. These teams are often organized into functional or cross-functional groups that work together to accomplish a certain set of tasks - deliver an application; design, implement, maintain, augment the infrastructure that supports an application; rinse & repeat. As such, there are many methodologies that define how teams work together to accomplish their set of tasks. A non-exhaustive list would include:

Agile
Kanban
Waterfall
Scrum (which, I think is Agile?)
Lean
Extreme Programming
GTD
And many…many…more (so many more)

These methodologies, for better or worse, define how a project is managed. They establish a set of standards and working cadence that facilitates a few expected outcomes: delivery, collaboration, acceptance criteria, estimation, and scheduling. When utilized correctly, these methodolgies can be powerful tools to manage a team toward an efficent and euphoric high-functioning juggernaut!

The question that can often get overlooked, even on these high-functioning championship teams is: How do I manage MY time? How do I prioritize and maximize MY productivity?

With that question in mind, we will take a look at a method of personal productivity management that can be beneficial for anyone working in an engineering/application/devops capacity.

Priority vs Productivity

In all my research, tweaks, failures, and successes, I’ve identified that there is a big distinction to be made between priority and productivity. They work hand in hand, and are equally important toward an efficient workflow. Priorty identifies what to do at a given moment; Productivity is how you do it. Being proficient at both is the key to maximizing our time and potential as engineers.

More often that not, external factors have much influence on prioritization. Deadlines, dependencies, budget contraints, outages, threats, public shamings, etc… can all impact which tasks get pulled into our “hot list”. Often times this is out of our control.

Prioritization can be summed up by using the common computer science analogy of the Stack vs Heap.

In this analogy, “the Stack” involves things that are done quickly, with very little congitive load. This could involve answering an email, documenting a new feature, commenting on a pull-request, reviewing a pull-request (sometimes), and other “quick” tasks. The key here is the cognitive load. A coworker of mine is an all-time champ at answering questions on the Network to Code Slack channel. During downtime he can engage with someone, diagnose their issue, and respond, almost asyncronously from his other tasks. He’ll do this if he has 5 minutes between meetings, waiting on a response from one of his employees, etc… He is able to do this based on experience and muscle memory - most questions require very little research for him. If a question does require additional research or investigation, he may refrain from responding or file it away for a later time …when he has time for “the Heap”.

“The Heap” in computer science refers to operations requiring memory allocations of a non-finite size. As such, it is more flexible, but more expensive than operations involving “the Stack”. As technologists, examples of using “The Heap” could include: reviwing documentation, writing code, debugging, troubleshooting, researching a new topic, reviewing a design request, and code review.

The Stack vs Heap analogy highlights the importance of properly identifying a task or request’s scope, and what is involved in fulfilling/resolving it. A good question to ask is: “Will this take me more than 5 minutes?” If so, it is normally left for an allocated time of focus, where the task can be given the attention it needs. Otherwise, it can be handled immediately, or during a time requiring less focus. Some answer emails and handle other “quick” tasks at pre-allocated times during the day (more on this later); others handle these tasks as they come in, but only for a short period of time, before returning to a more involved task (this is often the case for those in an operations role).

Proper Prioritization Prevents Poor Performance

The biggest takeaway from the Productivity vs Priority distinction is that it is imperitive that we identify our priorities and make a plan to execute on them. Conversely, it is important that we identify secondary responsiblities and potential distractions that can negatively impact our priorities.

If our main job responsibility is to deliver a feature by the end of a sprint, will our team care how many emails we have answered if the release is late? If we are asked to resolve tickets quickly to improve our team’s MTTR, does researching new features contribute to, or detract from that goal?

Though those questions may sound bleak, the good news is that we normally have more time for productivity that we think. This is especially true if we work in an environment with reasonable deadlines and efficient estimation - e.g., Network to Code (shameless plug). Normally, we have time to deliver on our primary, and additional responsiblities and look for ways to improve efficiency - if we improve our productivity.

Pomodoro Technique - Embrace the Tomato

The method I have personally found to be most productive FOR MYSELF is called “Pomodoro”. I have found great success utilizing this method because it forces you to allocate both focused and unfocused time, along with breaks. In short, the Pomodoro Technique consists of the following:

A “Pomodoro” is a CONSECUTIVE unit of focused time - normally 25 minutes
Each pomodoro is followed by a short (3-5 minute) or long (15-30 minute) break
Breaks increase in length as successful pomodoros are completed
Tasks are handled in a First In First Out method
If a tasks takes longer than 25 minutes, it can be completed after a break during the next pomodoro
If a task takes less than 25 minutes the remaining time can be left for unfocused (“Stack”) activities
WRITE EVERYTHING DOWN!! Take notes on tasks, successful/failed pomodoros, timing, etc…

It also promotes good prioritization, planning, estimation, and self-reflection. At the end of a day, week, or sprint, it is easy to identify where time was spent, and what optimizations are needed.

Here are some things I’ve done to utilize the pomodoro technique effectively:

Assign every task a due date as you receive it
At the beginning of the week define the week’s goals and the total time you’re allocating to work (time available minus meetings, appts, personal time, PTO)
At the beginning of the day write down Things To Do, keeping in mind the time available for that day
Identify which tasks are most difficult, and/or least desireable, and stagger them with more desireable/easy tasks
Always start with a desireable/easy task to build momentum
Schedule long breaks, lunches, and personal time, by adding these things to your calendar
Mute Slack, email, and other notifications (if possible). Move your phone to another room, or away from your reach
Have fun & execute

With all of these productivity “tips”, it is important to remember that “to err is human”. Some days I am a productivity machine. Some days I can barely type my own name. However, I can personally attest to the success of the pomodoro method. When I use it effectively, at the end of each day, week, sprint, my output is optimal, and I feel great about it. When everything is “on fire”, time seems to fly by, with little impact.

In Conclusion …Just Pick Something

Is the Pomodoro Technique the “silver bullet” for productivity? Surely - just like Agile, DevOps, and Kubernetes solve everything. This technique may not be the best for you. But, the principles of Prioritization and Productivity can help to contribute to an improved workflow, and hopefully make us all more effective and efficient. The most important thing is to be more deliberate with our time, and to pick a technique, method, and workflow, that works for each of us individually. Increasing productivity is the closest we can get to building a time machine, in that it can help us waste less of it. Maybe that justifies my obsession?

-Daryn

Automation Principles - DRY

2020-01-07T00:00:00+00:00

This is part of a series of posts focused on Network Automation Principles.

DRY in Computer Science

The phrase Do Not Repeat Yourself or DRY is a rather simple concept. If you are writing a piece of code multiple times, modularize the code in a way to ensure you are not copying code multiple times. The definition of DRY states:

“Every piece of knowledge must have a single, unambiguous, authoritative representation within a system”.

Once there are multiple copies, inevitably, one of the pieces of code will not function the same as others, based on some missed step. Additionally if you want to change functionality in one place, you would have to remember to change it in all places.

To put in context, Bart Simpson writing the same phrase on the chalkboard over and over is certainly not DRY.

Example

In this first contrived example you will see the duplication of code in how the VLAN data is being validated.

>>> def add_vlan(vlan_id):
...     if not isinstance(vlan_id, int):
...         raise TypeError("VLAN ID is not an integer")
...     elif not (vlan_id >=1 and vlan_id <= 4096):
...         raise ValueError("Invalid VLAN ID, which must be between 1-4096")
...     return "vlan {vlan_id}".format(vlan_id=vlan_id)
...
>>> def configure_port_vlan(interface, vlan_id):
...     if not isinstance(vlan_id, int):
...         raise TypeError("VLAN ID is not an integer")
...     elif not (vlan_id >=1 and vlan_id <= 4096):
...         raise ValueError("Invalid VLAN ID, which must be between 1-4096")
...     return "interface {interface}\n switchport access vlan {vlan_id}".format(interface=interface, vlan_id=vlan_id)
...
>>>

A common method to to modularize code is building functions. A function can be thought of as a piece of code that can be called to run that single action, and functions can call other functions. Here you can see how the validation functionality is broken out.

>>> def validate_vlan(vlan_id):
...     if not isinstance(vlan_id, int):
...         raise TypeError("VLAN ID is not an integer")
...     elif not (vlan_id >=1 and vlan_id <= 4096):
...         raise ValueError("Invalid VLAN ID, which must be between 1-4096")
...
>>>

The duplicated functionality is removed from the initial functions, and you can see it was tested to work the same.

>>> def add_vlan(vlan_id):
...     validate_vlan(vlan_id)
...     return "vlan {vlan_id}".format(vlan_id=vlan_id)
...
>>> def configure_port_vlan(interface, vlan_id):
...     validate_vlan(vlan_id)
...     return "interface {interface}\n switchport access vlan {vlan_id}".format(interface=interface, vlan_id=vlan_id)
...
>>> print(add_vlan(500))
vlan 500
>>> print(add_vlan(5000))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in add_vlan
  File "<stdin>", line 5, in validate_vlan
ValueError: Invalid VLAN ID, which must be between 1-4096
>>> print(configure_port_vlan("GigabitEthernet1/0/1", 50))
interface GigabitEthernet1/0/1
 switchport access vlan 50
>>> print(add_vlan("500"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in add_vlan
  File "<stdin>", line 3, in validate_vlan
TypeError: VLAN ID is not an integer
>>>

Each time you need to validate a VLAN ID, you simply can call the same function. If at a later date you decided that you wanted to validate the VLAN name for length as well, you can simply add that testing to the one function, instead of having to remember all of the multiple places the code can be.

DRY Data

You will note in the official definition, there is reference to “knowledge”, which as quoted from Pragmatic Programmer

“A system’s knowledge is far broader than just its code. It refers to database schemas, test plans, the build system, even documentation”.

Simply put, code is not the only thing DRY applies to. One example understood by network engineers is managing data for a subnet mask. I often run into solutions of data and associated Jinja files that similar to:

subnets:
  - network: 10.1.1.0
    cidr: 24
    subnet_mask: 255.255.255.0
    wildcard_mask: 0.0.0.255
  - network: 10.1.2.0
    cidr: 24
    subnet_mask: 255.255.255.0
    wildcard_mask: 0.0.0.255

{% if ansible_network_os = "ios" %}
  ip address {{ subnet['network'] | ipaddr('add', 1 ) }} {{ subnet['subnet_mask'] }}{# example result: 10.1.1.1 255.255.255.0 #}
{% elif ansible_network_os = "nxos" %}
  ip address {{ subnet['network'] | ipaddr('add', 1 ) }}/{{ subnet['cidr'] }}{# example result: 10.1.1.1/24 #}
{% endif %}

This solution certainly has benefits, however, given the sheer amount of subnets being managed, it is likely at some point there will be a user that creates a cidr of 25 and subnet_mask as 255.255.255.0. However, the subnet and wildcard mask can be ascertained from the cidr, and maintaining the data multiple times is superfluous and error prone. The one-time cost of building a translation ensures that you are not repeating yourself and not falling into the issues being described here.

Instead you can manage your data a bit more DRY, such as:

subnets:
  - network: 10.1.1.0/24
  - network: 10.1.2.0/24

{% if ansible_network_os = "ios" %}
  ip address {{ subnet['network'] | ipaddr('add', 1 ) }} {{ subnet['network'] | ipaddr('netmask') }}{# example result: 10.1.1.1 255.255.255.0 #}
{% elif ansible_network_os = "nxos" %}
  ip address {{ subnet['network'] | ipaddr('add', 1 ) }}/{{ subnet['network'] | ipaddr('prefix') }}{# example result: 10.1.1.1/24 #}
{% endif %}

While this is a trivial example, the potential issue with data duplication is compounded for every feature that needs to be configured on a given device.

Getting WET in a DRY World

If DRY is not repeating any piece of knowledge, the reverse would be “Write Every Time” (WET). Now why would you want to repeat yourself? Well, as always, it depends, and there are always design considerations that should be taken into account.

As an example, a common pattern we at NTC end up seeing is customers that build a single Jinja template for IOS, NXOS, and EOS. Since these three OS’s have a lot of commonalities, from a DRY perspective you may be inclined to write your templates in consolidated templates.

.
├── bgp.j2
├── ntp.j2
└── vlan.j2

0 directories, 3 files

With an example Jinja file such as

{% for vlan in vlans %}
vlan {{ vlan['id'] }}
 name {{ vlan['name'] }}
{% endfor %}
!

This works out well for this case, however, as you move on to more complicated use cases, such as BGP, you will end with greater differences between the various OS’s.

{% if ansible_network_os == 'ios' %}
router bgp {{ bgp['asn'] }}
 bgp router-id {{ bgp['id'] }}
 bgp log-neighbor-changes
{% for neighbor in bgp['neighbors'] %}
 neighbor {{ neighbor['ip'] }} remote-as {{ neighbor['asn'] }}
 neighbor {{ neighbor['ip'] }} description {{ neighbor['description'] }}
{% endfor %}
 address-family ipv4
{% for net in bgp['networks'] %}
  network {{ net | ipaddr('network') }} mask {{ net | ipaddr('netmask') }}
{% endfor %}
{% for neighbor in bgp['neighbors'] %}
  neighbor {{ neighbor['ip'] }} activate
{% endfor %}
 exit-address-family
{# 
--------------
SWITCH TO NXOS
--------------
#}
{% elif ansible_network_os == 'nxos' %}
router bgp {{ bgp['asn'] }}
  router-id {{ bgp['id'] }}
  address-family ipv4 unicast
{% for net in bgp['networks'] %}
    network {{ net }}
{% endfor %}
{% for neighbor in bgp['neighbors'] %}
  neighbor {{ neighbor['ip'] }} remote-as {{ neighbor['asn'] }}
    description {{ neighbor['description'] }}
    address-family ipv4 unicast
{% endfor %}
{% endif %}

As you can see, it very quickly get’s complicated. A few basic options, each with pros and cons.

Keep a flat structure, as originally shown, and leverage the similarities of the multiple OS’s to alleviate re-writing them, but add complexity to the use cases where there are differences.
Keep a nested structure, one for each OS, and have the code copied over from one OS to the other, where it is the same configuration.
Attempt to merge the two, and only leverage OS folder when complicated, and a default folder when not.

In this use case, my personal preference is to have a well defined OS structure as the next developer can review the tree structure (as depicted in the second option), and likely figure out where the configuration templates should go. Essentially, make it easier for the next developer to understand the layout, rather than worry about the duplication of code.

.
├── eos
│   ├── bgp.j2
│   ├── ntp.j2
│   └── vlan.j2
├── ios
│   ├── bgp.j2
│   ├── ntp.j2
│   └── vlan.j2
└── nxos
    ├── bgp.j2
    ├── ntp.j2
    └── vlan.j2

3 directories, 9 files

Situations like this will come up all the time, and you will simply have to use experience/intuition to understand the impact and make the best decision from there.

Conclusion

Managing your code or data should in most use cases be done in a DRY manner, as this method cuts down on errors and drives computational optimizations. That being said, there are some design considerations where it may make sense to take a WET approach.

-Ken

Metrics for Everyone

2019-12-31T00:00:00+00:00

This year we heard from a lot of folks that have started down their network automation journey. They are seeing initial success, but are finding it difficult to get buy-in from other teams, management, or both. Although frustrating, this experience is not unique to network automation.

DevOps teams have largely experienced the same issue, and looking back to the beginning of DevOps there was a framework put together by John Willis and Damon Edwards in 2010 known as CAMS. This framework set out to define the key pillars of DevOps:

Culture
Automation
Measurement
Sharing

Using this framework we can break down the buy-in problem (and finally get to the point of the blog, which is metrics):

You or your team have built some piece of automation, and are rightfully excited about it. The culture of your organization is however resistant to change. There are a variety of reasons for this resistance: long history of doing things manually, the expense of building and learning new things, poor results from similar efforts in the past.

Know your audience

Measuring the impact of your automation provides tangible data to combat resistance. It is important however to have the right data for your audience. There are layers to any organization, and each layer has drivers that are relevant to their goals.

For the purpose of this blog we will define three layers of the organization and show examples of metrics that may allay the concerns of that audience - know your audience.

Individual Contributors (Engineers)

This may be network engineers, operations, architects, etc. These are the people most directly related to doing the work. Their primary concerns tend to be:

Level of effort to learn something new
Time available to do something different
Reliability and performance of solutions

The metrics we want to show here revolve around the state of the infrastructure and the health of the overall solution. What systems are being affected? What failures are we seeing? How is the performance of the automation?

Direct Management

This level of the organization sits in the middle, balancing the drivers and concerns of the others

Capacity and expenditure of the teams resources
Well being of the team
Ensuring business objectives are articulated bi-bidirectionally and achieved

At this middle level we want to show the impact of automation on the team, the adoption of automation by other teams, and the impact on the business.

Senior Leadership

The focus at the top of the organization is big picture - the overarching goals of the company

Overall strategy
Financial implications
Long term growth and stability

Conclusion

It is important to empathize with all aspects of the organization and provide them the data they need to make informed decisions. DevOps is a cultural shift, as organizations that have started down this path are at various stages of maturity. Data drives business decisions, and the more relevant the data, the better the decisions.

The 2018 “State of DevOps Report” speaks to that journey, and has some great insight:

When it comes to measurement, we have found that expansion from teams to departments manifests as a shift from manually gathered IT system metrics to automated measurement of business objectives. The most sophisticated teams we’ve seen not only improved their IT processes and practices, but also managed to focus on delivering business value rather than just technology. These teams have applied their existing cultures of automation and measurement to business objectives.

-Rick (@shermdog01)

Exploring Jinja2 Variable Syntax in Ansible

2019-12-24T00:00:00+00:00

When working with Ansible playbooks and jinja2, you might notice different jinja2 syntax to access data. Some use square brackets (backup_commands['ios']) while others use dot notation (backup_commands.ios) when accessing data.

Both ways are valid and work fairly well, but some would say it’s safer to use [] bracket rather than . dot notation to access data in a production environment. For demo purposes, I tend to use the. notation just to type quicker but that won’t always work if accessing data from a variable.

Note: Other reasons to use square brackets:

When a variable name includes a hyphen (-)

When dynamically generating a dictionary key

Let’s look at a few examples and see why it’s not always possible to access data using the dot(.) notation and it’s always possible to access data using the square brackets([]).

Using Ansible to Access Dictionary Values

First let’s test this playbook called brackets_vs_dot.yml and run it.

A variable has been created under vars called backup_commands and nested, there are a few { "key": "value" } pairs with different vendors as keys and commands as values to access.

backup_commands:
  ios:  "show run"
  nxos:  "show run"
  junos:  "show configuration"

Four debug tasks have also been added that will print out each key in a nested dict of dicts. Two of the tasks will use bracket [] notation and the other two will use dot . notation.

---

    - name: Accessing values through dict keys
      hosts: localhost
      connection: local
      gather_facts: no

      vars:
        backup_commands:
          ios:  "show run"
          nxos:  "show run"
          junos:  "show configuration"

      tasks:

        - debug:
            var: backup_commands['ios']
        
        - debug:
            var: backup_commands['nxos']
        
        - debug:
            var: backup_commands.ios
        
        - debug:
            var: backup_commands.nxos

After running the playbook, you should see the following output with the values being accessed using the debug module.

ntc@jump-host:ansible$ brackets_vs_dot.yml
PLAY [Accessing values through dict keys] **********************************************************************************************************************************************************************************

TASK [debug] ***************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "backup_commands['ios']": "show run"
}

TASK [debug] ***************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "backup_commands['nxos']": "show run"
}

TASK [debug] ***************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "backup_commands.ios": "show run"
}

TASK [debug] ***************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "backup_commands.nxos": "show run"
}

PLAY RECAP *****************************************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Based on the output you can see that both bracket [] and . dot notation worked just fine with the same output results.

However, let’s say this time we want to access that same data, but through a variable. The results on this test will be a little different and will show why the . notation doesn’t always work.

This time vars_prompt has been added in a second play with a variable vendor_os to allow the user to pick an os and access the data through the debug tasks.

vars_prompt:
  - name: vendor_os
    prompt: "Enter Vendor OS"
    private: no

In this second play two tasks have been added using [] brackets and . dot notation to access the data based on the variable value entered through the prompt.

    - name: Accessing values through variables
      hosts: localhost
      connection: local
      gather_facts: no
      
      vars_prompt:
        - name: vendor_os
          prompt: "Enter Vendor OS"
          private: no
      
      vars:
        backup_commands:
          ios:  "show run"
          nxos:  "show run"
          junos:  "show configuration"

      tasks:

        - debug:
            var: backup_commands[vendor_os]
        
        - debug:
            var: backup_commands.vendor_os 

Accessing values through variables

Pay close attention to the second debug tasks with the results of "VARIABLE IS NOT DEFINED!".

Enter Vendor OS: junos

After entering junos in the prompt the following output show be displayed:

PLAY [Accessing values through variables] **********************************************************************************************************************************************************************************

TASK [debug] ***************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "backup_commands[vendor_os]": "show configuration"
}

TASK [debug] ***************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "backup_commands.vendor_os": "VARIABLE IS NOT DEFINED!"
}

PLAY RECAP *****************************************************************************************************************************************************************************************************************
localhost                  : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

If you’re still not convinced why we should stick to using [] bracket notation vs . dot notation then let’s try this out with jinja2.

Comparing Dot Notation vs Square Brackets using Jinja2

For this test the main tool being used is Template Designer for Automation, feel free to use this data and template to test it out yourself:

DATA

Using the same data structure from the previous Ansible test.

backup_commands:
  ios:  "show run"
  nxos:  "show run"
  junos:  "show configuration"

JINJA2 TEMPLATE - Accessing values through dict keys

A similar test to the first play of Ansible will be ran. This time instead of debug tasks, build a jinja2 template and add it in the TEMPLATE column of td4a.

In this example, each key will be accessed with square([]) bracket and dot (.) notation.

Accessing Keys

Bracket Notation:
  ios: {{ backup_commands['ios'] }}
  nxos: {{ backup_commands['nxos'] }}
  junos: {{ backup_commands['junos'] }}

Dot Notation:
  ios:  {{ backup_commands.ios }}
  nxos: {{ backup_commands.nxos }}
  junos: {{ backup_commands.junos }}

After adding the following jinja2 data into the TEMPLATE column, press the RENDER button on the top right of the middle column to view the results.

RESULT

You should see the following results, as expected from the jinja2 template showing that it has successfully accessed each value using both [] brackets and . notation.

Accessing Keys

Bracket Notation:
  ios: show run
  nxos: show run
  junos: show configuration

Dot Notation:
  ios:  show run
  nxos: show run
  junos: show configuration

JINJA2 TEMPLATE - Accessing values through variables

This time run a test with jinja2 for loops, the first one accessing the values through [] brackets and the second one commented out with . notation for now so it doesn’t dispaly an error.

Accessing Variables

Bracket Notation:
{% for vendor_os in backup_commands.keys() %}
  {{ vendor_os }}: {{ backup_commands[vendor_os]  }}
{% endfor %}

Dot Notation:
{% for vendor_os in backup_commands.keys() %}
  {#{{ vendor_os }}: {{ backup_commands.vendor_os }}#}
{% endfor %}

After pressing on the RENDER button again –> the following output should show up:

RESULT

The first two tests don’t change, the second two show the output as expected using the [] bracket and the . notation doesn’t show anything since it’s commented out.

Accessing Keys

Bracket Notation:
  ios: show run
  nxos: show run
  junos: show configuration

Dot Notation:
  ios:  show run
  nxos: show run
  junos: show configuration

Accessing Variables

Bracket Notation:
  ios: show run
  nxos: show run
  junos: show configuration

Dot Notation:

Finally it’s time to test the dot notation so remove the {##} comments from the jinja2 template to view the results using the . notation.

Dot Notation:
{% for vendor_os in backup_commands.keys() %}
  {{ vendor_os }}: {{ backup_commands.vendor_os }}
{% endfor %}

After pressing on the RENDER button one last time and viewing the results:

You should see the following error: Message: Issue found loading template. Details: Object has no attribute 'vendor_os' Line number: 22

Finally it has been proven that for good practice and sake of muscle memory, that maybe we should stick to using [] square brackets rather than . dot notation because you will always be able to access the data whether if it’s a key or a variable without any errors.

-Hector