The NTC Mag

How to Monitor Your VPN Infrastructure with Netmiko, NTC-Templates, and a Time Series Database

2020-03-30T00:00:00+00:00

With many people being asked to work from home, we have heard several customers looking to enhance the visibiility and monitoring of their VPN infrastructure. In this post I will show you how you can quickly collect information from your Cisco ASA firewall leveraging Netmiko, NTC-Templates (TextFSM), combined with Telegraf/Prometheus/Grafana. The approach would work on other network devices, not just Cisco ASAs. Considering the recent demand for ASA information, we will use this as an example.

Here is what the data flow will look like:

Users will connect to the ASA for remote access VPN services
Python: Collects information from the device via CLI and gets structured data by using a NEW template to parse the CLI output. This is presented via stdout in Influx data format
Telegraf: Generic collector that has multiple plugins to ingest data and that can send data to many databases out there.
- INPUT: Execute the Python script every 60s and read the results from stdout.
- OUTPUT: Expose the data over HTTP in a format compatible with Prometheus
Prometheus: Time Series DataBase (TSDB). Collects the data from Telegraf over HTTP, stores it, and exposes an API to query the data
Grafana: Solution to build dashboards, natively support Prometheus to query data.

An alternative to creating this Python script, you could have looked at using the Telegraf SNMP plugin as well. An SNMP query would be quicker than using SSH and getting data if you want basic counts. In this you will see that you can get custom metrics into a monitoring solution without having to use only SNMP.

Execution of Python

If executing just the Python script without being executed by Telegraf, this is what you would see:

$ python3 asa_anyconnect_to_telegraf.py --host 10.250.0.63
asa connected_users=1i,anyconnect_licenses=2i

This data will then get transformed by Telegraf into an output that is usable by Prometheus. It is possible to remove the requirement for Telegraf and have Python create the Prometheus Metrics. We wanted to keep the Python execution as simple as possible. To use the prometheus_client library check out their Github page.

Python Script

In this post we have the following components being used:

Python:
- Netmiko to SSH into an ASA, gather command output, and leverage the corresponding NTC Template
- NTC Template which is a TextFSM template for parsing raw text output into structured data
Telegraf: Takes the output of the Python script as an input and translates it to Prometheus metrics as an output

Python Requirements

The Python script below will have the following requirements set up before hand:

ENV Variables for authentication into the ASA
- ASA_USER: Username to log into the ASA
- ASA_PASSWORD: Password to log into the ASA
- ASA_SECRET (Optional): Enable password for the ASA, if left undefined will pick up the ASA_PASSWORD variable
Required Python Packages:
- Netmiko: For SSH and parsing
- Click: For argument handling - to get the hostname/IP address of the ASA
Github Repository for NTC-Templates setup in one of two ways:
- Cloned to user home directory cd ~ and git clone https://github.com/networktocode/ntc-templates.git
- NET_TEXTFSM Env variable set NET_TEXTFSM=/path/to/ntc-templates/templates/

The specific template is the newer template for cisco asa show vpn-sessiondb anyconnect introduced March 18, 2020

Python Code

There are two functions used in this quick script:

"""
(c) 2020 Network to Code
Licensed under the Apache License, Version 2.0 (the "License").
You may not use this file except in compliance with the License.
You may obtain a copy of the License at
http: // www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Python application to gather metrics from Cisco ASA firewall and export them as a metric for Telegraf
"""
from itertools import count
import os
import sys
import re
import click
from netmiko import ConnectHandler

def print_influx_metrics(data):
    """
    The print_influx_metrics function takes the data collected in a dictionary format and prints out
    each of the necesary components on a single line, which matches the Influx data format.

    Args:
        data (dictionary): Dictionary of the results to print out for influx
    """
    data_string = ""
    cnt = count()
    for measure, value in data.items():
        if next(cnt) > 0:
            data_string += ","
        data_string += f"{measure}={value}i"

    print(f"asa {data_string}")

    return True


def get_anyconnect_license_count(version_output):
    """
    Searches through the `show version` output to find all instances of the license and gets the
    output into integers to get a license count.

    Since there could be multiple ASAs in a cluster or HA pair, it is necessary to gather multiple data
    points for the license count that the ASAs are licensed for. This function uses regex to find all of
    the instances and returns the total count based on the `show version` command output.  

    Args:
        version_output (String): Output from Cisco ASA `show version`
    """
    pattern = r"AnyConnect\s+Premium\s+Peers\s+:\s+(\d+)"
    re_list = re.findall(pattern, version_output)

    total_licenses = 0
    for license_count in re_list:
        total_licenses += int(license_count)

    return total_licenses


# Add parsers for output of data types
@click.command()
@click.option("--host", required=True, help="Required - Host to connect to")
def main(host):
    """
    Main code execution
    """
    # Get ASA connection Information
    try:
        username = os.environ["ASA_USER"]
        password = os.environ["ASA_PASSWORD"]
        secret = os.getenv("ASA_SECRET", os.environ["ASA_PASSWORD"])
    except KeyError:
        print("Unable to find Username or Password in environment variables")
        print("Please verify that ASA_USER and ASA_PASSWORD are set")
        sys.exit(1)

    # Setup connection information and connect to host
    cisco_asa_device = {
        "host": host,
        "username": username,
        "password": password,
        "secret": secret,
        "device_type": "cisco_asa",
    }
    net_conn = ConnectHandler(**cisco_asa_device)

    # Get command output for data collection
    command = "show vpn-sessiondb anyconnect"
    command_output = net_conn.send_command(command, use_textfsm=True)

    # Check for no connected users
    if "INFO: There are presently no active sessions" in command_output:
        command_output = []

    # Get output of "show version"
    version_output = net_conn.send_command("show version")

    # Set data variable for output to Influx format
    data = {"connected_users": len(command_output), "anyconnect_licenses": get_anyconnect_license_count(version_output)}

    # Print out the metrics to standard out to be picked up by Telegraf
    print_influx_metrics(data)


if __name__ == "__main__":
    main()

Telegraf

Now that the data is being output via the stdout of the script, you will need to have an application read this data and transform it. This could be done in other ways as well, but Telegraf has this function built in already.

Telegraf will be setup to execute the Python script every minute. Then the output will be transformed by defining the output.

Telegraf Configuration

The configuration for this example is as follows:

# Globally set tags that shuld be set to meaningful tags for searching inside of a TSDB
[agent]
hostname = "demo"

[global_tags]
  device = "10.250.0.63"
  region = "midwest"

[[inputs.exec]]
  ## Interval is how often the execution should occur, here every 1 min (60 seconds)
  interval = "60s"
  # Commands to be executed in list format
  # To execute against multiple hosts, add multiple entries within the commands
  commands = [
      "python3 asa_anyconnect_to_telegraf.py --host 10.250.0.63"
  ]

  ## Timeout for each command to complete.
  # Tests in lab environment next to the device with local authentication has been 6 seconds
  timeout = "15s"

  ## Measurement name suffix (for separating different commands)
  name_suffix = "_parsed"

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## More about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "influx"

# Output to Prometheus Metrics format
# Define the listen port for which TCP port the web server will be listening on. Metrics will be
# available at "http://localhost:9222/metrics" in this instance.
# There are two versions of metrics and if `metric_version` is omitted then version 1 is used
[[outputs.prometheus_client]]
  listen = ":9222"
  metric_version = 2

Telegraf Output Example

Here is what the metrics will look like when exposed, without the default Telegraf information metrics.

# HELP asa_parsed_anyconnect_licenses Telegraf collected metric
# TYPE asa_parsed_anyconnect_licenses untyped
asa_parsed_anyconnect_licenses{device="10.250.0.63",host="demo",region="midwest"} 2
# HELP asa_parsed_connected_users Telegraf collected metric
# TYPE asa_parsed_connected_users untyped
asa_parsed_connected_users{device="10.250.0.63",host="demo",region="midwest"} 1

There are two metrics of anyconnect_licenses and connected_users that will get scraped. There are a total of 2 Anyconnect licenses available on this firewall with a single user connected. This can now get scraped by the Prometheus configuration and give insight to your ASA Anyconnect environment.

Prometheus Installation

There are several of options for installing a Prometheus TSDB (Time Series DataBase)including:

Precompiled binaries for Windows, Mac, and Linux
Docker images
Building from source

To get more details on installation options take a look at the Prometheus Github page.

Once installed you can navigate to the Prometheus API query page by going to http://<prometheus_host>:9090. You will then be presented with a search bar. This is where you can start a query for your metric that you wish to graph, such as start typing asa. Prometheus will help with an autocomplete set of options in the search bar. Once you have selected what you wish to query you can select Execute. This will give you a value at this point. To see what the query looks like over time you can select Graph next to the world console to give you a graph over time. Grafana will then use the same query language to add a graph.

Once up and running, you add your Telegraf host to the scraping configuration and Prometheus will start to scrape the HTML page provided and add the associated metrics into its TSDB.

A good video tutorial for getting started with Prometheus queries with network equipment can be found on YouTube from NANOG 77

Grafana Installation

Grafana is the dashboarding component of choice in the open source community. Grafana is able to use several sources to create graphs including modern TSDBs of InfluxDB and Prometheus. With the latest release, Grafana can even use Google Sheets as a datasource.

As you get going with Grafana there are pre-built dashboards available for download.

TYou will want to download Grafana to get started. There are several installation methods available on their download page including options for:

Linux
Windows
Mac
Docker
ARM (Raspberry Pi)

The Prometheus website has an article that is helpful for getting started with your own Prometheus dashboards.

If the video just above this link does not show up, you can see the video on the Network to Code YouTube.

-Josh

Our Top Work from Home Tips

2020-03-24T00:00:00+00:00

Whether you’re a seasoned home office pro or new to working from home, there’s no doubt that working in the place you live can present some unique challenges. Here at Network to Code, we’re no strangers to working from home – we’ve had remote workers across the US and EU for several years now. But given the sudden increase in people working from home, our team wanted to share some of our best tips and tricks for being productive outside a traditional office setting.

Working from home as an extrovert (Stephen Kiely, Network Automation Engineer)

My number one recommendation would be to switch to video calls for all your meetings. Video calls help you engage in conversation more effectively (just make sure to resist the urge to read emails or engage with other distractions during calls).

I also recommend finding a way to engage in “water cooler chat” remotely. Since you can’t get up and have conversations with colleagues, make sure you have an appropriate place where these conversations are accepted (Slack channels are great for this). At Network to Code, we also use virtual donuts – a service that sets you up with a new colleague each week for a conversation. Leave work out of these chats and use this as an opportunity to meet your coworkers!

Communication is key (Josh VanDeraa, Network Automation Engineer)

Working from home means setting boundaries, both with the other people you work with and with your colleagues. If possible, try to have a separate workspace to enforce this boundary. When you are working, you are working – communicating this to those you share a space with is key to your success.

It’s also critical that you speak up on the communication mediums provided to you (Slack, email, telephone, etc.) Make sure to regularly reach out to others and pay attention to building bonds (something those in office buildings may take for granted).

Keep healthy snacks around (Daniel Himes, Network Automation Engineer)

Keep something healthy around that you can snack on. You will, at some point during the day, find yourself drawn to the kitchen. Try to make sure it’s much easier to reach for the carrots than donuts.

Remember to take breaks (Jere Julian, Managing Director & Principle Architect)

Don’t forget to take a break every once in a while – you do it in the office and it’s important to make space for free time at home as well. Something as simple as starting laundry can give you the mental break you need, and it probably takes less time than when someone randomly stops by your desk at the office “for a chat.”

If you normally go out to lunch when in the office, make sure to completely step away when working from home too. Eat lunch in the kitchen instead of your office. Your mind needs that change of pace and scenery.

Separate your home and your office and don’t get too comfortable! (Jeremy Stretch, Sr. Network Automation Engineer)

When you work from home, “work time” and “home time” tend to blend together. It’s crucial to establish a dedicated workspace to help demarcate the two. Dedicating a specific room within your home is ideal, but even if you’re short on space, try carving out a corner of the living room or a specific chair and desk that you reserve for work functions. Be sure to avoid the bedroom: working where you sleep can cause difficulty sleeping.

Also be sure to bear in mind that work is still work. Don’t succumb to the temptation to “settle in” at home. If you’ve just recently made the switch from an office job, be sure to keep up your previous morning routine: shower, get dressed, eat breakfast, etc. It also helps to designate a specific start time each day and abide by it. There are no excuses for being late when you work at home!

Working from home with kids and setting up your home office (Bryan Culver, Sr. Network Automation Engineer)

When it comes to working from home with kids, I can’t say this enough times: SCHEDULE! SCHEDULE! SCHEDULE! Everyone else has already keyed into why to split your time for your own mental health but also this isn’t your every-day working from home. Odds are your kids are home now, and for most people both parents still need to work. This is OK. Communicate with your respective bosses and coworkers what your current working situation is and what you’re willing to do to work around it. But you can’t do that without at least identifying what your children’s needs. Younger ones need more structure than older ones, so work on identifying the times you need to be away from your desk. Everyone is going to have to move around schedules and communicating that before it becomes a problem will help diffuse a lot of frustration.

Setting up an appropriate workspace is also key. Working on top of milk cartons wouldn’t fly in an office, and the same applies at home. There’s no need to break the bank and there are a bunch of DIY standing desk instructions you can find online. Define your space and respect it. Work with your employer to see if they will help purchase or offset some of these costs as well. If your business has a lot of synchronous conversations (calls, video chats, etc.) invest in better equipment. This means a better microphone, better webcam, and more comfortable headphones. If you have a noisier home office setting, I suggest looking at cardioid style microphones which are very directional (I’m personally a fan of this one—they help cut out audio not directly in front of them.

Conclusion

We hope these tips have been helpful for those of you who are working from home. Make sure to take care of yourself and those around you! We’ll close with a tip near and dear to all our hearts: If you have a pet (or office manager as we refer to the around these parts) make sure to schedule regular ~~walks~~ meetings. They’ll love it, you’ll love it.

-The NTC Team

Leveraging Network Automation for Proactive Diagnostics and Visibility

2020-03-18T00:00:00+00:00

By now, COVID-19 has impacted all of us in some way, shape, or form. We truly hope everyone is following the guidelines and recommendations from WHO and their local authorities.

Most of our clients have already implemented a mandatory “work from home” policy for the foreseeable future (all of our employees are fully remote too). What we are seeing is a major impact to the technology infrastructure supporting their businesses. Often it is the front-line workers or customers that are consuming the infrastructure.

For example, after speaking to just a few of our clients, we’ve heard the following:

There is a greater need for visibility and dashboards for all types of traffic, performance, and capacity for remote and VPN infrastructure. There is increased troubleshooting happening in remote access, security, and VPN infrastructure.
There is increased use of VPN concentrators that may not be sized for performance, so they are actively disconnecting users that are idle for more than a certain period of time.
There is increased use of collaboration systems like WebEx Teams, Slack, and MS Teams.
With the general public at home, video games and streaming are putting even more demand on the network infrasructure and supporting systems.

A common approach to these problems we’ve taken in the past (way before COVID-19) is to automate proactive troubleshooting and diagnostics ensuring there isn’t degraded performance and also to ensure there is enough capacity on remote access, security, wireless, and collaboration systems. Automating these types of tasks can easily be integrated to Chat programs as well ensuring anyone on the team can execute the task at hand.

Please do not hesitate to reach out, even if it is for high level guidance. We can try and help. You can email us at info@networktocode.com and get community input on the Network to Code Slack Workspace too.

-Jason

Automation Principles - Atomicity

2020-03-10T00:00:00+00:00

This is part of a series of posts focused on Network Automation Principles.

Atomicity in Computer Science

In simple terms, an atomic operation guarantees that an action succeeds completely or fails completely. Going a step deeper, the included quote provides context and accuracy to describe the principle, in proper terminology:

In concurrent programming, an operation (or set of operations) is atomic, linearizable, indivisible or uninterruptible if it appears to the rest of the system to occur instantaneously.

Atomicity is a guarantee of isolation from concurrent processes. Additionally, atomic operations commonly have a succeed-or-fail definition — they either successfully change the state of the system, or have no apparent effect.

Atomicity is commonly enforced by mutual exclusion, whether at the hardware level building on a cache coherency protocol, or the software level using semaphores or locks. Thus, an atomic operation does not actually occur instantaneously.

In computer science, atomicity is often associated with databases and represents the “A” in ACID, which describes a set of properties for database transactions.

Example

A common example used to illustrate the effect of the principle is transferring money from one account to another. When transferring $100 from Bob’s to Sally’s bank account, it would be disastrous if money was removed from Bob’s account but not deposited to Sally’s account for Bob and Sally, as it would result in a missing $100 to Sally. The transaction would be equally problematic if the money was added to Sally’s account without being removed from Bob’s account for the bank.

Achieving Atomicity

There are several mechanisms to achieve atomicity, but for simplicity, the focus will be on locking. A simplified view of this can be seen as:

Identify memory locations that will be accessed.
Create locks on each location.
Perform a linear set of transactions and save to a new memory location.
Wait for all transactions to complete.
In a single transaction, swap the memory locations.

This strategy would likely result in the inability for the operation to be multi-threaded, but as alluded to, there are other methods.

Atomic Operations in Python

In researching examples, one blog referred to two key explanations. The official Python documentation provides insight into what operations are/are not atomic. Even more to the point is Google’s styling guide on Threading which provides the below insight.

While Python’s built-in data types such as dictionaries appear to have atomic operations, there are corner cases where they aren’t atomic (e.g. if hash or eq are implemented as Python methods) and their atomicity should not be relied upon. Neither should you rely on atomic variable assignment (since this in turn depends on dictionaries).

Rather interesting to note that even variable assignment are not guaranteed to be safe and atomic.

Atomic Operation in Networking

Using a traditional Cisco IOS CLI, applying configurations in a linear fashion can be problematic. Take, for example, creating a VTY ACL. Adding a single line to the new ACL, can enforce an implicit deny and could potentially lock the the operator out. Naturally, the ability to apply configurations atomically has obvious benefits.

Luckily, since at least 2007, there has been a feature called “configure replace”, to replace an entire Cisco IOS configuration. Even better, this is used within the NAPALM Python library. In order to illustrate the effect, the first attempt will fail to push configurations and demonstrate a proper rollback, and the second attempt will work as intended.

Initially, the “intended” configuration will contain this contrived incorrect configuration.

vlan 499
 name printers
!
vlan 5000
 name users

The mistyped VLAN 5000 instead of 500 will be problematic. Let’s observe how NAPALM reacts.

>>> import getpass
>>> from napalm import get_network_driver
>>> driver = get_network_driver('ios')
>>> hostname = 'ios-sw1'
>>> username = 'ntc'
>>> password = getpass.getpass()
Password:
>>> device = driver(hostname, username, password)
>>> device.open()
>>> device.load_replace_candidate(filename='ios-sw1')
>>> device.compare_config()
'+vlan 499\n  +name printers\n+vlan 5000\n +name users'
>>> device.commit_config()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/dist-packages/napalm/ios/ios.py", line 533, in commit_config
    raise ReplaceConfigException(msg)
napalm.base.exceptions.ReplaceConfigException: Candidate config could not be applied
Command rejected: Bad VLAN list - character #5 (EOL) delimits a VLAN
number which is out of the range 1..4094.
Failed to apply command vlan 5000
Aborting Rollback.

Rollback failed.Reverting back to the original configuration: flash:-Dec-14-13-05-59.491-0 ...

Total number of passes: 1
Rollback Done

The original configuration has been successfully restored.

>>>

As you can see, the comparison of the configuration indicates an attempt to add VLANs 499 and 5000. However, when a configuration commit is attempted, the configuration aborts with the following message Command rejected: Bad VLAN list - character #5 (EOL) delimits a VLAN number which is out of the range 1..4094. The failure forces a rollback and ensures that both VLANs did not get created.

Updating the VLAN 5000 to the correct VLAN 500 and rerunning the process results in a successful operation.

>>> device = driver(hostname, username, password)
>>> device.open()
>>> device.load_replace_candidate(filename='ios-sw1')
>>> device.compare_config()
'+vlan 499\n  +name printers\n+vlan 500\n +name users'
>>> device.commit_config()
>>>

This can be further verified on the switch, by viewing the configuration.

vlan 499
 name printers
!
vlan 500
 name users

While the “configure replace” process likely lacks the integrity that a PostgreSQL or MySQL database will have, it is largely effective in creating an atomic operation.

Conclusion

An atomic operation is generally safer and implies your intent–outside of rare use cases. In databases, this is often handled in transactions and in practice creating your own atomic actions often requires locks. Perhaps Yoda can best describe atomicity succinctly with his quote, “do or do not. There is no try.”

-Ken

Using Ansible through a Bastion Host

2020-03-05T00:00:00+00:00

This blog is going to go over the concepts of using Ansible to interact with a private network through a bastion host. The use of bastion hosts is not new to the industry—they have been used for a while by many companies that need to give users access to private networks. Bastion hosts are typically public facing, hardened systems that work as an entrypoint to systems that are behind a firewall or any other restricted locations.

Set up SSH public key authentication with bastion host

When interacting with a bastion host, the recommended first step is to set up SSH public keys and authenticate with the bastion host.

If an SSH key doe not already exist, create it by issuing the command ssh-keygen -t rsa

  root@eb54369adc49:/ntc# ssh-keygen -t rsa
  Generating public/private rsa key pair.
  Enter file in which to save the key (/root/.ssh/id_rsa):
  Created directory '/root/.ssh'.
  Enter passphrase (empty for no passphrase):
  Enter same passphrase again:
  Your identification has been saved in /root/.ssh/id_rsa.
  Your public key has been saved in /root/.ssh/id_rsa.pub.
  The key fingerprint is:
  SHA256:T5YDoF91QrJk2ZN1b7OYq9Pfn3OWVW518iCJHGIQKeQ root@eb54369adc49
  The key's randomart image is:
  +---[RSA 2048]----+
  |   .. +++++oo .  |
  |   ....+++=o . . |
  |    E. .+o + . .o|
  |     . . .o.o =.*|
  |      . S =  + *+|
  |         + .  . =|
  |          . .. .o|
  |           ... o=|
  |           .. .+*|
  +----[SHA256]-----+
  root@eb54369adc49:/ntc#
  root@eb54369adc49:/ntc#

Since this is just for demonstration, I’ve kept everything as default when creating the keys.

Note: Click here if you want to know more about how to set up an SSH public key with more details.

After creating the public key, the ssh-copy-id Linux command can be used to transfer the id_rsa.pub public key to the bastion host.

  root@eb54369adc49:/ntc#
  root@eb54369adc49:/ntc# ssh-copy-id ntc@bastion
  /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
  /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are   already installed
  /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to   install the new keys
  ntc@bastion password:
  
  Number of key(s) added: 1
  
  Now try logging into the machine, with:   "ssh 'ntc@bastion'"
  and check to make sure that only the key(s) you wanted were added.
  
  root@eb54369adc49:/ntc#

Finally, attempt to SSH into the bastion host and make sure it works as expected. The workstation should log in through SSH without having to provide a password.

  root@eb54369adc49:/ntc# ssh ntc@bastion
  Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-59-generic x86_64)
  
   * Documentation:  https://help.ubuntu.com
   * Management:     https://landscape.canonical.com
   * Support:        https://ubuntu.com/advantage
  
  466 packages can be updated.
  321 updates are security updates.
  
  New release '18.04.3 LTS' available.
  Run 'do-release-upgrade' to upgrade to it.
  
  Last login: Mon Jan 13 13:44:58 2020 from 71.52.24.123
  ntc@bastion:~$
  ntc@bastion:~$ exit
  logout
  Connection to bastion closed.
  root@eb54369adc49:/ntc#

There are many different ways of creating a public key and transferring it to the remote host. This transfer can also be handled via Ansible, as the below playbook shows:

---
    - name: Generate Key
      hosts: localhost
      connection: local
      gather_facts: no

      tasks: 
          - name: Generate an OpenSSH keypair with the default values (4096 bits, rsa)
            openssh_keypair:
                path: /root/.ssh/id_rsa
                owner: root
                group: root

          - name: Deploy public key to remote host
            shell: ssh-copy-id ntc@bastion

Setting up Ansible to Interact with a Cisco IOS Device through a Bastion Host

Now that it is possible to authenticate into the bastion host only using SSH keys and not require user/passwords, it’s time to build the Ansible inventory file with the devices that are going to be targeted by Ansible. Before creating the inventory file, notice how the only way to interact with the Cisco csr1 device is through the bastion host.

 # Topology
 ┏━━━━━━━━━━━━━━┓          ┏━━━━━━━━━━━━━━┓          ┏━━━━━━━━━━━━┓         
 ┃  Container   ┃──────────┃ Bastion host ┃──────────┃    csr1    ┃
 ┗━━━━━━━━━━━━━━┛          ┗━━━━━━━━━━━━━━┛          ┗━━━━━━━━━━━━┛

This first test is using the ping command to ping the csr1 device from the local container. The results should come back with 100% packet loss.

  root@eb54369adc49:/ntc#
  root@eb54369adc49:/ntc#
  root@eb54369adc49:/ntc# ping csr1 -c 5
  PING csr1 (54.84.81.49) 56(84) bytes of data.
  
  --- csr1 ping statistics ---
  5 packets transmitted, 0 received, 100% packet loss, time 4176ms

After using the SSH command to access the bastion host, the ping command is used again to show that there is IP connectivity between the bastion host and the networking device.

  root@eb54369adc49:/ntc#
  root@eb54369adc49:/ntc# ssh ntc@bastion
  Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-59-generic x86_64)
  
   * Documentation:  https://help.ubuntu.com
   * Management:     https://landscape.canonical.com
   * Support:        https://ubuntu.com/advantage
  
  466 packages can be updated.
  321 updates are security updates.
  
  New release '18.04.3 LTS' available.
  Run 'do-release-upgrade' to upgrade to it.
  
  Last login: Mon Jan 13 14:38:26 2020 from 71.52.24.123
  ntc@bastion:~$
  ntc@bastion:~$ ping csr1 -c 5
  PING csr1 (10.0.0.51) 56(84) bytes of data.
  64 bytes from csr1 (10.0.0.51): icmp_seq=2 ttl=255 time=2.34 ms
  64 bytes from csr1 (10.0.0.51): icmp_seq=3 ttl=255 time=2.26 ms
  64 bytes from csr1 (10.0.0.51): icmp_seq=4 ttl=255 time=1.99 ms
  64 bytes from csr1 (10.0.0.51): icmp_seq=5 ttl=255 time=2.23 ms
  
  --- csr1 ping statistics ---
  5 packets transmitted, 4 received, 20% packet loss, time 4013ms
  rtt min/avg/max/mdev = 1.994/2.209/2.342/0.138 ms
  ntc@bastion:~$

Now that you can verify direct connectivity from the container to the csr1 device is only possible through the bastion host, the ProxyCommand can be used to gain access to the Cisco device from the container.

Note: ProxyCommand is an SSH builtin command feature to provide support for proxy use cases.

  root@eb54369adc49:/ntc#
  root@eb54369adc49:/ntc# ssh -o ProxyCommand="ssh -W %h:%p ntc@bastion" ntc@csr1
  The authenticity of host 'csr1 (<no hostip for proxy command>)' can't be established.
  RSA key fingerprint is SHA256:BaRERnJrQ4vzALKQELc6lIs7Kujbe3UCPffPcAhcRKg.
  Are you sure you want to continue connecting (yes/no)? yes
  Warning: Permanently added 'csr1' (RSA) to the list of known hosts.
  Password:
  
  
  
  csr1#
  csr1#

Playbook

For this test the ios_command and ios_config will be used. The backup parameter has been enabled to show that the backup files are stored in the local machine and not on the bastion host.

---

    - name: Cisco IOS Playbook
      hosts: ios
      connection: network_cli
      gather_facts: no

      tasks:

        - name: SHOW VERSION
          ios_command:
            commands: show version

        - name: CONFIGURE SNMP COMMUNITY
          ios_config:
            commands: snmp-server community ntc-blog RW
            backup: true

Inventory

For now, only the basic credentials and device type are required to target the device.

  [all:vars]
  ansible_user=ntc
  ansible_ssh_password=ntc123
  
  [ios]
  csr1 
  
  [ios:vars]
  ansible_network_os=ios

Note: There is a dedicated group vars for IOS only for now, since a Juniper device will be tested later in this blog too.

First Execution of Playbook

Similar to the previous tests, the local machine should not be able to communicate with the Cisco device, since it does not have direct access to the private network.

  root@eb54369adc49:ntc$ ansible-playbook -i inventory bastion_playbook.yml
  
  PLAY [Cisco IOS Playbook] ******************************************************************
  
  TASK [SHOW VERSION] ************************************************************************
  fatal: [csr1]: FAILED! => {"changed": false, "msg": "timed out"}
  
  PLAY RECAP *********************************************************************************
  csr1 : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
  
  root@eb54369adc49:ntc$

Executing a Playbook against a Cisco IOS Device through a Bastion Host

To be able to communicate with the Cisco device the ProxyCommand needs to be executed when trying to SSH into the device. To start using the ProxyCommand with Ansible the ansible_ssh_common_args, a variable needs to be added to the inventory file.

  [all:vars]
  ansible_user=ntc
  ansible_ssh_password=ntc123
  
  [ios]
  csr1 
  
  [ios:vars]
  ansible_network_os=ios
  ansible_ssh_common_args=ProxyCommand="ssh -W %h:%p ntc@bastion"

More details can be found in the Ansible documentation about how to use the ansible_ssh_common_args and ProxyCommand. For now, I’ve added a device to the inventory file with the needed credentials, and the ProxyCommand used earlier and tested through the Linux terminal.

The Ansible playbook can now be executed normally using the standard ansible-playbook command. The only modification needed to interact with the networking device was enabling SSH keys and using the ansible_ssh_common_args in the inventory file.

  root@eb54369adc49:/ntc# ansible-playbook -i inventory bastion_playbook.yml -v
  Using /ntc/ansible.cfg as config file
  
  PLAY [Cisco IOS Playbook] ***********************************************************
  
  TASK [SHOW VERSION] ***************************************************************
  
  ok: [csr1] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}
  "changed": false, "stdout": ["Cisco IOS XE Software, 
  Version 16.06.02\nCisco IOS Software [Everest], Virtual XE Software
  (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.6.2, RELEASE
  SOFTWARE (fc2)\nTechnical Support: http://www.cisco.com
  techsupport\nCopyright (c) 1986-2017 by Cisco Systems, Inc
  \nCompiled Wed 01-Nov-17 07:24 by mcpre\n\n\nCisco IOS-X
  software, Copyright (c) 2005-2017 by cisco Systems, Inc
  \nAll rights reserved.  Certain components of Cisco IOS-XE
  software are\nlicensed under the GNU General Public License" ...omitted]]}
  
  TASK [CONFIGURE SNMP COMMUNITY] ***********************************************************
  changed: [csr1] => {"backup_path": "/ntc/backup/csr1_config.2020-01-13@22:02:16", "banners": 
  {}, "changed": true, "commands": ["snmp-server community ntc-blog RW"],
  "date": "2020-01-13", "filename": "csr1_config.2020-01-13@22:02:16", "shortname": "/ntc
  backup/csr1_config", "time": "22:02:16", "updates": ["snmp-server community ntc-blog RW"]}
  
  PLAY RECAP *********************************************************************************
  csr1    : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Setting up Ansible to Interact with a Juniper vMX Device using NETCONF through a Bastion Host

To interact with a Juniper device via a bastion host that uses NETCONF, a different variable is required and a configuration file needs to be built. This link gives some details on different ways to enable the feature, but for this test I’ll be adding the ansible_netconf_ssh_config=./ntc/netconf_proxy_config.cfg variable with the path to my netconf_proxy_config.cfg file.

Note: Even thought the Ansible documentation is not entirely clear why ansible_ssh_common_args does not work with NETCONF devices, maybe this link can provide insight to that question.

  [all:vars]
  ansible_user=ntc
  ansible_ssh_password=ntc123
  
  [ios]
  csr1 
  
  [ios:vars]
  ansible_network_os=ios
  ansible_ssh_common_args=ProxyCommand="ssh -W %h:%p ntc@bastion"
  
  [vmx]
  vmx1
  
  [vmx:vars]
  ansible_network_os=junos
  ansible_netconf_ssh_config=./ntc/netconf_proxy_config.cfg

Setting up the netconf_proxy_config.cfg and ansible.cfg file

The Ansible documentation shows how to build a proper netconf_proxy_config.cfg file. However, since I’m only testing with a single device, the only requirement is the ProxyCommand that will establish the SSH connection with the bastion host and the Juniper device.

ProxyCommand ssh -W %h:%p ntc@bastion

The next step will be to add a second play using the correct modules to interact with a Juniper device.

Executing a Playbook against a Juniper vMX Device using NETCONF through a Bastion Host

The example playbook has been extended to include not only the original Cisco IOS tasks, but an additional play to run the equivalent on Juniper devices. The goal is to run a show version and make a configuration change to add an SNMP community to the device configuration.

---

    - name: Cisco IOS Playbook
      hosts: ios
      connection: network_cli
      gather_facts: no
      tags: cisco

      tasks:

        - name: SHOW VERSION
          ios_command:
            commands: show version

        - name: CONFIGURE SNMP COMMUNITY
          ios_config:
            commands: snmp-server community ntc-blog RW
            backup: true

    - name: Juniper vMX Playbook
      hosts: vmx
      connection: netconf
      gather_facts: no
      tags: juniper
      
      tasks:

        - name: SHOW VERSION
          junos_command:
            commands: show version

        - name: CONFIGURE SNMP COMMUNITY
          junos_config:
            commands: set snmp community ntc-blog authorization read-write

After building the playbook, it’s time to execute it and see the results. In this case I ran the playbook and added a tag so only the Juniper play gets executed.

  root@eb54369adc49:/ntc# ansible-playbook -i inventory bastion_playbook.yml --tags juniper -v
  Using /ntc/ansible.cfg as config file
  
  PLAY [Cisco IOS Playbook] *******************************************************************
  
  PLAY [Juniper IOS Playbook] *****************************************************************
  
  TASK [SHOW VERSION] *************************************************************************
  ok: [vmx1] => {"changed": false, "stdout": ["Hostname: vmx1\nModel: vmx\nJunos: 15.1F4.15
  \nJUNOS Base OS boot [15.1F4.15]\nJUNOS Base OS Software Suite [15.1F4.15]\nJUNOS Crypto 
  Software Suite [15.1F4.15]\nJUNOS OnlineDocumentation [15.1F4.15]\nJUNOS 64-bit Kernel Software 
  Suite [15.1F4.15]\nJUNOS Routing Software Suite [15.1F4.15]\nJUNOS Runtime Software Suite [15.1F4.  15]
  \nJUNOS 64-bit Runtime Software Suite [15.1F4.15] \nJUNOS Services AACL PIC package [15.1F4.15]  \nJUNOS 
  Services Application Level Gateway (xlp64) [15.1F4.15]\nJUNOS Services ..omitted"]]}
  
  TASK [CONFIGURE SNMP COMMUNITY] *************************************************************
  changed: [vmx1] => {"changed": true}
  
  PLAY RECAP **********************************************************************************
  vmx1  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Summary

In order to use Ansible through a bastion host with core modules, SSH keys are recommended. When interacting with devices that support NETCONF, the ansible_netconf_ssh_config variable is required to be enabled and for traditional SSH devices the ansible_ssh_common_args is required.

Other third party modules like ntc_show_command or NAPALM can also be used but will not be able to support ansible_ssh_common_args or ansible_netconf_ssh_config. In this case, the delegate_to argument will be needed to use a third party module with a jump-host/bastion-host.

-Hector

Python Objects

2020-02-25T00:00:00+00:00

“Everything in Python is an object” has been a common refrain among the Python community. What is an object? A simple definition is: An object is an instance of a class. For example, strings are instances of the str class.

# Creating a string variable
>>> device_type = "switch"
# device_type is an instance of the str class
>>> type(device_type)
<class 'str'>
>>>

Classes themselves are instances of type.

# Creating a class object normally
>>> class Cisco:
...     vendor = "cisco"
...
>>> Cisco
<class '__main__.Cisco'>
>>> Cisco.vendor
'cisco'
>>> isinstance(Cisco, type)
True
# Creating a class object explicitly from type
>>> Arista = type("Arista", (), {"vendor": "arista"})
>>> Arista
<class '__main__.Arista'>
>>> Arista.vendor
'arista'
>>>

In addition to being an instance of a class, objects consist of attributes defining state and behavior. It is common to refer to state attributes as data attributes or variables, and behaviors are called methods. The Python builtin classes generally have many methods, but do not have data attributes. The str class provides methods for:

Manipulating the string (capitalize, format, etc.)
Finding information about the string (startswith, isnumeric, etc.)
Segmenting the string (partition, split, etc.)

>>> example = "this is a string"
# Capitalize the first letter of each word 
>>> example.title()
'This Is A String'
# Count the number of times the letter i appears
>>> example.count("i")
3
# Convert the string into a list of words
>>> example.split()
["this", "is", "a", "string"]

It is more common for custom classes to have data attributes that store information about the objects they imitate. For an object representing a switch, some example data attributes might be: modules, uptime, and reachability. Some example methods might be: reboot, and ping.

>>> switch = Arista(hostname="nyc-hq-rt1")
# Example of a data attribute
>>> switch.uptime
'4 years, 3 days'
# Example of a method attribute
>>> ping_stats = switch.ping("10.1.1.1", 5)
'pinging 10.1.1.1 times 5'
>>> ping_stats
{
    'attempts': 5,
    'success': 5,
    'fail': 0,
    'messages': ['pinged 10.1.1.1 in 0.21ms', 'pinged 10.1.1.1 in 0.14ms', ...]
}
>>>

Data Attributes

Data attributes are distinguished between class variables and instance variables. Class variables assign values that are the same for all objects of the class. Instance variables assign values for an instance of the class. Class variables are generally used for metadata fields, such as vendor in the first example. Instance variables contain more interesting information that the object methods act upon. Using the previous example, the reachability attribute might be used in a report to identify all switches that are unreachable. The Cisco class above is redefined to create instance variables upon initialization for hostname, connection status, and reachability information.

class Cisco:
    # Class variable
    vendor = "cisco"

    def __init__(self, hostname):
        # Instance variables
        self.hostname = hostname
        self.connected = False
        ping_reachability = self.ping(hostname, 2)
        if ping_reachability["success"]:
            self.connect()
            if self.connected:
                self.reachability = "authorized"
            else:
                self.reachability = "unauthorized"
        else:
            self.reachability = "unreachable"

    def connect(self):
        ...
        self.connected = True if connected else False

    def ping(self, destination, count):
        ...
        return {
            "attempts": count,
            "success": success_count,
            "fail": fail_count,
            "messages": [message for message in ping_results],
        }

Python uses dictionaries to store class and instance variables. Class variables share the same dictionary across all instances of the class, and instance variables are stored in a unique dictionary per instance. The class dict is stored in <class>.__dict__, and this is referenced on an instance with <obj>.__class__.__dict__. The instance dict is stored in <obj>.__dict__. Here are some examples showing how these two dictionaries behave.

# View the class dict
>>> Cisco.__dict__
mappingproxy({'vendor': 'cisco', ...})
# Create two objects of the Cisco class
>>> rt1 = Cisco("nyc-hq-rt1")
>>> rt2 = Cisco("nyc-dc-rt1")
# Viewing the instance dict
>>> rt1.__dict__
{'hostname': 'nyc-hq-rt1', 'connected': True, 'reachability': 'authorized'}
# Reachability is an instance variable and can have different values across objects
>>> rt1.reachability
'authorized'
>>> rt2.reachability
'unreachable'
# The class dict is referenced within instances
>>> Cisco.__dict__ == rt1.__class__.__dict__
True
# Changing the vendor on the class is reflected on all instances
>>> Cisco.vendor = "CSCO"
>>> rt1.vendor
'CSCO'
>>> rt2.vendor
'CSCO'
>>>

Attribute Precedence

This distinction between class and instance variables is important for understanding Python’s attribute access behavior. The standard attribute lookup uses the following precedence until the attribute is found, or an Error is raised:

Attributes defined on the instance
Attributes defined on the class
Attributes defined on inherited classes

Practically, this means that overriding a class attribute on an instance will only affect the single instance and not other instances.

>>> rt1 = Cisco("nyc-hq-rt1")
>>> rt2 = Cisco("nyc-dc-rt1")
# rt1 and rt2 both have a value of "cisco" for the vendor attribute
>>> rt1.vendor == rt2.vendor == "cisco"
True
# rt1's instance dict does not have an attribute for `vendor`
>>> rt1.__dict__
{'hostname': 'nyc-hq-rt1', 'connected': True, 'reachability': 'authorized'}
>>> rt1.vendor = "Cisco"
>>> rt1.vendor
'Cisco'
# rt1's instance dict was updated with a `vendor` attribute
>>> rt1.__dict__
{'hostname': 'nyc-hq-rt1', 'connected': True, 'reachability': 'authorized', 'vendor': 'Cisco'}
# rt2's vendor attribute remains unchanged
>>> rt2.vendor
'cisco'

Mutable Attributes

The above demonstrates the built-in safety provided when naming instance variables collides with class variables. However, this behavior might have surprising results when class variables point to mutable objects (such as lists). The Arista class is rewritten below to have a class variable defining the modules, with the value using a list for modular chassis.

class Arista:
    vendor = "arista"
    modules = ["7500E-48S", "DCS-7500E-SUP"]

>>> rt1 = Arista("nyc-hq-rt1")
>>> rt2 = Arista("nyc-dc-rt1")
>>> rt1.modules
["7500E-48S", "DCS-7500E-SUP"]
# Appending to rt2's `modules` attribute affects rt1
>>> rt2.modules.append("7500E-72S")
>>> rt1.modules
["7500E-48S", "DCS-7500E-SUP", "7500E-72S"]
# Using the `+=` operator on rt2's `modules` attributed affects rt1
>>> rt2.modules += ["7500E-36Q"]
>>> rt1.modules
["7500E-48S", "DCS-7500E-SUP", "7500E-72S", "7500E-36Q"]
# Both modifications to rt2's `modules` attribute affect the class attribute
>>> Arista.modules
["7500E-48S", "DCS-7500E-SUP", "7500E-72S", "7500E-36Q"]
# All references to `modules` are the same shared object
>>> Arista.modules is rt1.modulues is rt2.modules
True
>>>

Conclusion

Understanding how objects work is critical to working with Python. Fundamental to working with objects is managing and using data attributes. Data attributes can be created on the class, or on each instance of the class. Instance level attributes have a higher precedence than class level attributes. Finally, when creating attributes on a class, it is important to consider how they will be used, and that mutable attributes can lead to unexpected behavior.

-Jacob

Backing up device configurations using Nornir and Ansible

2020-02-20T00:00:00+00:00

In this blog, I’m going to go over how to run a configuration backup on a Cisco IOS device using Ansible and Nornir. I’ve heard many people say, “well which tool should I use to run against my infrastructure?” That’s a great question but it depends :). There are many factors that could affect the decision of using a specific tool i.e skill level, time, access to tools, team preference…the list goes on.

Hopefully, the examples shown here can help you determine which tool best fits the need. I’m not going deep into the details on the different technologies or explaining all the components of each of the files, but I’ll provide some links to learn more about them if needed.

Installation

If you want to follow along with the blog, Nornir and Ansible need to be installed in the machine locally or you will need to use tools that isolate dependencies like Docker and virtual environments. I ended up using Docker so if you have Docker installed in your machine and know how to use it, then feel free to use the Dockerfile below:

FROM python:3.7-stretch

WORKDIR /ntc

RUN apt-get update \
&& apt-get install tree
RUN pip install black \
ansible \
nornir

After adding and saving the commands to the Dockerfile run the following commands to build the container image and start running the container.

docker build -t nornir .
docker run -it --name nornir_ansible -v ${PWD}:/ntc nornir_ansible:latest /bin/bash

Backing up with Nornir

Nornir is an automation tool used to interact with networking devices. This tool is a bit different compared to other tools in the way that you write your own Python code to control the automation. For example, Ansible is written in Python but uses its own DSL which you use to describe what you want to have done. Nornir does not need a DSL instead, you can write everything in Python.

Configuration File

Now that everything has been installed, the first file to create is the Nornir configuration file. This file is similar to the ansible.cfg file which helps set all the system defaults, but in this case it’s going to be used to find the inventory file. Click on the following links to learn more about Nornir and Ansible configuration files:

Create a file called nornir_config.yml and inside the file store the following content:

---
inventory:
    plugin: nornir.plugins.inventory.ansible.AnsibleInventory
    options:
        hostsfile: "./inventory"

Note: Since this container has access to the local machine, any tool or editors can be used with the same files the container is able to see. I’ll be using Visual Studio Code in my local machine.

Inside the container run the cat command to make sure changes took place in the file.

root@c2e446b364a8:/ntc# cat nornir_config.yml
---
inventory:
    plugin: nornir.plugins.inventory.ansible.AnsibleInventory
    options:
        hostsfile: "./inventory"

Note: The plugin used here AnsibleInventory allows us to source an inventory file that would normally be consumed by Ansible. There are other plugins that can be used like simple, NSOT and NetBox.

Inventory File

The inventory file built here will be used for both Ansible and Nornir. The main differences here will be the hostvars names. For now Nornir will be using hostname, platform, username and password. Later, when the Ansible section comes, up a few other variables will be added. Create a file called inventory and place it in the same directory path as the nornir_config.yml file. Save it and run the cat command to make sure it saved properly and it’s seen by the container.

root@c2e446b364a8:/ntc# cat inventory
[all]
csr1 hostname=csr1 platform=ios username=ntc_user password=pass123

Click on the following links to learn more about Nornir and Ansible inventory:

Backup Python Script

Now that the configuration and inventory file has been created it’s time to build the script that will backup the device configuration. The script will consist of functions that will build the necessary directory, backup files, and two different functions using the netmiko and napalm modules to backup the device configurations.

The first step to building the script is to import all the needed libraries. In this case, the libraries needed are:

os: This library is going to be used to help interact with the local environment and build the backup directory and file names for the configurations.
nornir: Nornir is an automation framework written in Python to be used with Python.
- InitNornir: Object used to provide the configuration file containing system defaults. In this case, it’s being used to provide the inventory file.
- netmiko_send_command: It’s a netmiko function used to send arbitrary commands to networking devices.
- napalm_get: Function that comes from the NAPALM library to get facts from the device. In this case, it will be used to retrieve the device configuration.

import os
import logging
from nornir import InitNornir
from nornir.plugins.tasks.networking import netmiko_send_command, napalm_get

Two global variables need to be created. The first one, BACKUP_DIR, is the directory name used to store the backed up files and the second,nr, is used to carry the data stored in the nornir_config.yml file. In this case, it will contain the inventory file data.

BACKUP_DIR = "backups/"

nr = InitNornir(config_file="./nornir_config.yml")

The create_backups_dir is used to create a local directory called backups and it’s using the os library to create a directory with the name stored in the BACKUP_DIR variable.

def create_backups_dir():
    if not os.path.exists(BACKUP_DIR):
        os.mkdir(BACKUP_DIR)

The save_config_to_file function is used to create the backed up device configuration files inside the backups directory. Notice the parameters used will take as input method, hostname and config, these parameters will get their data from the get_netmiko_backups() and get_napalm_backups() functions that will be created in the next few steps.

def save_config_to_file(method, hostname, config):
    filename =  f"{hostname}-{method}.cfg"
    with open(os.path.join(BACKUP_DIR, filename), "w") as f:
        f.write(config)

The get_netmiko_backups function will run the netmiko_send_command module to send a “show run” command to the network device stored in the nr variable and it will be stored in the backups_results variable.

def get_netmiko_backups():
    backup_results = nr.run(
        task=netmiko_send_command, 
        command_string="show run"
        )

    for hostname in backup_results:
        save_config_to_file(
            method="netmiko",
            hostname=hostname,
            config=backup_results[hostname][0].result,
        )

The backups_results variable will end up with a value that stores the hostname and configuration as a result looking something like this:

>>> print(backup_results)
AggregatedResult (netmiko_send_command): {'csr1': MultiResult: [Result: "netmiko_send_command"]}
>>>
>>>

Note: AggregatedResult is a dict-like object that aggregates the results for all devices. You can access each individual result by doing my_aggr_result["hostname_of_device"]

The data inside backups_results can be extracted by using a for loop and stored in hostname. The data iterated through with the for loop will return the hostname information and backup configuration.

The get_napalm_backups function will be built the same way, except this time it will be using the napalm_get module to extract the device configuration.

def get_napalm_backups():
    backup_results = nr.run(task=napalm_get, getters=["config"])

    for hostname in backup_results:
        config = backup_results[hostname][0].result["config"]["startup"]
        save_config_to_file(method="napalm", hostname=hostname, config=config)

The last function main() will execute all the functions and the code entry point is added to start the script when executed.

def main():
    create_backups_dir()
    get_netmiko_backups()
    get_napalm_backups()

if __name__ == "__main__":
    main()

Once the script has been built, execute the command black to clean up and make sure all formatting is correct and use the command cat to make sure everything has been saved correctly. Click on the following link to learn more about black.

root@c2e446b364a8:/ntc# black backups.py
All done! ✨ 🍰 ✨
1 file left unchanged.

root@6b91330f9674:/ntc# cat backups.py
import os
from nornir import InitNornir
from nornir.plugins.tasks.networking import netmiko_send_command, napalm_get

BACKUP_DIR = "backups/"

nr = InitNornir(config_file="./nornir_config.yml")


def create_backups_dir():
    if not os.path.exists(BACKUP_DIR):
        os.mkdir(BACKUP_DIR)


def save_config_to_file(method, hostname, config):
    filename =  f"{hostname}-{method}.cfg"
    with open(os.path.join(BACKUP_DIR, hostname), "w") as f:
        f.write(config)


def get_netmiko_backups():
    backup_results = nr.run(
        task=netmiko_send_command, 
        command_string="show run"
        )

    for hostname in backup_results:
        save_config_to_file(
            method="netmiko",
            hostname=hostname,
            config=backup_results[hostname][0].result,
        )


def get_napalm_backups():
    backup_results = nr.run(task=napalm_get, getters=["config"])

    for hostname in backup_results:
        config = backup_results[hostname][0].result["config"]["startup"]
        save_config_to_file(method="napalm", hostname=hostname, config=config)


def main():
    create_backups_dir()
    get_netmiko_backups()
    get_napalm_backups()


if __name__ == "__main__":
    main()

Running the Python Script

Now that the script is complete, it’s time to execute it by using the Python command and file name right next to it python backups.py.

root@c2e446b364a8:/ntc# python backups.py
root@c2e446b364a8:/ntc#

The tree command can be used to view the configuration files stored in the directory backups configuration files stored inside of it with the specified file names built in the save_config_to_file function.

root@c2e446b364a8:/ntc# tree
.
├── Dockerfile
├── backups
│   ├── csr1-napalm.cfg
│   └── csr1-netmiko.cfg
├── backups.py
├── inventory
├── nornir.log
└── nornir_config.yml

1 directory, 7 files

Note: Take a look at the backed up configuration files using the cat command or an editor of choice. They are left off from this as they take up a lot of screen space.

By default, Nornir automatically configures logging when InitNornir is called and a file is created locally that can be viewed what functions where ran that belong to the Nornir library.

root@c2e446b364a8:/ntc# cat nornir.log
2020-01-02 21:33:26,519 -  nornir.core -     INFO -        run() - Running task 'netmiko_send_command' with args {'command_string': 'show run'} on 1 hosts
2020-01-02 21:33:32,134 -  nornir.core -     INFO -        run() - Running task 'napalm_get' with args {'getters': ['config']} on 1 hosts

Backing up with Ansible

Ansible is a configuration management tool originally created to interact with Linux servers, eventually it started gaining traction in the networking industry to manage networking device configurations. Ansible was built using Python but it uses YAML to configure all the automation tasks. The idea is to simplify the work that it takes to start automating.

Configuration file

The ansible.cfg configuration file is used to set some of the system default values. Create the ansible.cfg file in the root of the directory next to all the other files created previously. Then use the cat command to make sure everything was stored correctly.

inventory: Sets the location of the inventory file, by default it will look for it in etc/ansible/hosts or while running the playbook the -i flag can be used to point to the location of the file. In this case, it is pointed to the inventory file we will create in the next step within the same directory.
host_key_checking: This has been disabled to prevent host key checking and prevent issues when running the playbook.
interpreter_python: Has been set in the ansible.cfg file to prevent a new warning added to the new version of Ansible and specify what interpreter to use.
gathering: Is set to smart to gather facts by default, but don’t regather if already gathered.

root@c2e446b364a8:/ntc# cat ansible.cfg
[defaults]
inventory      = ./inventory
host_key_checking = False
interpreter_python = /usr/bin/python
gathering = smart

Inventory

Edit the same inventory file that was used for Nornir, except this time add the following variables for Ansible.

ansible_user=ntc_user
ansible_password=pass123
ansible_network_os=ios

Again use the cat command to make sure the changes took effect.

root@c2e446b364a8:/ntc# cat inventory
[all]
csr1 hostname=csr1 platform=ios username=ntc_user password=pass123 ansible_user=ntc_user ansible_password=pass123 ansible_network_os=ios

Playbook

The first part to the Playbook is the play definition. The play definition will contain a list of keys such as:

name: Any arbitrary name for the Playbook.
hosts: The devices that will be targeted in the scope of the play.
connection: Type of connection. In this case network_cli which is basically SSH.
tasks: The key that contains all the automation steps that belong to the play.
gather_facts: Before Ansible version 2.9, this key was normally disabled when interacting with networking devices. Now, it has the ability to gather device facts at the play definition level rather than at the task level.

The second part of the Playbook nested under the tasks key will have a list of key:value pair tasks.

name: Optional but recommended key that can have any arbitrary name for the task.
ios_config: This is the Python module used to run configuration commands against ios devices.
backup: Module parameter used to backup the device configuration. By default, if the backup directory is not created then it will create one in the same location as the Ansible playbook and store the configuration files in the backup directory.

Note: A recent parameter was added as backup_options with sub-options to change the directory path rather than the default which is backup and the ability to change the filename of the device configuration. In this case, the default value is set with this format <hostname>_config.<current-date>@<current-time>

Create the Ansible Playbook and call it backup_playbook.yml. Use the cat command to make sure the content is stored correctly:

root@c2e446b364a8:/ntc# cat backup_playbook.yml
---
    - name: BACKUP PLAYBOOK
      hosts: csr1
      connection: network_cli
      gather_facts: yes
      tasks:

        - name: BACKUP USING IOS_CONFIG
          ios_config:
            backup: yes

Running the Playbook

To run the Ansible Playbook use the command ansible-playbook backup_playbook.yml. The output is shown below:

Note: There is an extra task that says Gathering Facts which is a new feature added to Ansible that now gathers facts from networking devices at the play definition level. Take a look at this blog Ansible gather_facts for Networking Devices to learn a little more about it.

root@c2e446b364a8:/ntc# ansible-playbook backup_playbook.yml

PLAY [BACKUP PLAYBOOK] **********************************************************************************

TASK [Gathering Facts] **********************************************************************************
[WARNING]: Ignoring timeout(10) for ios_facts

ok: [csr1]

TASK [BACKUP USING IOS_CONFIG] ***************************************************************************
changed: [csr1]

PLAY RECAP ***********************************************************************************************
csr1                 : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Use the tree command to view the content inside and notice how the device configuration is stored in the backup directory:

root@c2e446b364a8:/ntc# tree
.
├── Dockerfile
├── ansible.cfg
├── backup
│   └── csr1_config.2020-01-03@15:43:16
├── backup_playbook.yml
├── backups
│   ├── csr1-napalm.cfg
│   └── csr1-netmiko.cfg
├── backups.py
├── inventory
├── nornir.log
└── nornir_config.yml

2 directories, 10 files

Backing up with ios_facts

There’s another way of backing up the device configurations by using ios_facts, except this time rather than creating a new task to gather facts we can leverage the gathering of facts that takes place at the play definition level and add a task that will create a new directory to store the device configurations and another task to copy the content from ansible_facts as a configuration file.

---
    - name: BACKUP PLAYBOOK
      hosts: csr1
      connection: network_cli
      gather_facts: yes
      tasks:

        - name: BACKUP USING IOS_CONFIG
          ios_config:
            backup: yes
            
        - name: CREATE BACKUP DIRECTORY
          file:
            path: ./ansible_backup
            state: directory
       
        - name: BACKUP USING IOS_FACTS
          copy:
            content: "{{ ansible_net_config }}"
            dest: ./ansible_backup/{{ inventory_hostname }}_config.{{ now() }}.cfg

In the playbook above, the copy module is used to copy content stored in the ansible_net_config key that is found inside ansible_facts. Any net_* key under ansible_facts can be accessed directly by using the ansible_net_* key rather than accessing the nested data structure. Now the new tasks have been added the playbook can be run again.

root@c2e446b364a8:/ntc# ansible-playbook backup_playbook.yml

PLAY [BACKUP PLAYBOOK] **********************************************************************************

TASK [Gathering Facts] **********************************************************************************
[WARNING]: Ignoring timeout(10) for ios_facts

ok: [csr1]

TASK [BACKUP USING IOS_CONFIG] ***************************************************************************
changed: [csr1]

TASK [CREATE BACKUP DIRECTORY] ****************************************************************************
changed: [csr1]

TASK [BACKUP USING IOS_FACTS] *****************************************************************************
changed: [csr1]

PLAY RECAP *************************************************************************************************
csr1            : ok=4    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Notice how every time the first task is run a new backup configuration is created since it’s been run with a different time stamp. The same can be done with the ios_facts backup method.

root@c2e446b364a8:/ntc# tree
.
├── Dockerfile
├── ansible.cfg
├── ansible_backup
│   └── csr1_config.2020-01-03 19:01:43.277008.cfg
├── backup
│   ├── csr1_config.2020-01-03@15:43:16
│   └── csr1_config.2020-01-03@19:01:42
├── backup_playbook.yml
├── backups
│   ├── csr1-napalm.cfg
│   └── csr1-netmiko.cfg
├── backups.py
├── csr1_config.2020-01-03@15:44:07
├── inventory
├── nornir.log
└── nornir_config.yml

3 directories, 13 files

Summary

There are many automation tools and modules that can help automate the backup of device configurations. The ones shown in this blog are some of the basic ways to do it, which ever method you choose will do the job. There can be different modifications that can be done to the Nornir script or even the Ansible playbook to better fit your application and hopefully what I have shown here can give you a start on what tool to use.

-Hector

My first REST API work with Python

2020-02-11T00:00:00+00:00

I come from a network engineering background, so when I started my network automation journey, it felt like I was banging my head against new technology with very little help. When I first started using APIs, our network team had just started with this brand new ACI “thing,” and I couldn’t find a well written “What is an API” blog with code I would call “appropriate.” In this blog, I will attempt to provide that for those of you that are just getting started with APIs. If you have been using APIs for a while, this may not be as useful.

One of the hardest parts of writing this was choosing an API to discuss. Some of the things I really wanted from an API were:

A Swagger page: I always thought of this as a webpage that has the API laid out nicely and gives the opportunity to run the API calls directly on it. This is more technically known as an Open API spec.
A platform that was common in the networking world that people would be likely to run across.
A platform that was easily available and didn’t require you to build a new system.

Sadly, this combo doesn’t seem to exist. In the end, I decided to go over 2 APIs:

NetBox
- “An open source web application designed to help manage and document computer networks.”
- Documentation that is easy for a beginner with its Swagger page.
- https://netbox.readthedocs.io/en/stable/
ACI
- A Network policy-based automation model for data centers.
- Cisco DevNet has an always on sandbox where you can access a working ACI environment at any time.
- When I was working on this API about a year ago, I found the documentation hard to navigate. I can see how it would work well for someone with more experience, but it didn’t work well for me at the time, and I haven’t kept up to date to know if their documentation has gotten more friendly.
- https://learningnetwork.cisco.com/docs/DOC-32331

REST API Core Concepts

While I’m not going to go to in depth, I want to walk through a few examples to give you the general idea of how a REST API works.

There are two main methods of authentication. One where the API is provided a username and password and the system gives back a temporary token. This process is often used in systems where you are running a system such as ACI. In the other method, a token you generate from a UI serves as your authentication. These are often used for web applications such as Slack, Twitter, Facebook, and fortunately NetBox, so we can see how to do that.

The general idea behind the REST API is you are going to an URL rather than typing in commands. Let’s look at this like a Cisco CLI. For example, your command might be show run interface gi1. The equivalent URL might be something like device.com/api/conf/interface/gi1 so to work with gi2 the URL would just change device.com/api/conf/interface/gi2.

The main types of calls are below.

GET
- Pulling data.
- You can’t really cause an issue doing this as long as you don’t pull too much at one time, e.g. pulling 1,000,000 records may bog down the system.
- Equivalent of a show command.
PUT/POST/PATCH
- Changing or creating data.
- Equivalent of configuration changes.
DELETE
- Delete data.

With the same examples from above, we will do a GET on device.com/api/conf/interface/gi1. It returns the dictionary below:

{
    description: "Disconnected"
    status: "shut"
    mode: "access"
    vlans: 110
}

We now know the description on the interface, the VLAN, and that it is shut down. Let’s say we wanted to make a change, just update it as below and PUT it to the same URL.

{
    description: "VM Server Bob"
    status: "no shut"
    mode: "trunk"
    vlans: "10-100"
}

The port is now up, with a new description, as a trunk allowing VLANs 10-100 though. While not quite as easy to understand the url above this URL https://device.com/restconf/data/Cisco-IOS-XE-native:native/interface=GigabitEthernet/1 would work for the RESTCONF API for IOS-XE devices and has an easy to read URL.

NetBox

I am running an instance of NetBox in my lab, the IP is 192.168.0.159, so that’s what all my examples will be of. It’s not hardened or using https, so all of my examples are using http on port 8000. Your box may vary, adjust accordingly.

First, let’s build and grab a token from http://192.168.0.159:8000/user/api-tokens/.

Next, we’ll look at the swagger documentation page at http://192.168.0.159:8000/api/docs/. Login to the session

and find an action to do, (e.g. get a device from DCIM,) select that, then select “Try it out.”

We can pull all devices if we were to look at another option, but the API call gives us lots of options. A screenshot would be overly busy, so I am just going to pull one device in this example. Please note the required tag.

The API call gives us all the data for that we asked for. Please pay special note to the cURL script. That is the call it made to the NetBox API. It sent a GET request to http://192.168.0.159:8000/api/dcim/devices/5/ please note the initial line we chose was /dcim/devices/{id}/

The cURL script is running an HTTP request to the URL listed, and you could run the same request on nearly all systems. For example, right now you can run “curl http://www.google.com” and make the HTTP call to Google. One of the really nice things about that cURL script is we can go to https://curl.trillworks.com, paste in the cURL script, and it will give us the Python equivalent. NOTE that if the request has the token in it you should be careful. Swap out the token before posting it to the website unless you like giving admin credentials to strangers.

Honestly there’s not much to be scared of here. Just working though Swagger and putting a few functions together from what you get from the website. Super easy… Now let’s look at ACI.

ACI

In this example, the Cisco DevNet ACI emulator will be used. Set the username, password, and the base URL. This URL will be in every call made. To compare to the NetBox example, the base_url would have been http://192.168.0.159:8000/api.

username = "admin"
password = "ciscopsdt"
base_url = "https://sandboxapicdc.cisco.com/api/"

The login script is honestly one of the most difficult parts of the whole process, as documentation is often sparse and there is no industry standard for field names across REST APIs. The process involves simply logging in, getting a token, and returning it. This is how many APIs work, and there are generally small differences for every API. Most of this was from sample code obtained from DevNet a few years ago.

def get_token(username, password, base_url):
    # Disable warning messages that come with using self signed certs
    requests.packages.urllib3.disable_warnings()
    # create credentials structure
    name_pwd = {"aaaUser": {"attributes": {"name": username, "pwd": password}}}
    json_credentials = json.dumps(name_pwd)
    # Create the full URL we need in order to login
    # You'll find we do this a lot
    login_url = base_url + "aaaLogin.json"
    # log in to API
    # requests is a library that can do HTTP transactions, in this case we POSTing up (Telling it) our
    # credentials to get the token back
    # Verify=False ignores self signed certs, but would kick off errors if we hadn't disabled them up above
    post_response = requests.post(login_url, data=json_credentials, verify=False)
    # getting the token from login response structure
    auth = json.loads(post_response.text)
    login_attributes = auth["imdata"][0]["aaaLogin"]["attributes"]
    auth_token = login_attributes["token"]
    # create token dictionary that we will use for authentication from here on out for auth cookie
    cookies = {}
    cookies["APIC-Cookie"] = auth_token
    return cookies

Let’s review the documentation for the ACI API. Once again, this is how it worked back in late 2018, the documentation may have changed since. Access the API inspector,

then proceed to the interesting area in the GUI, paying close attention to your last click. The example will return with the switch information, so I went to Fabric (top tap) -> Inventory -> POD1 (expanded). Sadly, the URL alone doesn't always give you exactly what you need. In this example I had to cut off the &subscription=yes` section. Often, determinng the URL can be the hardest part.

You can also see that the URL is asking for JSON in the pod-1.json? section. For ACI, it will return either XML or JSON. Personally, I find JSON easier to deal with in Python. Using the JSON library it’s not hard to convert JSON to a dictionary; https://www.w3schools.com/python/python_json.asp does a better job at explaining the details. Let’s take a look at this process.

import json
import requests
from pprint import pprint

def pull_pod1(cookies, base_url):
    sensor_url = (
        base_url + '/node/mo/topology/pod-1.json?query-target=children&target-subtree-class=fabricNode&query-target-filter=and(not(wcard(fabricNode.dn,%22__ui_%22)),and(ne(fabricNode.role,"controller")))'
    )
    get_response = requests.get(sensor_url, cookies=cookies, verify=False)
    return get_response.json()


username = "admin"
password = "ciscopsdt"
base_url = "https://sandboxapicdc.cisco.com/api/"
## Calling the authentication function above to get the auth cookie
cookies = get_token(username, password, base_url)
pprint(pull_pod1(cookies, base_url))

This gives us all the devices in POD1. I’ll just be putting in one device for the sake of space

{
  "imdata": [
    {
      "fabricNode": {
        "attributes": {
          "adSt": "on",
          "address": "10.0.112.64",
          "annotation": "",
          "apicType": "apic",
          "childAction": "",
          "delayedHeartbeat": "no",
          "dn": "topology/pod-1/node-101",
          "extMngdBy": "",
          "fabricSt": "active",
          "id": "101",
          "lastStateModTs": "2019-12-12T14:22:14.238+00:00",
          "lcOwn": "local",
          "modTs": "2019-12-12T14:23:02.315+00:00",
          "model": "N9K-C9396PX",
          "monPolDn": "uni/fabric/monfab-default",
          "name": "leaf-1",
          "nameAlias": "",
          "nodeType": "unspecified",
          "role": "leaf",
          "serial": "TEP-1-101",
          "status": "",
          "uid": "0",
          "vendor": "Cisco Systems, Inc",
          "version": ""
        }
      }
    }
  ]
}

Note: Output omitted and modified for the sake of brevity and clarity.

You can observe several key pieces of information about this switch. The DN info, ID, model number, serial number, etc. With ACI, I often had to pull one piece of data so that I could understand how to pull another piece of data. Finding the DN was almost always an important step in my programs.

There is an easier, albeit less stable way (based on my observations) to find the URLs we want. Right-click on an interesting object and select “Open in Object Store Browser.” That will direct to a login screen. After login, the UI brings up a page where it should display the query and give the output.

To see the URL it used we click on “Show URL and response of last query.”

You can navigate the API structure and find what you are looking for. It may take some practice to develop a full understanding of this method.

I hope this has been helpful. I know it can be overwhelming, but the benefits of working with an API vs clicking though a web-page are wonderful. For example, before I left my last job I wrote a Python program that could be run on demand. It pulled what VLANS went to what VM Servers and looked for errors on interfaces. We got a call that there were lots of outages going on. The other network engineers were manually logging into everything looking for issues, but I just ran my program and was able to find a missing VLAN on a trunk to a VM server. The VM failed because it could no longer communicate out-side of the VM server. It took approximately five minutes to find and fix the issue. With dozens of VM servers and very little info on what the actual issue was, I believe that outage would have been much longer if people were looking for the issue by hand.

-Daniel

Using NTC Templates in Ansible

2020-02-04T00:00:00+00:00

Network to Code has a repository created for maintaining templates that help to convert unstructured output from network devices into structured data. Google created TextFSM that provides a custom DSL to take command line output and return as structured data. Examples range from getting an output from a show lldp neighbor command and returning the list of neighbors to getting a list of addresses in an ARP table. This post is going to dive deeper into how to use these to get the data with Ansible.

You can find more about the ntc-templates on our Github page. These are templates that have been created by both Network to Code and by the community. It is an open source project that anyone is able to contribute a template to if there is not a template already created. Take a look at the readme for how you can help contribute to NTC Templates!

Other Methods

Ansible is not the only method for using TextFSM to get structured data returned from network device output. Several Python libraries such as Netmiko and Nornir are able to use TextFSM to return structured data by setting a flag to use TextFSM as well.

Using NTC-Templates with Ansible

There are two primary methods for sending data through a TextFSM parser in Ansible. You will get to see examples for both of these. The first method available is to use the TextFSM Filter. The second is in conjunction with the Ansible Galaxy role Network Engine.

Getting Started

The first step towards leveraging these templates is getting the templates on to a compute device that is running Ansible. For this demo, we’ll be using a local machine and Git to clone the repository to a local directory.

git clone git@github.com:networktocode/ntc-templates.git

Next you will need to install TextFSM to the same Python interpreter that your Ansible installation is installed in. This demo will leverage Python3. For the installation you will also use the --user flag to install as the local user and --upgrade to ensure that the latest version of TextFSM is installed.

pip3 install textfsm --user --upgrade

For the second method of using Ansible Galaxy’s Network Engine you will also need to install the role as follows:

ansible-galaxy install ansible-network.network-engine

Now that the environment is setup it’s time to take a look at the Playbook and the execution.

Playbook Setup and Execution

This lab is setup as a 3 router traingle. RTR-1 will connect to one interface on RTR-2 and RTR-3. RTR-2 will have one connection to RTR-1 and RTR-3. RTR-3 will have the corresponding connections to RTR-1 and RTR-2.

TextFSM CLI Parser - Ansible Built In

---
- name: "PLAY 1: DEMO OF TEXTFSM"
  hosts: routers
  connection: network_cli
  gather_facts: no
  tasks:
    - name: "TASK 1: GET COMMAND OUTPUT"
      ios_command:
        commands:
          - show lldp neighbors
      register: lldp_output

    - name: "TASK 2: REGISTER OUTPUT TO DEVICE_NEIGHBORS VARIABLE"
      set_fact:
        device_neighbors: "{{ lldp_output.stdout[0] | parse_cli_textfsm('~/ntc-templates/templates/cisco_ios_show_lldp_neighbors.textfsm') }}"

    - name: "TASK 3: PRINT OUTPUT"
      debug:
        msg: "{{ device_neighbors }}"

    - name: "TASK 4: PRINT NEIGHBORS"
      debug:
        msg: "{{ item['LOCAL_INTERFACE'] }}: {{ item['NEIGHBOR'] }}"
      loop: "{{ device_neighbors }}"
      loop_control:
        label: "{{ item['LOCAL_INTERFACE'] }}"

Walking through this first play, in Task 1 you are connecting to the device and running the command show lldp neighbors. This is saved to a variable named lldp_output that will be used later.

In Task 2 the output is being sent through the parse_cli_textfsm filter. The filter is being provided the library file to be used in the parsing. This is explicitely called out. The output from going through the TextFSM parser is then getting assigned to the variable device_neighbors, with each host in the execution having their own local instance of this variable.

In Task 3 you are getting to see the output of the variable. This shows the structured data. With this particular template you get the keys of CAPABILITIES, LOCAL_INTERFACES, NEIGHBOR, and NEIGHBOR_INTERFACE back. These are all defined within the TextFSM template.

In Task 4 you get to see a practical output to a screen if you want to audit and understand the neighbor relationships seen by LLDP on the network devices.

In this example you need to set a fact to be able to access the data later or continue to send the output through the parse_cli_textfsm filter every time you want to get at the structured data, such as a line number 15 in TASK 2.

TextFSM CLI Parser Output

Here is the output that corresponds with the play above.

ansible-playbook demo_ansible_textfsm.yml

PLAY [PLAY 1: DEMO OF TEXTFSM WITH CLI PARSER] *************************************************************************

TASK [TASK 1: GET COMMAND OUTPUT] **************************************************************************************
ok: [rtr-2]
ok: [rtr-3]
ok: [rtr-1]

TASK [TASK 2: REGISTER OUTPUT TO DEVICE_NEIGHBORS VARIABLE] ************************************************************
ok: [rtr-1]
ok: [rtr-2]
ok: [rtr-3]

TASK [TASK 3: PRINT OUTPUT] ********************************************************************************************
ok: [rtr-2] => {
    "msg": [
        {
            "CAPABILITIES": "R",
            "LOCAL_INTERFACE": "Gi0/0",
            "NEIGHBOR": "rtr-3",
            "NEIGHBOR_INTERFACE": "Gi0/0"
        },
        {
            "CAPABILITIES": "R",
            "LOCAL_INTERFACE": "Gi0/1",
            "NEIGHBOR": "rtr-1",
            "NEIGHBOR_INTERFACE": "Gi0/1"
        }
    ]
}
ok: [rtr-1] => {
    "msg": [
        {
            "CAPABILITIES": "R",
            "LOCAL_INTERFACE": "Gi0/2",
            "NEIGHBOR": "rtr-3",
            "NEIGHBOR_INTERFACE": "Gi0/1"
        },
        {
            "CAPABILITIES": "R",
            "LOCAL_INTERFACE": "Gi0/1",
            "NEIGHBOR": "rtr-2",
            "NEIGHBOR_INTERFACE": "Gi0/1"
        }
    ]
}
ok: [rtr-3] => {
    "msg": [
        {
            "CAPABILITIES": "R",
            "LOCAL_INTERFACE": "Gi0/1",
            "NEIGHBOR": "rtr-1",
            "NEIGHBOR_INTERFACE": "Gi0/2"
        },
        {
            "CAPABILITIES": "R",
            "LOCAL_INTERFACE": "Gi0/0",
            "NEIGHBOR": "rtr-2",
            "NEIGHBOR_INTERFACE": "Gi0/0"
        }
    ]
}

TASK [TASK 4: PRINT NEIGHBORS FROM CLI PARSER] *************************************************************************
ok: [rtr-1] => (item=Gi0/2) => {
    "msg": "Gi0/2: rtr-3"
}
ok: [rtr-1] => (item=Gi0/1) => {
    "msg": "Gi0/1: rtr-2"
}
ok: [rtr-2] => (item=Gi0/0) => {
    "msg": "Gi0/0: rtr-3"
}
ok: [rtr-2] => (item=Gi0/1) => {
    "msg": "Gi0/1: rtr-1"
}
ok: [rtr-3] => (item=Gi0/1) => {
    "msg": "Gi0/1: rtr-1"
}
ok: [rtr-3] => (item=Gi0/0) => {
    "msg": "Gi0/0: rtr-2"
}

PLAY RECAP *************************************************************************************************************
rtr-1                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
rtr-2                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
rtr-3                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Ansible Role - Network Engine

The Ansible Role - Network Engine is a role that extracts information about network devices into Ansible Facts. You can use either TextFSM syntax or YAML (command_parser option) to extract information about a network device from the command output. You can read more about using the role at the network-engine Github page.

Here you will see a second method to get the same output with Network Engine instead of the CLI Parser. The biggest difference is that in this method, the Network Engine role will register the returned data to the ansible_facts instead of requiring you to set it to your own variable.

---
- name: "PLAY 2: DEMO OF TEXTFSM WITH NETWORK ENGINE"
  hosts: routers
  connection: network_cli
  gather_facts: no
  roles:
    - ansible-network.network-engine
  tasks:
    - name: "TASK 1: GET COMMAND OUTPUT"
      ios_command:
        commands:
          - show lldp neighbors
      register: lldp_output

    - name: "TASK 2: RUN THROUGH THE PARSER"
      textfsm_parser:
        file: "/Users/ntcblog/ntc-templates/templates/cisco_ios_show_lldp_neighbors.textfsm"
        content: "{{ lldp_output.stdout[0] }}"
        name: lldp_output

    - name: "TASK 3: SHOW ANSIBLE FACTS OUTPUT"
      debug:
        msg: "{{ ansible_facts }}"

    - name: "TASK 4: PRINT NEIGHBORS FROM ANSIBLE NETWORK ENGINE"
      debug:
        msg: "{{ item['LOCAL_INTERFACE'] }}: {{ item['NEIGHBOR'] }}"
      loop: "{{ ansible_facts['lldp_output'] }}"
      loop_control:
        label: "{{ item['LOCAL_INTERFACE'] }}"

There is a new key that we needed to add to import the role that was installed with the ansible-galaxy command earlier. This is the roles: key that you see under gather_facts and contains a list of roles to import into the play. Here you see the import of the ansible-network.network-engine role.

In Task 1 you once again have the same command to gather the LLDP neighbors from the device.

In Task 2 instead of registering to a fact and sending through a filter, you now use the module textfsm_parser that takes file as a parameter that is the file of the TextFSM template. This is the same template referenced in the filter on the first play. You also pass content of what output you want to send through the parser. The last parameter is the name of the fact that you are registering to ansible_facts.

In Task 3, you once again get to see the output for ansible_facts. This will show that there are more details available about the device.

In Task 4 you get to see the same output as Play 1 where you get the neighbors printed out.

Network Engine Output

ansible-playbook demo_ansible_textfsm.yml

PLAY [PLAY 2: DEMO OF TEXTFSM WITH NETWORK ENGINE] *********************************************************************

TASK [TASK 1: GET COMMAND OUTPUT] **************************************************************************************
ok: [rtr-2]
ok: [rtr-3]
ok: [rtr-1]

TASK [TASK 2: RUN THROUGH THE PARSER] **********************************************************************************
ok: [rtr-1]
ok: [rtr-2]
ok: [rtr-3]

TASK [TASK 3: SHOW ANSIBLE FACTS OUTPUT] *******************************************************************************
ok: [rtr-1] => {
    "msg": {
        "discovered_interpreter_python": "/usr/bin/python",
        "lldp_output": [
            {
                "CAPABILITIES": "R",
                "LOCAL_INTERFACE": "Gi0/2",
                "NEIGHBOR": "rtr-3",
                "NEIGHBOR_INTERFACE": "Gi0/1"
            },
            {
                "CAPABILITIES": "R",
                "LOCAL_INTERFACE": "Gi0/1",
                "NEIGHBOR": "rtr-2",
                "NEIGHBOR_INTERFACE": "Gi0/1"
            }
        ]
    }
}
ok: [rtr-2] => {
    "msg": {
        "discovered_interpreter_python": "/usr/bin/python",
        "lldp_output": [
            {
                "CAPABILITIES": "R",
                "LOCAL_INTERFACE": "Gi0/0",
                "NEIGHBOR": "rtr-3",
                "NEIGHBOR_INTERFACE": "Gi0/0"
            },
            {
                "CAPABILITIES": "R",
                "LOCAL_INTERFACE": "Gi0/1",
                "NEIGHBOR": "rtr-1",
                "NEIGHBOR_INTERFACE": "Gi0/1"
            }
        ]
    }
}
ok: [rtr-3] => {
    "msg": {
        "discovered_interpreter_python": "/usr/bin/python",
        "lldp_output": [
            {
                "CAPABILITIES": "R",
                "LOCAL_INTERFACE": "Gi0/1",
                "NEIGHBOR": "rtr-1",
                "NEIGHBOR_INTERFACE": "Gi0/2"
            },
            {
                "CAPABILITIES": "R",
                "LOCAL_INTERFACE": "Gi0/0",
                "NEIGHBOR": "rtr-2",
                "NEIGHBOR_INTERFACE": "Gi0/0"
            }
        ]
    }
}

TASK [TASK 4: PRINT NEIGHBORS FROM ANSIBLE NETWORK ENGINE] *************************************************************
ok: [rtr-1] => (item=Gi0/2) => {
    "msg": "Gi0/2: rtr-3"
}
ok: [rtr-1] => (item=Gi0/1) => {
    "msg": "Gi0/1: rtr-2"
}
ok: [rtr-2] => (item=Gi0/0) => {
    "msg": "Gi0/0: rtr-3"
}
ok: [rtr-3] => (item=Gi0/1) => {
    "msg": "Gi0/1: rtr-1"
}
ok: [rtr-2] => (item=Gi0/1) => {
    "msg": "Gi0/1: rtr-1"
}
ok: [rtr-3] => (item=Gi0/0) => {
    "msg": "Gi0/0: rtr-2"
}

PLAY RECAP *************************************************************************************************************
rtr-1                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
rtr-2                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
rtr-3                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Summary

This shows that there are multiple ways of pairing TextFSM templates with Ansible. Getting structured data out of unstructured output is extremely valuable when it comes to automating a network environment. There are over 300 different templates currently in the NTC-Templates repository to help get structured data out of your unstructured data.

-Josh

Versionable Database

2020-01-28T00:00:00+00:00

One of the most important components of a network automation strategy is how to manage your data and/or Source of Truth (SoT). In networking, the data elements are often massive. As an example, most Fortune 500 type companies will have to manage a few hundred thousand to potentially millions of switchports, each with several data points (VLAN, MTU, IP, etc) per interface. The scale and criticality of this data present several challenges. The intention of this blog is describe the need for a versionable database in network automation and serve as an primer to a versionable database.

Problem Space

When managing and storing your Source of Truth data, there are two primary mechanisms, each with it’s own pros and cons.

Source Control (Git)

Provides the ability to version data in such a way the exact state of the data at any point in the past is known.
Provides the ability to know who is the owner of the data.
Provides the ability to populate data in a staging area, without modifying the production data.
Tooling integration with things such as CI Systems.
Decentralized storage of the data.

Database

Provides ACID-based transactions of the data.
Has a native querying language to obtain, filter, and all around work with the data.
Provides schema enforcement of the data.
The ability to scale to large datasets.

There is a clear dichotomy between these two choices. On one hand there is great integrations with all standard DevOps tooling, on the other hand there is an enterprise-grade manager of the data.

History

For several years, a few of us have been searching for solutions that combine these two concepts. If one could build a database with Git constructs, this would allow versionable management of your data, with the ability to query, effectively store, and manage at scale. Through various searches it would seem that this has not been solved in any meaningful way. The closest I have come across was a project called NOMS, but it has limitations.

Recently a startup company called liquidata was founded and is creating a solution–a library called Dolt. Dolt is is written on top of the NOMS project, in Go. The intention is to support all Git semantics (such as branch, add, commit, etc..), as well as all MySQL semantics (such as insert, create, update, etc…).

By supporting all MySQL semantics, applications that use a MySQL server should be able to simply use a Dolt-SQL server with no disruption. Whether you are using an ODBC driver or raw SQL, in theory it should still work. You would naturally have to build in the capability for your application to take advantage of the Git capabilities or use traditional “Git-like” workflows.

By supporting all Git semantics, data should be able to be managed in a decentralized fashion using Git workflows to branch, add, diff, and merge the data. In a similar vein to GitHub, they have developed DoltHub, to provide that level of tooling expected in a standard Git user interface, such as forking, API’s, webhooks, and CI integrations.

Primer

In this brief introduction to the technology, we will create a Dolt data repository, called “simple-inventory” and run it on a dolt-sql server. If you care to follow along the only requirements are to have Docker and internet access.

Setup

Start the Docker container.

docker run -it --rm --name dolt golang:1.12.14

Install Dolt by running curl -L https://github.com/liquidata-inc/dolt/releases/download/v0.12.0/install.sh | bash

root@2a29f8a34dd3:/go# curl -L https://github.com/liquidata-inc/dolt/releases/download/v0.12.0/install.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   601    0   601    0     0   2647      0 --:--:-- --:--:-- --:--:--  2647
100  3038  100  3038    0     0   8392      0 --:--:-- --:--:-- --:--:--  8392
Downloading: https://github.com/liquidata-inc/dolt/releases/download/v0.12.0/dolt-linux-amd64.tar.gz
Installing dolt, git-dolt and git-dolt-smudge to /usr/local/bin.
root@2a29f8a34dd3:/go#

You can create a Dolt data repository by creating a folder, and then initializing the repository. Just like Git, in Dolt, you need to add yourself to the Dolt config. If you are familiar with Git, these commands will be familiar, only changing the command from git to dolt, but keeping all of the other options the same.

root@2a29f8a34dd3:/go# mkdir simple-inventory
root@2a29f8a34dd3:/go# cd simple-inventory
root@2a29f8a34dd3:/go/simple-inventory# dolt config --global --add user.email "ken@celenza.org"
Config successfully updated.
root@2a29f8a34dd3:/go/simple-inventory# dolt config --global --add user.name "itdependsnetworks"
Config successfully updated.
root@2a29f8a34dd3:/go/simple-inventory# dolt init
Successfully initialized dolt data repository.
root@2a29f8a34dd3:/go/simple-inventory# dolt status
On branch master
nothing to commit, working tree clean
root@2a29f8a34dd3:/go/simple-inventory# dolt log
commit hc8gq0434ckkjeo36rccn55mo14gk87q
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:33 +0000 2020

	Initialize data repository

root@2a29f8a34dd3:/go/simple-inventory#

Creating Tables

Ideally, if you are familiar with MySQL the only difference should be using Dolt engine instead of a MySQL server, everything should be the same. To enter the Dolt SQL server, simply use the command dolt sql.

root@2a29f8a34dd3:/go/simple-inventory# dolt sql
# Welcome to the DoltSQL shell.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit.
doltsql>

Create tables as you normally would, in this example, we will create two tables. The table device_inventory will have the inventory of the devices, with columns for hostname and IP address. The table vlan will have the hostname (with foreign key relationaship to the device_inventory table) column, VLAN , and name of the VLAN.

Note: Though the foreign key relationaship syntax is defined, it is not a currently supported Dolt feature.

doltsql> CREATE TABLE device_inventory (
      ->     hostname varchar(32) NOT NULL,
      ->     ip_address varchar(15) NOT NULL,
      ->     primary key (`hostname`)
      -> );
doltsql> CREATE TABLE vlan (
      ->     hostname varchar(32),
      ->     vlan int NOT NULL,
      ->     name varchar(32) NOT NULL,
      ->     PRIMARY KEY (hostname, vlan),
      ->     FOREIGN KEY (hostname) REFERENCES device_inventory(hostname)
      -> );
doltsql> exit

Here is where it starts to get interesting–we can combine these two different concepts and see how they interact. So far, the data has not been committed into master, only staged into our local environment. We can view this by issuing standard Git-like commands when we return back to the command line from the dolt-sql server.

Run the dolt status, dolt diff, and dolt diff -q commands.

root@2a29f8a34dd3:/go/simple-inventory# dolt status
On branch master
Untracked files:
  (use "dolt add <table>" to include in what will be committed)
	new table:      device_inventory
	new table:      vlan
root@2a29f8a34dd3:/go/simple-inventory# dolt diff
diff --dolt a/device_inventory b/device_inventory
added table
diff --dolt a/vlan b/vlan
added table
root@2a29f8a34dd3:/go/simple-inventory# dolt diff -q
CREATE TABLE `device_inventory` (
  `hostname` LONGTEXT NOT NULL COMMENT 'tag:0',
  `ip_address` LONGTEXT NOT NULL COMMENT 'tag:1',
  PRIMARY KEY (`hostname`)
);
CREATE TABLE `vlan` (
  `hostname` LONGTEXT NOT NULL COMMENT 'tag:0',
  `vlan` BIGINT NOT NULL COMMENT 'tag:1',
  `name` LONGTEXT NOT NULL COMMENT 'tag:2',
  PRIMARY KEY (`hostname`,`vlan`)
);
root@2a29f8a34dd3:/go/simple-inventory#

The dolt status command shows that the schema has not “taken effect” by being merged into master. The dolt diff with optional -q flag, shows that a new table has been created. Note the additional tag parameters, this is logic to track column names, which may change over time and cause name conflicts. The new schema can be committed into master.

Issue the commands dolt add -a, dolt commit -m 'initial schema', and dolt log.

root@2a29f8a34dd3:/go/simple-inventory# dolt add -a
root@2a29f8a34dd3:/go/simple-inventory# dolt commit -m 'initial schema'
commit 4n29eaqb1v9pmnpnda2r43fc91p1vp9l
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:41 +0000 2020

	initial schema

root@2a29f8a34dd3:/go/simple-inventory# dolt log
commit 4n29eaqb1v9pmnpnda2r43fc91p1vp9l
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:41 +0000 2020

	initial schema

commit hc8gq0434ckkjeo36rccn55mo14gk87q
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:33 +0000 2020

	Initialize data repository

root@2a29f8a34dd3:/go/simple-inventory#

Awesome! The first piece of information of the database was committed.

Adding data

Defining schema is not all the valuable without the data. Again, using standard SQL constructs we can commit the data. Let’s use proper branching strategies this time.

Create a branch and Enter dolt-sql

root@2a29f8a34dd3:/go/simple-inventory# dolt checkout -b intial_data
Switched to branch 'intial_data'
root@2a29f8a34dd3:/go/simple-inventory# dolt sql
# Welcome to the DoltSQL shell.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit.
doltsql>

Add data

doltsql> INSERT INTO device_inventory (hostname, ip_address) VALUES ("nyc-sw01", "10.1.1.1");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO device_inventory (hostname, ip_address) VALUES ("nyc-sw02", "10.1.1.2");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO vlan (hostname, vlan, name) VALUES ("nyc-sw01", 10, "user");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO vlan (hostname, vlan, name) VALUES ("nyc-sw01", 20, "printer");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO vlan (hostname, vlan, name) VALUES ("nyc-sw01", 30, "wap");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO vlan (hostname, vlan, name) VALUES ("nyc-sw02", 10, "user");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO vlan (hostname, vlan, name) VALUES ("nyc-sw02", 20, "printer");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> INSERT INTO vlan (hostname, vlan, name) VALUES ("nyc-sw02", 30, "wap");
+---------+
| updated |
+---------+
| 1       |
+---------+
doltsql> exit
Bye

Now that we have staged the data, we can view the date in two different ways. The first is via raw sql entries, and the other is more akin to a unix diff.

View the diff. In the console, diffs are actually color coded green and red for add and remove respectively.

root@2a29f8a34dd3:/go/simple-inventory# dolt diff
diff --dolt a/device_inventory b/device_inventory
--- a/device_inventory @ tktob1spsoos6isfdj4o9benp9c57iic
+++ b/device_inventory @ b37haqr0n9j1e5bckj35sqt4kntojlp9
+-----+----------+------------+
|     | hostname | ip_address |
+-----+----------+------------+
|  +  | nyc-sw01 | 10.1.1.1   |
|  +  | nyc-sw02 | 10.1.1.2   |
+-----+----------+------------+
diff --dolt a/vlan b/vlan
--- a/vlan @ 7f8lqlv2k9cpdth68kob9hl8atuejrpn
+++ b/vlan @ e2kovelv2sfjuo584o0v6hp76207k720
+-----+----------+------+---------+
|     | hostname | vlan | name    |
+-----+----------+------+---------+
|  +  | nyc-sw01 | 10   | user    |
|  +  | nyc-sw01 | 20   | printer |
|  +  | nyc-sw01 | 30   | wap     |
|  +  | nyc-sw02 | 10   | user    |
|  +  | nyc-sw02 | 20   | printer |
|  +  | nyc-sw02 | 30   | wap     |
+-----+----------+------+---------+
root@2a29f8a34dd3:/go/simple-inventory# dolt diff -q
INSERT INTO `device_inventory` (`hostname`,`ip_address`) VALUES ("nyc-sw01","10.1.1.1");
INSERT INTO `device_inventory` (`hostname`,`ip_address`) VALUES ("nyc-sw02","10.1.1.2");
INSERT INTO `vlan` (`hostname`,`vlan`,`name`) VALUES ("nyc-sw01",10,"user");
INSERT INTO `vlan` (`hostname`,`vlan`,`name`) VALUES ("nyc-sw01",20,"printer");
INSERT INTO `vlan` (`hostname`,`vlan`,`name`) VALUES ("nyc-sw01",30,"wap");
INSERT INTO `vlan` (`hostname`,`vlan`,`name`) VALUES ("nyc-sw02",10,"user");
INSERT INTO `vlan` (`hostname`,`vlan`,`name`) VALUES ("nyc-sw02",20,"printer");
INSERT INTO `vlan` (`hostname`,`vlan`,`name`) VALUES ("nyc-sw02",30,"wap");
root@2a29f8a34dd3:/go/simple-inventory#

Personally, I’m pretty impressed with what is happening here, but there is still more to do to complete this workflow. The data needs to be committed, and merged from the “feature” branch into master branch.

Commit to the feature branch.

root@2a29f8a34dd3:/go/simple-inventory# dolt add -a
root@2a29f8a34dd3:/go/simple-inventory# dolt commit -m 'initial data'
commit gbt6r21mtbd0cvs5iahog912essjhn0n
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:50 +0000 2020

	initial data

root@2a29f8a34dd3:/go/simple-inventory#

Merge into the master branch.

root@2a29f8a34dd3:/go/simple-inventory# dolt checkout master
Switched to branch 'master'
root@2a29f8a34dd3:/go/simple-inventory# dolt branch
  intial_data
* master
root@2a29f8a34dd3:/go/simple-inventory# dolt merge intial_data
Updating 4n29eaqb1v9pmnpnda2r43fc91p1vp9l..gbt6r21mtbd0cvs5iahog912essjhn0n
Fast-forward
root@2a29f8a34dd3:/go/simple-inventory# dolt log
commit gbt6r21mtbd0cvs5iahog912essjhn0n
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:50 +0000 2020

	initial data

commit 4n29eaqb1v9pmnpnda2r43fc91p1vp9l
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:41 +0000 2020

	initial schema

commit hc8gq0434ckkjeo36rccn55mo14gk87q
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:22:33 +0000 2020

	Initialize data repository

root@2a29f8a34dd3:/go/simple-inventory#

So far this is showing the “Create” part of standard CRUD operations is working, meaning we can add data to the repository.

Update Data

Naturally, as soon as data is entered, you will want to modify it. The process is the same to modify: branch, sql statements, add, commit, and merge.

Checkout a new branch, and enter Dolt SQL.

root@2a29f8a34dd3:/go/simple-inventory# dolt checkout -b change_vlan
Switched to branch 'change_vlan'
root@2a29f8a34dd3:/go/simple-inventory# dolt sql
# Welcome to the DoltSQL shell.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit.
doltsql>

Update the data.

doltsql> update vlan set name = "prnt" where name = "printer";
+---------+---------+
| matched | updated |
+---------+---------+
| 2       | 2       |
+---------+---------+
doltsql> exit
Bye

View the diff.

root@2a29f8a34dd3:/go/simple-inventory# dolt diff
diff --dolt a/vlan b/vlan
--- a/vlan @ bjg5ou111lmu03ciu6aisphf7m6jbk1r
+++ b/vlan @ 7f8lqlv2k9cpdth68kob9hl8atuejrpn
+-----+----------+------+---------+
|     | hostname | vlan | name    |
+-----+----------+------+---------+
|  <  | nyc-sw01 | 20   | printer |
|  >  | nyc-sw01 | 20   | prnt    |
|  <  | nyc-sw02 | 20   | printer |
|  >  | nyc-sw02 | 20   | prnt    |
+-----+----------+------+---------+
root@2a29f8a34dd3:/go/simple-inventory# dolt diff -q
UPDATE `vlan` SET `name`="prnt" WHERE (`hostname`="nyc-sw01" AND `vlan`=20);
UPDATE `vlan` SET `name`="prnt" WHERE (`hostname`="nyc-sw02" AND `vlan`=20);
root@2a29f8a34dd3:/go/simple-inventory#

A keen eye will note the captured update statements shown in the diff are not the same as the update statement inputted in the SQL server. This conversion makes sense, since the data could be merged on a different set of data that had more or less data in it. It also illustrates the complexity of building this technology.

Commit to a feature branch and merge to master.

root@2a29f8a34dd3:/go/simple-inventory# dolt add -a
root@2a29f8a34dd3:/go/simple-inventory# dolt commit -m 'update data'
commit jqe3vb84fhfunn7uapksdt5thb4rr7r7
Author: itdependsnetworks <ken@celenza.org>
Date:   Tue Jan 28 04:23:03 +0000 2020

	update data

root@2a29f8a34dd3:/go/simple-inventory# dolt checkout master
Switched to branch 'master'
root@2a29f8a34dd3:/go/simple-inventory# dolt merge change_vlan
Updating gbt6r21mtbd0cvs5iahog912essjhn0n..jqe3vb84fhfunn7uapksdt5thb4rr7r7
Fast-forward
root@2a29f8a34dd3:/go/simple-inventory# dolt sql
# Welcome to the DoltSQL shell.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit.
doltsql>

View the data from dolt sql.

doltsql> select * from vlan;
+----------+------+------+
| hostname | vlan | name |
+----------+------+------+
| nyc-sw01 | 10   | user |
| nyc-sw01 | 20   | prnt |
| nyc-sw01 | 30   | wap  |
| nyc-sw02 | 10   | user |
| nyc-sw02 | 20   | prnt |
| nyc-sw02 | 30   | wap  |
+----------+------+------+
doltsql>

You can also view who is the owner of the data by tracking to the commit, which includes the auther, time, and commit hash metadata as well, using the dolt blame <table> command.

root@2a29f8a34dd3:/go/simple-inventory# dolt blame vlan
+----------+------+--------------+-------------------+------------------------------+-------------------+
| HOSTNAME | VLAN | COMMIT MSG   | AUTHOR            | TIME                         | COMMIT            |
+----------+------+--------------+-------------------+------------------------------+-------------------+
| nyc-sw01 | 30   | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i |
| nyc-sw02 | 10   | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i |
| nyc-sw02 | 20   | update data  | itdependsnetworks | Tue Jan 28 04:23:03 UTC 2020 | jqe3vb84fhfunn7ua |
| nyc-sw02 | 30   | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i |
| nyc-sw01 | 10   | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i |
| nyc-sw01 | 20   | update data  | itdependsnetworks | Tue Jan 28 04:23:03 UTC 2020 | jqe3vb84fhfunn7ua |
+----------+------+--------------+-------------------+------------------------------+-------------------+
root@2a29f8a34dd3:/go/simple-inventory#

Summary

This is just a primer, and there is still a lot of work to be done to truly have feature parity with both MySQL and Git, but this is a great step in the right direction. I plan to continue to monitor and follow up with examples, use cases, and library updates over time. Specifically, next time, I want to extend the workflow to include DoltHub to review those capabilities as well.

-Ken