The NTC Mag

Introduction to Diffing and Syncing Data with DiffSync

2021-05-11T00:00:00+00:00

In the world of network automation, we rarely are so fortunate as to have only a single authoritative source of data to work with. More often, we are responsible for and responsive to data in multiple distinct systems. When these systems overlap in their domains of responsibility, or in the information that they manage, we find it necessary to compare data between them and resolve any differences. Doing so manually is tedious and error-prone — what’s needed here is automation! This is what has led us at Network to Code to develop the Python library, DiffSync.

A few examples of this kind of problem:

In a network that we manage, Nautobot stores our device inventory and IPAM information, but BGP configuration of these devices is managed by a set of YAML files stored in a Git repository. We need to identify whether there are any discrepancies between the devices represented in these two data sets. And every time there’s a new Git commit, we need to repeat this check.
Our device inventory in our Nautobot source of truth (SoT) needs to be replicated into our IT Service Management (ITSM) platform so that devices can be associated with tickets. As devices are deployed, configured, and eventually decommissioned, both systems need to accurately reflect these changes.
We have a legacy database system that’s slated to be decommissioned, and we need to periodically pull the latest updates in the legacy system over into the new database that’s starting to take over.

These and countless other real-world data comparison and synchronization problems are what led us to develop DiffSync. In short, DiffSync is a generic utility library that can be used to compare (“diff”) and synchronize (“sync”) different data sets. It’s free and open source, and we’ve been using it for some time now to save development time on projects both internal and external. DiffSync is the engine underlying tools such as Network Importer and the Nautobot NetBox Importer plugin for Nautobot.

This will be the first in a short series of blog posts about DiffSync — in this post I’ll introduce what DiffSync is and the basics of how it works, and walk you through a short example of writing a Python script that will use DiffSync to identify, report, then resolve the differences between a pair of JSON files.

When to Use DiffSync

DiffSync is at its most useful when you have multiple sources or sets of data to compare and/or synchronize, and especially if any of the following are true:

If you need to repeatedly compare or synchronize the data sets as one or both change over time.
If you need to account for not only the creation of new records, but also changes to and deletion of existing records as well.
If various types of data in your data set naturally form a tree-like or parent-child relationship with other data.
If the different data sets have some attributes in common and other attributes that are exclusive to one or the other.

Overview of DiffSync

DiffSync acts as an intermediate translation layer between all of the data sets you are diffing and/or syncing. In practical terms, this means that to use DiffSync, you will define a set of data models as well as the “adapters” needed to translate between each base data source and the data model. In Python terms, the adapters will be subclasses of the DiffSync class, and each data model class will be a subclass of the DiffSyncModel class.

Once you have used each adapter to load each data source into a collection of data model records, you can then ask DiffSync to “diff” the two data sets, and it will produce a structured representation of the difference between them. In Python, this is accomplished by calling the diff_to() or diff_from() method on one adapter and passing the other adapter as a parameter.

You can also ask DiffSync to “sync” one data set onto the other, and it will instruct your adapter as to the steps it needs to take to make sure that its data set accurately reflects the other. In Python, this is accomplished by calling the sync_to() or sync_from() method on one adapter and passing the other adapter as a parameter.

A Basic Example

Let’s start with an intentionally basic example. Our “data sets” will be two JSON files, each of which contains a random subset of the numbers 1 through 50. We can generate these files with a little Python:

>>> import json
>>> import random
>>>
>>> data1 = [i for i in range(1, 50) if random.randint(0, 1) == 1]
>>> with open("file1.json", "w") as file_handle:
...     json.dump(data1, file_handle)
...
>>> data2 = [i for i in range(1, 50) if random.randint(0, 1) == 1]
>>> with open("file2.json", "w") as file_handle:
...     json.dump(data2, file_handle)
...
>>>

Installing DiffSync

DiffSync is available on PyPI, so all that’s needed is to create a Python virtual environment and install DiffSync into it:

$ python3.6 -m venv diffsync_virtualenv
$ source diffsync_virtualenv/bin/activate
(diffsync_virtualenv) $ pip install --upgrade pip
(diffsync_virtualenv) $ pip install diffsync

Note that DiffSync requires Python 3.6 or later!

Defining the Data Model

When creating a new data model (subclass of DiffSyncModel) there are a few metadata properties, prefixed with _, that you may need to define for it to work properly. For this simple data model, the only two we need to be concerned about are _modelname (a descriptive label for this model) and _identifiers (a tuple of data fields that uniquely identify a single record of this model).

Here we’re defining a model called Number, which has one attribute, value, that also serves as the unique identifier of each instance of this model.

from diffsync import DiffSyncModel

class Number(DiffSyncModel):
    """Simple data model, storing only a number."""

    # DiffSync metadata fields
    _modelname = "number"
    _identifiers = ("value",)  # must be a tuple, not a single stand-alone value!

    # Data attributes on each instance of this model, including the above-listed identifier(s):
    value: int

In case you’re unfamiliar with the value: int syntax, this is the syntax used in Python 3.6 and later for variable type annotations. DiffSync is based on a library called Pydantic, which uses these type annotations to actually construct a data model that enforces the data types you’ve declared.

Defining the Adapter

In this case, the two “data sets” we are going to compare share the same underlying “data source”, a simple JSON file. So here we actually only need to create a single adapter class that we can use for both data sets. In a more complex example involving differing databases or systems, you would need to create a separate adapter class for each one, but the same concepts apply.

Here we’re defining an adapter called NumbersJSONAdapter, specifying that it knows about Number data records, and implementing logic for it to load these from the specified JSON file.

import json

from diffsync import DiffSync

class NumbersJSONAdapter(DiffSync):

    # Tell DiffSync that when we refer to a model called "number", we will use the Number class
    number = Number

    # Tell DiffSync which base/parent model(s) it should start with when diffing or syncing
    top_level = ["number"]

    def __init__(self, filename, *args, **kwargs):
        super().__init__(*args, **kwargs)

        with open(filename, "r") as source_file:
            data = json.load(source_file)

        for input_value in data:
            # Create a Number record representing this value
            record = Number(value=input_value)
            # Add this record to our internal data store
            self.add(record)

Generating a Diff

Let’s put it all together into a single self-contained Python script:

# numbers_script.py
import json
import pprint

from diffsync import DiffSync, DiffSyncModel
from diffsync.logging import enable_console_logging


class Number(DiffSyncModel):
    """Simple data model, storing only a number."""

    # DiffSync metadata fields
    _modelname = "number"
    _identifiers = ("value",)  # must be a tuple, not a single standalone value!

    # Data attributes on each instance of this model, including the above-listed identifier(s):
    value: int


class NumbersJSONAdapter(DiffSync):

    # Tell DiffSync that when we refer to a model called "number", we will use the Number class
    number = Number

    # Tell DiffSync which base/parent model(s) it should start with when diffing or syncing
    top_level = ["number"]

    def __init__(self, filename, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self.filename = filename

        with open(filename, "r") as source_file:
            data = json.load(source_file)

        for input_value in data:
            # Create a Number record representing this value
            record = Number(value=input_value)
            # Add this record to our internal data store
            self.add(record)


def load_data():
    """Load both data sets and return the populated DiffSync adapter objects."""
    data1 = NumbersJSONAdapter("file1.json", name="file1")
    data2 = NumbersJSONAdapter("file2.json", name="file2")

    return (data1, data2)


def diff_data():
    """Generate and print the diff between two data sets."""
    data1, data2 = load_data()

    diff = data1.diff_to(data2)
    print(diff.str())
    pprint.pprint(diff.dict())


if __name__ == "__main__":
    enable_console_logging(verbosity=1)
    diff_data()

If you save this script and execute it, you should get output similar to the following:

(diffsync_virtualenv) $ python numbers_script.py
2021-05-03 13:32.47 [info     ] Beginning diff calculation     [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
2021-05-03 13:32.47 [info     ] Diff calculation complete      [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
number
  number: 9 MISSING in file2
  number: 11 MISSING in file2
  number: 12 MISSING in file2
  number: 13 MISSING in file2
  number: 20 MISSING in file2
  number: 22 MISSING in file2
  number: 32 MISSING in file2
  number: 38 MISSING in file2
  number: 44 MISSING in file2
  number: 48 MISSING in file2
  number: 49 MISSING in file2
  number: 1 MISSING in file1
  number: 2 MISSING in file1
  number: 4 MISSING in file1
  number: 5 MISSING in file1
  number: 8 MISSING in file1
  number: 15 MISSING in file1
  number: 18 MISSING in file1
  number: 21 MISSING in file1
  number: 23 MISSING in file1
  number: 30 MISSING in file1
  number: 34 MISSING in file1
  number: 39 MISSING in file1
  number: 42 MISSING in file1
  number: 45 MISSING in file1
  number: 47 MISSING in file1
{'number': {'1': {'-': {}},
            '11': {'+': {}},
            '12': {'+': {}},
            '13': {'+': {}},
            '15': {'-': {}},
            '18': {'-': {}},
            '2': {'-': {}},
            '20': {'+': {}},
            '21': {'-': {}},
            '22': {'+': {}},
            '23': {'-': {}},
            '30': {'-': {}},
            '32': {'+': {}},
            '34': {'-': {}},
            '38': {'+': {}},
            '39': {'-': {}},
            '4': {'-': {}},
            '42': {'-': {}},
            '44': {'+': {}},
            '45': {'-': {}},
            '47': {'-': {}},
            '48': {'+': {}},
            '49': {'+': {}},
            '5': {'-': {}},
            '8': {'-': {}},
            '9': {'+': {}}}}

The exact numbers reported will, of course, be different since we randomly generated these two files!

As you can see, DiffSync has identified the numbers that differ between the two files as a DiffSync Diff object, which can be converted to a string representation or a dictionary representation as needed.

DiffSync’s built-in logging uses structlog to generate log messages with structured data attached. By using diffsync.logging.enable_console_logging() in our script, we’re converting those log messages to standard Python log messages, but it’s possible in more advanced integrations to access the structured logs directly, allowing you to do various forms of log processing without needing to parse free-text log messages.

Syncing Changes

For reporting and human-directed analysis, the above diff generation might be all that you need. But for automation, the next step is to be able to automatically resolve the diff and bring the two data sets into sync. Doing this as a dry run (without actually making any changes to the data sets) is as simple as calling sync_to() instead of diff_to():

# number_script.py

# ...

def sync_data():
    """Sync the changes from data1 onto data2."""
    data1, data2 = load_data()

    data1.sync_to(data2)

    # Show that *within DiffSync* there is now no longer any diff between the two data sets!
    print(data1.diff_to(data2).str())

if __name__ == "__main__":
    enable_console_logging(verbosity=1)
    sync_data()

(diffsync_virtualenv) $ python numbers_script.py
2021-05-03 13:49.35 [info     ] Beginning diff calculation     [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
2021-05-03 13:49.35 [info     ] Diff calculation complete      [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
2021-05-03 13:49.35 [info     ] Beginning sync                 [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
2021-05-03 13:49.35 [info     ] Created successfully           [diffsync.helpers] action=create diffs={'+': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=9
2021-05-03 13:49.35 [info     ] Created successfully           [diffsync.helpers] action=create diffs={'+': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=11
2021-05-03 13:49.35 [info     ] Created successfully           [diffsync.helpers] action=create diffs={'+': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=12
2021-05-03 13:49.35 [info     ] Created successfully           [diffsync.helpers] action=create diffs={'+': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=13
2021-05-03 13:49.35 [info     ] Created successfully           [diffsync.helpers] action=create diffs={'+': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=20
2021-05-03 13:49.35 [info     ] Created successfully           [diffsync.helpers] action=create diffs={'+': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=22
2021-05-03 13:49.35 [info     ] Created successfully           [diffsync.helpers] action=create diffs={'+': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=32
2021-05-03 13:49.35 [info     ] Created successfully           [diffsync.helpers] action=create diffs={'+': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=38
2021-05-03 13:49.35 [info     ] Created successfully           [diffsync.helpers] action=create diffs={'+': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=44
2021-05-03 13:49.35 [info     ] Created successfully           [diffsync.helpers] action=create diffs={'+': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=48
2021-05-03 13:49.35 [info     ] Created successfully           [diffsync.helpers] action=create diffs={'+': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=49
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=1
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=2
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=4
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=5
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=8
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=15
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=18
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=21
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=23
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=30
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=34
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=39
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=42
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=45
2021-05-03 13:49.35 [info     ] Deleted successfully           [diffsync.helpers] action=delete diffs={'-': {}} dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> model=number src=<NumbersJSONAdapter "file1"> status=success unique_id=47
2021-05-03 13:49.35 [info     ] Sync complete                  [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
2021-05-03 13:49.35 [info     ] Beginning diff calculation     [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
2021-05-03 13:49.35 [info     ] Diff calculation complete      [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
(no diffs)

Note again that this is essentially a dry run — because we haven’t yet written any code telling DiffSync how to actually write changes back to the dataset, the “Created successfully” and “Deleted successfully” log messages are referring only to successful changes within DiffSync’s own representation of the data! We can see at the end that, at least within DiffSync itself, there are no longer any diffs between the two data set adapters.

Note also that because DiffSync identifies the diff set before performing a sync, only the numbers that differ between the two data sets are being modified — numbers that exist in both data sets are left completely untouched! This is a key feature of DiffSync, as particularly on subsequent re-syncs between data sets, existing and fully synchronized data should not be modified unnecessarily.

To make this more useful, of course, we need to write some code to actually output back to disk the changed data set. In this example, since our data set is a flat file rather than a collection of individual database records, we won’t implement individual create(), update(), and delete() methods on the Number class. We will instead implement sync_complete(), which is a callback function that DiffSync automatically calls at the end of a successful sync operation, providing us the opportunity to perform a bulk write of the entire, fully updated data set back to disk:

# numbers_script.py

from diffsync import Diff, DiffSync, DiffSyncModel, DiffSyncFlags

# ...

class NumbersJSONAdapter(DiffSync):
    # ...

    def sync_complete(self, source: DiffSync, diff: Diff, flags=DiffSyncFlags.NONE, logger=None):
        """Callback after a sync has completed, updating the model data of this instance.

        Note: this callback is **only** triggered if the sync actually resulted in data changes.
        """
        numbers = self.get_all("number")
        data = sorted([number.value for number in numbers])

        target_filename = f"{self.filename}.new"
        logger.info("Creating new output file", filename=target_filename)

        with open(target_filename, "w") as target_file:
            json.dump(data, target_file)

        logger.info("Output file created successfully", filename=target_filename)

After adding the above callback function, if you run the script again, you’ll see a few more lines of output from the new logging calls you added, and can then confirm that your new data file was successfully created and is identical to file1.json:

(diffsync_virtualenv) $ python numbers_script.py
2021-05-03 14:00.07 [info     ] Beginning diff calculation     [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
...
2021-05-03 14:00.07 [info     ] Sync complete                  [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
2021-05-03 14:00.07 [info     ] Creating new output file       [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> filename=file2.json.new flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
2021-05-03 14:00.07 [info     ] Output file created successfully [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> filename=file2.json.new flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
2021-05-03 14:00.07 [info     ] Beginning diff calculation     [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
2021-05-03 14:00.07 [info     ] Diff calculation complete      [diffsync.helpers] dst=<NumbersJSONAdapter "file2"> flags=<DiffSyncFlags.NONE: 0> src=<NumbersJSONAdapter "file1">
(no diffs)

(diffsync_virtualenv) $ diff file1.json file2.json.new
(diffsync_virtualenv) $

Conclusion

I hope this has been a helpful introduction to why and how to use DiffSync. In a future blog post we’ll dive into more advanced examples, including how to handle data sets consisting of multiple distinct database records or files, how to handle hierarchical or tree-like data sets, and how to handle data sets for which not all data attributes apply to all data sets.

-Glenn

Installing Nautobot - A Complete Walk-Through

2021-05-06T00:00:00+00:00

Nautobot 1.0 is here! With the release of Nautobot 1.0, we thought it’d be appropriate to post an article that walks a user through the entire Nautobot installation process.

This post is intended for those who may be new to Nautobot, those who may be installing it for the first time, those who may not have a lot of experience with Unix, those for whom it’s helpful to see the same information in a slightly different format to assist learning, or any combination of those.

This article follows and expands upon the Nautobot installation documentation. For example, this post will not just tell you which specific commands to run. It will also explain why you are running the command and what the command is doing, hopefully giving you a better understanding of the mechanics of the installation. Since this article is written for those of basic to intermediate knowledge with Unix and Nautobot installation, more advanced users may prefer the installation docs instead of this article.

This post will cover a basic installation, with all of Nautobot’s processes running on a single server running Ubuntu 20.04.2 LTS.

tim@nautobot10:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal
tim@nautobot10:~$

Each step in this process will show the command to be run, as well as an example of that command, and the resulting output from my actual install.

$ this is a command

nautobot@nautobot10:~$ this is a command
This is all the resulting output from the command from my actual install.
Showing the terminal commands and output hopefully provides context.

Nautobot OS Requirements

Nautobot supports installation on Ubuntu 20.04+, CentOS 8.2+, or Red Hat Enterprise Linux (RHEL) 8.2+. Other POSIX-compliant systems including practically any flavor of Linux, BSD, or even macOS should work, but are not officially supported.

Again, this article will document the install on a single server running Ubuntu 20.04.2.

Nautobot Dependency Requirements

Note: This article will feature several CLI snippets showing the actual installation. Any CLI action with the tim user will signify a user with root access. Any CLI action with the nautobot user will signify the user specific to the Nautobot environment.

Nautobot has the following minimum version requirements for its dependencies:

Python: 3.6
PostgreSQL: 9.6
Redis: 4.0

Optional Dependencies

Nautobot can operate without the following optional packages, but will not be suitable for use in most production environments without them:

uWSGI WSGI server or equivalent
NGINX HTTP server or equivalent
External authentication (this post will not cover external authentication)

Installation Overview

Nautobot’s installation process was designed to be simple. It has the following four main sections:

Installing the dependencies and infrastructure
Installing Nautobot
Configuring the web service and worker
HTTP server configuration

The next four sections will cover each of those areas in detail.

Install Nautobot Dependencies and Infrastructure

Nautobot must have the required packages along with an installed database infrastructure. This section will cover setting up that environment.

Install System Packages

First, update the package sources list:

$ sudo apt update -y

tim@nautobot10:~$ sudo apt update -y
[sudo] password for tim: 
Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu focal-security InRelease
Reading package lists... Done
Building dependency tree       
Reading state information... Done
57 packages can be upgraded. Run 'apt list --upgradable' to see them.
tim@nautobot10:~$

The following command will install the dependencies:

$ sudo apt install -y git python3 python3-pip python3-venv python3-dev postgresql redis-server

tim@nautobot10:~$ sudo apt install -y git python3 python3-pip python3-venv python3-dev postgresql redis-server
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3 is already the newest version (3.8.2-0ubuntu2).
python3 set to manually installed.
git is already the newest version (1:2.25.1-1ubuntu3.1).
git set to manually installed.
The following additional packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-9 dpkg-dev fakeroot g++ g++-9 gcc gcc-9 gcc-9-base libalgorithm-diff-perl
  libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan5 libatomic1 libbinutils libc-dev-bin libc6 libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdpkg-perl
 libexpat1-dev libfakeroot libfile-fcntllock-perl libgcc-9-dev
 . . . 
< --- snip --- >
 . . . 
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
tim@nautobot10:~$ 

Database Setup

Nautobot 1.0.0 supports the PostgreSQL database. MySQL support is on the road map.

Create the Nautobot Database

The following steps demonstrate how to create Nautobot’s database (named nautobot), create a user named nautobot, and then assign the nautobot user permissions to the nautobot database.

Sudo to the postgres user (the Postgres installation creates this user) and get to the postgres prompt: $ sudo -u postgres psql

tim@nautobot10:~$ sudo -u postgres psql
psql (12.6 (Ubuntu 12.6-0ubuntu0.20.04.1))
Type "help" for help.
 
postgres=#

Create the nautobot database:

CREATE DATABASE nautobot;

postgres=# CREATE DATABASE nautobot;
CREATE DATABASE
postgres=#

Create the nautobot user and assign a password:

CREATE USER nautobot WITH PASSWORD '<--don't use this password-->';

postgres=# CREATE USER nautobot WITH PASSWORD 'gipsy_danger_S45T%23}>@T2[?';
CREATE ROLE
postgres=#

WARNING: Don’t use the passwords from the examples.

Grant privileges for the nautobot user in the nautobot database:

postgres=# GRANT ALL PRIVILEGES ON DATABASE nautobot to nautobot;
GRANT
postgres=#

Exit Postgres:

postgres=# \q
tim@nautobot10:~$

Verify Database Authentication and Connection

Follow the example below to test database user nautobot’s authentication and connection to the nautobot database:

$ psql --username nautobot --password --host localhost nautobot

tim@nautobot10:~$ psql --username nautobot --password --host localhost nautobot
Password: 
psql (12.6 (Ubuntu 12.6-0ubuntu0.20.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
 
nautobot=> \conninfo
You are connected to database "nautobot" as user "nautobot" on host "localhost" (address "127.0.0.1") at port "5432".
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
nautobot=> \q
tim@nautobot10:~$ 

TIP: If something goes wrong (such as an authentication problem) with your database, you can easily delete the nautobot user and database with the following procedure:
tim@nautobot10:~$ sudo -u postgres psql
psql (12.6 (Ubuntu 12.6-0ubuntu0.20.04.1))
Type "help" for help.
 
postgres=# DROP DATABASE nautobot;
DROP DATABASE
postgres=# DROP USER nautobot;
DROP ROLE
postgres=# \q
tim@nautobot10:~$
You can then recreate it using the steps outlined in the Create the Nautobot Database section above.

Validate Redis

Redis was already installed in the prior apt install action. Verify that it’s working:

$ redis-cli ping

tim@nautobot10:~$ redis-cli ping
PONG
tim@nautobot10:~$ 

Install Nautobot

Select the `NAUTOBOT_ROOT` Directory and Create the `nautobot` System User

NAUTOBOT_ROOT is the directory where all Nautobot-related items will be installed.

NOTE: This install will use /opt/nautobot as NAUTOBOT_ROOT, but you may choose any directory you wish.

The following command:

Creates the nautobot user
Creates the NAUTOBOT_ROOT directory
Sets NAUTOBOT_ROOT as nautobot’s home directory

$ sudo useradd --system --shell /bin/bash --create-home --home-dir /opt/nautobot nautobot

tim@nautobot10:~$ sudo useradd --system --shell /bin/bash --create-home --home-dir /opt/nautobot nautobot
[sudo] password for tim: 
tim@nautobot10:~$ 

Create the Python Virtual Environment

The following command creates the Python virtual environment (venv) for the nautobot user in NAUTOBOT_ROOT (/opt/nautobot in this example):

$ sudo -u nautobot python3 -m venv /opt/nautobot

tim@nautobot10:~$ sudo -u nautobot python3 -m venv /opt/nautobot
tim@nautobot10:~$ 

Up to this point, we’ve not actually set NAUTOBOT_ROOT as a variable referring to a specific directory in the environment. Here is where that actually gets done:

$ echo "export NAUTOBOT_ROOT=/opt/nautobot" | sudo tee -a ~nautobot/.bashrc

tim@nautobot10:~$ echo "export NAUTOBOT_ROOT=/opt/nautobot" | sudo tee -a ~nautobot/.bashrc
export NAUTOBOT_ROOT=/opt/nautobot
tim@nautobot10:~$ 

The above command updates ~/.bashrc for nautobot so that any time you become nautobot, your NAUTOBOT_ROOT will be set automatically.

If you want you can verify the change in ~/.bashrc:

$ cat /opt/nautobot/.bashrc

tim@nautobot10:~$ cat /opt/nautobot/.bashrc 
# ~/.bashrc: executed by bash(1) for non-login shells.
. . . 
< ---- BIG snip ---- >
. . . 
fi
export NAUTOBOT_ROOT=/opt/nautobot
tim@nautobot10:~$ 

Sudo to `nautobot` and Explore the Virtual Environment

Nautobot must be installed by the nautobot user to prevent permissions problems.

$ sudo -iu nautobot

tim@nautobot10:~$ sudo -iu nautobot
nautobot@nautobot10:~$ 

Verify NAUTOBOT_ROOT:

$ echo $NAUTOBOT_ROOT

nautobot@nautobot10:~$ echo $NAUTOBOT_ROOT 
/opt/nautobot
nautobot@nautobot10:~$ 

Explore the virtual environment. Check the $PATH variable:

$ echo $PATH

nautobot@nautobot10:~$ echo $PATH
/opt/nautobot/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
nautobot@nautobot10:~$ 

Verify your pip3:

$ which pip3

nautobot@nautobot10:~$ which pip3
/opt/nautobot/bin/pip3
nautobot@nautobot10:~$ 

Automatic Access to the Virtual Environment

Earlier, we created the nautobot user’s venv using sudo -u nautobot python3 -m venv /opt/nautobot.

Calling venv for /opt/nautobot/ (NAUTOBOT_ROOT) created a directory structure within /opt/nautobot, including the bin, include, and lib directories therein. The /opt/nautobot/bin directory will hold all the packages for the venv.

NOTE: There are a number of directories within NAUTOBOT_ROOT that get created during Nautobot installation, including static, media, git, and jobs.

The nautobot user will automatically access the packages in the venv without need to explicitly initiate it because the first path in nautobot’s $PATH is /opt/nautobot/bin, which has all the packages that Nautobot requires.

Prepare the Virtual Environment

Update Pip to the latest version. We also want to install the wheel library, telling Pip to try to install wheel packages when they are available. Installing a package in wheel format can significantly improve the installation time and eliminates some requirements for development libraries.

$ pip3 install --upgrade pip wheel

nautobot@nautobot10:~$ pip3 install --upgrade pip wheel
Collecting pip
  Downloading pip-21.1-py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 3.1 MB/s 
Collecting wheel
  Downloading wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Installing collected packages: pip, wheel
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-21.1 wheel-0.36.2
nautobot@nautobot10:~$ 

Install Nautobot!

The time has come! Now install the Nautobot package as the nautobot user:

$ pip3 install nautobot

nautobot@nautobot10:~$ pip3 install nautobot
Collecting nautobot
  Downloading nautobot-1.0.0-py3-none-any.whl (11.1 MB)
     |████████████████████████████████| 11.1 MB 4.5 MB/s 
 . . . 
< --- snip --- >
Successfully installed Django-3.1.8 . . . nautobot-1.0.0 . . . urllib3-1.26.4
nautobot@nautobot10:~$

The nautobot-server command is your single gateway to all things Nautobot. Use it to verify your version:

$ nautobot-server --version

nautobot@nautobot10:~$ nautobot-server --version
1.0.0
nautobot@nautobot10:~$ 

Configure Nautobot

Nautobot’s configurations are stored in a file called nautobot_config.py. Running nautobot-server init creates this file by default in $NAUTOBOT_ROOT:

$ nautobot-server init

nautobot@nautobot10:~$ nautobot-server init
Configuration file created at '/opt/nautobot/nautobot_config.py'
nautobot@nautobot10:~$ 

There are a couple of changes in nautobot_config.py required to get Nautobot working. As the nautobot user, edit the file.

nautobot@nautobot10:~$ vi /opt/nautobot/nautobot_config.py

The first required change is the ALLOWED_HOSTS value. You can set this to ['*'] for a quick start, but that is not suitable for a production environment.

# This is a list of valid fully-qualified domain names (FQDNs) for the Nautobot server. Nautobot will not permit write
# access to the server via any other hostnames. The first FQDN in the list will be treated as the preferred name.
#
# Example: ALLOWED_HOSTS = ['nautobot.example.com', 'nautobot.internal.local']
ALLOWED_HOSTS = ['*']

WARNING: Attempting to access Nautobot via the Web UI via a URL not listed here will result in a Bad Request (400) return. We’ve not made it to the Web UI access part yet, but remember this for context.

You will also need to configure the DATABASES section of nautobot_config.py, updating the NAUTOBOT_USER and NAUTOBOT_PASSWORD settings with the values you defined in the prior psql command sequence. These are the minimum changes needed to run Nautobot on a local server; you can of course make additional changes as suits your particular environment.

# PostgreSQL database configuration. See the Django documentation for a complete list of available parameters:
#   https://docs.djangoproject.com/en/stable/ref/settings/#databases
DATABASES = {
    "default": {
        "NAME": os.getenv("NAUTOBOT_DATABASE", "nautobot"),  # Database name
        "USER": os.getenv("NAUTOBOT_USER", "nautobot"),  # Database username
        "PASSWORD": os.getenv("NAUTOBOT_PASSWORD", "gipsy_danger_S45T%23}>@T2[?"),  # Database password
        "HOST": os.getenv("NAUTOBOT_DB_HOST", "localhost"),  # Database server
        "PORT": os.getenv("NAUTOBOT_DB_PORT", ""),  # Database port (leave blank for default)
        "CONN_MAX_AGE": os.getenv("NAUTOBOT_DB_TIMEOUT", 300),  # Database timeout
        "ENGINE": "django.db.backends.postgresql",  # Database driver (Postgres only supported!)
    }
}

Specify Optional Packages Within `local_requirements.txt`

All required Python packages will be installed when running pip3 install nautobot.

Nautobot also supports optional packages that improve its functionality. For example, NAPALM is a powerful automation tool for network elements because it abstracts away the underlying OS, allowing the user to leverage a unified API when interacting with a network element.

NOTE: If you do not need to install any optional packages at this point, you can safely skip this step and proceed to Prepare the Database.

Optional packages should be listed in $NAUTOBOT_ROOT/local_requirements.txt. The example below will specify napalm as a local requirement.

To install NAPALM, add napalm to the local_requirements.txt file for later use.

NOTE: NAPALM_USERNAME and NAPALM_PASSWORD must be configured in the $NAUTOBOT_ROOT/nautobot_config.py file for Nautobot to successfully use NAPALM.

$ echo napalm >> $NAUTOBOT_ROOT/local_requirements.txt

nautobot@nautobot10:/opt$ echo napalm >> $NAUTOBOT_ROOT/local_requirements.txt
nautobot@nautobot10:/opt$ cat $NAUTOBOT_ROOT/local_requirements.txt 
napalm
nautobot@nautobot10:/opt$ 

Prepare the Database

Nautobot’s database must be migrated prior to use. This creates the database tables and relationships:

$ nautobot-server migrate

nautobot@nautobot10:~$ nautobot-server migrate
Wrapping model clean methods for custom validators failed because the ContentType table was not available or populated. This is normal during the execution of the migration command for the first time.
Operations to perform:
  Apply all migrations: admin, auth, circuits, contenttypes, dcim, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0001_initial... OK
  Applying auth.0002_alter_permission_name_max_length... OK
 . . . 
< --- snip --- >
 . . . 
   Applying extras.0004_populate_default_status_records...
    Model dcim.Device
      Adding and linking status Offline
      Adding and linking status Active
      Adding and linking status Planned
      Adding and linking status Staged
      Adding and linking status Failed
      Adding and linking status Inventory
      Adding and linking status Decommissioning
 . . . 
< --- snip --- >
 . . . 
    Model circuits.Circuit
      Linking to existing status Planned
      Adding and linking status Provisioning
      Linking to existing status Active
      Linking to existing status Offline
      Adding and linking status Deprovisioning
      Adding and linking status Decommissioned
 . . . 
< --- snip --- >
 . . . 
    Added 19, linked 29 status records
  Applying taggit.0003_taggeditem_add_unique_index... OK
nautobot@nautobot10:~$ 

NOTE: In the above output, notice the multiple database tables being created; a few are shown for context

Create a Superuser and the Static Directories

The next step is to create a Nautobot superuser. This will be the account that you will use to log into the Nautobot Web UI for the first time.

The superuser will be an administrative account that will allow you to create other users, set permissions, create security tokens, etc.

NOTE: The email address field is not required, but be sure to use a very strong password.

$ nautobot-server createsuperuser

nautobot@nautobot10:~$ nautobot-server createsuperuser
Username: tim
Email address: 
Password: 
Password (again): 
Superuser created successfully.
nautobot@nautobot10:~$ 

Nautobot has several directories within $NAUTOBOT_ROOT (default location) that store static files for operations such as:

Git repos (git directory)
Custom jobs (jobs directory)
Media such as images and attachments (media directory)
CSS stylesheets for Web rendering (static directory)

Here is a view of all the directories in $NAUTOBOT_ROOT

nautobot@nautobot10:~$ ls -ld $NAUTOBOT_ROOT/*/
drwxrwxr-x  3 nautobot nautobot 4096 Apr 30 19:26 /opt/nautobot/bin/
drwxrwxr-x  2 nautobot nautobot 4096 Apr 28 20:35 /opt/nautobot/git/
drwxrwxr-x  2 nautobot nautobot 4096 Apr 28 19:55 /opt/nautobot/include/
drwxrwxr-x  2 nautobot nautobot 4096 Apr 28 20:35 /opt/nautobot/jobs/
drwxrwxr-x  3 nautobot nautobot 4096 Apr 28 19:55 /opt/nautobot/lib/
drwxrwxr-x  3 nautobot nautobot 4096 Apr 28 19:55 /opt/nautobot/lib64/
drwxrwxr-x  4 nautobot nautobot 4096 Apr 28 20:35 /opt/nautobot/media/
drwxrwxr-x  3 nautobot nautobot 4096 Apr 28 19:55 /opt/nautobot/share/
drwxrwxr-x 19 nautobot nautobot 4096 Apr 28 21:53 /opt/nautobot/static/
nautobot@nautobot10:~$ 

The collectstatic command will create these directories if they don’t exist; it will also copy the required files to the static directory:

$ nautobot-server collectstatic

nautobot@nautobot10:~$ nautobot-server collectstatic
 
960 static files copied to '/opt/nautobot/static'.
nautobot@nautobot10:~$ 

Install Local Requirements

NOTE: If you did not specify a local_requirements.txt file prior, you can skip to the Check Your Configuration section.

A prior section detailed how to specify optional packages for install within $NAUTOBOT_ROOT/local_requirements.txt. Now it’s time to install those optional packages as the nautobot user.

$ pip3 install -r $NAUTOBOT_ROOT/local_requirements.txt

nautobot@nautobot10:~$ pip3 install -r $NAUTOBOT_ROOT/local_requirements.txt
Collecting napalm
  Downloading napalm-3.2.0-py2.py3-none-any.whl (230 kB)
     |████████████████████████████████| 230 kB 3.1 MB/s 
Requirement already satisfied: pyYAML in ./lib/python3.8/site-packages (from napalm->-r /opt/nautobot/local_requirements.txt (line 1)) (5.4.1)
 . . . 
< --- snip --- >
 . . . 
Successfully installed bcrypt-3.2.0 ciscoconfparse-1.5.30 colorama-0.4.4 dnspython-2.1.0 future-0.18.2 junos-eznc-2.6.0 lxml-4.6.3 napalm-3.2.0 ncclient-0.6.9 netmiko-3.4.0 ntc-templates-2.0.0 paramiko-2.7.2 passlib-1.7.4 pyeapi-0.8.4 pynacl-1.4.0 pyserial-3.5 scp-0.13.3 tenacity-7.0.0 textfsm-1.1.0 transitions-0.8.8 yamlordereddictloader-0.4.0
nautobot@nautobot10:~$ 

Check Your Configuration**

Nautobot uses Django’s native system check framework to validate the configuration, detect common problems, and provide hints for how to fix them.

These checks run automatically when using nautobot-server runserver (which we will get to in a moment), but not when running in production using WSGI. Nevertheless, it’s good to get into the habit of running the checks before deployments!

$ nautobot-server check

nautobot@nautobot10:~$ nautobot-server check
System check identified no issues (0 silenced).
nautobot@nautobot10:~$ 

** I really wanted to title this section “Check Yourself Before You Wreck Yourself”; please consider that to be the alternate title.

Test the Application

Now it’s time to see this installation’s Web UI for the first time! Start Nautobot’s development server on port 8000.

$ nautobot-server runserver 0.0.0.0:8000 --insecure

nautobot@nautobot10:~$ nautobot-server runserver 0.0.0.0:8000 --insecure
Performing system checks...
 
System check identified no issues (0 silenced).
April 28, 2021 - 21:55:50
Django version 3.1.8, using settings 'nautobot_config'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
[28/Apr/2021 21:56:20] "GET / HTTP/1.1" 200 18684

DANGER: DO NOT USE THIS SERVER IN A PRODUCTION SETTING. The development server is for development and testing purposes only. It is neither performant nor secure enough for production use.

From your local host, connect to the name or IP of the server (as defined in ALLOWED_HOSTS) on port 8000; for example, http://127.0.0.1:8000/.

NOTE: The URL that you use in order to reach the UI must be included in the ALLOWED_HOSTS list in nautobot_config.py; if not, you will receive a Bad Request (400) return.

You can also reach Nautobot from a remote host. The example below shows reaching the development server UI via the remote server’s IP address.

Here is some sample output you’ll see from the development server as you access the Web UI:

[28/Apr/2021 21:56:20] "GET /static/materialdesignicons-5.4.55/css/materialdesignicons.min.css HTTP/1.1" 200 269370
[28/Apr/2021 21:56:20] "GET /static/bootstrap-3.4.1-dist/css/bootstrap.min.css HTTP/1.1" 200 121457
[28/Apr/2021 21:56:20] "GET /static/jquery-ui-1.12.1/jquery-ui.css HTTP/1.1" 200 37326
[28/Apr/2021 21:56:20] "GET /static/select2-4.0.13/dist/css/select2.min.css HTTP/1.1" 200 14966
[28/Apr/2021 21:56:20] "GET /static/flatpickr-4.6.3/themes/light.css HTTP/1.1" 200 18996
[28/Apr/2021 21:56:20] "GET /static/select2-bootstrap-0.1.0-beta.10/select2-bootstrap.min.css HTTP/1.1" 200 16792
[28/Apr/2021 21:56:20] "GET /static/css/base.css?v1.0.0 HTTP/1.1" 200 8528
[28/Apr/2021 21:56:20] "GET /static/jquery/jquery-3.6.0.min.js HTTP/1.1" 200 89501
[28/Apr/2021 21:56:20] "GET /static/jquery-ui-1.12.1/jquery-ui.min.js HTTP/1.1" 200 253669
[28/Apr/2021 21:56:21] "GET /static/bootstrap-3.4.1-dist/js/bootstrap.min.js HTTP/1.1" 200 39680
[28/Apr/2021 21:56:21] "GET /static/clipboard.js/clipboard-2.0.6.min.js HTTP/1.1" 200 10454
[28/Apr/2021 21:56:21] "GET /static/img/nautobot_logo.svg HTTP/1.1" 200 13318
[28/Apr/2021 21:56:21] "GET /static/js/forms.js?v1.0.0 HTTP/1.1" 200 18603
[28/Apr/2021 21:56:21] "GET /static/flatpickr-4.6.3/flatpickr.min.js HTTP/1.1" 200 48518
[28/Apr/2021 21:56:21] "GET /static/select2-4.0.13/dist/js/select2.min.js HTTP/1.1" 200 70891
[28/Apr/2021 21:56:21] "GET /static/materialdesignicons-5.4.55/fonts/materialdesignicons-webfont.woff2?v=5.8.55 HTTP/1.1" 200 319984
[28/Apr/2021 21:56:21] "GET /static/img/favicon.ico HTTP/1.1" 200 15086

Notice that the UI will be locked down until you authenticate. Click the Log in button on the upper right. You will see the login screen.

When you authenticate in you’ll see more debug output at the development server’s command line:

[28/Apr/2021 21:58:17] "GET /login/?next=/ HTTP/1.1" 200 8332
[28/Apr/2021 21:58:34] "POST /login/ HTTP/1.1" 302 0
[28/Apr/2021 21:58:34] "GET / HTTP/1.1" 200 58444

Once you authenticate, you will see an unlocked UI with no data:

Back in your terminal, type Ctrl-C to stop the development server. Now you’re ready to proceed to starting Nautobot as a system service.

Configure Nautobot’s Web Service and Worker

Nautobot runs as a Web Server Gateway Interface (WSGI) application behind an HTTP server.

Nautobot comes pre-installed with uWSGI to use as the WSGI server, but other WSGI servers should work equally as well. This demo will use the pre-installed uWSGI server, guiding you through configuring uWSGI and establishing Nautobot to run on system startup.

Configure uWSGI

As the nautobot user, copy and paste the following into $NAUTOBOT_ROOT/uwsgi.ini:

[uwsgi]
; The IP address (typically localhost) and port that the WSGI process should listen on
socket = 127.0.0.1:8001

; Fail to start if any parameter in the configuration file isn’t explicitly understood by uWSGI
strict = true

; Enable master process to gracefully re-spawn and pre-fork workers
master = true

; Allow Python app-generated threads to run
enable-threads = true

;Try to remove all of the generated file/sockets during shutdown
vacuum = true

; Do not use multiple interpreters, allowing only Nautobot to run
single-interpreter = true

; Shutdown when receiving SIGTERM (default is respawn)
die-on-term = true

; Prevents uWSGI from starting if it is unable load Nautobot (usually due to errors)
need-app = true

; By default, uWSGI has rather verbose logging that can be noisy
disable-logging = true

; Assert that critical 4xx and 5xx errors are still logged
log-4xx = true
log-5xx = true

;
; Advanced settings (disabled by default)
; Customize these for your environment if and only if you need them.
; Ref: https://uwsgi-docs.readthedocs.io/en/latest/Options.html
;

; Number of uWSGI workers to spawn. This should typically be 2n+1, where n is the number of CPU cores present.
; processes = 5

; If using subdirectory hosting e.g. example.com/nautobot, you must uncomment this line. Otherwise you'll get double paths e.g. example.com/nautobot/nautobot/.
; See: https://uwsgi-docs.readthedocs.io/en/latest/Changelog-2.0.11.html#fixpathinfo-routing-action
; route-run = fixpathinfo:

This is a basic configuration that should suffice for most initial installations.

Setup systemd

The systemd suite will control uWSGI as well as Nautobot’s background worker processes.

Configure Nautobot Service

With root permissions, copy and paste the following into /etc/systemd/system/nautobot.service:

[Unit]
Description=Nautobot WSGI Service
Documentation=https://nautobot.readthedocs.io/en/stable/
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment="NAUTOBOT_ROOT=/opt/nautobot"

User=nautobot
Group=nautobot
PIDFile=/var/tmp/nautobot.pid
WorkingDirectory=/opt/nautobot

ExecStart=/opt/nautobot/bin/nautobot-server start --pidfile /var/tmp/nautobot.pid --ini /opt/nautobot/uwsgi.ini
ExecStop=/opt/nautobot/bin/nautobot-server start --stop /var/tmp/nautobot.pid
ExecReload=/opt/nautobot/bin/nautobot-server start --reload /var/tmp/nautobot.pid

Restart=on-failure
RestartSec=30
PrivateTmp=true

[Install]
WantedBy=multi-user.target

NOTE: Notice the above configuration leverages the nautobot-server start management command that directly invokes uWSGI. The nautobot-server command is meant to server as the single entrypoint into the Nautobot application.

Configure Nautobot Worker Service

To configure the Nautobot Worker, with root permissions, copy and paste the following into /etc/systemd/system/nautobot-worker.service:

[Unit]
Description=Nautobot Request Queue Worker
Documentation=https://nautobot.readthedocs.io/en/stable/
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment="NAUTOBOT_ROOT=/opt/nautobot"

User=nautobot
Group=nautobot
WorkingDirectory=/opt/nautobot

ExecStart=/opt/nautobot/bin/nautobot-server rqworker

Restart=on-failure
RestartSec=30
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Reload `systemd` and Verify the Nautobot Service

Reload the systemd daemon to pick up the changes in the new services:

$ sudo systemctl daemon-reload

tim@nautobot10:~$ sudo systemctl daemon-reload 
tim@nautobot10:~$ 

Now start the nautobot and nautobot-worker services and enable them to initiate at boot:

$ sudo systemctl enable --now nautobot nautobot-worker

tim@nautobot10:~$ sudo systemctl enable --now nautobot nautobot-worker
Created symlink /etc/systemd/system/multi-user.target.wants/nautobot.service → /etc/systemd/system/nautobot.service.
Created symlink /etc/systemd/system/multi-user.target.wants/nautobot-worker.service → /etc/systemd/system/nautobot-worker.service.
tim@nautobot10:~$ 

NOTE: The /etc/systemd/system/multi-user.target in the output above refers to the Runlevel (0-6) that the machine transits when powering on. Runlevel 0 is poweroff.target, Runlevel 5 is graphical.target (full user access with graphical display and networking). The symlinks created above start /etc/systemd/system/nautobot.service and /etc/systemd/system/nautobot-worker.service as the machine transits Runlevel 2 (multi-user.target).

Now use systemctl status nautobot.service to verify that the Nautobot service is running:

$ systemctl status nautobot.service

tim@nautobot10:~$ systemctl status nautobot.service 
● nautobot.service - Nautobot WSGI Service
     Loaded: loaded (/etc/systemd/system/nautobot.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-04-28 22:39:54 UTC; 1min 3s ago
       Docs: https://nautobot.readthedocs.io/en/stable/
   Main PID: 2234541 (nautobot-server)
      Tasks: 2 (limit: 4585)
     Memory: 91.5M
     CGroup: /system.slice/nautobot.service
             ├─2234541 /opt/nautobot/bin/python3 /opt/nautobot/bin/nautobot-server start --pidfile /var/tmp/nautobot.pid -->
             └─2234793 /opt/nautobot/bin/python3 /opt/nautobot/bin/nautobot-server start --pidfile /var/tmp/nautobot.pid -->
 
Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: --- Python VM already initialized ---
Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: Python main interpreter initialized at 0x18df740
Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: python threads support enabled
Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: your server socket listen backlog is limited to 100 connections
Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: your mercy for graceful operations on workers is 60 seconds
Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: mapped 145808 bytes (142 KB) for 1 cores
Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: *** Operational MODE: single process ***
Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x18df740>
Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: spawned uWSGI master process (pid: 2234541)
Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: spawned uWSGI worker 1 (pid: 2234793, cores: 1)

Because we like being thorough, let’s also check the Nautobot worker process:

$ systemctl status nautobot-worker.service

tim@nautobot10:~$ systemctl status nautobot-worker.service 
● nautobot-worker.service - Nautobot Request Queue Worker
     Loaded: loaded (/etc/systemd/system/nautobot-worker.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-05-04 22:10:11 UTC; 1min 4s ago
       Docs: https://nautobot.readthedocs.io/en/stable/
   Main PID: 1122 (nautobot-server)
      Tasks: 2 (limit: 4585)
     Memory: 110.6M
     CGroup: /system.slice/nautobot-worker.service
             └─1122 /opt/nautobot/bin/python3 /opt/nautobot/bin/nautobot-server rqworker

May 04 22:10:11 nautobot10 systemd[1]: Started Nautobot Request Queue Worker.
May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 Worker rq:worker:f24092394e3e4217a7bfafc300ddbac9: started, ver>
May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 Subscribing to channel rq:pubsub:f24092394e3e4217a7bfafc300ddba>
May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 *** Listening on default, check_releases, custom_fields, webhoo>
May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 Cleaning registries for queue: default
May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 Cleaning registries for queue: check_releases
May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 Cleaning registries for queue: custom_fields
May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 Cleaning registries for queue: webhooks

NOTE: The nautobot.service process is responsible for managing the WebUI and API; the nautobot-worker.service process subscribes to Redis Queue (RQ) and manages jobs, consumes tasks, and publishes results to the dababase.

Troubleshooting

If the Nautobot service fails to start, issue the command journalctl -eu nautobot.service to check for log messages that may indicate the problem.

For further troubleshooting steps, refer to the Nautobot documentation WSGI troubleshooting section.

Configure HTTP Server

We are almost there, I promise! This last section covers HTTP server configuration, which is the last step!

Get an SSL Certificate

To run Nautobot with HTTPS you will need a certificate, preferably one from a trusted commercial provider. To get off the ground, and depending on your organization’s policies, you may be able to use a self-signed certificate for testing and then later implement a trusted certificate.

WARNING: A certificate from a trusted authority is highly recommended for production.

Two files will be needed: the public certificate (nautobot.crt) and the private key (nautobot.key).

This post will initially use a self-signed certificate for testing purposes, and will later cover the results of obtaining a trusted certificate from Let’s Encrypt in an addendum at the end of the article.

Self-Signed Cert

The command below will generate a self-signed certificate, used for testing (root permissions required!):

$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout /etc/ssl/private/nautobot.key \
  -out /etc/ssl/certs/nautobot.crt

NOTE: It is not necessary to answer the questions in the self-signed certificate generation procedure.

tim@nautobot10:~$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
>   -keyout /etc/ssl/private/nautobot.key \
>   -out /etc/ssl/certs/nautobot.crt
[sudo] password for tim: 
Generating a RSA private key
..........................................................................................+++++
..........+++++
writing new private key to '/etc/ssl/private/nautobot.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:CO
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:
tim@nautobot10:~$ 

HTTP Server Installation

Nautobot will support any HTTP server. Nautobot’s documentation covers setup instructions for NGINX, and this post will reprise those.

NGINX Installation

First, install NGINX (root permissions required):

$ sudo apt install -y nginx

tim@nautobot10:~$ sudo apt install -y nginx
Reading package lists... Done
Building dependency tree       
Reading state information... Done
 . . . 
< --- snip --- >
 . . . 
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
tim@nautobot10:~$ 

NGINX Configuration

Once NGINX is installed, copy and paste the following NGINX configuration into /etc/nginx/sites-available/nautobot.conf (root permissions required):

server {
    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;
 
    server_name _;
 
    ssl_certificate /etc/ssl/certs/nautobot.crt;
    ssl_certificate_key /etc/ssl/private/nautobot.key;
 
    client_max_body_size 25m;
 
    location /static/ {
        alias /opt/nautobot/static/;
    }
 
    # For subdirectory hosting, you'll want to toggle this (e.g. `/nautobot/`).
    # Don't forget to set `FORCE_SCRIPT_NAME` in your `nautobot_config.py` to match.
    # location /nautobot/ {
    location / {
        include uwsgi_params;
        uwsgi_pass  127.0.0.1:8001;
        uwsgi_param Host $host;
        uwsgi_param X-Real-IP $remote_addr;
        uwsgi_param X-Forwarded-For $proxy_add_x_forwarded_for;
        uwsgi_param X-Forwarded-Proto $http_x_forwarded_proto;
 
        # If you want subdirectory hosting, uncomment this. The path must match
        # the path of this location block (e.g. `/nautobot`). For NGINX the path
        # MUST NOT end with a trailing "/".
        # uwsgi_param SCRIPT_NAME /nautobot;
    }
}
 
server {
    # Redirect HTTP traffic to HTTPS
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    return 301 https://$host$request_uri;
}

NOTE: This config redirects HTTP to HTTPS

Enable Nautobot

To enable the Nautobot Web UI, you’ll need to delete /etc/nginx/sites-enabled/default and create a symbolic link in the sites-enabled directory to the configuration file you just created:

As user with root permissions:

$ sudo rm -f /etc/nginx/sites-enabled/default

$ sudo ln -s /etc/nginx/sites-available/nautobot.conf /etc/nginx/sites-enabled/nautobot.conf

tim@nautobot10:~$ sudo rm -f /etc/nginx/sites-enabled/default 
tim@nautobot10:~$ sudo ln -s /etc/nginx/sites-available/nautobot.conf /etc/nginx/sites-enabled/nautobot.conf
tim@nautobot10:~$ ls -l /etc/nginx/sites-enabled/
total 0
lrwxrwxrwx 1 root root 40 Apr 29 16:03 nautobot.conf -> /etc/nginx/sites-available/nautobot.conf
tim@nautobot10:~$ 

Restart NGINX

Restart NGINX to use the new configuration (in /etc/nginx/ above):

$ sudo systemctl restart nginx

tim@nautobot10:~$ sudo systemctl restart nginx
tim@nautobot10:~$ 

If the restart fails, you will see some type of notification at the CLI when you run the restart command.

Although we received no error message above, let’s run a check to see NGINX’s status:

tim@nautobot10:~$ systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-04-29 20:53:49 UTC; 17min ago
       Docs: man:nginx(8)
    Process: 2738959 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 2738960 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 2738961 (nginx)
      Tasks: 3 (limit: 4585)
     Memory: 12.0M
     CGroup: /system.slice/nginx.service
             ├─2738961 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─2738962 nginx: worker process
             └─2738963 nginx: worker process
 
Apr 29 20:53:49 nautobot10 systemd[1]: Starting A high performance web server and a reverse proxy server...
Apr 29 20:53:49 nautobot10 systemd[1]: Started A high performance web server and a reverse proxy server.
tim@nautobot10:~$ 

The Home Stretch - Connect to Nautobot!

In a browser, connect to Nautobot. You should not have to specify a port.

REMEMBER: You must connect using a valid URL from ALLOWED_HOSTS in $NAUTOBOT_ROOT/nautobot_config.py.

If you used a self-signed certificate, your browser will likely warn you that attempting to connect to the site you just created is very dangerous and will likely result in puppies being harmed if you connect.

Connect anyway, but not because you hate puppies.

NOTE: In Google Chrome, depending upon your settings, you may have to explicitly type ‘thisisunsafe’ into the browser while seeing the warning screen below in order to connect. While doing so, you won’t see your typing anywhere in the browser.

You should now see something like this:

Congratulations! This completes the Nautobot installation on Ubuntu 20.04.2.

Troubleshooting

If you do not see a successful result, you can find troubleshooting actions here, including actions for a 502 Bad Gateway error.

Wrapping Up

This blog covered Nautobot installation on a fresh version of Ubuntu 20.04.2. It walked you through a complete installation, providing context for each step and real output from a real installation. I hope this was helpful! If you have any comments on anything that did not work, was not clear, something else that should have been included, or any other feedback please comment below or reach out to me on NtC’s public Slack channel.

Have a great day!

-Tim

Addendum: Getting a Trusted Certificate

If you used a self-signed certificate to get to this point, you may have questions on how to obtain a trusted commercial certificate. In a separate installation, I initially used a self-signed certificate, but went back and used Let’s Encrypt to get a trusted certificate. Although obtaining a trusted certificate is beyond the formal scope of this article, I wanted to briefly detail my experience going through that process. This section will cover the highlights.

Go to Let’s Encrypt
Since we have shell access on the Nautobot server, select the option to use the Certbot client
Select NGINX software on Ubuntu 20.04 OS, as in the picture below:

That page above had some directions to follow
Following those directions, choose to get a certificate and have Certbot edit the NGINX configuration (/etc/nginx/sites-available/nautobot.conf)
- The other option is simply to receive a certificate and install it yourself

After that process, comment out the original self-signed certificate info in /etc/nginx/sites-available/nautobot.conf
Here is what the /etc/nginx/sites-available/nautobot.conf file looked like after that process:
- Notice the sections within that say managed by Certbot

# THIS IS MY ORIGINAL SELF-SIGNED CERT INFO I COMMENTED OUT
#server {
#    listen 443 ssl http2 default_server;
#    listen [::]:443 ssl http2 default_server;
#
#    server_name _;
#
#    ssl_certificate /etc/ssl/certs/nautobot.crt;
#    ssl_certificate_key /etc/ssl/private/nautobot.key;
#
#    client_max_body_size 25m;
#
#    location /static/ {
#        alias /opt/nautobot/static/;
#    }
#
#    # For subdirectory hosting, you'll want to toggle this (e.g. `/nautobot/`).
#    # Don't forget to set `FORCE_SCRIPT_NAME` in your `nautobot_config.py` to match.
#    # location /nautobot/ {
#    location / {
#        include uwsgi_params;
#        uwsgi_pass  127.0.0.1:8001;
#        uwsgi_param Host $host;
#        uwsgi_param X-Real-IP $remote_addr;
#        uwsgi_param X-Forwarded-For $proxy_add_x_forwarded_for;
#        uwsgi_param X-Forwarded-Proto $http_x_forwarded_proto;
#
#        # If you want subdirectory hosting, uncomment this. The path must match
#        # the path of this location block (e.g. `/nautobot`). For NGINX the path
#        # MUST NOT end with a trailing "/".
#        # uwsgi_param SCRIPT_NAME /nautobot;
#    }
#
#}
 
server {
    # Redirect HTTP traffic to HTTPS
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    return 301 https://$host$request_uri;
}
server {
    listen 443 ssl http2 ;
    listen [::]:443 ssl http2 ;
    server_name fiola-nautobot.cloud.networktocode.com; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/fiola-nautobot.cloud.networktocode.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/fiola-nautobot.cloud.networktocode.com/privkey.pem; # managed by Certbot
 
    client_max_body_size 25m;
 
    location /static/ {
        alias /opt/nautobot/static/;
    }
 
    # For subdirectory hosting, you'll want to toggle this (e.g. `/nautobot/`).

Connect to Nautobot in a new browser window, note that the Not Secure warning in the URL window is now a lock, indicating a trusted certificate:

Ansible Become Logging

2021-05-04T00:00:00+00:00

This blog post aims to assist Ansible users in providing documentation to security teams when using privileged escalation (become) in Ansible playbooks.

The Problem Statement

Ansible does not use a specific command when using privileged escalation on remote machines see here for more details. As an example, if you wanted to restart the httpd service, the command you run on the system would be sudo systemctl restart httpd but when Ansible executes the task, you would see something like /bin/sh BECOME-SUCCESS ; /usr/bin/python /home/myuser/.ansible/tmp/ansible-tmp-abcdef/AnsiballZ_12345.py. Here is a link explaining the Ansiballz framework. This presents two concerns to most security teams. First, they cannot limit the commands used by Ansible. Second, they cannot see which commands Ansible executes. We will be addressing the second concern.

The Solution

Ansible provides Callback Plugins that can be used when Ansible responds to various events inside a playbook. Ansible provides a small set of default callback plugins out of the box, but none that address our problem.

Custom Callback Plugin

We will be creating a custom callback plugin that hooks into the use of become and logs to stdout. This will not log the actual command used by Ansible but the module and arguments passed to the module. As an example, instead of systemctl restart httpd the log would show the ansible.builtin.service module with the name set to httpd and the state set to restarted.

First we need to build out our directory structure. We are choosing the name “become” for our plugin, but you could choose your own name. By convention, the filename and plugin name should match, but they do not have to. This assumes the callback plugin will be added to an existing Ansible project:

├── ansible.cfg                 // Ansible Configuration file.
├── example_pb.yaml             // Ansible playbook.
├── callback_plugins                  
│   └── become.py               // Our Become Callback Plugin.

mkdir callback_plugins
touch callback_plugins/become.py

Then add the callback plugin to your ansible.cfg file.

# ansible.cfg
[defaults]
callback_whitelist = become

From there we will create a sample playbook that restarts httpd:

# example_pb.yaml

---
- hosts: all
  tasks:
    - name: "ESCALATED TASK: RESTART HTTPD"
      ansible.builtin.service:
        name: httpd
        state: restarted
      become: true
    
    - name: "NON-ESCALATED TASK: DEBUG"
      ansible.builtin.debug:
        msg: "This task does not require escalation"

Next we will add the basic structure to our callback plugin. Ansible provides an example plugin that we will modify.

# become.py
# Make coding more python3-ish, this is required for contributions to Ansible
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

# not only visible to ansible-doc, it also 'declares' the options the plugin requires and how to configure them.
DOCUMENTATION = '''
  callback: become
  requirements:
    - whitelist in configuration
  short_description: When become is used, log to stdout.
  description:
      - This callback will log the become, become method, task name, action and arguments to stdout.
'''

from ansible.plugins.callback import CallbackBase


class CallbackModule(CallbackBase):
    """
    This callback module tells you when become is used.
    """
    CALLBACK_VERSION = 2.0
    CALLBACK_NAME = 'become'

    # only needed if you ship it and don't want to enable by default
    CALLBACK_NEEDS_WHITELIST = True

    def __init__(self):

        # make sure the expected objects are present, calling the base's __init__
        super(CallbackModule, self).__init__()

We have now added the skeleton that Ansible requires, but we have not added any new functionality to our plugin. Callback plugins listen for events that Ansible emits. Check out the callback init file in the Ansible repository for a list of all the events supported. The event that we will be looking at hooking into is the v2_runner_on_start, since it passes not only the task but also the host, and it executes at the start of a task. Let’s add our first callback now.

# become.py
class CallbackModule(CallbackBase):
...
    def v2_runner_on_start(self, host, task):
        """
        Process the runner on start event.
        """
        if task.become:
            print(f"Hostname: {host.name}")
            print(f"Task: {task.name}")
            print(f"Action: {task.action}")
            print(f"Arguments: {task.args}")

We are accessing attributes of task and host here. In this example we are showing just a few, but you can use dir(task) or dir(host) to find many others. When we execute the playbook, we should see the following:

TASK [Gathering Facts]****************************
ok: [my_hostname]
TASK [ESCALATED TASK: RESTART HTTPD]**************
Hostname: my_hostname
Task: ESCALATED TASK: RESTART HTTPD
Action: ansible.builtin.service
Arguments: {'name': 'httpd', 'state': 'restarted'}
changed: [my_hostname]
TASK [NON-ESCALATED TASK: DEBUG]
ok: [my_hostname] => {
    "msg": "This task does not require escalation"
}

Supporting Variables

At this point, We have a callback plugin that will log to stdout the use of become in our playbooks. But if our playbook uses variables, we will not see those variables. Let’s modify our example to show:

# example_pb.yaml

---
- hosts: all
  tasks:
    - name: "ESCALATED TASK: RESTART HTTPD"
      ansible.builtin.service:
        name: "{{ service }}"
        state: restarted
      become: true
      vars:
        service: httpd
    
    - name: "NON-ESCALATED TASK: DEBUG"
      ansible.builtin.debug:
        msg: "This task does not require escalation"

Now let’s run the playbook:

TASK [Gathering Facts]****************************
ok: [my_hostname]
TASK [ESCALATED TASK: RESTART HTTPD]**************
Hostname: my_hostname
Task: ESCALATED TASK: RESTART HTTPD
Action: ansible.builtin.service
Arguments: {'name': '{{ service }}', 'state': 'restarted'}
changed: [my_hostname]
TASK [NON-ESCALATED TASK: DEBUG]
ok: [my_hostname] => {
    "msg": "This task does not require escalation"
}

In order to access variables passed into a task (or from group vars), we need to add the Templar class to our callback plugin. We will need to add a few more hooks to get all of the play data and the hostvars.

# become.py

from ansible.plugins.callback import CallbackBase
from ansible.template import Templar


class CallbackModule(CallbackBase):
...

    def v2_playbook_on_start(self, playbook):
        """
        Initialize self.playbook.
        """
        self.playbook = playbook

    def v2_playbook_on_play_start(self, play):
        """
        Get the variable manager of the current play from the playbook.
        """
        self.play = play
        self.vm = play.get_variable_manager()

    def _all_vars(self, host=None, task=None):
        """
        Load all variables for the given inputs.
        """
        return self.vm.get_vars(
            play = self.play,
            host = host,
            task = task
        )

    def v2_runner_on_start(self, host, task):
        """
        Process the runner on start event.
        """
        templar = Templar(loader=self.playbook.get_loader(),
                  variables=self._vars(host=host, task=task))

        if task.become:
            print(f"Hostname: {host.name}")
            print(f"Task: {task.name}")
            print(f"Action: {task.action}")
            print(f"Arguments: {templar.template(task.args, fail_on_undefined=False)}")

With the above changes, running the playbook will now fill in the variables:

TASK [Gathering Facts]****************************
ok: [my_hostname]
TASK [ESCALATED TASK: RESTART HTTPD]**************
Hostname: my_hostname
Task: ESCALATED TASK: RESTART HTTPD
Action: ansible.builtin.service
Arguments: {'name': 'httpd', 'state': 'restarted'}
changed: [my_hostname]
TASK [NON-ESCALATED TASK: DEBUG]
ok: [my_hostname] => {
    "msg": "This task does not require escalation"
}

The Next Step

Now that we know how to access the task attributes and variables, we could extend any of the other popular callback plugins, such as syslog_json, Logstash, or Splunk, to send the data directly to the security team.

There are a few Ansible playbook options that are not covered here, such as loop or with_items, become_method, or become_user, but this should be enough to get you started. Another suggested change would be to set CALLBACK_NEEDS_WHITELIST = False. With this change, we would no longer need to whitelist become in the ansible.cfg file. This, paired with Ansible Tower or AWX, would allow the security team to ensure conformance to standards. For this to work, the plugin would need to be moved to the environment created by Ansible Tower or AWX. See Adding a plugin locally for more details.

Conclusion

Ansible callback plugins give us the ability to provide detailed evidence of our privileged escalation to answer some of the security team’s concerns.

Thanks for taking the time to read!

-Stephen

Here is the full code from become.py:

# become.py
# Make coding more python3-ish, this is required for contributions to Ansible
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

# not only visible to ansible-doc, it also 'declares' the options the plugin requires and how to configure them.
DOCUMENTATION = '''
  callback: become
  requirements:
    - whitelist in configuration
  short_description: Logs become use to stdout
  description:
      - This callback will log the become, become method, task name, action and arguments to stdout.
'''

from ansible.plugins.callback import CallbackBase
from ansible.template import Templar


class CallbackModule(CallbackBase):
    """
    This callback module tells you when become is used.
    """
    CALLBACK_VERSION = 2.0
    CALLBACK_NAME = 'become'

    # only needed if you ship it and don't want to enable by default
    CALLBACK_NEEDS_WHITELIST = True

    def __init__(self):

        # make sure the expected objects are present, calling the base's __init__
        super(CallbackModule, self).__init__()

    def v2_playbook_on_start(self, playbook):
        """
        Initialize self.playbook.
        """
        self.playbook = playbook

    def v2_playbook_on_play_start(self, play):
        """
        Get the variable manager of the current play from the playbook.
        """
        self.play = play
        self.vm = play.get_variable_manager()

    def _all_vars(self, host=None, task=None):
        """
        Load all variables for the given inputs.
        """
        return self.vm.get_vars(
            play = self.play,
            host = host,
            task = task
        )

    def v2_runner_on_start(self, host, task):
        """
        Process the runner on start event.
        """
        templar = Templar(loader=self.playbook.get_loader(),
                  variables=self._vars(host=host, task=task))

        if task.become:
            print(f"Hostname: {host.name}")
            print(f"Task: {task.name}")
            print(f"Action: {task.action}")
            print(f"Arguments: {templar.template(task.args, fail_on_undefined=False)}")

References

Nautobot GraphQL Requests via Postman and Python

2021-04-29T00:00:00+00:00

GraphQL is a powerful query tool because it allows efficient queries for specific information that can span multiple resources that would otherwise require literally dozens of individual REST queries and extensive data filtering, post-processing, and isolation.

This article builds on the prior articles in this series, which describe how to craft GraphQL queries in Nautobot’s GraphiQL interface and how to use GraphQL aliasing to customize the returned key names and do multiple queries in a single GraphQL request.

The previous post in this series demonstrated how to leverage the Pynautobot Python package for programmatic GraphQL queries, showing a contrasting way (from the Postman-centric way described in the post below) to construct programmatic GraphQL queries in Python against Nautobot. The Pynautobot Python path provides a very clean, customized way to programmatically run simple or sophisticated GraphQL requests against Nautobot. The methodology described in this post examines a more general way to go about that, but at the expense of additional steps. Depending on your use case, one way or the other may be preferable.

This post will demonstrate three different techniques for configuring token authentication within Postman and two different data formats in the request body.

This post uses Postman version 8.0.6.

The examples in this post all use the public Nautobot demo site and the free Postman app. Readers are encouraged to follow along.

Authentication

Security tokens are typically required for programmatic access to Nautobot’s data. The following sections cover how to obtain a token and how to leverage it within Postman to craft GraphQL API calls.

Nautobot security tokens are created in the Web UI. To view your token(s), navigate to the API Tokens page under your user profile. If there is no token present, create one or get the necessary permissions to do so.

Setting Up Postman Token Authentication

There are multiple ways to authenticate your Nautobot API queries in Postman:

Directly inserting the token information in the request header
The Authentication tab for the request
The Authorization tab for the collection

Authentication Option 1: Editing the Header

Inserting the token authentication information directly in the header allows individual queries to authenticate, but the user must manually populate the header with the authentication info on each new query.

To start a query that uses this method within Postman, create a new Postman collection:

Click on the ‘+’ sign on the upper left of the app.
Name the collection by clicking on the pencil/edit button by the default New Collection name.

NOTE: A Postman Collection is a group of saved requests, oftentimes with a common theme, workflow, or function

Name the collection Authentication Testing.

Create a new API request by clicking on the ‘+’ sign to the right of the new collection tab.

Before we create the actual request, we’ll edit the header by adding the authentication/token info:

Go to the Headers tab
View the auto-generated headers (click on the eyeball)
In the next available row, type Authorization in the Key column (it should allow you to auto-complete)
In the Value column, type in the word Token<space><key>
- In this example I entered Token aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

The Nautobot public sandbox key is aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Your Authorization portion is now complete.

Query via GraphQL in Body

The next steps will guide you through crafting the rest of the query. This query will retrieve the name of each device along with the device’s site name.

This example will use the GraphQL format in the body.

Change the request type to POST
Enter the Nautobot server + GraphQL endpoint URL <server>/api/graphql/
Go to the Body section
Select GraphQL
Input the specific query shown below

query {
 devices {
   name
   site {
    name
  }
 }
}

Click the Send button and wait for the return data. You should see output similar to the picture below if you are using the public-facing Nautobot demo site.

Save the request in the collection.

Authentication Option 2: The Authorization Tab

This section will show an example of authentication in the request’s Authorization tab. This example will reside in the same Authentication Testing collection.

To create the new request:

Right-click on the collection name
Select Add Request
Name the request bkk devices; uses Authorization Tab

Configure the Authorization tab first, using the settings shown below; the Token aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa entry is the same as in the prior section.

Go to the Headers tab. If the hidden headers are not already being shown, click on the hidden headers eyeball to view the hidden headers. The Authorization key-value pair should be visible now. This was auto-generated from info in the Authorization tab.

Query via Raw Data

Finish out the query, but this time use raw in the Body section. Be sure to specify the format as JSON.

This query will return the device names in the bkk site, along with each device’s rack name and device role name. Be sure to escape the double-quotes in the site query parameter as shown below.

{
   "query": "query {devices(site:\"bkk\") {name device_role { name } rack { name } } }"
}

NOTE: Raw format is really cumbersome to work with, in part because of the requirement to escape quotes

Notice the potentially tedious nature of crafting the query above using the raw data option. It may be best to use the GraphQL format in the body for more complex queries.

If you check the Headers tab again, you’ll notice that changing the body format to JSON sets the Content-Type value to application/json.

Authentication Option 3: Via the Collection Environment

Within a Postman collection, it is more practical to configure authentication in the collection’s environment instead of configuring authentication for each request. Configuring authentication for the entire collection lets the user configure authentication just once.

This next section will describe how to set authentication at the collection level.

To the right of the collection name there are 3 dots (1); click on them and select Edit (2).

From here

Select the Authorization tab
Select API Key for Type
Set Key to Authorization
Set Value to Token<space><key>
Save

Any added requests in this collection can now use the Inherit auth from parent setting.

To configure a request to use the collection’s authentication:

Select the bkk devices request that uses the Authorization tab (for example)
Select the Authorization tab within the request
Switch the Type to Inherit auth from parent
Click Send

Save this updated request if you wish.

Converting Postman Queries to Python

Postman has helped us construct our GraphQL requests. This next section describes how to leverage those requests programmatically in Python.

This section will continue on from where we left off above, with the bkk devices request.

Postman has a very useful capability to export your crafted requests into a code format of your choice. On the right-hand side of the app, select the Code snippet button, marked as </>.

Within the search box that appears you can type Python or select from any of the languages that appear in the dropdown menu. This example will use the Python requests library.

Selecting the Python - Requests option, you will now see Python code that will programmatically make the request using Python’s requests library and return the response.

If cookie information appears in the code and you do not wish this to be present, you can manually delete it from the code or prevent it from being generated by clicking on Cookies (located directly beneath the Send button) to open the Cookie Manager; delete any cookies configured and then regenerate the Python code.

This Python code snippet can be added to any script or simply inserted into a Python interpreter:

~ % python3
Python 3.9.2 (v3.9.2:1a79785e3e, Feb 19 2021, 09:06:10)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>>
>>> url = "https://demo.nautobot.com/api/graphql/"
>>>
>>> payload="{\n    \"query\": \"query {devices(site:\\\"bkk\\\") {name device_role { name } rack { name } } }\"\n}"
>>> headers = {
...   'Authorization': 'Token aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
...   'Content-Type': 'application/json'
... }
>>>
>>> response = requests.request("POST", url, headers=headers, data=payload)
>>>
>>> print(response.text)
{"data":{"devices":[{"name":"bkk-edge-01","device_role":{"name":"edge"},"rack":{"name":"bkk-101"}},{"name":"bkk-edge-02","device_role":{"name":"edge"},"rack":{"name":"bkk-102"}},{"name":"bkk-leaf-01","device_role":{"name":"leaf"},"rack":{"name":"bkk-101"}},{"name":"bkk-leaf-02","device_role":{"name":"leaf"},"rack":{"name":"bkk-102"}},{"name":"bkk-leaf-03","device_role":{"name":"leaf"},"rack":{"name":"bkk-103"}},{"name":"bkk-leaf-04","device_role":{"name":"leaf"},"rack":{"name":"bkk-104"}},{"name":"bkk-leaf-05","device_role":{"name":"leaf"},"rack":{"name":"bkk-105"}},{"name":"bkk-leaf-06","device_role":{"name":"leaf"},"rack":{"name":"bkk-106"}},{"name":"bkk-leaf-07","device_role":{"name":"leaf"},"rack":{"name":"bkk-107"}},{"name":"bkk-leaf-08","device_role":{"name":"leaf"},"rack":{"name":"bkk-108"}}]}}
>>>

NOTE: Being that there are multiple ways to use the requests library, you can also use the more concise requests.post method: response = requests.post(url, headers=headers, data=payload).

Here is a Python script that uses the requests.post method. The script executes the GraphQL request, prints the text response, and then pretty prints the returned json data:

import json
import requests

from pprint import pprint

print("Querying Nautobot via graphQL API call.")
print()

url = "https://demo.nautobot.com/api/graphql/"
print("url is: {}".format(url))
print()

payload="{\n    \"query\": \"query {devices(site:\\\"bkk\\\") {name device_role { name } rack { name } } }\"\n}"
print("payload is: {}".format(payload))
print()

headers = {
  'Authorization': 'Token aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
  'Content-Type': 'application/json'
}
print("headers is: {}".format(headers))
print()

response = requests.post(url, headers=headers, data=payload)

response_data = response.json()

print("Here is the response data in json:")
pprint(response_data)

Note the tedious nature of crafting payload variable in the script above - it’s fairly complex with all the escaping backslashes. Luckily, the Postman app will create the object in the code for us, whether we use GraphQL or raw for the Body format.

Wrapping Up

To fully leverage GraphQL’s efficiency, it must be used programmatically. Prior posts in this series demonstrated using Nautobot’s GraphiQL interface to craft GraphQL queries and how to use GraphQL aliasing. This post builds on that by showing how to convert those GraphQL queries into remote requests in Postman and going on to export those requests to Python code for programmatic use. Also be sure to check out the prior post on using programmatic GraphQL requests with Nautobot via the pynautobot package and how that stacks up against the methodology in this post. Generally, the pynautobot path has fewer moving parts and offers customized APIs for Nautobot that allow sophisticated GraphQL queries. The Postman+requests path offers a more general framework but with more steps.

Cisco NSO Development Environment in Docker

2021-04-27T00:00:00+00:00

Developing services for Cisco NSO requires a development environment, which can be set up locally on your laptop or on a dedicated server. When a developer starts working on an NSO service, it has to install a correct NSO version, all required NEDs, and dependencies. This is where Docker can come in handy. It is a tool to make packaging easy, which means that you can pre-build a Docker image, which contains NSO, NEDs, and all dependencies.

In this blog post, I will show you how to create an NSO Docker image that can be used for development. There are a couple of advantages of using Docker containers for development. The most important is that everything can be packaged together in a single unit.

To build a Docker image, Docker requires a Dockerfile, where you put build instructions to create an image. I can create Dockerfile from scratch. But there is a better way. NSO in Docker from Cisco is a repository that contains scripts and files to create a base NSO Docker image. Using the NSO in Docker project will produce a base image, which can be extended with additional packages.

This blog post is divided in three sections:

Prepare a base image
Extend the base image
Run a Docker container

Prepare a Base Image

The first step in the process is to build a base image, which is a bare NSO installation that contains NSO only. The NSO in Docker project is used in this step.

Clone the Repository

The NSO in Docker repository should be cloned first.

$ git clone git@github.com:NSO-developer/nso-docker.git
Cloning into 'nso-docker'...
remote: Enumerating objects: 4597, done.
remote: Counting objects: 100% (1579/1579), done.
remote: Compressing objects: 100% (635/635), done.
remote: Total 4597 (delta 1043), reused 1395 (delta 871), pack-reused 3018
Receiving objects: 100% (4597/4597), 1.36 MiB | 2.08 MiB/s, done.
Resolving deltas: 100% (2932/2932), done.
$
$ cd nso-docker/
nso-docker$

Download Installation Files

To install NSO, you need the installation file, which is not included in the NSO in Docker project. You can download the installation file from the Cisco DevNet page. The installation file is an executable script, which verifies a signature and creates the installer file when executed.

You should put the file into the ./nso-install-files directory, which is a default path, where the build script will take a look for the NSO installer.

nso-docker$ cd nso-install-files
nso-install-files$ ls
nso-5.5.linux.x86_64.signed.bin

The script should be executed to extract the installer file.

nso-install-files$ sh nso-5.5.linux.x86_64.signed.bin
Unpacking...
Verifying signature...
Retrieving CA certificate from http://www.cisco.com/security/pki/certs/crcam2.cer ...
Successfully retrieved and verified crcam2.cer.
Retrieving SubCA certificate from http://www.cisco.com/security/pki/certs/innerspace.cer ...
Successfully retrieved and verified innerspace.cer.
Successfully verified root, subca and end-entity certificate chain.
Successfully fetched a public key from tailf.cer.
Successfully verified the signature of nso-5.5.linux.x86_64.installer.bin using tailf.cer

This produces multiple files. All these files should be removed and you should only keep the actual installer in the directory.

nso-install-files$ ls
nso-5.5.linux.x86_64.installer.bin

Build the Base Image

Now that the installer is in the ./nso-install-files you are ready to build the base image. You need to run the make command in the root of the project.

nso-install-files$ cd ..
nso-docker$ make
...
OMITTED FOR BREVITY
...

This creates a base image. It is as simple as that.

nso-docker$ docker image ls
REPOSITORY            TAG               IMAGE ID       CREATED       SIZE
cisco-nso-base        5.5-foobar        856ffdeef133   6 days ago    564MB
cisco-nso-dev         5.5-foobar        04f1f5da57b3   6 days ago    1.4GB

The make command creates two images:

a production image, which contains only required packages
a development image, which also contains documentation, compilers, and other tools

The make command adds the username to the image tag (5.5-foobar), but you should add another tag that only includes the NSO version.

Add a New Tag to the Base Image

The Makefile comes with another directive (tag-release), which will be used to tag the image.

nso-docker$ make NSO_VERSION=5.5 tag-release
docker tag cisco-nso-dev:5.5-foobar cisco-nso-dev:5.5
docker tag cisco-nso-base:5.5-foobar cisco-nso-base:5.5
nso-docker$ docker image ls
REPOSITORY            TAG               IMAGE ID       CREATED       SIZE
cisco-nso-base        5.5               856ffdeef133   6 days ago    564MB
cisco-nso-base        5.5-foobar        856ffdeef133   6 days ago    564MB
cisco-nso-dev         5.5               04f1f5da57b3   6 days ago    1.4GB
cisco-nso-dev         5.5-foobar        04f1f5da57b3   6 days ago    1.4GB

The base image is now ready.

Extend the Base Image

The base image will be extended with additional packages such as NEDs. The cisco-nso-base image will be used as a base image. All instructions to extend the base image will be added into the Dockerfile.

Create Dockerfile

Let’s first create Dockerfile. The following Docker image is pretty simple, but it can be more complex than that. The file contains the instructions to extend the cisco-nso-dev:5.5 base image. In the build process, Docker will create a directory and copy NED files from local neds directory to /var/opt/ncs/packages/ inside the container.

nso-docker$ cd ../nso-dev-image
nso-dev-image$ cat Dockerfile
FROM cisco-nso-dev:5.5

RUN mkdir -p /var/opt/ncs/packages/
COPY neds/*.tar.gz /var/opt/ncs/packages/

Download NEDs

You can download NEDs from DevNet and add these as compressed .tar.gz files to the neds directory.

nso-dev-image$ ls neds
cisco-ios-cli-6.69.tar.gz cisco-nx-cli-5.21.tar.gz

Build Docker Image

Now all components are prepared to build an NSO Docker image.

nso-dev-image$ docker build -t nso:5.5-neds .
[+] Building 2.2s (8/8) FINISHED
 => [internal] load build definition from Dockerfile                                                                        0.0s
 => => transferring dockerfile: 144B                                                                                        0.0s
 => [internal] load .dockerignore                                                                                           0.0s
 => => transferring context: 2B                                                                                             0.0s
 => [internal] load metadata for docker.io/library/cisco-nso-dev:5.5                                                        0.0s
 => [1/3] FROM docker.io/library/cisco-nso-dev:5.5                                                                          0.1s
 => [internal] load build context                                                                                           1.4s
 => => transferring context: 83.84MB                                                                                        1.4s
 => [2/3] RUN mkdir -p /var/opt/ncs/packages/                                                                               0.7s
 => [3/3] COPY neds/*.tar.gz /var/opt/ncs/packages/                                                                         0.2s
 => exporting to image                                                                                                      0.4s
 => => exporting layers                                                                                                     0.4s
 => => writing image sha256:2ee95dcc017a77e3725f984de62c9f896f5c73d8fc00130fd43ca6c0fc2aee6e                                0.0s
 => => naming to docker.io/library/nso:5.5-neds                                                                             0.0s

After a couple of moments the image is ready. The name of the image is nso and it is tagged as 5.5-neds.

nso-dev-image$ docker image ls
REPOSITORY            TAG               IMAGE ID       CREATED          SIZE
nso                   5.5-neds          2ee95dcc017a   30 seconds ago   1.48GB
cisco-nso-base        5.5               856ffdeef133   6 days ago       564MB
cisco-nso-base        5.5-foobar        856ffdeef133   6 days ago       564MB
cisco-nso-dev         5.5               04f1f5da57b3   6 days ago       1.4GB
cisco-nso-dev         5.5-foobar        04f1f5da57b3   6 days ago       1.4GB

Push the Image to the Registry

In order to allow other developers to use the same image, the image can be pushed to a registry, such as Docker Hub. To do that, another tag must be added to the image.

nso-dev-image$ docker tag nso:5.5-neds networktocode/nso:5.5-neds
nso-dev-image$ docker image ls
REPOSITORY            TAG               IMAGE ID       CREATED         SIZE
nso                   5.5-neds          2ee95dcc017a   3 minutes ago   1.48GB
networktocode/nso     5.5-neds          2ee95dcc017a   3 minutes ago   1.48GB
cisco-nso-base        5.5               856ffdeef133   6 days ago      564MB
cisco-nso-base        5.5-foobar        856ffdeef133   6 days ago      564MB
cisco-nso-dev         5.5               04f1f5da57b3   6 days ago      1.4GB
cisco-nso-dev         5.5-foobar        04f1f5da57b3   6 days ago      1.4GB

Once the image is tagged, it can be pushed to the registry.

nso-dev-image$ docker push networktocode/nso:5.5-neds

From this point, the image is available to anyone that has access to the registry.

Run a Docker Container

To make everything easier and repeatable, you can create a Makefile, which contains commands to start and purge NSO containers.

Create Makefile

Let’s first create a Makefile. There are two directives we’ll add to the Makefile. The first one starts a container, while the other is used to stop and remove the container.

There are a couple of options in the run directive. The one that should be pointed out is the -v option. This option maps a directory on a local filesystem to a directory in a container. You can map an existing git repository with NSO services to the directory inside the container. NSO loads services from that directory. Because files are originally located on a local filesystem, you can use preferred tools for development, while using Docker as a runtime environment. In this example the services directory is mapped.

nso-dev-image$ cat Makefile

.PHONY: run
run:
	docker run -itd --name nso \
                   -v ${PWD}/../services:/nso/run/packages \
                   -e ADMIN_PASSWORD=admin \
                   -p 2024:22 \
                   networktocode/nso:5.5-neds

.PHONY: purge
purge:
	docker stop nso
	docker rm nso

Start a Container

You can now start a container using the make run command.

nso-dev-image$ make run

And voila, the container with NSO is running. After a couple of moments the container is ready to use.

nso-dev-image$ docker ps
CONTAINER ID   IMAGE                        COMMAND         CREATED         STATUS                   PORTS                                                      NAMES
3623e63ab2a6   networktocode/nso:5.5-neds   "/run-nso.sh"   4 minutes ago   Up 4 minutes (healthy)   80/tcp, 443/tcp, 830/tcp, 4334/tcp, 0.0.0.0:2024->22/tcp   nso

Connect to NSO

You can now use ssh to connect to the NSO instance. The port 22 in the container is mapped to the port 2024 on the host. Let’s open an ssh session to port 2024.

nso-dev-image$ ssh admin@localhost -p 2024
admin@localhost's password:

User admin last logged in 2021-04-26T21:31:21.009543+00:00, to 3623e63ab2a6, from 172.17.0.1 using cli-ssh
admin connected from 172.17.0.1 using ssh on 3623e63ab2a6
admin@ncs>

Verify Packages

You can now work with NSO as you would with any other instance running natively on the system or running on a dedicated server. The following example displays the packages that are deployed in NSO.

admin@ncs> switch cli
admin@ncs# show packages package oper-status
                                                                                                      PACKAGE
                        PROGRAM                                                                       META     FILE
                        CODE     JAVA           PYTHON         BAD NCS  PACKAGE  PACKAGE  CIRCULAR    DATA     LOAD   ERROR
NAME                UP  ERROR    UNINITIALIZED  UNINITIALIZED  VERSION  NAME     VERSION  DEPENDENCY  ERROR    ERROR  INFO
-----------------------------------------------------------------------------------------------------------------------------
cisco-ios-cli-6.69  X   -        -              -              -        -        -        -           -        -      -
cisco-nx-cli-5.21   X   -        -              -              -        -        -        -           -        -      -
l3vpn               X   -        -              -              -        -        -        -           -        -      -

There are two NEDs that were added during the build process. Another package called l3vpn is included in my services directory, which is mapped to the container. You can make changes to the l3vpn packages locally on the host, and when you are ready, you can reload packages in NSO and test the new functionality.

Cleanup

It is also quite simple to remove the development environment. Since you have the purge directive in the Makefile, you can simply run the make purge command to stop and remove the container.

nso-dev-image$ make purge
docker stop nso
nso
docker rm nso
nso

Conclusion

Using Docker for NSO development brings many advantages. It is easy to set up an environment. Developers only need to run a Docker container, while an image is automatically pulled from a Docker registry. The same image can be used later in test and development environments, and you can be pretty sure that all packages and dependencies are installed. You can avoid “it works on my laptop” issues, because the same runtime environment can be used in development and in production.

The build process requires more steps, but it can be easily automated. Typically, the process will be running as a job in a CI/CD pipeline.

Hopefully this post shows you how to build an NSO development environment using Docker containers.

-Uros

References

Setting Up Nautobot ChatOps with Microsoft Teams

2021-04-22T00:00:00+00:00

Network to Code has released a number of amazing plugins for Nautobot. One of which, adding ChatOps functionality, can be found here on GitHub. This plugin adds ChatOps capabilities directly into your existing ChatOps client, in the form of a chat bot, and supports four of the more popular services available right now. The four services currently supported are Slack, Microsoft Teams, Webex Teams, and Mattermost.

If this is your first time hearing about ChatOps or this plugin, you can see the ChatOps demo on YouTube or join slack.networktocode.com and try it out for yourself in the #nautobot-chat channel.

Today, I’ll be going over how to get this plugin working in Nautobot and how to get a chat bot up and running for Microsoft Teams. The process is fairly different from the other three providers listed, and slightly more complex, but the end results are amazing. Let’s dive right in!

Getting Started

There are two main parts to getting the ChatOps plugin working with any ChatOps service: configuring it on the ChatOps service directly, and installing and configuring it on your Nautobot server. Microsoft Teams splits the first part into two sections: creating the service in Azure, and installing the app in the Teams client.

For simplicity, I will already assume you have the base Nautobot server installed and working. If not, you can find the full documentation over on readthedocs.io, or join our public Slack channel #nautobot at slack.networktocode.com and ask for assistance.

Part 1: Configuring Microsoft Teams SaaS

Azure and Permissions

To start off, I will be configuring a brand-new bot for Microsoft Teams from scratch. Microsoft runs their bots differently from Slack, Webex Teams, or Mattermost, in that their bot service runs on Azure. If you don’t have a Microsoft Azure account, you will need to create one or get access to it through your company before continuing.

According to the Microsoft docs, you will need “Contributor access either in the subscription or in a specific resource group. A user with the Contributor role in a resource group can create a new bot in that specific resource group. A user in the Contributor role for a subscription can create a bot in a new or existing resource group.”

Configuring Azure

There are three main parts to configuring a bot in Azure:

1) Create a Bot Service and Resource Group

2) Configure the Bot Service channel

3) Create a Client Secret for the Bot Service

I’ll break down each part individually, with step-by-step instructions and screenshots along the way.

1 - Create a Bot Service and Resource Group

First, log into the Azure Portal at https://portal.azure.com.

At the top of the screen is a search bar. Search for “Bot Channels Registration”, then select the option with the same name under “Marketplace” on the right side. This will take you to the page to create a new Bot Service.

NOTE: You may need to activate this service first within your company’s Azure subscription, which is not covered in this post.

A few key fields to fill out when creating a new Bot Service are:

Bot Handle - What you want your bot handle to be. This is not what your bot is called in the MS Teams client, or how users will interact with your bot, but it is unique (case-insensitive) within the overall Azure Bot Framework.
Resource Group - If there’s an existing one you want to use, select it. Otherwise, select the “Create new” link and create a new resource group. In this example, I’m creating a new Resource Group called “RG_nautobot_ntcblog”.
Pricing Tier - I’m using F0, which is Azure’s basic free tier.
Messaging Endpoint - Enter your Nautobot service URL in this format: https://<server>/api/plugins/chatops/ms_teams/messages/.

In this demo example, I’m using the Ngrok service. For a production Nautobot server, you would enter in the publicly facing DNS endpoint for inbound webhooks into your Nautobot server.

Subscription, Location, and Application Insights - Use whatever is appropriate for your company and situation.

Once this is all filled out, select the Create button at the bottom of the screen.

Wait until the deployment is done and successful before continuing. You can monitor its progress in the upper right of the Azure dashboard, under the alerts icon (looks like a bell).

2 - Configure the Bot Service Channel

Back on the Azure Portal homepage, in the main search bar, search for “Bot Services”, then click on the corresponding link under the Services section on the left side.

Select the name of the bot handle you just created.

Select the Microsoft Teams client icon, as circled in the screenshot below.

All of the options on the next Configure Microsoft Teams page should be ok when left to default, but should be reviewed anyway for your specific use case.

Once done, click Save at the bottom of the page, and review and Agree to any ToS popups.

3 - Create a Client Secret for the Bot Service

Next, on the left sidebar, select Configuration under the Settings section. Save the Microsoft App ID somewhere else, as you’ll need this later.

Select the “Manage” link directly above the App ID.

This will take you to the Certificates & Secrets page.

Click New client secret. Name it something descriptive, configure the expiration setting as necessary, and click Add.

Once it’s created, it will appear in the Client Secrets table at the bottom of the page. Copy and save the newly generated secret for later, as there’s no way to recover it once you leave the page.

NOTE: If you lose the key or copy it incorrectly, you will have to return to this page and generate a new secret.

Azure Recap

At this point, the Nautobot ChatOps plugin is fully set up within Azure. You should have two pieces of information for later use: the App ID and the Client Secret.

Part 2: Installing and Configuring the Nautobot ChatOps Plugin

Next, you must install and configure the Nautobot ChatOps plugin on your Nautobot server. Luckily, the fine folks at Network to Code have made this process incredibly simple!

Installing the Plugin

First, log into your Nautobot server and change to the user account Nautobot is running as. From there, it’s as simple as installing the plugin via a pip install command.

$ sudo -iu nautobot
$ pip3 install nautobot-chatops

Once the package is installed, the plugin will need to be enabled in your nautobot_config.py. If Nautobot was originally set up according to the default installation docs, this file will be located at /opt/nautobot/nautobot_config.py. In this file, add in the name of the plugins to the PLUGINS variable, then configure the required settings in the PLUGINS_CONFIG variable below it.

PLUGINS = ["nautobot_chatops"]

PLUGINS_CONFIG = {
    "nautobot_chatops": {
        "enable_ms_teams": True,
        "microsoft_app_id": "<app_id>",
        "microsoft_app_password": "<client_secret>"
    }
}

Make sure to replace <app_id> and <client_secret> with the App ID and Client Secret saved from Azure in the previous steps. Then save the file and restart the Nginx and Nautobot services.

sudo systemctl restart nginx
sudo systemctl restart nautobot-worker.service

Configuring the Plugin in Nautobot

Next, we need to configure the plugin in Nautobot to accept commands. For most deployments, open and unrestricted access to the bot from any chat account is undesirable. Therefore, access to the chatbot defaults to “deny all” when initially installed. Permissions for individual organizations, channels, and users must be set up here. For the purposes of this blog post, we will grant all access.

First, log into your Nautobot server. If this is the first plugin installed, a new menu option will appear at the top called Plugins. Under it, in section Nautobot ChatOps, select Access Grants.

Select the Add button to create a new access grant.

Command - You can specify permissions on a command-by-command basis, or specify all commands with an asterisk * as a wildcard. Example commands: nautobot or clear
Subcommand - You can specify permissions for subcommands as well, or all subcommands with an asterisk *. Example subcommands: get-devices or help
Grant Type - You need to create permissions for all three options: Organization, Channel, and User.
- Organization - This is for permissions specific to your organization. This is good for configuring allowed/blocked commands organization-wide.
- Channel - This is for configuring permissions on a per-channel basis.
- User - This is for configuring permissions on a per-user basis.
Name - The corresponding name of the organization, channel, or user. This is used more like a description, whereas the value below is used when interacting with the MS Teams SaaS API on the backend.
Value - Corresponding ID value to grant access to. Enter an asterisk * to grant access to all organizations, channels, or users.

Once all three permissions are created, the plugin is done being set up in Nautobot. The minimum amount of permissions required are three. You can allow everyone in your organization access to all commands (not recommended) by using wildcards for organization, channel, and user permissions.

In the above example, here’s how I’ve set it up:

Organization - The org only has access to the nautobot command. It does not have access to clear, or any future commands the plugin may end up supporting.
User - Anyone can run just the nautobot get-devices command, however user John Doe can run any command. Note that he cannot run clear, as that is restricted at the Organization permission above.
Channel - Anyone can accss the bot from any channel, but again, only the nautobot get-devices command. However, anyone in channel bot-admins can access any command available to them.

To summarize, anyone can run nautobot get-devices, whereas John Doe and anyone in the channel Bot Admins can run any nautobot subcommand. Nobody can run clear or any command that doesn’t start with nautobot.

The last step is configuring the Microsoft Teams client.

Part 3: Installing and Configuring the App in Teams

The last main step needed is uploading and installing the app into your Microsoft Teams client for use within your organization.

Before continuing, you need to download a single ZIP file from the ChatOps plugin repo, found here. This will be used later for ease of installing the app into the client.

The ZIP file contains three files:

1) manifest.json - Pre-configured information for the bot

2) color.png - Image to use for the bot

3) outline.png - Transparent image to use for the bot

Open up your Microsoft Teams desktop client and log in if you haven’t already.

On the left sidebar and select Apps. Use the Apps search bar to search for “App Studio”, then select the tile and click Open to open the App Studio. If it isn’t currently installed, install it first, then open it.

If not already selected, select Manifest Editor on the top horizontal menu bar, then select Import an existing app. Upload the Nautobot_ms_teams.zip file that you downloaded earlier.

Once imported, the Edit an app page will appear, allowing you to configure the settings for the bot.

Required Setting Changes

The App ID setting must be updated in two different locations. This ZIP file comes pre-loaded with an example App ID, but it must be replaced with the one for your specific bot, as created and saved from Azure in the previous steps. All other settings can be left as is, but feel free to review them as desired.

1) Under section 1 Details, select App details. Update the App ID to the value that you saved from Azure earlier.

2) Under section 2 Capabilities, select page Bots. Select the Edit button next to the Bot at the top. In the pop-out window, in the field Connect to a different bot id, update the App ID to the value that you saved from Azure earlier. Then click Save.

3) Under section 3 Finish, select page Test and distribute. Select the Download button to download the app as a .zip file to your computer.

NOTE: You can attempt to Install the app instead of downloading it and re-uploading it in step 4, however it requires permissions to do so, which I had trouble with even when I was the administrator of the Microsoft Teams environment (e.g. in my free-tier personal environment).

4) The app, with the now-updated App ID’s, will download to the ~/Downloads folder (or equivalent) on your computer. The file should be named Nautobot.zip (or, if you changed the name of the bot in the manifest, the name you gave it).

Once downloaded, select Apps on the bottom of the left-hand sidebar, then scroll to the bottom and select Upload a Custom App. If your company requires approval of custom apps, it will be submitted for approval before being installed.

5) The app package will then upload into MS Teams as an install app, and you’ll be taken to the Apps page with the bot (app) listed under Built by your org. Select the bot/app tile, then click the blue Add button.

Done!

That’s it! Your new Nautobot ChatOps plugin should now be installed for your Microsoft Teams client, and usable by anyone with the appropriate permissions (configured earlier in part 2).

You can do some really cool things with the bot once it’s up and running, and you have some data in Nautobot. You can send the message nautobot help to the app (no / forward slash) to see a list of all supported commmands.

Interacting with Nautobot in Microsoft Teams

There are currently a couple of ways to interact with the Nautobot plugin by default directly in the Microsoft Teams client, although these can be modified in the app permissions in the same area where you installed the app originally (in part 3). They are:

1) Chat - In the main left sidebar, select Chat, then search for “Nautobot” (or whatever you renamed the bot to). You can message the bot directly here.

2) App - In the main left sidebar, select the three dots, then in the pop-out menu, search for “Nautobot” and select it. I recommend right-clicking the icon in the left sidebar once the window opens to pin it for future interactions.

Forward Looking

Here at Network to Code, as we continue developing Nautobot, we will be adding functionality to this ChatOps plugin as well. With the code publicly available here on GitHub, we encourage anyone looking to contribute to do so and join our growing open-source community around Nautobot!

Additionally, there’s a blog post from earlier this month around creating your own custom chat commands within this plugin. If interested, you can read it here.

Thanks for reading, and I hope you enjoy ChatOps as much as I do!

-Matt

Ansible - BGP Networking Troubleshooting Guide

2021-04-20T00:00:00+00:00

Network troubleshooting is a common automation use case. Network outages are costly and time-consuming and often require the network engineers to log into network equipment and manually investigate network issues. Working on network operations teams, I quickly noticed that troubleshooting network problems is a playbook of repeatable steps, hence the rationale for automating network troubleshooting with Ansible.

Use Case - BGP

Troubleshooting Layer 3 connectivity tends to lead an operations engineer to jump into multiple routers and check routing. Let’s say internet access has been lost from the WAN edge. If I were troubleshooting this, my instincts would tell me to go to my edge router(s) and check the BGP neighbor going towards my ISP.

east-rtr#show ip bgp summary

<...output omitted...>

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
4.4.4.4      4   400       0       0        0    0    0 08:11    Idle

From the output of show ip bgp summary the issue can determined, BGP is down toward the ISP. How can Ansible help? This is a simplified example with one router and one WAN connection, but what happens if you have 10, 15, or more BGP relationships you need to check. It is costly to manually log in to each router to check the status of BGP. How can Ansible help?

Checking BGP with Ansible

Here is a sequential listing of what the Ansible playbook is doing.

Run show ip bgp summary outputs from ISP routers.
Use ansible-napalm to get BGP facts on the neighbors for easy reporting.
Create an easy-to-consume report using a Jinja2 template to create a report with BGP neighbor status.
Assemble all the device reports into a single overview report.
Iterate through the neighbors and if a neighbor is down, attempt to ping the destination IP to verify Layer 3 reachability using napalm-ping.

Pre-req

There needs to be a valid Ansible inventory, either a static inventory file or dynamic inventories utilizing an existing SoT (Source of Truth). For demonstration purposes a static file will be used.

inventory.cfg

[isp_routers]

[isp_routers:vars]
ansible_network_os=ios

[isp_routers:children]
east_isp
west_isp

[east_isp]
east-rtr

[west_isp]
west-rtr

For help building an inventory file. See Ansible Inventory

Step 1

Create a simple playbook to execute show ip bgp neighbors on all of the routers in the group called isp_routers.

---
- name: "PLAY:1 - GET BGP SUMMARY"
  gather_facts: False
  connection: "network_cli"
  hosts: "isp_routers"
  tasks:
  - name: "TASK:1 - 'SHOW IP BGP SUMMARY'"
    ios_command:
      commands: "show ip bgp summary"
    register: "output_ios"
  - name: "TASK:2 - PRINT BGP OUTPUT"
    debug:
      msg: "{{ output_ios.stdout[0] }}"

Running the playbook results in the following output.

▶ ansible-playbook pb.yml -u ntc -k
SSH password: 

PLAY [PLAY:1 - GET BGP SUMMARY] **************************************************************************************************************************************************************************************

TASK [TASK:1 - 'SHOW IP BGP SUMMARY'] ********************************************************************************************************************************************************************************

ok: [east-rtr]
ok: [west-rtr]

TASK [TASK:2 - PRINT BGP OUTPUT] *************************************************************************************************************************************************************************************
ok: [east-rtr] => {
    "msg": "BGP router identifier 1.1.1.1, local AS number 100\nBGP table version is 416, main routing table version 416\n28 network entries using 6944 bytes of memory\n41 path entries using 5576 bytes of memory\n8/7 BGP path/bestpath attribute entries using 2304 bytes of memory\n4 BGP AS-PATH entries using 128 bytes of memory\n0 BGP route-map cache entries using 0 bytes of memory\n0 BGP filter-list cache entries using 0 bytes of memory\nBGP using 14952 total bytes of memory\nBGP activity 124/96 prefixes, 232/191 paths, scan interval 60 secs\n32 networks peaked at 23:40:21 Jan 7 2021 UTC (6w5d ago)\n\nNeighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd\n4.4.4.4      4   400       0       0        0    0    0 08:21    Idle"
}
ok: [west-rtr] => {
    "msg": "BGP router identifier 2.2.2.2, local AS number 100\nBGP table version is 579, main routing table version 579\n28 network entries using 6944 bytes of memory\n41 path entries using 5576 bytes of memory\n8/7 BGP path/bestpath attribute entries using 2304 bytes of memory\n4 BGP AS-PATH entries using 128 bytes of memory\n0 BGP route-map cache entries using 0 bytes of memory\n0 BGP filter-list cache entries using 0 bytes of memory\nBGP using 14952 total bytes of memory\nBGP activity 158/130 prefixes, 267/226 paths, scan interval 60 secs\n32 networks peaked at 23:40:21 Jan 7 2021 UTC (6w5d ago)\n\nNeighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd\n8.8.8.8      4   400       0       0        0    0    0 18:52    1"
}

PLAY RECAP ***********************************************************************************************************************************************************************************************************
east-rtr   : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
west-rtr   : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0    

At this point you have a single pane to quickly check all the BGP neighbors; however, it’s hard to read the output. To take this playbook to the next level, we can easily take command output and create structured data using one of the various cli parsing modules.

Ansible Parsing
Parsing Strategies on NTC Blog by Mikhail Yohman.
- Parsing Strategies - An Introduction

What if more information is needed? You could check route counts or layer 3 reachability.

Let’s dig into this use case further.

Step 2

Use napalm-ansible module to run get-facts on BGP.

Note: For readability the rest of the task will be in a PLAY:2 of the playbook.

- name: "PLAY:2 - USE NAPALM BGP FACTS"
  gather_facts: False
  connection: "network_cli"
  hosts: "isp_routers"
  tasks:
  - name: "TASK:1 - 'GET BGP FACTS'"
    napalm_get_facts:
      filter:
        - "bgp_neighbors"
    register: "bgp"
  - debug: var=bgp

Results in:

▶ ansible-playbook napalm_pb.yml -u ntc -k
SSH password: 

PLAY [PLAY:2 - USE NAPALM BGP FACTS] *********************************************************************************************************************************************************************************

TASK [TASK:1 - 'GET BGP FACTS'] **************************************************************************************************************************************************************************************
ok: [east-rtr]
ok: [west-rtr]

TASK [debug] *********************************************************************************************************************************************************************************************************
ok: [east-rtr]] => {
    "bgp": {
        "ansible_facts": {
            "napalm_bgp_neighbors": {
                "global": {
                    "peers": {
                        "4.4.4.4": {
                            "address_family": {
                                "ipv4 unicast": {
                                    "accepted_prefixes": 0,
                                    "received_prefixes": 0,
                                    "sent_prefixes": 0
                                }
                            },
                            "description": "",
                            "is_enabled": true,
                            "is_up": false,
                            "local_as": 100,
                            "remote_as": 400,
                            "remote_id": "4.4.4.4",
                            "uptime": 0
                        },
                    "router_id": "1.1.1.1"
                }
            }
        },
        "changed": false,
        "failed": false
    }
}
ok: [west-rtr] => {
    "bgp": {
        "ansible_facts": {
            "napalm_bgp_neighbors": {
                "global": {
                    "peers": {
                        "8.8.8.8": {
                            "address_family": {
                                "ipv4 unicast": {
                                    "accepted_prefixes": 1,
                                    "received_prefixes": 1,
                                    "sent_prefixes": 11
                                }
                            },
                            "description": "",
                            "is_enabled": true,
                            "is_up": true,
                            "local_as": 100,
                            "remote_as": 400,
                            "remote_id": "8.8.8.8",
                            "uptime": 1641600
                        },
                    "router_id": "2.2.2.2"
                }
            }
        },
        "changed": false,
        "failed": false
    }
}

PLAY RECAP ***********************************************************************************************************************************************************************************************************
east-rtr   : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
west-rtr   : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

The playbook returns structured operational data on the BGP neighbors. This data can easily be used to build a report.

Step 3

Create a report with the validation of BGP.

We can take the registered data from TASK:1 and pass it to the template module where a Jinja2 template can be used to create a report.

We will add TASK:2 to our PLAY.

---
- name: "TASK:2 - 'GENERATE REPORTS'"
  template:
    src: "./templates/bgp_report.j2"
    dest: "./build/{{ inventory_hostname }}.txt"

An example of a Jinja2 template can be seen below:

bgp_report.j2

Hostname: {{ inventory_hostname }}
-----------------
{% for neighbor, details in bgp["ansible_facts"]["napalm_bgp_neighbors"]["global"]["peers"].items() %}
Neighbor:           {{ neighbor }}
Enabled:            {{ details["is_enabled"] }}
Neighbor_UP:        {{ details["is_up"] }}
Accepted Prefixes:  {{ details['address_family']['ipv4 unicast']['accepted_prefixes'] }}
Received Prefixes:  {{ details['address_family']['ipv4 unicast']['received_prefixes'] }}
Sent Prefixes:      {{ details['address_family']['ipv4 unicast']['sent_prefixes'] }}

{% endfor %}

The Jinja2 template will render the structured data into a human-readable format.

Example:

Hostname: east-rtr
-----------------
Neighbor:          4.4.4.4
Enabled:           True
Neighbor_UP:       False
Accepted Prefixes: 0
Received Prefixes: 0
Sent Prefixes:     0

Step 4

With multiple devices in our inventory group, a file per device will be written. Parsing through multiple files can slow down the time to resolution; therefore, merging all these files together into one all-encompassing report will be done in the next task.

The Ansible assemble module will be used to merge all the reports together.

- name: "TASK:3 - ASSEMBLE REPORTING FROM HOST DETAILS"
    assemble:
    src: "./build"  # Directory with files to merge.
    dest: "./reports/report.txt"  # Merged output filename.

Once TASK:3 executes, one report is generated with the following output:

Hostname: east-rtr
-----------------
Neighbor:          4.4.4.4
Enabled:           True
Neighbor_UP:       False
Accepted Prefixes: 0
Received Prefixes: 0
Sent Prefixes:     0

Hostname: west-rtr
-----------------
Neighbor:          8.8.8.8
Enabled:           True
Neighbor_UP:       True
Accepted Prefixes: 8
Received Prefixes: 8
Sent Prefixes:     22

Now a single easy-to-read file exists to look at neighbors. We see east-rtr has a BGP neighbor that is DOWN.

Step 5

Check whether any DOWN neighbor is reachable via ping.

- name: "TASK:4 - PING BGP NEIGHBORS THAT ARE DOWN"
  napalm_ping:
    hostname: "{{ inventory_hostname }}"
    username: "{{ ansible_user }}"
    password: "{{ ansible_password }}"
    dev_os: "{{ ansible_network_os }}"
    destination: "{{ item.key }}"
  loop: "{{ bgp['ansible_facts']['napalm_bgp_neighbors']['global']['peers'] | dict2items }}"
  when: "not item.value.is_up"
  register: neighbor_down

After the reachability check is completed, print the results for the DOWN neighbors.

- name: "TASK:5 - PRINT PING RESULTS FOR DOWN NEIGHBORS"
  debug:
    msg: "{{ item['ping_results'] }}"
  loop: "{{ neighbor_down['results'] }}"
  when: "item['ping_results'] is defined"

TASK:5 example output:

    "msg": {
        "success": {
            "packet_loss": 5,
            "probes_sent": 5,
            "results": [],
            "rtt_avg": 0.0,
            "rtt_max": 0.0,
            "rtt_min": 0.0,
            "rtt_stddev": 0.0
        }
    }
}

Playbook Summary

Valuable troubleshooting data was gathered by running this playbook. A BGP neighbor is down on east-rtr. Details about all neighbors were also collected, including: enabled state, current neighbor state, and sent/received route counts. Finally, for any DOWN neighbors a reachability check using ping was performed. Most importantly, all this data was assembled across all our isp_routers in just seconds. This was still a simplified example with only two routers, but extrapolating this across tens, hundreds, or more routers is very powerful.

It is important to mention that additional tasks could be added to this playbook to troubleshoot further, for example:

Check the routing to the neighbor IP.
Grab the next-hop IP from the route entry.
Verify that the ARP table for the next-hop IP has a MAC entry.

Full Playbook

- name: "PLAY:1 - GET BGP SUMMARY"
  gather_facts: False
  connection: "network_cli"
  hosts: "isp_routers"
  tasks:
    - name: "TASK:1 - 'SHOW IP BGP SUMMARY'"
      ios_command:
        commands: "show ip bgp summary"
      register: "output_ios"
    - name: "TASK:2 - PRINT BGP OUTPUT"
      debug:
        msg: "{{ output_ios.stdout[0] }}"
- name: "PLAY:2 - USE NAPALM BGP FACTS"
  gather_facts: False
  connection: "network_cli"
  hosts: "isp_routers"
  tasks:
    - name: "TASK:1 - 'GET BGP FACTS'"
      napalm_get_facts: filter="bgp_neighbors"
      register: "bgp"
    - debug: var=bgp
    - name: "TASK:2 - 'GENERATE REPORT'"
      template:
        src: "./templates/bgp_report.j2"
        dest: "./build/{{ inventory_hostname }}.txt"
    - name: "TASK:3 - ASSEMBLE REPORTING FROM HOST DETAILS"
      assemble:
        src: "./build"
        dest: "./reports/report.txt"
    - name: "TASK:4 - PING BGP NEIGHBORS THAT ARE DOWN"
      napalm_ping:
        hostname: "{{ inventory_hostname }}"
        username: "{{ ansible_user }}"
        password: "{{ ansible_password }}"
        dev_os: "{{ ansible_network_os }}"
        destination: "{{ item['key'] }}"
      with_dict: "{{ bgp['ansible_facts']['napalm_bgp_neighbors']['global']['peers'] }}"
      when: "not item['value']['is_up']"
      register: "neighbor_down"
    - name: "TASK:5 - PRINT PING RESULTS FOR DOWN NEIGHBORS"
      debug:
        msg: "{{ item['ping_results'] }}"
      loop: "{{ neighbor_down['results'] }}"
      when: "item['ping_results'] is defined"

Summary

BGP troubleshooting is one of a multitude of operational troubleshooting playbooks that could be executed for troubleshooting connectivity issues. Taking these same steps to other use cases can greatly improve MTTR on network issues and outages. Furthermore, these playbooks can be extended using a module to update ITSM ticket notes, or even for use during an existing daily network readiness task.

-Jeff

Programmatic Nautobot GraphQL Requests via Pynautobot

2021-04-15T00:00:00+00:00

Thanks to its ability to efficiently allow the request of specific information that can span multiple resources, GraphQL is a powerful query tool. A single GraphQL API request can return information that would otherwise require literally dozens of individual REST queries and extensive data filtering, post-processing, and isolation.

A prior blog post in this series covered how to craft Nautobot GraphQL queries using the Nautobot GraphiQL interface. Recall that the GraphiQL interface is just for testing the GraphQL queries. This post will build on that knowledge, showing you how to leverage those queries to craft remote GraphQL requests for programmatic use via the open source pynautobot Python library. The pynautobot project page is here; the GitHub repository can be found here.

The pynautobot Python library is an API wrapper that allows easy interactions with Nautobot. This specific article will focus the GraphQL capabilities within the library.

The examples in this post all use the public Nautobot demo site https://demo.nautobot.com/. Readers are encouraged to follow along.

Authentication

Tokens

Getting Started with Pynautobot

Installing Pynautobot

pynautobot is a Python library. From your environment install it via pip3:

% pip3 install pynautobot
Collecting pynautobot
Downloading pynautobot-1.0.1-py3-none-any.whl (29 kB)
. . . 
Successfully installed pynautobot-1.0.1
% 

Using Pynautbot

This first example will walk through a pynautobot GraphQL query in a Python interpreter.

Start a Python shell and import pynautobot:

% python3
Python 3.9.2 (v3.9.2:1a79785e3e, Feb 19 2021, 09:06:10) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> 
>>> import pynautobot
>>> 

Taking a quick look at Classes within pynautobot, api is the one we will want to use. The help tells us that the api Class can take url and token as parameters.

>>> dir(pynautobot)
['AllocationError', 'ContentError', 'DistributionNotFound', 'RequestError',  << dunders snipped >>, 'api', 'core', 'get_distribution', 'models']
>>> 
>>> help(pynautobot.api)


Help on class Api in module pynautobot.core.api:

class Api(builtins.object)
 |  Api(url, token=None, threading=False)
 . . . 

NOTE: the token from the https://demo.nautobot.com server is aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Define a value token with the token value from your Nautobot implementation and a value url for your Nautobot server.

>>> url = "https://demo.nautobot.com"
>>> 
>>> token = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
>>> 

This live example will use a query from a previous post. Define query in Python using the exact text from the query used in example 3 in the GraphiQL post:

NOTE: the query text can be found in the last section of the blog, under Text Query Examples and Results

>>> query = """
... query {
...  devices(site:"ams") {
...    name
...  }
... }
... """
>>> 
>>> query
'\nquery {\n devices(site:"ams") {\n   name\n }\n}\n'
>>> 
>>> print(query)

query {
 devices(site:"ams") {
   name
 }
}
>>> 

Now construct the pynautobot.api Class with the parameters we have:

>>> 
>>> nb = pynautobot.api(url, token)
>>> 

Notice that the nb Object instance has a graphql attribute . . .

>>> dir(nb)
['__class__', << dunders snipped >>, 'base_url', 'circuits', 'dcim', 'extras', 'graphql', 'headers', 'http_session', 'ipam', 'openapi', 'plugins', 'status', 'tenancy', 'threading', 'token', 'users', 'version', 'virtualization']
>>> 

. . . and graphql has a query method:

>>> dir(nb.graphql)
['__class__', << dunders snipped >>, 'api', 'query', 'url']
>>> 
>>> help(nb.graphql.query)

Help on method query in module pynautobot.core.graphql:

query(query: str, variables: Optional[Dict[str, Any]] = None) -> pynautobot.core.graphql.GraphQLRecord method of pynautobot.core.graphql.GraphQLQuery instance
    Runs query against Nautobot Graphql endpoint.
    
    Args:
        query (str): Query string to send to the API
        variables (dict): Dictionary of variables to use with the query string, defaults to None
    
    Raises:
        GraphQLException:
            - When the query string is invalid.
        TypeError:
            - When `query` passed in is not of type string.
            - When `variables` passed in is not a dictionary.
        Exception:
            - When unknown error is passed in, please open an issue so this can be addressed.
    
    Examples:
        >>> try:
        ...   response.raise_for_status()
        ... except Exception as e:
        ...   variable = e
        ... 
        >>> variable
        >>> variable.response.json()
        {'errors': [{'message': 'Cannot query field "nae" on type "DeviceType". Did you mean "name" or "face"?', 'locations': [{'line': 4, 'column': 5}]}]}
    
        >>> variable.response.status_code
        400
    
    Returns:
        GraphQLRecord: Response of the API call

Using the structure above, create the query and explore the results.

>>> 
>>> response = nb.graphql.query(query=query)
>>> 
>>> dir(response)
['__class__', << dunders snipped >>, 'json', 'status_code']
>>> 
>>> response.status_code
200
>>> 
>>> response.json
{'data': {'devices': [{'name': 'ams-edge-01'}, {'name': 'ams-edge-02'}, {'name': 'ams-leaf-01'}, {'name': 'ams-leaf-02'}, {'name': 'ams-leaf-03'}, {'name': 'ams-leaf-04'}, {'name': 'ams-leaf-05'}, {'name': 'ams-leaf-06'}, {'name': 'ams-leaf-07'}, {'name': 'ams-leaf-08'}]}}
>>> 

Here is response in a more readable format:

>>> from pprint import pprint
>>> pprint(response.json)
{'data': {'devices': [{'name': 'ams-edge-01'},
                      {'name': 'ams-edge-02'},
                      {'name': 'ams-leaf-01'},
                      {'name': 'ams-leaf-02'},
                      {'name': 'ams-leaf-03'},
                      {'name': 'ams-leaf-04'},
                      {'name': 'ams-leaf-05'},
                      {'name': 'ams-leaf-06'},
                      {'name': 'ams-leaf-07'},
                      {'name': 'ams-leaf-08'}]}}
>>> 

The response object can be parsed, making the data accessible to your Python code:

>>> 
>>> devices = response.json['data']['devices']
>>> 
>>> devices[2]
{'name': 'ams-leaf-01'}
>>> 

A Full Pynautobot Script

This next example features a full script, using an example from the GraphQL Aliasing with Nautobot post in this series. You can also see the YouTube video that accompanies the post here.

The script below features a query that uses GraphQL aliasing to request device names for multiple sites in a single query. The device names in each site will be returned grouped by the site name; each device name will be alaised with an inventory_hostname key (instead of name) for use in an Ansible environment.

The query text for this script was pulled directly from the Aliasing Solves the Problem section of the referenced blog post and copied directly into the query variable.

import json
import pynautobot

from pprint import pprint

print("Querying Nautobot via pynautobot.")
print()

url = "https://demo.nautobot.com"
print("url is: {}".format(url))
print()

query = """
query {
  ams_devices:devices(site:"ams") {
    inventory_hostname:name
  }
  sin_devices:devices(site:"sin") {
    inventory_hostname:name
  }
  bkk_devices:devices(site:"bkk") {
    inventory_hostname:name
  }
}
"""

print("query is:")
print(query)
print()

token = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"

nb = pynautobot.api(url, token)

response = nb.graphql.query(query=query)
response_data = response.json

print("Here is the response data in json:")
pprint(response_data)
print()

print("Here are the bkk devices:")
pprint(response_data['data']['bkk_devices'])

Here is the script’s output:

% python3 -i graphql_ams_query_pynautobot.py
Querying Nautobot via pynautobot.

url is: https://demo.nautobot.com

query is:

query {
  ams_devices:devices(site:"ams") {
    inventory_hostname:name
  }
  sin_devices:devices(site:"sin") {
    inventory_hostname:name
  }
  bkk_devices:devices(site:"bkk") {
    inventory_hostname:name
  }
}


Here is the response data in json:
{'data': {'ams_devices': [{'inventory_hostname': 'ams-edge-01'},
                          {'inventory_hostname': 'ams-edge-02'},
                          {'inventory_hostname': 'ams-leaf-01'},
                          {'inventory_hostname': 'ams-leaf-02'},
                          {'inventory_hostname': 'ams-leaf-03'},
                          {'inventory_hostname': 'ams-leaf-04'},
                          {'inventory_hostname': 'ams-leaf-05'},
                          {'inventory_hostname': 'ams-leaf-06'},
                          {'inventory_hostname': 'ams-leaf-07'},
                          {'inventory_hostname': 'ams-leaf-08'}],
          'bkk_devices': [{'inventory_hostname': 'bkk-edge-01'},
                          {'inventory_hostname': 'bkk-edge-02'},
                          {'inventory_hostname': 'bkk-leaf-01'},
                          {'inventory_hostname': 'bkk-leaf-02'},
                          {'inventory_hostname': 'bkk-leaf-03'},
                          {'inventory_hostname': 'bkk-leaf-04'},
                          {'inventory_hostname': 'bkk-leaf-05'},
                          {'inventory_hostname': 'bkk-leaf-06'},
                          {'inventory_hostname': 'bkk-leaf-07'},
                          {'inventory_hostname': 'bkk-leaf-08'}],
          'sin_devices': [{'inventory_hostname': 'sin-edge-01'},
                          {'inventory_hostname': 'sin-edge-02'},
                          {'inventory_hostname': 'sin-leaf-01'},
                          {'inventory_hostname': 'sin-leaf-02'},
                          {'inventory_hostname': 'sin-leaf-03'},
                          {'inventory_hostname': 'sin-leaf-04'},
                          {'inventory_hostname': 'sin-leaf-05'},
                          {'inventory_hostname': 'sin-leaf-06'},
                          {'inventory_hostname': 'sin-leaf-07'},
                          {'inventory_hostname': 'sin-leaf-08'}]}}

Here are the bkk devices:
[{'inventory_hostname': 'bkk-edge-01'},
 {'inventory_hostname': 'bkk-edge-02'},
 {'inventory_hostname': 'bkk-leaf-01'},
 {'inventory_hostname': 'bkk-leaf-02'},
 {'inventory_hostname': 'bkk-leaf-03'},
 {'inventory_hostname': 'bkk-leaf-04'},
 {'inventory_hostname': 'bkk-leaf-05'},
 {'inventory_hostname': 'bkk-leaf-06'},
 {'inventory_hostname': 'bkk-leaf-07'},
 {'inventory_hostname': 'bkk-leaf-08'}]
>>> 

Wrapping Up

To fully leverage GraphQL’s efficiency, it must be used programmatically. The first post in this series demonstrated using Nautobot’s GraphiQL interface to craft GraphQL queries. This post builds on that by showing how to convert those GraphQL queries into remote requests in Python code for programmatic use.

To find out more about pynautobot, including the additional capabilities not described in this post, start with the pynautobot Github repo.

Deploying Services with Docker, NGINX, Route 53 & Let’s Encrypt

2021-04-13T00:00:00+00:00

Docker is a power tool for deploying applications or services, and there are numerous Docker orchestration tools available that can help to simplify the management of the deployed containers. But what if you are wanting to deploy a small number of services and not wanting to undertake setting up and managing another application stack just to run a handful of containers. I will cover how I deployed a handful of services on a single Docker host. The services I deployed are LetsEncrypt to generate a wildcard certificate, Route 53 to register A and CNAME records, and NGINX to do reverse proxy with SNI encapsulation. I previously had some of these services deployed in containers on a Raspberry Pi as part of my Aquarium Controller, but I wanted to provide better flexibility for deployment and not pigeonhole myself to deploying only ARM-compatible containers. That, combined with wanting persistent services deployed at home, is what led me to building a new physical Linux host running Ubtunu 20.04 LTS & Docker.

This post is meant to show the flexibility of Docker not a production guide on deploying with Docker. I fully recommend using orchestration when deploying containers and would have deployed via container orchestration if I were working with multiple hosts or was managing more services.

In this post, I will be using my InfluxDB service as the example service. This service is used from both outside the Docker host via NGINX reverse proxy and for east/west container-to-container communication. Also, all of my services are defined as code via docker-compose to provide an easier experience vs raw docker run commands.

Communication

Below is an example of a communication flow for a user consuming a service deployed in Docker that then consumes another backend service on the same host. The end user is none the wiser on how the traffic is flowing, and as the administrator you are able to take advantage of container-to-container communication.

Route 53

I am using Route 53 to register all of my DNS records; this simplifies the amount of services I am managing on premise by not running a service like Bind 9 DNS. All of the records I am publishing in Route53 resolve to private IP addresses and are not routeable from outside my network. In my examples I have a single A record for the physical host itself and all services are CNAME records pointing to the server’s A record. My domain name was registered with Route 53, which helps to streamline the process.

Docker Container Name Resolution

Docker provides networks that are internal to the Docker daemon and the ability to perform container name resolution for containers that are on the same Docker network. To simplify the declaration of these supporting services, I am using docker-compose; and to communicate east/west within containers I only have to send traffic to the adjacent service name. In the above diagram, the Grafana service communicates to the InfluxDB service via http://influxdb:8086/. Docker resolves the hostname influxdb to the IP address of the InfluxDB container.

Let’s Encrypt

Although the services are deployed on my local home network and are behind an appropriate firewall, I prefer to deploy services with TLS from day one. This is a best practice that was instilled in me from day one of my days in enterprise environments. In my example, I am using CertBot to request and manage my certificate. To simplify certificate management I am using a wildcard cert of *.whitej6.com for all services. Also, an extension of TLS is server name indication or SNI

NGINX and SNI

All the services that are meant to be consumed from outside the Docker daemon are only exposed to localhost within the port definition of the service. This ensures ALL traffic that I want to allow into Docker must come from the physical host or an application that is deployed to the host. Each of the services that I need to expose have their own definition in an NGINX configuration file. The configuration tells NGINX which certificate to use, which requested server name maps to which underlying localhost port number. When NGINX receives an HTTPS request it first determines which service is requested via SNI and then performs a reverse proxy to the correct port on localhost. This allows me to terminate TLS on the physical host and run plain text protocols from NGINX to the underlying Docker service. By performing my own TLS termination in a secure manner outside containers, it can simplify container deployment, reduce the need to customize a vendor-provided container, and/or figure out how each vendor performs TLS termination in their containers.

Example Deployment

The example used is an actual depiction of services I am hosting within my home.

Prerequisites

This example will not be covering how to install Ubuntu, Docker, docker-compose, CertBot, or NGINX. These items have well documented installations and should be referenced. The example will cover the consumption of these services.

Route 53

I am using Route 53 to host DNS records needed for the applications deployed in the Docker stack. Below is a table of the two records that are needed for deploying. At a later step in the example, I will create a third record that is required for generating my certificate.

Record Type	Source	Destination
A	ubuntu-server.whitej6.com	10.0.0.16
CNAME	influxdb.whitej6.com	ubuntu-server.whitej6.com

You can see in the output that influxdb.whitej6.com is a CNAME for ubuntu-server.whitej6.com that resolves to 10.0.0.16

➜  ~ dig influxdb.whitej6.com

; <<>> DiG 9.10.6 <<>> influxdb.whitej6.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61162
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;influxdb.whitej6.com.		IN	A

;; ANSWER SECTION:
influxdb.whitej6.com.	236	IN	CNAME	ubuntu-server.whitej6.com.
ubuntu-server.whitej6.com. 238	IN	A	10.0.0.16

;; Query time: 21 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Apr 06 13:44:41 CDT 2021
;; MSG SIZE  rcvd: 93

➜  ~ 

Generating Certificate

Using Cerbot to generate my Let’s Encrypt certificate provides a simple end-to-end solution for generating and managing a signed certificate. There are a few different challenges that can be used to validate domain ownership. Since I have minimal experience with Certbot and I can easily administer my DNS records in my domain, I chose to use a DNS challenge. In the example below you will see I issue the command and Certbot responds with a TXT record I need to create for _acme-challenge.whitej6.com with a value of XXXXXX-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX. After I create the record, I can then hit Enter to continue and Certbot will perform the challenge and validate whether the value of the TXT record matches what was expected. Let’s Encrypt certificates are short-lived certificates that last 90 days. Certbot supports renewing the certificates via certbot renew command.

➜  ~ sudo certbot -d "*.whitej6.com" --manual --preferred-challenges dns certonly
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Requesting a certificate for *.whitej6.com
Performing the following challenges:
dns-01 challenge for whitej6.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please deploy a DNS TXT record under the name
_acme-challenge.whitej6.com with the following value:

XXXXXX-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Before continuing, verify the record is deployed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/whitej6.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/whitej6.com/privkey.pem
   Your certificate will expire on 2021-07-05. To obtain a new or
   tweaked version of this certificate in the future, simply run
   certbot again. To non-interactively renew *all* of your
   certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

➜  ~ 

Deploying Containers

The deployment and management of containers is done via docker-compose to simplify the management of Docker.

~/influxdb/docker-compose.yml

---
version: "3"
services:
  influxdb:
    image: influxdb:latest
    restart: always
    ports:
      - "127.0.0.1:8086:8086"
    volumes:
      - "/influxdb:/var/lib/influxdb"

I want to ensure each deployed service can communicate with other services as needed. To accomplish this, I am using the same compose project name in the deployments. If each service is deployed in another compose project, the name resolution and east/west communication becomes complicated. In the output below, you will see Docker is not too terribly happy to see services declared outside the docker-compose.yml file, this is to be expected.

➜  ~ export COMPOSE_PROJECT_NAME="server"
➜  ~ docker-compose -f influxdb/docker-compose.yml up -d
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
WARNING: Found orphan containers (server_grafana_1, server_gitlab_1, server_redis_1, server_registry_1, server_netbox_1, server_postgres_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Starting server_influxdb_1  ... done
➜  ~ 

Configuring NGINX

I personally like keeping my base NGINX configuration file default and overload what I need with individual .conf files in the conf.d folder. Each override I declare in it’s own file to make it easier for me to identify and make changes. The file paths below are based on the default install location of NGINX. Your install may use different paths, but the concept is the same.

/etc/nginx/conf.d/redirect.conf

Forces all HTTP traffic to redirect to HTTPS and keeps the requested host intact.

server {
    listen 80 default_server;
    return 301 https://$host$request_uri;
}

/etc/nginx/conf.d/influxdb.conf

map $http_x_forwarded_proto $thescheme {
    default $scheme;
    https https;
}

server {
    # Inbound requested hostname
    server_name influxdb.whitej6.com;

    listen 443 ssl;

    # Let's Encrypt certificate location
    ssl_certificate /etc/letsencrypt/live/whitej6.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/whitej6.com/privkey.pem;

    client_max_body_size 25m;
    proxy_set_header X-Forwarded-Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto $thescheme;
    add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';

    # Where to reverse proxy HTTP traffic to
    location / {
        proxy_pass http://localhost:8086;
    }

}

Restart NGINX

Once the configuration is complete for NGINX, you need to restart the NGINX service. I am using systemctl to manage running system services on the host.

➜  ~ sudo systemctl restart nginx
➜  ~

Validating Stack

➜  influxdb echo "curl from outside the docker network"
curl from outside the docker network
➜  influxdb curl https://influxdb.whitej6.com/health
{"checks":[],"message":"ready for queries and writes","name":"influxdb","status":"pass","version":"1.8.4"}%
➜  influxdb
➜  influxdb echo "curl from inside the docker network"
curl from inside the docker network
➜  influxdb docker exec -it server_gitlab_1 sh -c "curl http://influxdb:8086/health"
{"checks":[],"message":"ready for queries and writes","name":"influxdb","status":"pass","version":"1.8.4"%
➜  influxdb 

With all the services deployed and verified I have a fully functioning multi-tenanted Docker host securely serving each service over HTTPS with DNS records to improve useability. This has enabled me to move the Grafana and InfluxDB service off the Raspberry Pi hosting my aquarium controller to the Docker host to improve user experience. Hopefully this post helps you in securely deploying services to a Docker host.

GraphQL Aliasing with Nautobot

2021-04-08T00:00:00+00:00

GraphQL aliasing is a very potent feature that allows the user to customize the query results by specifying key names in the returned data.

GraphQL is very efficient, in large part because it allows a single query to access info from multiple resources and returns only the specific information the user requests. This greatly reduces the number of required queries and eliminates the need for post-query data filtering.

GraphQL aliasing is another feature within GraphQL that increases efficiency. This post will demonstrate GraphQL aliasing with two real-world use cases and examples.

A recent Network to Code blog post covered how to construct GraphQL queries using Nautobot’s GraphiQL interface. This post will build on that prior post, so if you have not read that prior post yet and are not familiar with GraphQL, I’d recommend reading that before going further because this article builds on those concepts.

What Is GraphQL Aliasing?

GraphQL aliasing allows the user to customize the names of data keys in the returned data. This is helpful in situations where the user is working in a pre-existing data architecture that expects specific key names or where it’s helpful to have key names that conform to the existing architecture’s naming conventions.

This is perhaps better explained with examples, so let’s get to those. The reader can follow along with these examples at the Nautobot public demo sandbox at https://demo.nautobot.com/. Log in with the posted username and password on the site.

An Ansible Context

Ansible is a powerful abstraction and orchestration tool. Working in that context, it’s helpful to have Nautobot return data that conforms to Ansible’s naming conventions.

A Baseline Query

Here is a GraphQL query in Nautobot that uses a query parameter and returns all device names in a site:

query {
  ams_devices:devices(site:"ams") {
    name
  }
}

NOTE: The query keyword is optional. The above query can also be conducted with query omitted, as such:
{
  ams_devices:devices(site:"ams") {
    name
  }
}

Here is the returned data for this baseline example:

{
  "data": {
    "ams_devices": [
      {
        "name": "ams-edge-01"
      },
      {
        "name": "ams-edge-02"
      },
      {
        "name": "ams-leaf-01"
      },
      {
        "name": "ams-leaf-02"
      },
      {
        "name": "ams-leaf-03"
      },
      {
        "name": "ams-leaf-04"
      },
      {
        "name": "ams-leaf-05"
      },
      {
        "name": "ams-leaf-06"
      },
      {
        "name": "ams-leaf-07"
      },
      {
        "name": "ams-leaf-08"
      }
    ]
  }
}

Query with Aliasing

Since we’re in an Ansible context in this example, it’s likely more useful to have the device names come back with an inventory_hostname key instead of the name key since inventory_hostname matches up well with Ansible terminology and use cases.

Here is an example that does that. Notice the aliasing, done by prepending inventory_hostname: to the queried name parameter:

{
  ams_devices:devices(site:"ams") {
    inventory_hostname:name
  }
}

This is the data that gets returned:

{
  "data": {
    "ams_devices": [
      {
        "inventory_hostname": "ams-edge-01"
      },
      {
        "inventory_hostname": "ams-edge-02"
      },
      {
        "inventory_hostname": "ams-leaf-01"
      },
      {
        "inventory_hostname": "ams-leaf-02"
      },
      {
        "inventory_hostname": "ams-leaf-03"
      },
      {
        "inventory_hostname": "ams-leaf-04"
      },
      {
        "inventory_hostname": "ams-leaf-05"
      },
      {
        "inventory_hostname": "ams-leaf-06"
      },
      {
        "inventory_hostname": "ams-leaf-07"
      },
      {
        "inventory_hostname": "ams-leaf-08"
      }
    ]
  }
}

Pretty slick! Again, there is no need to post-process the data since GraphQL returned only the data we asked for and gave it the specific key name we want.

Executing Multiple Targeted Queries

Aliasing is also useful for executing what would be multiple targeted queries in a single request.

The Problem Statement

Building on the example above, imagine that you need inventory_hostname values from multiple specific sites. You want the inventory_hostname data grouped by each queried site.

Querying for multiple device sets, each with different parameters, with only a single query produces errors:

GraphQL does not like the multiple devices queries because each query has a different parameter (site, in this example).

Aliasing Solves the Problem

Aliasing allows the user to execute those multiple queries in a single request. In this use case, we will alias each devices query, making it unique within the query set:

query {
  ams_devices:devices(site:"ams") {
    inventory_hostname:name
  }
  sin_devices:devices(site:"sin") {
    inventory_hostname:name
  }
  bkk_devices:devices(site:"bkk") {
    inventory_hostname:name
  }
}

Notice each devices query is aliased to <site name>_devices (ams_devices, sin_devices, and bkk_devices), making it unique within the query set. Aliasing solves the uniqueness problem within the multiple query set, making the request valid.

Here are the returned results from Nautobot (snipped for brevity):

{
  "data": {
    "ams_devices": [
      {
        "inventory_hostname": "ams-edge-01"
      },
      {
        "inventory_hostname": "ams-edge-02"
      },
      {
        "inventory_hostname": "ams-leaf-01"
      },
    <---- snip ---->
      {
        "inventory_hostname": "ams-leaf-08"
      }
    ],
    "sin_devices": [
      {
        "inventory_hostname": "sin-edge-01"
      },
      {
        "inventory_hostname": "sin-edge-02"
      },
      {
        "inventory_hostname": "sin-leaf-01"
      },
    <---- snip ---->
      {
        "inventory_hostname": "sin-leaf-08"
      }
    ],
    "bkk_devices": [
      {
        "inventory_hostname": "bkk-edge-01"
      },
      {
        "inventory_hostname": "bkk-edge-02"
      },
      {
        "inventory_hostname": "bkk-leaf-01"
      },
    <---- snip ---->
      {
        "inventory_hostname": "bkk-leaf-08"
      }
    ]
  }
}

Notice how each site’s inventory_hostname list is a value for the site’s alias, making this data programmatically easy to leverage.

GraphQL Efficiency

Aliasing is another tool GraphQL offers that allows the user to efficiently query for specific data with a minimal amount of requests and little to no required post-processing of the returned data.

Be sure to check out the accompanying YouTube video for this post here, or click the image below.

The NTC Mag

Introduction to Diffing and Syncing Data with DiffSync

When to Use DiffSync

Overview of DiffSync

A Basic Example

Installing DiffSync

Defining the Data Model

Defining the Adapter

Generating a Diff

Syncing Changes

Conclusion

Installing Nautobot - A Complete Walk-Through

Nautobot OS Requirements

Nautobot Dependency Requirements

Optional Dependencies

Installation Overview

Install Nautobot Dependencies and Infrastructure

Install System Packages

Database Setup

Create the Nautobot Database

Verify Database Authentication and Connection

Validate Redis

Install Nautobot

Select the NAUTOBOT_ROOT Directory and Create the nautobot System User

Create the Python Virtual Environment

Sudo to nautobot and Explore the Virtual Environment

Automatic Access to the Virtual Environment

Prepare the Virtual Environment

Install Nautobot!

Configure Nautobot

Specify Optional Packages Within local_requirements.txt

Prepare the Database

Create a Superuser and the Static Directories

Install Local Requirements

Check Your Configuration**

Test the Application

Configure Nautobot’s Web Service and Worker

Configure uWSGI

Setup systemd

Configure Nautobot Service

Configure Nautobot Worker Service

Reload systemd and Verify the Nautobot Service

Troubleshooting

Configure HTTP Server

Get an SSL Certificate

Self-Signed Cert

HTTP Server Installation

NGINX Installation

NGINX Configuration

Enable Nautobot

Restart NGINX

The Home Stretch - Connect to Nautobot!

Troubleshooting

Wrapping Up

Addendum: Getting a Trusted Certificate

Ansible Become Logging

The Problem Statement

The Solution

Custom Callback Plugin

Supporting Variables

The Next Step

Conclusion

References

Nautobot GraphQL Requests via Postman and Python

Authentication

Setting Up Postman Token Authentication

Authentication Option 1: Editing the Header

Query via GraphQL in Body

Authentication Option 2: The Authorization Tab

Query via Raw Data

Authentication Option 3: Via the Collection Environment

Converting Postman Queries to Python

Wrapping Up

Cisco NSO Development Environment in Docker

Prepare a Base Image

Clone the Repository

Download Installation Files

Build the Base Image

Add a New Tag to the Base Image

Extend the Base Image

Select the `NAUTOBOT_ROOT` Directory and Create the `nautobot` System User

Sudo to `nautobot` and Explore the Virtual Environment

Specify Optional Packages Within `local_requirements.txt`

Reload `systemd` and Verify the Nautobot Service