The NTC Mag

Intro to Pandas (Part 2) - Exploratory data analysis for network traffic

2022-01-18T00:00:00+00:00

Data analytics is an important skill for every engineer and even more the Network Engineer that goes through large amounts of data for troubleshooting. Networking companies have been moving towards data science integrations for appliances and software. The Arista EOS Network Data Lake is a characteristic example where Artificial Intelligence and Machine Learning are used to analyze data from different resources and lead to actionable decisions.

This blog aims to develop these skills, and it is a part of a series related to data analysis for Network Engineers. The first part was a detailed introduction on how to use Pandas, a powerful Python Data Science framework, to analyze networking data. The second part included instructions on how to run the code in these blogs using Jupyter notebooks and the Poetry virtual environment. This third blog is going deeper into how we can explore black-box networking data with a powerful analysis technique, Exploratory Data Analysis (EDA). Naturally, we will be using Pandas and Jupyter notebooks in our examples.

Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) is an approach/philosophy for data analysis that employs a variety of statistical and graphical techniques to make sense of any type of black-box data.

Goals of EDA

EDA aims at accomplishing the following goals:

Tailor a good fitting: Matching your data as closely as possible to a distribution described by a mathematical expression has several benefits, such as predicting the next failure in your network.
Find outliers: Outliers are these odd data points that lie outside of a group of data. An example is a web server farm where all port requests are aimed at specific ports and every once in a while a random port is requested. An outlier may be caused by error or intentional testing and even adversarial attack behavior.
Create a list of ranked important factors: Removing unwanted features or finding the most important ones is called dimensionality reduction in data science terms. For example, with EDA you will be able to distinguish using statistical metrics a subset of the most important features or appliances that may affect your network performance, outages, and errors.
Discover optimal settings: How many times have you wondered how it would be if you could fine-tune your BGP timers, MTUs, or bandwidth allocation and not just guess these values? EDA helps discover the best value for networking settings.

Why EDA

EDA has been proven an invaluable tool for Data Scientists, why not for Network Engineers? We gather a lot of network data, such as packet captures, syslogs, etc., that we do not know how to make sense of or how to mine their value. Even though we use out-of-box data analytics tools, such as Splunk, the insight of the Network Engineer that stems from building a model and processing raw data, is invaluable.

How to Implement EDA

To implement EDA, you need tools that you probably use in your day-to-day network operations and did not know they were part of EDA:

Strong graphical analysis: from single variable plots to time series, and multi-variable plots, there is a graphical tool in EDA that fits your problem.
Statistics: this may include hypothesis testing, calculations of summary statistics, metrics for scale, and the shape of your data.

We will explore these techniques with a network dataset in the next section.

EDA for Network Data

In this section, we will review data preprocessing with graphical and statistical analysis EDA techniques.

Dataset

The dataset we will use is a 5GB packet capture of Operating System (OS) scans from the Kitsune Network Attack Dataset. You will find the code referenced below in the Pandas Blog GitHub repository.

Preprocessing

Pre-processing of the data includes cleaning and adding metadata. We will add useful metadata to our dataset.

We start with the necessary imports and reading the csv file to a Pandas data frame:

import numpy as np
import pandas as pd

os_scan_data = pd.read_csv("../data/OS_Scan_dataset.csv")
os_scan_data

That will print a subset of the data since the file is too large:

For more information about Pandas data frames, please check the Intro to Pandas blog post.

Then, we will create metadata timestamp objects using the to_datetime function:

import datetime

timestamps = pd.to_datetime(os_scan_data["Time"], format='%Y-%m-%d %H:%M:%S.%f')
os_scan_data["Time"] = timestamps
print("Timestamps")
print(timestamps)

The timestamps are shown below:

Timestamps
0         2017-08-07 08:17:12.597437
1         2017-08-07 08:17:12.597474
2         2017-08-07 08:17:12.597553
3         2017-08-07 08:17:12.597558
4         2017-08-07 08:17:12.597679
                     ...            
1697846   2017-08-07 09:09:25.354194
1697847   2017-08-07 09:09:25.354321
1697848   2017-08-07 09:09:25.354341
1697849   2017-08-07 09:09:25.354358
1697850   2017-08-07 09:09:25.354493
Name: Time, Length: 1697851, dtype: datetime64[ns]

Finally, we will calculate interesting derivative data, such as the packet interarrivals. To this end, we will use the numpy function np.diff that takes as input a column of numbers and subtracts its rows in pairs:

interarrival_times = np.diff(timestamps)
interarrival_times

The packet interarrival values are printed below:

array([ 37000,  79000,   5000, ...,  20000,  17000, 135000],
      dtype='timedelta64[ns]')

We append the array to the os_scan_data type, casting it to int, and print the columns of the dataset to verify that the Interarrivals column has been appended:

interarrival_times = np.append(interarrival_times, [0])
os_scan_data["Interarrivals"] = interarrival_times.astype(int)
os_scan_data.columns

Below are the column names of our data after the preprocessing:

Index(['No.', 'Time', 'Source', 'Destination', 'Protocol', 'Length', 'Info',
       'Src Port', 'Dst Port', 'Interarrivals'],
      dtype='object')

Now we are ready to create pretty graphs!

Graphical Analysis

In this section, we will focus on two graphical techniques from EDA: histograms and scatter plots. We will demonstrate how to combine the information with jointplots to analyze black-box datasets.

Histogram

The first graph that we will make may not be pretty, however it demonstrates the value and flexibility of Pandas and graphical analysis for data exploration:

os_scan_data.hist(column=["Length", "Interarrivals", "Src Port", "Dst Port"])

With a single line of code and the power of Pandas data frames, we already have a set of meaningful plots. The histogram offers a graphical summary of the distribution of a single variable dataset. In the above histograms, we see how the values of Length, Interarrivals, Src Port, and Dst Port are distributed, i.e., spread, into a continuous interval of values.

Histograms offer an insight to the shape of our data and they can be fine tuned to give us a better point of view. The main “ingredient” of the histogram is a bin; a bin corresponds to the bars that you see in the graphs above and its height indicates the number of elements that fall within a range of values. The default size of bins in the data frame hist function is 10. For example, a bin of size (width) 10 and height 1000 indicates that there are 1000 values x within the range: 0 <= x < 10. Modifying the bin size is a powerful technique to get additional granularity or a “big picture” view of the data distribution:

os_scan_data.hist(column='Length', bins=20)
os_scan_data.hist(column='Length', bins=100)

There is a whole science in how to fine-tune a histogram’s bin size. A good rule of thumb is that if you have dense data, a large size will give you a good “bird’s-eye view”. In the case of packet lengths, we have sparse of data. Therefore, the smaller bin helps us distinguish the data shape.

Scatter Plot

A scatter plot is another common graphical tool of EDA. Using a scatter plot, we are plotting two variables against each other with the goal of correlating their values:

os_scan_data.plot.scatter(x='Interarrivals', y='Src Port')
os_scan_data.plot.scatter(x='Interarrivals', y='Dst Port')

The story narrated by these two graphs is that packet interarrival values to source ports have a wider spread, i.e. 0..2 x 10^7, whereas for destination ports these values have half the spread. That may point to slow response or a high speed scan, such as an OS scan! Part of the story is a high usage of low source and destination port numbers. This may point to OS services running on these ports, that are targeted on a wide spread of intervals.

Joint Plots

Now let’s combine the scatter and histogram plots for additional insight into our data. We will use an additional plotting package, seaborn:

import matplotlib.pyplot as plt
import seaborn as sns

short_interarrivals = os_scan_data[(os_scan_data['Interarrivals'] < 10000) & (os_scan_data['Interarrivals'] > 0)]
sns.jointplot(x='Interarrivals', y='Dst Port', kind='hex', data=short_interarrivals)
sns.jointplot(x='Interarrivals', y='Dst Port', kind='kde', data=short_interarrivals)
plt.show()

Note that we used the power of Pandas data frame to define a new frame short_intervals, where we take interarrivals that are less than 10K nanoseconds. The hex type plot resembles a scatter plot with histograms on the sides. The color coding of the data points indicates higher concentration of values in this specific area. The kde (Kernel Distribution Estimate) gives a distribution similar to a histogram, however the centralizing values, i.e., kernels, are visualized as well. The three distinct parts of the graph in kde will be described with three different mathematical distributions.

Summary Statistics

Summary statistics as part of EDA are extremely useful when dealing with a large set of data:

short_interarrivals.describe()

With a single line of code, the describe Pandas function gives us several statistics such as percentiles, min, max values, etc. These statistics can lead to distribution fitting and additional insights into the data.

Autocorrelation

Finally, autocorrelation calculations show how much the values within a series, i.e., the length or interarrival values, are related:

length_series = os_scan_data["Length"]
length_series.autocorr()  
0.3938818297281779

interarrival_series = os_scan_data["Interarrivals"]
interarrival_series.autocorr() 
-0.031230988268827732

In this case the packet lengths are positively correlated, which means that if a value is above average, the next value will likely be above average. Negative autocorrelation such as the one that is observed for packet interarrivals, means that if an interarrival is above average, the next interarrival will likely be below average. This is a powerful metric for predictions.

Recap

We have reviewed how to use EDA techniques to extract useful information from black-box data. This part of the series data analytics for Network Engineers, offers a deeper understanding of the power of the Pandas library and the statistical techniques that you can implement with it. In the last part of the series, we will review some predictive models. Stay tuned!

-Xenia

Resources

GitHub repo with code examples.
Part 1: Introduction to Pandas for Network Development
Jupyter Notebooks for Development

How to Write Better Python Tests for Network Programming

2022-01-11T00:00:00+00:00

In this blog post, I will share with you a simple technique that helped me a lot in writing better, testable code: writing tests in parallel with developing the code.

Why Is It Worth Having Tests?

Have you heard any of these?

Writing tests slows down development. I will write tests when the code is ready.
It may still change; if I write tests now I will have to rewrite them, so I will write tests when the code is ready.

I have heard it countless times, and also said it myself. Today I think it is one of the most common mistakes to leave tests for later. It usually means that tests are not as good as they could be or there are no tests at all due to other priorities. Furthermore, if you expect your code to change that is actually a good argument to have tests. When you expect changes, you know that you will eventually have to retest. Perhaps you will have to amend your tests, but when some of your tests fail after the change, you get the extra verification that it’s only related to the change. Lack of decent tests results in technical debt. And like any debt, sooner or later you will have to pay it off. It usually happens when you go back to your code after a while to change/fix something, and all that time you could spend writing tests you will probably spend on manually retesting your code after changing or fixing something. If you still remember how you tested it before, this may be manageable; if not, you will spend even more time on it. You can even skip testing and rely on the grace of the gods that it will work well. But you may avoid all of this if you change just one thing!

How Do You Run Your Code?

python <your_file>.py Right? OK, time for the pro tip!

What if you avoid running code directly and run it with tests instead?

Development Through Tests

When developing code, we write functions, classes, methods. And we run them to test whether they give us what we expect. Running your code for the first time is the right time to develop tests! All you need to do is just run your code with pytest instead of running it directly; capture outputs which you normally check with print(); and gradually build your tests as you develop your code.

Let’s get our hands dirty by creating some practical examples. This is our project structure:

├── main.py
└── tests
    ├── __init__.py
    └── test_main.py

Create our first function in main.py, something simple.

# main.py
def simple_math_function(*args):
    """Sum arguments"""
    total = 0
    for arg in args:
        total += arg
    return total

Now we should test our function to check whether we get what we expect. But instead of running python main.py, we create a test in tests/test_main.py and we run pytest -s. Remember the -s option, as it gives all print() outputs on-screen. We use print in the test, but you can use it anywhere in your code. Now we just want to capture our print the same way we would by running python main.py and calling our function there.

# tests/test_main.py
import main

def test_simple_math_function():
    o = main.simple_math_function(1, 2, 3, 4, 5)
    print(o)

pytest -s
============================== test session starts ============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini
collected 1 item                                                                                                                                                                     

tests/test_main.py 15
.

============================== 1 passed in 0.01s ===============================

I usually use -k option to point to a specific test. This is convenient when you already have many tests, and you want to work on one. Let’s run tests again, limiting them to only the test we work on.

pytest -s -k simple_math_function
============================== test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini
collected 1 item                                                                                                                                                                     

tests/test_main.py 15
.

============================== 1 passed in 0.01s ===============================

Our output is 15, and it is indeed the sum of all the arguments we passed to our function. Now we can just replace print with assert and we now have a test that compares the function call result with our previously captured expected result. We have our first test completed, which will remain and will be executed automatically whenever we run our tests in the future.

# tests/test_main.py
import main

def test_simple_math_function():
    assert main.simple_math_function(1, 2, 3, 4, 5) == 15

pytest -s -v -k our_simple_function
============================== test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini
collected 1 item                                                                                                                                                                     

tests/test_main.py::test_our_simple_function PASSED

============================== 1 passed in 0.02s ===============================

Note -v option, which gives more verbose output. Let’s make one more function and test.

# main.py
def simple_hello(name):
    return f"Hello dear {name}!"

# tests/test_main.py
import main

def test_simple_hello():
    print(main.simple_hello("Guest"))

pytest -sv -k simple_hello
============================== test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini
collected 2 items / 1 deselected / 1 selected                                                                                                                                        

tests/test_main.py::test_simple_hello Hello dear Guest!
PASSED

========================== 1 passed, 1 deselected in 0.02s =======================

Again we modify print to assert, and we add the expected result and run the test again.

# tests/test_main.py
import main

def test_simple_hello():
    assert main.simple_hello("Guest") == "Hello dear Guest!"

pytest -sv -k simple_hello
============================== test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini
collected 2 items / 1 deselected / 1 selected                                                                                                                                        

tests/test_main.py::test_simple_hello PASSED

========================= 1 passed, 1 deselected in 0.03s =======================

As you see, the effort is comparable to typical testing with print, but with a little more effort, we have unit tests that will remain after we remove print statements. This is a huge benefit for the future and for anyone else who will work with our code.

Practice Makes Perfect

Let’s develop something more practical from the networking world. We will use netmiko to get software version from a device, and we develop that through tests.

# main.py
from netmiko import ConnectHandler

def get_running_version(driver, host, username="admin", password="admin"):
    with ConnectHandler(
        device_type=driver,
        host=host,
        username=username,
        password=password
    ) as device:
        version = device.send_command("show version", use_textfsm=True)
    return version

# tests/test_main.py
import main

def test_get_running_version():
    version = main.get_running_version("cisco_ios", "10.1.1.1")
    print(version)

Let’s run to see what we get from the device.

pytest -sv -k get_running_version
============================== test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini
collected 3 items / 2 deselected / 1 selected                                                                                                                                        

tests/test_main.py::test_get_running_version [{'version': '15.7(3)M5', 'rommon': 'System', 'hostname': 'LANRTR01', 'uptime': '1 year, 42 weeks, 4 days, 1 hour, 18 minutes', 'uptime
_years': '1', 'uptime_weeks': '42', 'uptime_days': '4', 'uptime_hours': '1', 'uptime_minutes': '18', 'reload_reason': 'Reload Command', 'running_image': 'c2951-universalk9-mz.SPA.157
-3.M5.bin', 'hardware': ['CISCO2951/K9'], 'serial': ['FGL2014508V'], 'config_register': '0x2102', 'mac': [], 'restarted': '10:48:48 GMT Fri Mar 6 2020'}]
PASSED

======================== 1 passed, 2 deselected in 6.01s =========================

We need index 0 and the version key. We modify the return in our function in main.py and run the test again.

# main.py
def get_running_version(driver, host, username="admin", password="admin"):
    with ConnectHandler(
        device_type=driver,
        host=host,
        username=username,
        password=password
    ) as device:
        version = device.send_command("show version", use_textfsm=True)
    return version[0]["version"]

pytest -sv -k get_running_version
============================== test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini
collected 3 items / 2 deselected / 1 selected                                                                                                                                        

tests/test_main.py::test_get_running_version 15.7(3)M5
PASSED

========================= 1 passed, 2 deselected in 9.02s =======================

Now we can modify our test: remove print and add assert and enter the returned value as the expected value, then we run the test again.

# tests/test_main.py
import main

def test_get_running_version():
    version = main.get_running_version("cisco_ios", "10.1.1.1")
    assert version == "15.7(3)M5"

pytest -sv -k get_running_version
============================== test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini
collected 3 items / 2 deselected / 1 selected                                                                                                                                        

tests/test_main.py::test_get_running_version PASSED

======================== 1 passed, 2 deselected in 8.01s =========================

Our test works fine, but it takes 8 sec to complete because we still connect to the real device. We need to mock up netmiko output. Under tests/conftest.py, we create FakeDevice class, where we overwrite netmiko send_command method, which we use to get structured output of show version, and we return the same output that we collected from the device with print. Because we call ConnectHandler with context manager, we also need to implement __enter__ and __exit__ methods. Next we create mock_netmiko fixture where we use pytest monkeypatch to patch ConnectHandler in our main.py module. This fixture we use as an argument in our test function. You can read more on how to mock/monkeypatch in pytest documentation.

# tests/conftest.py
import pytest
import main


class FakeDevice:
    def __init__(self, **kwargs):
        pass

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        pass

    def send_command(self, *args, **kwargs):
        return [
            {
                'version': '15.7(3)M5',
                'rommon': 'System',
                'hostname': 'LANRTR01',
                'uptime': '1 year, 42 weeks, 4 days, 1 hour, 18 minutes',
                'uptime_years': '1',
                'uptime_weeks': '42',
                'uptime_days': '4',
                'uptime_hours': '1',
                'uptime_minutes': '18',
                'reload_reason': 'Reload Command',
                'running_image': 'c2951-universalk9-mz.SPA.157-3.M5.bin',
                'hardware': ['CISCO2951/K9'],
                'serial': ['FGL2014508V'],
                'config_register': '0x2102',
                'mac': [],
                'restarted': '10:48:48 GMT Fri Mar 6 2020'
            }
        ]


@pytest.fixture()
def mock_netmiko(monkeypatch):
    """Mock netmiko."""
    monkeypatch.setattr(main, "ConnectHandler", FakeDevice)

# /tests/test_main.py
import main

def test_get_running_version(mock_netmiko):
    version = main.get_running_version("cisco_ios", "10.1.1.1")
    assert version == "15.7(3)M5"

We run the test again.

pytest -sv -k get_running_version
============================== test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini
collected 3 items / 2 deselected / 1 selected                                                                                                                                        

tests/test_main.py::test_get_running_version PASSED

=========================== 1 passed, 2 deselected in 0.02s =====================

This time it took only 0.02 sec to execute the test because we used mock and did not connect to the device anymore.

More on Developing Tests

Check out Netmiko Sandbox, where you can get more practice with structured command output from multiple vendor devices—all available as code, so you don’t even have to run any device! You can also easily collect command outputs for your mocks.

Also check out Adam’s awesome series of blog posts on pytest in the networking world, where Adam shares practical fundamentals of testing. Part 1 Part 2 Part 3 Pay attention to test parametrization and consider how we could extend our first two tests with more parameters.

Conclusion

It may seem like Test Driven Development, but is it really TDD? Well, TDD principles say that a test is developed first, before the actual code that makes the test pass. In this approach code and tests are developed in parallel, so formally it doesn’t strictly follow TDD principles. I would put this in between TDD and the typical code development followed by tests.

The presented approach to testing requires you to change your habits of how you run your code during development, but it has several significant advantages:

tests are developed in parallel with code, Will do it later is avoided
manual tests are input to automated tests, work on manual tests done once can be automatically executed later
better code quality, developed code is testable, you will not be able to develop tests for untestable code
increased test coverage right from the beginning as opposed to tests developed later
greater confidence after implementing changes/fixes as all tests can be performed instantaneously and automatically

-Patryk

Advanced Options for Building a Nautobot SSoT App

2022-01-06T00:00:00+00:00

In the first part of this series, we reviewed the building blocks of an SSoT app for Nautobot. We reviewed the design of DiffSyncModel classes, the CRUD methods on those classes, building your System of Record adapters to fill those models, and finally the Nautobot Job that executes the synchronization of data between your Systems of Record. In this second half, we’ll review advanced options available to you when architecting an SSoT app like controlling the order of processing for your data and handling special requirements for object deletion.

Please note: it is expected that you’ve read the Nautobot Plugin: Single Source of Truth (SSoT) and Building a Nautobot SSoT App posts and understand the framework terminology, such as Data Source and Data Target.

In the designing of your SSoT application you might find yourself in a situation where you want to define the processing order of your SoR objects. The standard method of processing a set of objects in the DiffSync process can’t be guaranteed but is typically a simple first in, first out queue defined by the order the objects were presented by your adapters. To change this behavior you can extend the Diff class itself and define the processing order. One option might be to process each class of your objects alphabetically, as shown in the example below:

from collections import defaultdict
from diffsync.diff import Diff


class CustomOrderingDiff(Diff):
    """Alternate diff class to list children in alphabetical order, except devices to be ordered by CRUD action."""

    @classmethod
    def order_children_default(cls, children):
        """Simple diff to return all children in alphabetical order."""
        for child_name, _ in sorted(children.items()):
            yield children[child_name]

In some cases you wish to have this done for a single object type, like Devices. This can be done by having a method in your custom Diff class named after the type of the object in the pattern order_children_<type>. It will utilize the order_children_default method for any other object classes that haven’t been explicitly defined. This option also allows you to control the order of CRUD operations that happen on a particular object, as shown in the example below:

    @classmethod
    def order_children_device(cls, children):
        """Return a list of device sorted by CRUD action, starting with deletion, then create, and update, along with being in alphabetical order."""
        children_by_type = defaultdict(list)

        # Organize the children's name by action create, update, or delete
        for child_name, child in children.items():
            action = child.action or "skip"
            children_by_type[action].append(child_name)

        # Create a global list, organized per action, with deletion first to prevent conflicts
        sorted_children = sorted(children_by_type["delete"])
        sorted_children += sorted(children_by_type["create"])
        sorted_children += sorted(children_by_type["update"])
        sorted_children += sorted(children_by_type["skip"])

        for name in sorted_children:
            yield children[name]

Once you’ve defined your custom Diff ordering class you simply need to pass it to the appropriate diff_from/diff_to or sync_from/sync_to methods, as shown below:

from diffsync import DiffSyncFlags
from diffsync.exceptions import ObjectNotCreated

    def sync_data(self):
        """SSoT synchronization from Device42 to Nautobot."""
        client = Device42API()
        d42_adapter = Device42Adapter(job=self, sync=self.sync, client=client)
        d42_adapter.load()
        nb_adapter = NautobotAdapter(job=self, sync=self.sync)
        nb_adapter.load()
        diff = nb_adapter.diff_from(d42_adapter, diff_class=CustomOrderingDiff)
        if not self.kwargs["dry_run"]:
            try:
                nb_adapter.sync_from(d42_adapter, diff_class=CustomOrderingDiff)
            except ObjectNotCreated as err:
                self.log_debug(message=f"Unable to create object. {err}")
            self.log_success(message="Sync complete.")

Custom Diff classes can come in handy when you need to ensure that an obsolete version of an object has been removed before a newer version being installed to prevent possible conflicts.

In addition to controlling the flow of your object processing, you might have a situation where the synchronization fails or you only want to consider objects that exist in one of or both of your Systems of Record. In these cases you would want to utilize a DiffSync Flag. The core DiffSync engine provides two sets of flags, allowing for modifying behavior of DiffSync at either the Global or Model level. As the name implies, global flags apply to all data and while model flags apply to a specific model. A list of the included global flag options (as of DiffSync 1.3) has been provided below:

Name	Description
`CONTINUE_ON_FAILURE`	Continue synchronizing even if failures are encountered when syncing individual models.
`SKIP_UNMATCHED_SRC`	Ignore objects that only exist in the source/”from” DiffSync when determining diffs and syncing. If this flag is set, no new objects will be created in the target/”to” DiffSync.
`SKIP_UNMATCHED_DST`	Ignore objects that only exist in the target/”to” DiffSync when determining diffs and syncing. If this flag is set, no objects will be deleted from the target/”to” DiffSync.
`SKIP_UNMATCHED_BOTH`	Convenience value combining both SKIP_UNMATCHED_SRC and SKIP_UNMATCHED_DST into a single flag
`LOG_UNCHANGED_RECORDS`	If this flag is set, a log message will be generated during synchronization for each model, even unchanged ones.

Like your custom Diff ordering class, utilizing the global flags simply requires applying them to the appropriate diff and sync methods in your Job, as below:

from diffsync.enum import DiffSyncFlags
flags = DiffSyncFlags.CONTINUE_ON_FAILURE
diff = nb_adapter.diff_from(d42_adapter, diff_class=CustomOrderingDiff, flags=flags)

Model flags are applied to individual DiffSyncModel instances, for example, you could apply them from the adapter’s load method, as shown in the example below:

from diffsync import DiffSync
from diffsync.enum import DiffSyncModelFlags
from models import Device

class NSOAdapter(DiffSync):

    device = Device

    def load(self, data):
        """Load all devices into the adapter and add the flag IGNORE to all non-ACI devices."""
        for device in data.get("devices"):
            obj = self.device(name=device["name"])
            if "ACI" not in device["name"]:
                obj.model_flags = DiffSyncModelFlags.IGNORE
            self.add(obj)

The DiffSync library, as of version 1.3, currently includes two options for Model Flags, as shown in the table below:

Name	Description
`IGNORE`	Do not render diffs containing this model; do not make any changes to this model when synchronizing. Can be used to indicate a model instance that exists but should not be changed by DiffSync.
`SKIP_CHILDREN_ON_DELETE`	When deleting this model, do not recursively delete its children. Can be used for the case where deletion of a model results in the automatic deletion of all its children.

Both global flags and model flags are stored as a binary representation. This allows for storage of multiple flags within a single variable and allows for additional flags to be added in the future. Due to the nature of each flag being a different binary value it is necessary to perform a bitwise OR operation when utilizing multiple flags at once. Imagine the scenario where you want to skip objects that don’t exist in either Systems of Record and log all object records regardless of their being changed. You would first need to define one flag and then perform the bitwise OR operation, as shown in the example:

>>> from diffsync.enum import DiffSyncFlags
>>> flags = DiffSyncFlags.SKIP_UNMATCHED_BOTH
>>> flags
<DiffSyncFlags.SKIP_UNMATCHED_BOTH: 6>
>>> bin(flags.value)
'0b110'
>>> flags |= DiffSyncFlags.LOG_UNCHANGED_RECORDS
>>> flags
<DiffSyncFlags.LOG_UNCHANGED_RECORDS|SKIP_UNMATCHED_BOTH|SKIP_UNMATCHED_DST|SKIP_UNMATCHED_SRC: 14>
>>> bin(flags.value)
>>> '0b1110'

Now that you’ve defined exactly how you want your SSoT application to handle the data from your Systems of Record you might have a requirement to perform some action on the data once the sync has completed. Luckily, the SSoT app makes this easy by looking for a sync_complete method in your DataTarget adapter and running it if found. A case where this could be used is one where deletion of objects in your Systems of Record needs to be handled in a specific manner due to inter-object dependence. An example of this would be something like a Site in Nautobot that can’t be deleted until all devices, racks, and other objects in that Site have been deleted or moved. To perform this operation you would need to define your object’s delete method, as below:

def delete(self):
    """Delete Site object from Nautobot.

    Because Site has a direct relationship with many other objects, it can't be deleted before anything else.
    The self.diffsync.objects_to_delete dictionary stores all objects for deletion and removes them from Nautobot
    in the correct order. This is used in the Nautobot adapter sync_complete function.
    """
    self.diffsync.job.log_warning(message=f"Site {self.name} will be deleted.")
    super().delete()
    site = Site.objects.get(id=self.uuid)
    self.diffsync.objects_to_delete["site"].append(site)  # pylint: disable=protected-access
    return self

The delete method is marking the object as deleted, but instead of deleting it immediately from Nautobot’s database, it is adding it to a list of objects to be removed once the synchronization has completed and the appropriate order of deleting objects can be performed, as shown in the following example:

from diffsync import DiffSync
from django.db.models import ProtectedError

class NautobotAdapter(DiffSync):
    """Nautobot adapter for DiffSync."""

    objects_to_delete = defaultdict(list)

    def sync_complete(self, source: DiffSync, *args, **kwargs):
        """Clean up function for DiffSync sync.

        Once the sync is complete, this function runs deleting any objects
        from Nautobot that need to be deleted in a specific order.

        Args:
            source (DiffSync): DiffSync
        """
        for grouping in (
            "ipaddr",
            "subnet",
            "vrf",
            "vlan",
            "cluster",
            "port",
            "device",
            "device_type",
            "manufacturer",
            "rack",
            "site",  # can't delete a site until all of its dependent objects, above, have been deleted
        ):
            for nautobot_object in self.objects_to_delete[grouping]:
                try:
                    nautobot_object.delete()
                except ProtectedError:
                    self.job.log_failure(obj=nautobot_object, message="Deletion failed protected object")
            self.objects_to_delete[grouping] = []

        return super().sync_complete(source, *args, **kwargs)

Just be aware that any changes made to your Systems of Record through the sync_complete method should be ones that won’t impact the data sets between your Systems of Record. This is essential to minimize unnecessary updates on subsequent runs of your sync Job. An example of this would be performing a DNS query of your devices and creating IP Addresses and interfaces on devices after the sync is complete. Doing so would cause these same IP Addresses and interfaces to be absent in the comparison of your Systems of Record and would thus be deleted and then re-added again after the sync finished. This would cause a repeated cycle of objects being removed and re-added. The best practice is to have any manipulation of your data sets in your Systems of Record performed within your adapters and prior to the diff and sync are performed.

Now that you know the basics of designing an SSoT app and have been enlightened to the power of global and model DiffSync flags, custom Diff classes, and the sync_complete method, the options for designing your Single Source of Truth application are limited only by your imagination. We at Network to Code look forward to seeing what you and the community creates.

-Justin

Working with Webhooks in Nautobot

2021-12-23T00:00:00+00:00

This week’s Nautobot blog deals with webhooks. In basic terms, a webhook is a method for one web application to programmatically provide information to another web app.

The use case for webhooks is predicated on events: when a certain event happens, another event should happen in response. If that response is an action by a different app, then a webhook can be sent to notify the app that it needs to take action. The webhook can also carry information to that receiving app, including:

Notification that a specific event happened
Information about the event

Let’s illustrate this with a practical use case. This post will describe how to create a webhook in Nautobot that will trigger when a new device is created. The webhook will notify Microsoft Teams to post a message about the event in a channel.

Getting a Target URL (MS Teams Example)

A webhook needs a target URL: this is a destination endpoint for the webhook to send its information to.

This example will use MS Teams, but most of the chat platforms have easy methods for creating incoming webhook endpoints.

It’s quite simple to get an incoming webhook URL for MS Teams:

1. In the desired Teams channel, click on the three dots (…) in the top-right corner of the channel.

2. Click on Connectors.

3. Search for incoming webhooks and then click on Add; this action allows you to set webhooks on the channel.

4. To create the webhook, go back to the three dots (…) in the top right of the channel and click on Connectors.

5. You will see Incoming Webhook listed as a connector; click on Configure.

6. On the configuration screen, give the webhook a name and click Create.

7. You will be taken to a new screen that has a webhook URL. Copy that URL: this is the target URL for your webhook you’ll configure in Nautobot.

Configuring the Webhook in Nautobot

Webhooks are an Extensibility feature in Nautobot, and configuration is quite simple. To create the webhook for this example:

1. Navigate to Extensibility –> Webhooks –> +

2. Fill out the form, including:

Name
Specify the object(s) in the multi-selection drop-down menu (in our example the only object is DCIM | Device)
Enable the webhook
Specify the criteria for sending the webhook (create/update/delete) - select create for our example
Populate the URL section with the target URL that MS Teams gave us
Populate the Body template section with the JSON { "text":"data" } for now (a later segment of this article will cover this section in more detail)
Click Create

At this point, we have a very simple webhook:

Let’s test it! Create a new device in Nautobot by navigating to Devices –> Devices –> + in the top-level Nautobot menu. Fill out the Add a new device form and click on the Create button. Your MS Teams channel should show a message that looks similar to this:

The Full Device Data Appendix has the complete JSON returned by the data context environment variable.

For the Body template section, we just included the data context variable for now, since that contains the richest set of information. The Nautobot documentation has more detailed information on webhook configuration and Jinja2 template support.

Since we sent all the data context variable to MS Teams, the output in our channel is a dump of the JSON within data. This is handy output to start because it gives us a visual of what attributes are present and what we can parse for. The next section digs into this and the notion of context variables a bit more.

The Body template Section

Let’s dig a bit deeper into the Body template section. This part of the webhook carries information to the receiving endpoint URL. Nautobot makes several context variables available for use in this section. The available context variables include: event, model, timestamp, username, request_id, and data.

The entire Body template section (as well as the Additional headers section) supports Jinja2 templating. The user can access and parse the context variables via Jinja2 formatting.

If this section is left blank, Nautobot will populate the request body with a raw dump of the webhook context. Many platforms will not accept a pure JSON dump, as they require specific formatting. For example, MS Teams and Slack require the webhook info to be in a key, value pair format, with "text" as the key.

So, as an example, to send MS Teams the device name, and the username that created the device, the body template would be:

{ "text":"{{ data.name }} created by {{ username }}"}

Using info within the data and username context variables, let’s craft a more descriptive message to send to MS Teams:

{ "text":"New device created in Nautobot. Device '{{ data.name }}' created in site '{{ data.site.name }}' with status '{{ data.status.value }}' on '{{ data.created }}' by user '{{ username }}'"}

Update your webhook body template section with this data and create another device in Nautobot; you will see a more informative message in MS Teams:

Webhooks and ChatOps Symmetry

For those familiar with Nautobot’s ChatOps app and for those who may not be yet, I’d like to point out a symmetry here:

ChatOps allows a user to query Nautobot for information and to initiate other actions within Nautobot from a chat platform (Slack, MS Teams, Webex Teams, Mattermost).
Nautobot’s webhooks allow Nautobot to proactively communicate to users in their chat platform(s) when specific events happen.

These two actions together form a back-and-forth synergy, making interaction with Nautobot more efficient.

Wrapping Up

Webhooks can play an important role in workflows because they coordinate activities between applications. As a central part of your automation infrastructure, Nautobot’s webhooks feature gives you another option for integration that best suits your environment.

Thank you for your time, and have a Happy New Year!

-Tim Fiola

Developer Advocate

Appendix: Full Device Data

This is the full json output for the newly created device:

{'id': '8d40f3b3-5a04-413f-bee0-c6ff312bbed1', 
  'url': '/api/dcim/devices/8d40f3b3-5a04-413f-bee0-c6ff312bbed1/', 
  'name': 'web-test-01', 
  'device_type': {
    'id': '63d6b13d-e2ab-42b9-a847-eb218708dc3a', 
    'url': '/api/dcim/device-types/63d6b13d-e2ab-42b9-a847-eb218708dc3a/', 
    'manufacturer': {
      'id': '9843c7a4-6139-480f-879c-e012bcb5ae34', 
      'url': '/api/dcim/manufacturers/9843c7a4-6139-480f-879c-e012bcb5ae34/', 
      'name': 'Cisco', 
      'slug': 'cisco', 
      'display': 'Cisco'
    }, 
    'model': 'Nexus 9Kv', 
    'slug': 'cisco-nx-osv-chassis', 
    'display': 'Cisco Nexus 9Kv'
  }, 'device_role': {
    'id': 'bc1fffae-e7e5-426c-a08f-b9b5a986bab3', 
    'url': '/api/dcim/device-roles/bc1fffae-e7e5-426c-a08f-b9b5a986bab3/', 
    'name': 'Backbone', 
    'slug': 'backbone', 
    'display': 'Backbone'
  }, 
  'tenant': None, 
  'platform': None, 
  'serial': '', 
  'asset_tag': None, 
  'site': {
    'id': '9117f79b-148b-47f6-9d71-e984d602f1ed', 
    'url': '/api/dcim/sites/9117f79b-148b-47f6-9d71-e984d602f1ed/', 
    'name': 'Jersey City', 
    'slug': 'jcy', 
    'display': 'Jersey City'
  }, 
  'rack': None, 
  'position': None, 
  'face': None, 
  'parent_device': None, 
  'status': {
    'value': 'active', 
    'label': 'Active'
  }, 
  'primary_ip': None, 
  'primary_ip4': None, 
  'primary_ip6': None, 
  'secrets_group': None, 
  'cluster': None, 
  'virtual_chassis': None, 
  'vc_position': None, 
  'vc_priority': None, 
  'comments': '', 
  'local_context_schema': None, 
  'local_context_data': None, 
  'tags': [], 
  'custom_fields': , 
  'created': '2021-12-14', 
  'last_updated': '2021-12-14T17:31:23.589832Z', 
  'display': 'web-test-01'}

Building a Nautobot SSoT App

2021-12-22T00:00:00+00:00

In a previous post we established the importance of having a single source of truth (SSoT), provided an overview of the Nautobot Single Source of Truth (SSoT) framework, and how the SSoT framework works to enable synchronization of your Systems of Record (SoR). In addition, we’ve also shown a Nautobot SSoT App for Arista CloudVision which extends the SSoT base framework. So now you ask, how do I synchronize my data to and from Nautobot using the Single Source of Truth framework? In this first part of a two-part series, I’ll be explaining the basics of creating your own SSoT app, and then next month, I’ll be following up with more advanced options available when building an SSoT app.

Please note: it is expected that you’ve read the Nautobot Plugin: Single Source of Truth (SSoT) post and understand the framework terminology, such as Data Source and Data Target.

The first thing to do when creating an SSoT app is to define the data shared between your SoR that you want to synchronize. For example, you might want to pull your Rooms from Device42 into Nautobot. You would then create a class that inherits from the DiffSyncModel class as shown below:

from diffsync import DiffSyncModel
from typing import List, Optional


class Room(DiffSyncModel):
    """Room model."""

    _modelname = "room"
    _identifiers = ("name", "building")
    _shortname = ("name",)
    _attributes = ("notes",)
    _children = {"rack": "racks"}
    name: str
    building: str
    notes: Optional[str]
    racks: List["Rack"] = list()

You’ll notice that there are both public and private class attributes defined. Each private attribute is used to help define the model itself within the DiffSync framework. There are two attributes required on every DiffSyncModel, the _modelname and _identifiers attributes. The _modelname attribute defines the type of the model and is used to identify the shared models between your SoR. The _identifier attribute specifies the public attributes used to generate a name for the objects created when loading data from your adapters. It’s essential to confirm that the identifiers used for the object make it globally unique to ensure an accurate sync.

The remaining attributes are optional but can be quite useful in the process. The _shortname attribute identifies an object apart from other objects of the same type allowing for use of a shorter name. The _attributes attribute specifies all attributes that are of interest for synchronization. You’ll notice that each of these public attributes is defined using pydantic typing syntax. These are essential for ensuring data integrity while performing the synchronization. Please note that you must use the Optional type for any attribute that you wish to allow to be None. The last private attribute is _children, which defines other models related to the model you’re creating. In this example, Rooms are children of the Building as you have many Rooms inside a Building. This allows you to define a hierarchy of models for importing. Please note that this is meant for a direct parent-to-child relationship and not multi-branching inheritance. The _children attribute is defined using the pattern of {<model_name>: <field_name>}.

The next step is to define the CRUD (Create, Update, Delete) methods for each model. These methods will handle taking the data, once loaded from your Data Source, and making the relevant changes to the object in your Data Target. Although you may add the CRUD methods for your object to the DiffSyncModel class that you created in the first step, best practice is to create new classes that inherit from that DiffSyncModel class, as shown below:

from django.utils.text import slugify
from nautobot.dcim.models import RackGroup as NautobotRackGroup
from nautobot.dcim.models import Site as NautobotSite


class NautobotRoom(Room):
    
    """Nautobot Room CRUD methods."""

    @classmethod
    def create(cls, diffsync, ids, attrs):
        """Create RackGroup object in Nautobot."""
        new_rg = NautobotRackGroup(
            name=ids["name"],
            slug=slugify(ids["name"]),
            site=NautobotSite.objects.get(name=ids["building"]),
            description=attrs["notes"] if attrs.get("notes") else "",
        )
        new_rg.validated_save()
        return super().create(ids=ids, diffsync=diffsync, attrs=attrs)

    def update(self, attrs):
        """Update RackGroup object in Nautobot."""
        _rg = NautobotRackGroup.objects.get(name=self.name, site__name=self.building)
        if attrs.get("notes"):
            _rg.description = attrs["notes"]
        _rg.validated_save()
        return super().update(attrs)

    def delete(self):
        """Delete RackGroup object from Nautobot."""
        self.diffsync.job.log_warning(f"RackGroup {self.name} will be deleted.")
        super().delete()
        rackgroup = NautobotRackGroup.objects.get(**self.get_identifiers())
        rackgroup.delete()
        return self

Each of the create(), update(), and delete() methods for an object are called once a diff is completed and the synchronization process is started. Which method is called depends upon the required changes to the object in your Data Target. When the create() method is called, the object’s identifier and other attributes are passed to it as the ids and attrs variables respectively. The diffsync variable is for handling interactions with the DiffSync Job, such as sending log messages. For the logging of the Job results to be accurate, it is essential that the object is returned to the create method with the variables passed. However, unlike the create() method, the update() method receives only the attributes that have been changed for an object. This means that it is required for the implementer to validate if attributes have been passed or not before making appropriate changes. The delete() method will receive only the class object itself.

When utilizing inheritance between models, ensure the related models have the update_forward_refs() method called. This is essential to establish the relationships between objects.

Once the models and their CRUD methods have been defined, the next step is to write the adapters that load the models you specified in the previous steps. It is in this sense that the adapter class adapts the data from your Data Source. The adapters are required to reference each model that you wish to have considered at the top of the DiffSync object along with a top_level list of your models in the order that you wish to have them processed, as you can see below:

from diffsync import DiffSync
from .models import Building, Room


class Device42Adapter(DiffSync):
    """DiffSync adapter for Device42 server."""

    building = Building
    room = Room

    top_level = ["building"]

from diffsync import DiffSync
from .models import Building, Room


class NautobotAdapter(DiffSync):
    """Nautobot adapter for DiffSync."""

    building = Building
    room = Room

    top_level = ["building"]

As you can see above, you will always have two Systems of Record in a diff so you will need an adapter for both. It is best practice to have them matching at the top to ensure that items are processed identically. As you can see in the examples above, only the Building model is in the top_level list as the Room model is a child and will be processed after the Building is. It is up to the implementer to determine how they wish to load the models they create in the adapters. While loading your models from the methods in your adapters, it is essential that you pass valid DiffSyncModel objects that adhere to what you specified in your models when passed to the add() function. Failing to do so will cause validation errors.

It’s advised to use a load() method to call your other model-specific methods to keep things concise.

from diffsync.exceptions import ObjectAlreadyExists

class Device42Adapter(DiffSync):
...

    def load_rooms(self):
        """Load Device42 rooms."""
        for record in self._device42.api_call(path="api/1.0/rooms")["rooms"]:
            if record.get("building"):
                room = self.room(
                    name=record["name"],
                    building=record["building"],
                    notes=record["notes"] if record.get("notes") else "",
                )
                try:
                    self.add(room)
                    _site = self.get(self.building, record.get("building"))
                    _site.add_child(child=room)
                except ObjectAlreadyExists as err:
                    self.job.log_warning(f"{record['name']} is already loaded. {err}")
            else:
                self.job.log_warning(f"{record['name']} missing Building, skipping.")
                continue

The example above shows how data is pulled from the Device42 API and creates the Room objects that were detailed in the first step. Once the object has been created, it is then added into the DiffSync set with the add() method. As a Room is a child object of a Building, there is an additional step of finding the parent Building object with the get() method, and then using the add_child() method to add the relationship between the objects. If there is an existing object with the same identifiers, the ObjectAlreadyExists exception will be thrown, so it’s advised to wrap the add() method in a try/except block.

With your adapters for each SoR created, the final step is to write your Nautobot Job. This will handle the loading of your models from the adapters, the diff of the objects once loaded, and the synchronization of data by calling the CRUD methods as appropriate. The Job class must be derived from either the DataSource or DataTarget class and is required to include a sync_data method to handle the synchronization process. Optionally, you can also add a config_information or data_mappings method to enrich the data presented to the end user in Nautobot.

from django.templatetags.static import static
from nautobot.extras.jobs import Job
from nautobot_ssot.jobs.base import DataSource

from diffsync.exceptions import ObjectNotCreated
from .device42 import Device42Adapter
from .nautobot import NautobotAdapter


class Device42DataSource(DataSource, Job):
    """Device42 SSoT Data Source."""

    class Meta:
        """Meta data for Device42."""

        name = "Device42"
        data_source = "Device42"
        data_source_icon = static("./d42_logo.png")
        description = "Sync information from Device42 to Nautobot"

    def sync_data(self):
        """Device42 Sync."""
        d42_adapter = Device42Adapter(job=self, sync=self.sync)
        d42_adapter.load()
        nb_adapter = NautobotAdapter(job=self, sync=self.sync)
        nb_adapter.load()
        diff = nb_adapter.diff_from(d42_adapter)
        self.sync.diff = diff.dict()
        self.sync.save()
        self.log_info(message=diff.summary())
        if not self.kwargs["dry_run"]:
            try:
                nb_adapter.sync_from(d42_adapter)
            except ObjectNotCreated as err:
                self.log_debug(f"Unable to create object. {err}")
            self.log_success(message="Sync complete.")


jobs = [Device42DataSource]

As is shown, the Job should create an instance of each of your adapters and call their load() methods to create the DiffSyncModel objects. Once that’s done, a diff and sync can be completed utilizing either the diff_from/diff_to and sync_from/sync_to methods on the adapter objects. Which you use depends upon which way you wish to have the synchronization performed. In the example above, once the models have been loaded, a diff from Device42 to Nautobot is done to report the required objects to be created, updated, or deleted. Finally, if the Job is not a dry-run, synchronization will be executed. Again, this is done with a sync_from from the Device42 adapter object to the Nautobot adapter object. Depending upon the results of an object being created, an ObjectNotCreated exception may be thrown, so it’s advised to use a try/except block when calling the sync methods to ensure it’s caught and handled appropriately. Once all of the objects have been processed, a final success log is sent and the GUI should be updated to reflect the changes.

In summary, the creation of an SSoT app requires the following steps to be completed:

Create one or more DiffSyncModel classes to define the data you wish to synchronize.
For each DiffSyncModel, define the CRUD methods to handle the requisite changes in your Data Target.
Write a DiffSync adapter class for each System of Record to load data from each into your DiffSyncModel classes.
Finally, write your Nautobot Job to perform the synchronization of data between your Data Source and Data Target.

In the next part of this series, we’ll look into how to customize the processing of objects and the use of global and model flags in the DiffSync process.

-Justin

Integrated Version Control in Nautobot

2021-12-16T00:00:00+00:00

In the past few months, many of us here at Network to Code have been investing a lot of time with Nautobot Version Control. The ultimate goal with adding version control to Nautobot is to make network automation safer by bringing proven Git- and GitHub-like workflows directly to Nautobot.

Along the way, we’ve created a lot of content in order to showcase how the version control capability can benefit Nautobot users and those looking for safer ways to manage data. This week’s post serves as an aggregation for all the Version Control related content to date.

Below, organized by content type, are hyperlinks to each piece of content that has been produced over the past few months.

Code and Documentation

Version Control repository
- This holds the open source code for the Version Control integration.
Version Control documentation on readthedocs.io
- These are the official docs for the integration.

Blogs

Database Version Control with Nautobot
- This blog discusses the value that Version Control brings with its Git-like workflows and how those workflows fit into existing Nautobot capabilities to keep network automation data clean.
Nautobots, Roll Back!
- This post discusses the business and operational value in allowing users to review changes to SoT data prior to pushing them to production, and then to quickly undo that set of changes if needed.
Keeping Data Clean with Nautobot Config Contexts, Schemas, and Git
- This post is not Version Control related explicitly, but it does discuss Nautobot features you can leverage that complement the Version Control workflows.
- These features, especially when paired with Nautobot Version Control app workflows, provide a layered, defense in depth to keep your automation data clean.

Videos

YouTube Version Control playlist
- Intro Series: This series of videos introduces the user to the Version Control with Nautobot, its use cases, and how to use the app.
  - Unit 1: Defining the problem at hand
  - Unit 2: Bringing version control to Nautobot’s database
  - Unit 3: Navigating the top-level Version Control menu
  - Unit 4: Managing pull requests with Version Control
- Demo Series: This series of videos provides demonstrations of how to set up a demo instance of Version Control with Nautobot, workflows enabled by it, and how to interact with Version Control programmatically.
  - Unit 1: Setting up a demo environment
  - Unit 2: Version Control enabled workflow demo
  - Unit 3: Version Control: Jenkins CI pipeline added
  - Unit 4: Using the Version Control API

In the coming months we hope to have more engagement to get feedback users and to further showcase the business and operational benefits a version-controlled Nautobot environment provides.

Thank you! Have a great day, and have a great New Year!

-Tim Fiola

Developer Advocate

How a Job as an Automation Engineer Has Changed Decisions I Make in My Personal Life

2021-12-14T00:00:00+00:00

Early on in my network journey I was tasked with data mining and bulk configuration change tasks, which were quite common tasks at the NOC I was working at that time. These weren’t tasks I was terribly pleased with doing after the third or fourth time doing the same exact thing over and over again. Things started to evolve from there, simple formulas in spreadsheets to transform data to a Java app that helped with incident management. With each solution I conjured up, my job became less repetitive and my perception on any repetitive tasks changed. A few chapters later in my career I moved on from one of solutions to building automation at scale.

As any automation engineer will admit, the first step in automating a known workflow is thinking differently. This typically starts with evaluating current requirements; then reevaluating legacy requirements helps to shape the outcome. So many times have I spent weeks developing a solution only to find that requirement X exists because an engineer said or did something years ago, and no one has questioned whether this is still applicable. Once we’ve settled on what the new workflow will look like, the second part to thinking differently is focusing on trying to say yes, instead of how many reasons there may be to say no. A wireless engineer once told me it was impossible to have help desk troubleshoot a wireless connection via automation. After talking through the most common issues, I was able to create a playbook that was tied to a Slack chat command that covered the most common use case and prevented those tickets from making it to network ops.

These simple working practices have now influenced decisions in my personal life. Now, when selecting almost any non-trivial purchase my first thought is can I automate some portion or use technology to enrich my experience? I have always had the mentality to solve with technology first, but typically that has led to shortsighted purpose-driven purchases.

Aquarium

There is a running joke with my coworkers that I can’t write a blog post or do a presentation without making it about my aquarium at some point. To make sure I don’t disappoint, automation has been front-of-mind for so many purchases for my aquarium. This started with creating a controller based on a RaspberryPi and industrial sensors so that I could have better visibility into telemetry metrics, such as temperature and pH. After starting to collect data, I quickly found myself rethinking the equipment I had selected. This opened my eyes to the art of the possibility.

About six months after starting my first aquarium build (in many years), I started to dream of what is next. How can I take what I learned to make more informed decisions that make my life easier and make a better ecosystem for my pets? I went from having to manually top off fresh water, cleaning filters every few days, and manually adding chemicals to a mostly hands-off approach to managing my aquarium.

Things Improved with Automation

Cleaning filter socks every few days -> Automated roller mat that changes the filter media based on an optical sensor and only requires once-a-month maintenance.
Daily water top offs of ~1 gallon of fresh water -> Auto top off that uses a 10-gallon reservoir and optical sensor to ensure a more stable salinity.
Telemetry statistics
- Reagent-based tests taking roughly half an hour to complete in total, which led to never being done -> Some done every 2 minutes while others are done every 6 hours, and all are stored in a time-series database with Grafana for visualization.
- Non-reagent test which takes seconds to perform BUT rarely did -> Just like reagent-based, performed every 2 minutes and sent to the same time-series database.
Lighting was done with hard on/off timers which could be a harsh transition -> Lighting based on a ramp sunrise/sunset/moonlight cycle to help promote happy pets and coral growth.

Fireworks

Growing up in a small town, every year the Fourth of July was a big event for my family, and I was fascinated with setting off fireworks. As adults, my brother and I still have that childlike fascination. His yearly Fourth of July part started off with him manually setting up each firework one by one for the whole family to marvel at, but he wanted something more.

Our first automated firework display was roughly 15 minutes long and was done the old-fashioned way. In the weeks leading up to the party my brother had spent countless hours watching YouTube videos of the specific fireworks he purchased and researching exact run times of each. When the time came to build the display, we had four sheets of plywood where he had mapped out each firework and the exact length of time delay fuse that was needed to trigger each one without overlap. It was a masterpiece and had a single point of failure…. One fuse path, nothing had multiple paths or a concept of failure domain. By some miracle, the whole show went from start to finish without issue, and only one mortar failed to launch.

The next two years the display was driven by a remote firing system. This gave us the ability to stop mid-show, if needed (fire safety reasons were a concern), and we went from one failure domain to sixteen failure domains. Evolving the display was a big step forward to ensure success, plus it gave two adult “children” the ability to press buttons to blow things up, so it was a big win.

Home Automation

This mindset of automation-first has also influenced purchases of light switches, garage door openers, TVs, audio systems, and even Christmas lights. Small tasks of having schedules for exterior lights or skills with Amazon Alexa have improved the ecosystem that is my home. With simply telling Alexa It's bedtime all of my interior lights & TVs shutoff. This was a huge win living with someone who always forgets to turn things off.

The choice for an audio system was something I pondered for quite some time. Finding a single room/purpose solution was easy but the right whole home audio experience that integrated with other ecosystems I had was the more difficult task. AirPlay2 compatibility along with a built in multi-zone audio, multiple available form factors including indoor & outdoor solutions, and Amazon Alexa integration were the tech-related requirements. In the end, the most comprehensive product that also had great audio quality ended up being on the higher end of pricing, but having the complete end-to-end integration was a huge deciding factor.

My latest child-like obsession that has been influenced by automation has been my Christmas light display. This year is my second year of having a display driven via automation and choreographed to music. When I was a child there was one house that had a few props that would have sections that would blink to look like rotating tires on a truck or Santa’s waving at people as they passed by. This brought me so much joy and being able to build a display using a RaspberryPi, a soldering iron, and FM transmitter just to potentially bring others the same joy it brought me as a child is worth the expense.

Summary

This post has been a departure from normal tech blogs, but I think it’s fascinating to see how what I do day in and day out for work has influenced my personal life. It’s very common to see people in tech use tech to make their lives easier, but I have found a higher concentration of an automation-first approach at home in automation engineers than others in tech. Always question why and how you do things—you may one day surprise yourself by realizing how automation could make things easier.

-Jeremy

Nautobot ChatOps for Grafana

2021-12-09T00:00:00+00:00

Two of the more intriguing topics I have heard lately that also seems to resonates with network engineers and network professionals is the insight telemetry provides, and the ease of use chat platforms such as Slack and Microsoft Teams deliver to your keyboard and fingertips. The Grafana ChatOps application is designed to provide the best of both worlds. Grafana ChatOps is a Nautobot extension used with the Nautobot ChatOps base framework to provide all the operational graphs provided by Grafana delivered via chat clients.

Today, we will walk through some of the features within the Grafana ChatOps integration, as well as some of the requirements and procedures to get up and running with Grafana ChatOps.

An important note on the architecture design choices with this ChatOps app (plugin) is that chat commands are defined dynamically based on the Grafana panels and dashboards (we’ll go into this a little later). When you launch the app for the first time, you will see that no chat commands have been defined yet. You can define commands automatically or manually and tie them to specific Grafana panels within a dashboard.

Installation

The package for the Grafana ChatOps app is available on PyPI and can be installed using pip.

Prior to installing the Nautobot Grafana Plugin, you should have the following installed:

Nautobot installed and configured.
Grafana application installed and configured with dashboards and panels.
Grafana Image Rendering Service installed.
Grafana Image Rendering Plugin for Grafana installed in your Grafana application.
Nautobot Plugin ChatOps installed and configured for your specific chat platform.

For the full installation guide, please refer to the Grafana ChatOps repo Install Guide.

Usage

Building Grafana ChatOps commands can be done using a manual or automated approach. The automated approach uses the DiffSync library to synchronize Grafana dashboards, panels, and variables with the Nautobot Grafana ChatOps plugin.

Defining Commands

To define a command within the Grafana plugin for use with your chat client, there are two main components that we need to have populated.

Define at least one Grafana Dashboard.
Define at least one Grafana Panel within the Dashboard.

This tutorial will take you through the steps noted above to get a chat command exposed in your chat client.

The first step is to define a dashboard so that the Grafana plugin is aware of the dashboard that exists within Grafana. You can define a dashboard in Grafana in two ways: defining a dashboard manually or using the “Sync” feature to synchronize your Grafana dashboards automatically.

Defining a Dashboard Manually

To define a dashboard manually, you can go to Plugins > Dashboards and click the + Add button located in the upper right of the screen. In the form for a new dashboard, you need to define the slug, uid, and Friendly Name.

NOTE: You can find the slug and uid info by navigating to your Grafana instance and going to the desired dashboard,

Defining a Dashboard Using the Sync Method

Alternatively, you can define a set of dashboards by synchronizing your Grafana dashboard configuration to the Grafana plugin. To synchronize dashboards, within Nautobot, navigate to Plugins > Dashboards and click the Sync button.

This process will utilize the DiffSync library to synchronize, create, update, and delete dashboards in Nautobot with the Dashboards that are defined in the Grafana application. Once complete, you will see all dashboards imported into Nautobot.

Defining Grafana Panels

The second step to defining Grafana commands in Nautobot for your chat client is to define the panels you wish to expose via chat.

Panels are closely associated to chat commands, where there will be a chat command for each panel defined.

Similar to dashboards, you can define panels in two ways within Nautobot.

Defining a Panel Manually

To define a panel manually, go to Plugins > Panels and click the + Add button located in the upper right of the screen. In the modal for a new panel, you need to select the dashboard that the panel is defined under, then add a command name, along with a friendly name, and define the Panel ID.

The Active checkbox will allow the command to show up in your chat client. If the panel is marked as inactive, it will still be defined in Nautobot, but restricted from being shown in the chat client.

NOTE: You can find the panel id by navigating to your desired panel, selecting View, then looking at the URL.

Defining Panels Using the Sync Method

Alternatively, you can define a set of panels by synchronizing your Grafana panels configuration for a given dashboard to the Grafana plugin. To synchronize panels for a dashboard, within Nautobot, navigate to Plugins > Panels and click the Sync button.

This process will utilize the DiffSync library to synchronize, create, update, and delete panels in Nautobot with the Dashboard Panels that are defined in the Grafana application. Once complete, you will see all panels for a dashboard imported into Nautobot.

Panels are synchronized on a per-dashboard basis. All panels synchronized will be INACTIVE by default, you will need to set them to active to see them in Chat.

Once your dashboard and panels have been defined, and you activate the panels you wish to expose to the chat client, you will be able to see the available chat commands, as well as run commands to generate your panels.

Advanced Usage

Additional functionality can be added to the Grafana ChatOps plugin if you have variables defined on your dashboards. Panel variables can also be imported via the “Sync” functionality and associated with a panel. Then you can go in and customize how the variables behave and even enrich the ChatOps experience using Nautobot as a Source of Truth for your variables!

To read more on the advanced usage of the Grafana ChatOps plugin with panel variables, refer to the Advanced Usage Guide in the repository.

Conclusion

ChatOps has given a conduit to retrieve and respond interactively using a platform that is already in place and used for communication across almost any device, while Grafana has provided a feature-rich observability platform. With the Nautobot Grafana integration, we can now have the best of both worlds. Let us know how you’re using the Grafana ChatOps or if you have any questions or issues in the GitHub repo.

-Josh Silvas

Nautobot Chatops for Cisco ACI

2021-12-07T00:00:00+00:00

We’re excited to announce the newest addition to our growing list of Nautobot Chatops Applications, the Cisco ACI Chatops Plugin! The Cisco ACI Chatops integration makes it possible to interact with the ACI controller, the APIC (Application Policy Infrastructure Controller), using chat commands in Slack, Mattermost, Cisco Webex, and Microsoft Teams. With this integration, network operations teams supporting Cisco ACI can use chat commands to:

execute commands against multiple different APIC clusters in different data centers and/or regions
register new leaf or spine switches in the fabric using chat commands
quickly glean useful data from an APIC for informational or troubleshooting purposes

Below, we’ll review some of the features and commands in this initial release.

Multi-Fabric Support

Multi-Fabric refers to the ability to provide support for multiple APIC clusters. While there is not currently support today for the Cisco Multi-Site Orchestrator (MSO), which provides a single point of management for multiple APIC clusters, the ACI Chatops app supports configuration of as many individual APIC controllers as needed. The chat platform will prompt for the APIC to execute a command against (it’s also open source, so contributions are more than welcome!):

In addition, the selection dialogue can be avoided, if desired, simply by providing the APIC cluster name as the second argument to the chat command. For example, to execute against the APIC cluster called ntcapic in our configuration, the command would be:

/aci get-tenants ntcapic

ntcapic is a friendly name assigned to our APIC cluster. Under the hood it informs the Chatops app which APIC hostname and credentials to use. The details can be found in the Installation Guide.

Node Registration

In Cisco ACI, when new Leaf and Spine switches are plugged into the ACI fabric for the first time, they are discovered using LLDP (Link Layer Discovery Protocol) and show up in the APIC as unregistered nodes. An administrator must then access the APIC GUI and register the new node by assigning it a name and unique ID number. The below set of chat commands could be used by network operators to view registered and pending nodes, and then register a newly discovered node in the fabric.

Get Nodes

Displays a list of all registered nodes in the fabric.

Get Pending Nodes

Displays a list of any unregistered nodes that have been discovered.

Register Node

Information Gathering

The below chat commands can be used to retrieve and display configuration and operational details from the APIC.

Display APIC Details

Don’t remember the hostname or IP addressing details of the APIC? Need to look up the serial number or model information? No problem, just run the aci get-controllers command!

Display Tenants

Get the list of tenants from an APIC using the command aci get-tenants.

Display Application Profiles

Get the list of Application Profiles in a specified tenant using the aci get-aps command.

You can also specify all for the tenant field to get a list of all Application Profiles in the fabric across all tenants.

Display Endpoint Groups (EPGs)

Get a list of all EPGs in a specified Application Profile using the aci get-epgs command.

You can also specify all for the Application Profile selection dialogue to see EPGs for all Application Profiles in a tenant.

How about all EPGs across all tenants? Sure, just specify all for the Tenant dialogue…

Display Endpoint Group Details

The aci get-epg-details chat command provides useful information about a specified EPG, consolidating information from several API calls.

Display VRFs

The aci get-vrfs chat command displays the VRFs in use in a specified tenant.

It is also possible to display all VRFs in the fabric by selecting all from the tenant selection dialogue.

Display Bridge Domains

The aci get-bds command displays Bridge Domains in a specified tenant and includes useful details from several API calls, such as the configured subnet, VRF, L2 Forwarding, and L3 Routing details.

You can also get a fabric-wide view of Bridge Domains by selecting all from the tenant selection dialogue.

Display Interface State

The aci get-interfaces command can be used to quickly view interface state on a specified node.

You can also filter for all operational or non-operational interfaces by selecting up or down from the Interface State selection dialogue.

Wrapping Up

With the commands developed so far, our main focus was on providing the ability to glean useful operational details from an ACI fabric; but we could easily implement any task using the extensible API that Cisco ACI provides. What other chat commands for Cisco ACI would you find useful? Please feel free to hit us up in the comments or in our Public Slack channel.

-Matt

Data Modeling for Network Engineers

2021-11-30T00:00:00+00:00

Structured Data, Schemas, Network as Code, and Sources of Truth—what do these all have in common? Data is central to all of them in concept and in practice. How do we work with all of that data? By modeling it of course! We’ll discuss how to start data modeling. Data modeling is not typically the focus of the conversation but is usually what you are working with, and it’s important to understand some ways to approach data modeling.

Data models are abstractions—they detail the information and relationships we need to take into account that, when combined with assumptions, come up with the entire description of the subject. Data modeling is more of an art than a science. Coming up with a data model will almost always be an iterative approach: having too much and paring back or having too little and adding.

Generally, there are two approaches to modeling:

Top-down: Starting with what you want and expanding/refining
Bottom-up: Starting with what you have and refining

In top-down modeling, you typically try to come up with the data that you think you’ll need; likely it won’t be all or the same data that you end up with. In this case, the data modeling exercise is much like developing a new design.

In bottom-up modeling, you will have many more points of data than your model will end up describing. The beginning of a bottom-up process would likely be a configuration itself. As you see as we begin to cover in our example, we start to peel the layers back as we make assumptions and take into account certain facts or axioms about our data, design, and configurations.

Diving into Topologies

We’re going to cover more of the bottom-up process today since that’s where a lot of NetDevOps are starting: with existing networks, data, and configurations that need to be reasoned over and modeled.

We’ll go over an example focused on modeling Devices’ Connections and what we need to model in order to derive their configurations. Consider the diagram below. It is a typical campus design where a location aggregation or distribution router aggregates all of the connections from the building and then connects to the core. This is a very common network design pattern. We’re interested in modeling the Layer 3 interfaces of Dist A in the diagram below. Generally, this modeling exercise should be applicable to most other Layer 3 interfaces in the network, but especially other Layer 3 interfaces connected to core devices on distribution switches.

Configurations

Because our design has been ruthlessly standardized, the topology above is a good example for all of our campus buildings. From the diagram, we can immediately determine that our two distribution switches should have some standard ports dedicated to certain functions: 1 uplink to the campus core, 2 cross-connects, and 2 downlinks to access switches for each distribution switch. Here’s a snippet of the configuration for dist-a.

interface 1/1/55 no shutdown mtu 9198 qos trust none description Connection to C.Core 1 ip mtu 9198 ip address 10.10.1.2/30 arp timeout 600

interface 1/1/56 no shutdown mtu 9198 qos trust none description Connection to C.Core 2 ip mtu 9198 ip address 10.10.1.5/30 arp timeout 600

Keep in mind that we’re trying to get the minimum amount of data that will be able to regenerate the configuration above.

From this configuration, we could start with a model like this:

[{
    "name": "1/1/55",
    "shutdown": False,
    "lag": None,
    "mtu": 9198,
    "qos-trust": "none",
    "description": "Connection to C.Core 1",
    "routing": True,
    "trunking-mode": None,
    "allowed-vlan": "all",
    "native-vlan": None,
    "ip-mtu": 9198,
    "ip-address": "10.10.1.2/30",
    "arp-timeout": 600
},
{
    "name": "1/1/56",
    "shutdown": False,
    "lag": None,
    "mtu": 9198,
    "qos-trust": "none",
    "description": "Connection to C.Core 2",
    "routing": True,
    "trunking-mode": None,
    "allowed-vlan": "all",
    "native-vlan": None,
    "ip-mtu": 9198,
    "ip-address": "10.10.1.5/30",
    "arp-timeout": 600
}]

For data above, it is very verbose, touching on each possible configuration across most of the interfaces. This would quickly get out of hand if we need to create and manage this for every interface across all the switches across a campus. Let’s pare the model back some for our specific interface above.

We can start with taking away properties we can assume at the point of use (in the template or when building out the data) or assumed defaults:

If there is an IP, routing will be enabled and trunking will not. 
All of our configured interfaces will be enabled.
The MTU will always be 9198. 
If there is an IP, we need an IP-MTU statement.
ARP Timeout will always be 600.

With these assumptions, our model would look something like this

[{
    "name": "1/1/55",
    "description": "Connection to C.Core 1",
    "ip-address": "10.10.1.2/30",
},
{
    "name": "1/1/56",
    "description": "Connection to C.Core 2",
    "ip-address": "10.10.1.5/30",
}]

Our resulting data model above is much less verbose, making it easier to read and manage. Keep in mind the data that we will put into the model for rendering must be stored somewhere, so the fewer points of data we need for each interface the better.

Usage

After developing the data model, we can easily input the data and use it in a template such as the interface template below. While this specific template may match a certain switch model, the data input should or could match across different models of switches.

interface {{ interface["name"] }}
   mtu 9198
{% if interface["description"] | length > 1 %}
   description {{ interface["description"] }}
{% endif %}
{% if interface["ip_addresses"] | length > 0 %}
   no switchport
{% endif %}
{% if interface["ip_addresses"] | length > 0 %}
{% for addr in interface["ip_addresses"] %}
{% if addr["address"] is defined %}
   ip address {{ addr["address"] }}
{% endif %}
{% endfor %}
   ip mtu 9198
   ip arp timeout 600
{% endif %}
   no shutdown

Process

While developing the model, it may be helpful to keep track of the model in a spreadsheet or table. Every property of the model can be kept in a spreadsheet, to make it easier to view the model all at once. Keeping the expected variable types, whether a property is required or is optional, the source of the variable in an instance of the model, and finally an example of the property value are all helpful to explain and work with the data model.

Here is an example of values to keep track of for the data model above.

property	attribute type	required	example	system of record	description/notes
name	string	required	1/1/55	nautobot
description	string	optional	Connection to access sw 1	nautobot	determined by cable connection in nautobot
ip_addresses	string	optional	10.10.1.2/30	nautobot	from nautobot device interface

Conclusion

We went over what data modeling is, how to get started with the data modeling process for network interfaces, how the data model could be used, and how to keep track of the model. In future blogs, we’ll go over the process for modeling other aspects of configuration and how the data model could be represented and/or derived from a Source of Truth. Thanks for reading!

Stephen Corry