The NTC Mag

Nautobot Plugin: Single Source of Truth (SSoT)

2021-08-05T00:00:00+00:00

In managing networks, there are systems of record (SoR) that own distinct sets of information about the network, and sources of truth (SoT) from which users and network automation tools pull their required data. All too commonly, these are one and the same, with each SoR serving as an SoT, which has the undesirable consequence that the user or automation needs to consult multiple SoT in order to do their job. Worse, in many cases there is overlap or intersection between various SoR – for example, device hardware inventory might be managed in one SoR, while device configuration might belong to a different SoR – which requires the consumer to cross-correlate information between SoTs in order to gain the full picture. Fortunately, there’s a solution – the introduction of a unified single source of truth (SSoT) that aggregates data from multiple SoR and provides a single unified point of access to this data. Towards this end, we are now introducing the Single Source of Truth (SSoT) plugin for Nautobot.

Overview

This open-source plugin is designed to enable you to use Nautobot as your network’s SSoT, unifying data from any number of SoR behind Nautobot’s UI and APIs. Additionally, this plugin makes it quick and easy to develop and integrate specialized Nautobot Jobs that use DiffSync to synchronize data from other systems (“Data Sources”) into Nautobot and/or from Nautobot to other systems (“Data Targets”) as desired. In short, whatever system(s) you need to synchronize with Nautobot, you can support via Jobs (either open-source or custom-built) that use this plugin.

Please note that there is a 1:1 correlation between Nautobot plugins and Nautobot Apps. The terms can be used interchangeably.

Key features of the Nautobot SSoT plugin include:

SSoT Dashboard

The Single Source of Truth Dashboard UI lists available Data Sources and Data Targets and summarizes recent synchronization history.

Data Source / Data Target Views

Each installed Data Source or Data Target automatically creates a corresponding detailed landing page that displays its configuration, data mappings between the source/target and Nautobot, and sync history.

SSoT Job Result Database and Views

After executing a data synchronization Job, its impact (diffs, number of changes overall), outcome (successes/failures/errors), and detailed logs are automatically saved in Nautobot’s database, and can be reviewed in detail according to your needs.

Installing the Plugin

The plugin is available as a Python package in PyPI and can be installed atop an existing Nautobot installation using pip:

pip install nautobot-ssot

This plugin is compatible with Nautobot 1.0.3 and higher.

Once installed, the plugin needs to be enabled in your nautobot_config.py:

# nautobot_config.py
PLUGINS = [
    # ...,
    "nautobot_ssot",
]

Using the Plugin

As initially installed, the plugin provides example Data Source and Data Target jobs for syncing some basic data (Regions and Sites) between your Nautobot installation and a remote Nautobot instance (such as demo.nautobot.com). We’ll use those example Jobs for this introduction.

Accessing the SSoT Dashboard

You can begin by selecting Plugins > Single Source of Truth > Dashboard from the navigation bar (or navigating to /plugins/ssot/) on your Nautobot instance to view the SSoT dashboard:

Running a Data Synchronization Job

From here you can click the Sync button next to any of the available Data Sources or Data Targets - for this example, let’s click the Sync button under Example Data Source to begin syncing data from a remote Nautobot instance to your local Nautobot instance. This will open a web form (which should look familiar if you have any prior experience with Nautobot Jobs) prompting you for any necessary inputs and options for the data sync:

Note that by default the Dry Run option is selected; if this is checked, the Job will report on what would be changed but will not actually make any database updates. For sake of a more interesting example, make sure to uncheck this option before clicking the Run Job button.

After submitting the Job, you will be redirected to a standard Nautobot Job Result view, which will refresh automatically until the Job has run to completion.

Viewing Synchronization Results

Once the Job has finished, a button labeled SSoT Sync Details will appear at the top right of the page:

Clicking this button will redirect you to the SSoT plugin’s detailed view of this sync, including the diffs between the two systems that DiffSync detected and the detailed logs of exactly what got changed:

This view describes in detail everything that occurred during the data synchronization attempt. The primary Data Sync tab summarizes the overall outcome of the sync attempt, including a view of the diffs (if any) identified by DiffSync and a summary of the actions taken (create, update, delete) and their outcomes (success, failure, error).

The Job Logs tab shows any general status messages generated by the data synchronization Job as it executed; this is roughly equivalent to the Nautobot Job Result view shown earlier.

The Sync Logs tab shows the logs captured from DiffSync regarding the individual data records being synchronized, details of any contents or changes of these records, and other detailed information. (Sync logs can also be accessed directly via the Plugins > Single Source of Truth > Logs menu item if desired.)

Reviewing History and Job Details

You can return to the dashboard view and see that it now summarizes the recent sync history, including this successful sync:

From here, you can click on the Example Data Source link to inspect the detailed view of this Job, including its execution history and the data mappings that it includes in the sync:

Running Again

If you run this Job a second time, you will see that, thanks to DiffSync, the Job detects that nothing has changed that needs to be resynced, and completes quickly:

Finding, Creating, and Enabling More Jobs

Now that you have some idea of how this plugin works and what it’s capable of, you probably want to investigate how to integrate your own specific SoR systems with Nautobot via this plugin.

Keep an eye on this space, as we’ll be open-sourcing several example Data Source and Data Target Jobs in the near future for use with this plugin! You may also browse the Nautobot App Ecosystem under the Single Source of Truth category to see what’s available already and what’s coming up on the horizon.

We also have documentation on how to develop your own custom Jobs from scratch to support whatever SoR you want to integrate.

In any case, once the desired Job(s) have been developed, you can install them into your Nautobot system like any other Nautobot Job:

by packaging Jobs into a Nautobot plugin, which can then be installed into Nautobot’s virtual environment
by committing Jobs into a Git repository, which can be configured in Nautobot and refreshed on demand
by manual installation of individual Job source files to Nautobot’s JOBS_ROOT directory

Next Steps

Hopefully you can now see the value that this plugin provides as a framework for integrating any number of other SoT and SoR with Nautobot as the unified SSoT. Keeping data synchronized between these systems, especially via automation such as this, makes our lives easier and reduces the likelihood of trouble resulting from out-of-sync information. If you do develop your own Data Source and/or Data Target Job with this plugin, we’d love to hear all about it!

-Glenn

Manipulating Data with Jinja Map and Selectattr

2021-08-03T00:00:00+00:00

Manipulating data with the Jinja filters, Map, Selectattr, and Select provides quick and efficient ways to create new data structures. This can simplify creating lookup dictionaries and avoids complex loops or Jinja templating options.

In many instances in ansible, I’ll use an ordered list of dictionaries as my data model; this specifically keeps the preferred ordering for the device and type of configuration as well as makes it easy to do operations on all the members of that list. For some operations that rely on specific dictionaries, it can be convoluted to dig that information out of the list, or work on a subset of the list, without going through loops and conditionals, without these tools.

Like all good overviews, it’s best explained with an example. We’ll start with this example data structure for some selected interfaces, a commonly used pattern.

---
interfaces:
  - name: "mgmt"
    enable: true
    dhcp: true
    ip_address: no
    description: "management"
  - name: "1/1/1"
    enable: true
    ip_address: 10.1.1.2/24
    description: "uplink to core-1"
  - name: "1/1/2"
    enable: true
    ip_address: 10.1.2.2/24
    description: "uplink to core-2"
  - name: "1/1/3"
    enable: true
    ip_address: 10.1.3.1/24
    description: "peerlink to agg-2"
  - name: "vlan42"
    ip_address: 10.42.0.2/24
    ip_helper: false
    description: "inband management"
  - name: "vlan44"
    ip_address: 10.44.0.3/24
    ip_helper: true
    fhrp: true
    description: "wifi aps"

Selecattr

Selectattr takes in a list of dictionaries and applies a test to each dictionary.

One of the first use cases might be to get all of the interfaces that have an IP address defined:

{{ interfaces | selectattr('ip_address') }}

With the selectattr filter, a Jinja test can be used as the second argument; this gives us a lot of flexibility when selecting our data. Additional arguments can be specified as inputs to the test specified in the second arg.

One thing to be careful of with the selectattr filter with this example: The attribute you use must be defined, but not all of our interfaces have ip_address defined. When the default test is applied to the value of the attribute (bool), the var doesn’t exist, which will result in an error. We’ll hit that error wherever ip_address is not defined in our interfaces data structure.

To get around this in our data, we specify the ‘defined’ parameter to test whether the attribute is there or not:

{{ interfaces | selectattr('ip_address', 'defined') }}

This returns a list of dictionaries where the ip_address attribute is present. Easy enough.

[
    {
        "enable": true,
        "ip_address": "10.1.1.2/24",
        "name": "1/1/1"
    },
    {...},
]

To be even more granular, we can chain the output to another selectattr filter and further test the ip_address or another attribute. For example, we can pass the list of interfaces with IP addresses and test whether the IP address on the interface is in another list of IP addresses:

{{ interfaces | selectattr('ip_address', 'defined') | selectattr('ip_address', 'in', '[10.1.3.1/24]') }}

Which gives us this output:

    [
        {
            "description": "peerlink to agg-2",
            "enable": true,
            "ip_address": "10.1.3.1/24",
            "name": "1/1/3"
        }
    ]

Map

Map allows us to turn a list of dictionaries into a simpler list or flatten the list of dictionaries. We specify the values of the list by giving Map the attribute we want to take from the dictionaries.

Now, let’s get a list of the IP addresses. We could easily use this in a route-map, prefix-list, or ACL.

{{ interfaces | map(attribute='ip_address', default='n/a') }}

Gives the output:

    [
        "n/a",
        "10.1.1.1/24",
        "10.1.2.1/24",
        "10.1.3.1/24",
        "10.42.0.1/24",
        "10.44.0.1/24"
    ]

Similar to the selectattr, Map expects the attribute to be defined. In this case we have to specify a default value in case the dictionary doesn’t contain the specified attribute, ip_address in this case.

It might be useful to ONLY have IP addresses in our list of IPs. Here we can combine the selectattr by filtering down the dictionaries where the ip_address attribute is defined and then using Map to filter the dictionaries down to the values of the ip_address attribute.

{{ interfaces | selectattr('ip_address', 'defined') | map(attribute='ip_address') }}

Output:

[
    "10.1.1.1/24",
    "10.1.2.1/24",
    "10.1.3.1/24",
    "10.42.0.1/24",
    "10.44.0.1/24"
]

The Map filter also gives us the ability to input another filter to use on the data in a list; this allows us to use a filter that doesn’t support working on lists to work on each element of the list without using a loop. Most of the Jinja built-in filters will work without the Map filter, however.

Other Useful Combinations

If we need to look up IP addresses by interface name, we can instead use the items2dict filter. This gives us the ability to map attributes of the dictionary to new keys and values. This is useful when specific addresses are used in specific ways, such as network-specific ACLs, prefix-lists, bgp neighbors, and others that need to reference interfaces in specific ways or different orders.

We first use selectattr to filter down to which dictionaries have an ip_address, then use items2dict to output a new dictionary where the interface name is the key and the ip_address is the value.

{{ interfaces | selectattr('ip_address', 'defined') | items2dict(key_name='name', value_name='ip_address') }}

This gives us a nice tidy lookup dictionary without any loops and a relatively readable call:

{
    "1/1/1": "10.1.1.1/24",
    "1/1/2": "10.1.2.1/24",
    "1/1/3": "10.1.3.1/24",
    "vlan42": "10.42.0.1/24",
    "vlan44": "10.44.0.1/24"
}

References:

Jinja Builtin Filters

All in all, these are very handy tools to use to get new views of the same data without having to resort to too much looping or other odd manipulations. I hope you find them useful!

-Stephen Corry

Nautobot, Beyond Network Source of Truth

2021-07-29T00:00:00+00:00

As the title suggests, this post will use Nautobot for something other than a Network Source of Truth. This post will use Nautobot to solve one of life’s most difficult questions, perhaps one of the most frequently asked questions. This question is known to ruin the evening, but with Nautobot’s help, this no longer has to be the case: The question Nautobot will answer for us is, “What’s for dinner?”

This post will walk through using one new feature introduced in Nautobot version 1.1 and one existing feature, in order to have each Site in Nautobot provide a recipe recommendation. The two features are independent, but often work hand in hand to provide dynamic data to a specific entry in Nautobot. The primary feature that will provide the Site with a recipe is Custom Links. The Custom Link will make use of a Custom Jinja2 Filter in order to generate a recommendation from a third-party API.

High-Level Architecture

The API used by the Custom Jinja2 Filter will be Spoonacular. Spoonacular requires an account and token to use the API, but it allows for several API requests per day for free. The API also has several filtering capabilities, the two that will be used here are: cuisine and ingredients. A Nautobot Plugin needs to be created in order to give Nautobot access to the Custom Jinja2 Filter, which will also provide access to the Spoonacular API token using PLUGINS_CONFIG in nautbot_config.py.

The Custom Link will be added to the dcim | site Content Type, and will use the Site object to pass the Jinja2 Filter a cuisine and an ingredient. The cuisine will be obtained from the Site’s parent Region’s slug field, and the Site’s slug field will be used to provide an ingredient. This means that:

Each Region that is associated with a Site must have a valid Spoonacular cuisine for its slug value
Each Site must be associated to a Region
Each Site must have an ingredient for its slug value

A list of acceptable cuisines for the API can be found at Spoonacular Cuisines.

Project Structure

This Plugin will only provide a Jinja2 Filter, so the file structure is very simple:

nautobot_plugin_recipe_filter directory for the Plugin App
nautobot_plugin_recipe_filter/__init__.py file with the PluginConfig definition
nautobot_plugin_recipe_filter/recipe_filters.py for providing the filter to get recipe recommendations
pyproject.toml, README.md, and LICENSE for installing the Plugin in Nautobot.

$ tree nautobot-plugin-recipe-filter
.
├── nautobot_plugin_recipe_filter
│     ├── __init__.py
│     └── recipe_filters.py
├── LICENSE
├── pyproject.toml
└── README.md

nautobot_plugin_recipe_filter/init.py

"""Plugin declaration for nautobot_plugin_recipe_filter."""

__version__ = "0.1.0"

from nautobot.extras.plugins import PluginConfig


class NautobotPluginRecipeFilterConfig(PluginConfig):
    """Plugin configuration for the nautobot_plugin_recipe_filter plugin."""

    name = "nautobot_plugin_recipe_filter"
    verbose_name = "Nautobot Plugin Recipe Filter"
    version = __version__
    author = "Network to Code, LLC"
    description = "Nautobot Plugin Jinja2 Filter for Recipe Recommendations"
    required_settings = ["spoonacular_token"]
    min_version = "1.1.0"
    max_version = "1.9999"
    default_settings = {}
    caching_config = {}
    jinja_filters = "recipe_filters.py"


config = NautobotPluginRecipeFilterConfig

pyproject.toml

[tool.poetry]
name = "nautobot-plugin-recipe-filter"
version = "0.1.0"
description = "Nautobot Plugin Jinja2 Filter for Recipe Recommendations"
authors = ["Network to Code, LLC <info@networktocode.com>"]
license = "Apache-2.0"
readme = "README.md"
homepage = "https://github.com/networktocode/nautobot-plugin-recipe-filter"
repository = "https://github.com/networktocode/nautobot-plugin-recipe-filter"
keywords = ["nautobot", "nautobot-plugin"]
include = ["LICENSE", "README.md"]
packages = [
    { include = "nautobot_plugin_recipe_filter" },
]

[tool.poetry.dependencies]
python = "^3.6"
nautobot = "^v1.1.0"
requests = "*"

Creating the Custom Jinja2 Filter

Creating a Custom Jinja2 Filter that can be consumed within Nautobot follows the standard Jinja2 process, and is auto-registered to the Nautobot Environment when a Plugin is installed and enabled on the Nautobot instance. Each Plugin can define the name of the file where Jinja2 Filters should be loaded from using the PluginConfig; the default location is jinja_filters.py. The above PluginConfig specified recipe_filters.py, so the filters made available by this plugin will go in that file.

In order to register the Jinja2 Filter with Nautobot, the Filter function is decorated with @library.filter, where library is imported from the django_jinja library. The django_jinja library is included with Nautobot in order to support the Custom Jinja2 Filters feature. This Filter function will use the requests library to make API calls to the Spoonacular API. The Jinja2 Filter will return the URL to the recipe recommended by Spoonacular. In order to interact with the Spoonacular API, the token will be retrieved using the spoonacular_token setting in the PLUGINS_CONFIG; this setting is marked as required in the above PluginConfig.

recipe_filters.py

"""Custom filters for nautobot_plugin_recipe_filter."""
import requests

from django.conf import settings

from django_jinja import library


SPOONACULAR_TOKEN = settings.PLUGINS_CONFIG["nautobot_plugin_recipe_filter"]["spoonacular_token"]
SPOONACULAR_URL = "https://api.spoonacular.com/recipes"
SPOONACULAR_RECOMMENDATION_URL = f"{SPOONACULAR_URL}/complexSearch"  # URL to get recommendations
SPOONACULAR_RECIPE_URL = f"{SPOONACULAR_URL}//information"  # URL to get details of recommendation


@library.filter
def get_recipe_url(cuisine:str, ingredient:str) -> str:
    """
    Get Recipe recommendation based on cuisine and single ingredient.
    
    Args:
        cuisine (str): The cuisine derived from Nautobot's Region.slug value.
        ingredient (str): The ingredient to include in the recipe based on Nautobot's Site.slug value.
    
    Returns:
        str: The URL of the recommended recipe.
    """
    recommendation_params = {
        "apiKey": SPOONACULAR_TOKEN,
        "number": 1,  # Limit number of responses to a single recipe
        "cuisine": cuisine,
        "includeIngredients": ingredient,
    }
    # Get recipe recommendation
    recommendation_response = requests.get(url=SPOONACULAR_RECOMMENDATION_URL, params=recommendation_params)
    recommendation_response.raise_for_status()
    recipe_id = recommendation_response.json()["results"][0]["id"]
    recipe_params = {"apiKey": SPOONACULAR_TOKEN}
    # Get recipe detailed information
    recipe_response = requests.get(url=SPOONACULAR_RECIPE_URL.format(id=recipe_id), params=recipe_params)
    recipe_response.raise_for_status()
    recipe_json = recipe_response.json()
    # Get URL to recipe on Spoonacular's website
    recipe_url = recipe_json["spoonacularSourceUrl"]

    return recipe_url

The Plugin is ready to be installed into the Nautobot environment. See the Plugin Installation Guide for details. The Database Migration and Collect Static Files steps can be skipped, since the Plugin does not use a database table and does not provide static files.

Example Config Settings

PLUGINS = ["nautobot_plugin_recipe_filter"]
PLUGINS_CONFIG = {
    "nautobot_plugin_recipe_filter": {
        "spoonacular_token": "abc123"
    }
}

The Jinja2 Filters provided by Plugins are available in Nautobot wherever Jinja2 text fields are used. This post will showcase using the above Jinja2 Filter to create a Custom Link; however, the Filter could also be used by the new Computed Fields feature added in v1.1.

The reason for using Custom Links in this instance is that the goal is to provide a link to another site with the recipe and cooking instructions. Computed Fields are designed to be text fields, and have HTML escaped; this means that an HTML hyperlink would be rendered as plain-text and not be clickable.

Creating a Custom Link

Nautobot’s Custom Link feature allows hyperlinks and groups of hyperlinks to be added to the upper-right side of pages displaying a single entry (DetailViews). Creating a Custom Link has several configuration options:

Content Type: Sets the Model that will have its entries display the hyperlink
Name: The name used to identify the Custom Link within Nautobot
Text: A Jinja2 expression that will be rendered and used as the text displayed for the hyperlink
URL: The URL that will be used for the hyperlink
Weight: Determines the ordering of the hyperlinks displayed
Group Name: Allows multiple hyperlinks to be displayed under a single drop-down button
Button Class: Determines the color of the button used for the hyperlink
New Window: Determines whether clicking the hyperlink should open the URL in a new tab or the current tab

Guide to Create a Custom Link

There are two steps to creating most objects in Nautobot:

Browsing to the form to create the object from the main Navigation Bar
Filling out and submitting the form

Browsing to Add Custom Link Form

Creating a Custom Link is done in the Web UI by browsing to Extensibility > Custom Links, and clicking the + icon to bring up the form to create a new Custom Link.

Filling Out the Add Custom Link Form

The Custom Link used to provide a hyperlink to Spoonacular’s recipe recommendation specifies a Content Type of dcim | site, and a Name and Text of Recipe Recommendation. The URL uses the Jinja2 Filter defined above to obtain the URL to the recipe, and clicks will open the hyperlink in a new tab.

The Jinja2 expression for the URL makes use of the variable obj, which is the Django ORM object for the database record. Since the Content Type for this Custom Link uses the Site model, it will be a record from the Site database table. Nautobot Site entries have foreign key fields to the Region, so both the Region’s slug (cuisine) and Site’s slug (ingredient) can be passed to the Custom Jinja2 Filter.

Once all required fields are filled out on the form, clicking Create will create the Custom Link.

Using the Custom Link

The Custom Link created above is displayed on every Site page. If the Jinja2 expression fails to render an HTML link, then the Custom Link button will not be clickable. In order to showcase the Custom Link, a guide is provided below to create a Region and Site, which results in a valid URL being created for the Site.

Guide to Create a Region

In order for a valid link to be shown, each Site must belong to a Region that has a slug value with a valid cuisine.

Browsing to Add Region Form

Creating a Region is done by browsing to Organization > Regions, and clicking the + icon to bring up the form to create a new Region.

Filling Out the Add Region Form

The Region used in this example is Spain, which correlates to the spanish cuisine used by the Spoonacular API.

Once all required fields have been filled out on the form, clicking Create will create the Region.

Guide to Create a Site

In addition to each Site being linked to a Region, the Site must also use an ingredient for the value of its slug field.

Browsing to Add Site Form

Creating a Site is done by browsing to Organization > Sites, and clicking the + icon to bring up the form to create a new Site.

Filling Out the Add Site Form

The Site in this example is Madrid, and is assigned to the Spain Region created above. The slug (ingredient) value uses rice. In Nautobot, Sites require the status to be set; this Site uses the built-in Active Status.

The Site form is longer than the previous forms, but scrolling to the bottom of the form displays the familiar Create button to create the Site.

Viewing the Recipe Recommendation

Clicking on the Create button above redirects to the newly created Site Page with the Custom Link added to it.

Viewing the Custom Link on the Site Page

The Custom Link for the recipe recommendation is displayed in the top-right corner of the page.

Clicking the Link to View the Recipe

Since the check box to open the hyperlink in a new tab was selected on the Create Custom Link form, clicking the Custom Link on the Site page opens the Spoonacular recipe page in a new tab. The recommendation returned in the example is Paella.

Conclusion

Nautobot is more than a Source of Truth application. The newly added Custom Jinja2 Filters provide an easy way to perform more complex operations for fields that are rendered through Jinja2. An example of a field that supports Jinja2 expressions is the URL field in Custom Links. Custom Links can be added to Pages dynamically, and provide a convenient way of giving Users access to related pages.

The recipe recommendation example used here is perhaps not practical, but something similar could be built for linking to other Source of Truth systems. Maybe Nautobot is not the System of Record for certain data points, but is instead used to aggregate multiple data points into a single place for read-only operations. The Jinja2 Filter could look up the URL to the data point in the Application used as the System of Record. It might be handy to have a link in Nautobot that is able to take users to that page in order to manage the data there.

Ansible Execution Strategies

2021-07-27T00:00:00+00:00

Adjusting Ansible playbook execution strategies is a use case that has come up several times. The default strategy Ansible uses, is called linear, and it works great a majority of the time, but what if you’re automating a task that need to be done in a specific order or way. Strategies can be used to define how a playbook should be executed. This includes the ability to run in serial, run in batches, and even, in extreme cases, allows the ability to write a custom strategy plugin.

In this blog post I will provide multiple examples demonstrating the different strategies and settings. The playbooks that will be used will be dramatically simplified to allow the focus to be on the strategies in use rather than the content of the playbooks.

Strategy Basics

Linear(Default): Playbook runs in a linear execution, each task is run in order and the next task is not started until all hosts are finished with the current task.
Debug: The execution is done in a debug mode that allows the user to interactively run the playbook for troubleshooting purposes.
Free: Playbook is executed linearly, but each host runs the tasks at its own pace and has no requirement for all hosts to finish a task before continuing to the next task.
Host Pinned: Operating like the free strategy, host_pinned will run tasks as fast as it can on the number of hosts specified in the serial keyword, immediately starting a host as soon as one ends.

Note: The strategy host_pinned may operate as one would expect free to run, so give it try to make sure you understand the subtle difference.

To see available strategies, use ansible-doc -t strategy -l command.

Strategies can be set per PLAY or under the [defaults] group in the ansible.cfg file.

Note: Remember, a playbook often has a single play in it, but can have multiple plays.

Other Execution Tune-ables

Ansible also provides some keywords that can be used to further tune how a playbook runs.

Forks: The number of simultaneous connections a task can use to perform work.
- Forks can be tuned based on the processing power available. The default is set to 5. If processing power on the Ansible control machine is not a concern this can be modified. This allows for playbook execution to be faster.
  
  Note: Setting the fork value to 1 effectively makes each task run on a single host before the next host starts the same task.
Run Once: This option is exactly as it sounds. It’s a task that should be run only once.
- Great for operations that need to be done only once for the group. Not once per host.
- Good example would be creating a directory on the control machine for storing configuration files. You do NOT need every host to create the directory, this needs to be completed only one time.
Serial: The number of hosts that Ansible should be running side by side.
- Great for playbooks that include redundant devices that should not be worked on at the same time.
- Great for when there is possible race conditions.
Throttle: Can be used in the task definition to limit the number of hosts the task is executed on. Similar to fork, but at a task level.
Order: Order specifies how the hosts for a given play are going to be executed. By default the order comes from the inventory file, but other options exist that can provide flexibility if inventory hostnames provide insights into a given environment.
- reverse_inventory: Is the opposite of the default mode explained previously. It reverses the order from the inventory file.
- shuffle: Random order.
- sorted: Alphabetical.
- reverse_sorted: Reverse alphabetical.

Strategies and Keyword Interrelationship

This is where I found myself struggling when learning these concepts. Ansible provides all the flexibility listed above which provides great power, but there is a relationship between the options that can cause an unexpected behavior.

The key relationship to be aware of is throttle. When using it in parallel with forks or serial, throttle must be less than forks or serial.

Now that I have summarized the options available, I will demo some of the options to solidify what the strategies accomplish.

Note: The default strategy linear, will not be demonstrated.

Free Strategy

The playbook has two tasks.

Print the hostname.
Print the network OS.

The playbook:

---
- name: "DEBUG INVENTORY"
  hosts: "all"
  gather_facts: False
  strategy: "free"
  connection: "network_cli"
  tasks:
    - name: "10010: GET DEVICE NAMES"
      debug:
        msg: "{{ inventory_hostname }}"

    - name: "10015: GET DEVICE OS"
      debug:
        msg: "{{ ansible_network_os }}"
...

▶ ansible-playbook -i inventory.cfg pb.yml

PLAY [DEBUG INVENTORY] ******************************************************************************************************************

TASK [10010: GET DEVICE NAMES] **********************************************************************************************************
ok: [nxos-spine1] => {
    "msg": "nxos-spine1"
}
ok: [vmx2] => {
    "msg": "vmx2"
}
ok: [vmx1] => {
    "msg": "vmx1"
}
ok: [nxos-spine2] => {
    "msg": "nxos-spine2"
}
ok: [vmx3] => {
    "msg": "vmx3"
}
ok: [csr1] => {
    "msg": "csr1"
}
ok: [csr2] => {
    "msg": "csr2"
}
ok: [csr3] => {
    "msg": "csr3"
}
ok: [eos-leaf1] => {
    "msg": "eos-leaf1"
}
ok: [eos-leaf2] => {
    "msg": "eos-leaf2"
}

TASK [10015: GET DEVICE OS] *************************************************************************************************************
ok: [vmx1] => {
    "msg": "junos"
}
ok: [nxos-spine1] => {
    "msg": "nxos"
}
ok: [nxos-spine2] => {
    "msg": "nxos"
}

TASK [10010: GET DEVICE NAMES] **********************************************************************************************************
ok: [eos-spine1] => {
    "msg": "eos-spine1"
}
ok: [eos-spine2] => {
    "msg": "eos-spine2"
}

TASK [10015: GET DEVICE OS] *************************************************************************************************************
ok: [vmx2] => {
    "msg": "junos"
}
ok: [vmx3] => {
    "msg": "junos"
}
ok: [csr1] => {
    "msg": "ios"
}
ok: [csr2] => {
    "msg": "ios"
}
ok: [csr3] => {
    "msg": "ios"
}
ok: [eos-leaf1] => {
    "msg": "eos"
}
ok: [eos-leaf2] => {
    "msg": "eos"
}
ok: [eos-spine1] => {
    "msg": "eos"
}
ok: [eos-spine2] => {
    "msg": "eos"
}

PLAY RECAP ******************************************************************************************************************************
csr1                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
csr2                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
csr3                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-leaf1                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-leaf2                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-spine1                 : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-spine2                 : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
nxos-spine1                : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
nxos-spine2                : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vmx1                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vmx2                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vmx3                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Looking into the execution of the playbook, the first thing I notice is each task isn’t completed before the next task is executed. free allows Ansible to run each host independently and execute the tasks as quick as it can. free is a great option when the tasks aren’t dependent on one another, or the hosts themselves aren’t dependent on one another. A good example might be collecting operation state data after a change.

Debug Strategy

Very helpful when a playbook is running into issues, and the failed output isn’t helpful enough to identify the problem.

The playbook:

---
- name: "DEBUG INVENTORY"
  hosts: "all"
  gather_facts: False
  strategy: "debug"
  connection: "network_cli"
  tasks:
    - name: "10010: GET DEVICE NAMES"
      debug:
        msg: "{{ inventory_hostnames }}"
      debugger: on_failed

Notice: For demonstration I misspelled the variable name by adding an s to the end.

▶ ansible-playbook -i inventory.cfg pb.yml

PLAY [DEBUG INVENTORY] ******************************************************************************************************************

TASK [10010: GET DEVICE NAMES] **********************************************************************************************************
fatal: [nxos-spine1]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'inventory_hostnames' is undefined\n\nThe error appears to be in '/Users/jeffkala/Documents/github-clones/ansible-examples/strategies/pb.yml': line 8, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: \"10010: GET DEVICE NAMES\"\n      ^ here\nThis one looks easy to fix. It seems that there is a value started\nwith a quote, and the YAML parser is expecting to see the line ended\nwith the same kind of quote. For instance:\n\n    when: \"ok\" in result.stdout\n\nCould be written as:\n\n   when: '\"ok\" in result.stdout'\n\nOr equivalently:\n\n   when: \"'ok' in result.stdout\"\n"}

[nxos-spine1] TASK: 10010: GET DEVICE NAMES (debug)> dir()
['host', 'play_context', 'result', 'task', 'task_vars']

[nxos-spine1] TASK: 10010: GET DEVICE NAMES (debug)> task_vars.keys()
dict_keys(['ansible_network_os', 'inventory_file', 'inventory_dir', 'inventory_hostname', 'inventory_hostname_short', 'group_names', 'ansible_facts', 'playbook_dir', 'ansible_playbook_python', 'ansible_config_file', 'ansible_role_names', 'ansible_play_role_names', 'ansible_dependent_role_names', 'role_names', 'ansible_play_name', 'groups', 'ansible_play_hosts_all', 'ansible_play_hosts', 'ansible_play_batch', 'play_hosts', 'omit', 'ansible_version', 'ansible_check_mode', 'ansible_diff_mode', 'ansible_forks', 'ansible_inventory_sources', 'ansible_skip_tags', 'ansible_run_tags', 'ansible_verbosity', 'hostvars', 'environment', 'vars', 'ansible_current_hosts', 'ansible_failed_hosts'])

[nxos-spine1] TASK: 10010: GET DEVICE NAMES (debug)> task_vars.get('inventory_hostname')
'nxos-spine1'

[nxos-spine1] TASK: 10010: GET DEVICE NAMES (debug)> exit()

This strategy is extremely helpful. Looking at the play execution, the first task fails on the first host. Ansible raises a fatal error and provides the message but then drops the user into a debug shell where standard Python can be used to troubleshoot. The first step I performed was to run dir() to see what objects I could look into. For this example, I next looked into the task_vars.key() to see the valid keys. Once I printed this information I notice that I needed to use inventory_hostname instead of inventory_hostnames. Of course, this is a simplified example, but on more complex playbooks it’s a valuable option to identify issues.

Host Pinned

Host pinned is a bit more tricky, and in the backend uses serial to determine the batch number used. If serial is not set, it defaults to all.

The playbook:

---
- name: "DEBUG INVENTORY"
  hosts: "all"
  gather_facts: False
  strategy: "host_pinned"
  serial: 2
  connection: "network_cli"
  tasks:
    - name: "10010: GET DEVICE NAMES"
      debug:
        msg: "{{ inventory_hostname }}"
    - name: "10015: GET DEVICE OS"
      debug:
        msg: "{{ ansible_network_os }}"
...

▶ ansible-playbook -i inventory.cfg pb.yml

PLAY [DEBUG INVENTORY] ******************************************************************************************************************

TASK [10010: GET DEVICE NAMES] **********************************************************************************************************
ok: [nxos-spine1] => {
    "msg": "nxos-spine1"
}
ok: [nxos-spine2] => {
    "msg": "nxos-spine2"
}

TASK [10015: GET DEVICE OS] *************************************************************************************************************
ok: [nxos-spine1] => {
    "msg": "nxos"
}
ok: [nxos-spine2] => {
    "msg": "nxos"
}

PLAY [DEBUG INVENTORY] ******************************************************************************************************************

TASK [10010: GET DEVICE NAMES] **********************************************************************************************************
ok: [vmx1] => {
    "msg": "vmx1"
}
ok: [vmx2] => {
    "msg": "vmx2"
}

TASK [10015: GET DEVICE OS] *************************************************************************************************************
ok: [vmx1] => {
    "msg": "junos"
}
ok: [vmx2] => {
    "msg": "junos"
}

PLAY [DEBUG INVENTORY] ******************************************************************************************************************

TASK [10010: GET DEVICE NAMES] **********************************************************************************************************
ok: [vmx3] => {
    "msg": "vmx3"
}
ok: [csr1] => {
    "msg": "csr1"
}

TASK [10015: GET DEVICE OS] *************************************************************************************************************
ok: [vmx3] => {
    "msg": "junos"
}
ok: [csr1] => {
    "msg": "ios"
}

PLAY [DEBUG INVENTORY] ******************************************************************************************************************

TASK [10010: GET DEVICE NAMES] **********************************************************************************************************
ok: [csr2] => {
    "msg": "csr2"
}
ok: [csr3] => {
    "msg": "csr3"
}

TASK [10015: GET DEVICE OS] *************************************************************************************************************
ok: [csr2] => {
    "msg": "ios"
}
ok: [csr3] => {
    "msg": "ios"
}

PLAY [DEBUG INVENTORY] ******************************************************************************************************************

TASK [10010: GET DEVICE NAMES] **********************************************************************************************************
ok: [eos-leaf1] => {
    "msg": "eos-leaf1"
}
ok: [eos-leaf2] => {
    "msg": "eos-leaf2"
}

TASK [10015: GET DEVICE OS] *************************************************************************************************************
ok: [eos-leaf1] => {
    "msg": "eos"
}
ok: [eos-leaf2] => {
    "msg": "eos"
}

PLAY [DEBUG INVENTORY] ******************************************************************************************************************

TASK [10010: GET DEVICE NAMES] **********************************************************************************************************
ok: [eos-spine1] => {
    "msg": "eos-spine1"
}
ok: [eos-spine2] => {
    "msg": "eos-spine2"
}

TASK [10015: GET DEVICE OS] *************************************************************************************************************
ok: [eos-spine1] => {
    "msg": "eos"
}
ok: [eos-spine2] => {
    "msg": "eos"
}

PLAY RECAP ******************************************************************************************************************************
csr1                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
csr2                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
csr3                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-leaf1                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-leaf2                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-spine1                 : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-spine2                 : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
nxos-spine1                : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
nxos-spine2                : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vmx1                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vmx2                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vmx3                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Pay close attention to each iteration. When serial is used, it’s looping over the PLAY itself with the number specified in the serial keyword. In the above output, the PLAY was executed six times, each time running through the tasks with two hosts.

Now that the main strategies available have been demonstrated, the next few demos will focus on some of the other keywords that can be used to change the execution of the playbook.

Serial

Using serial without tying it to a strategy allows for further flexibility. Running playbooks in batches defined by the administrator can help validate that a playbook is working as expected before executing it against a larger batch of host. Since the previous example showed what serial does, below I’ll provide a few options when specifying the serial option.

Using different batches.

---
- name: "DEBUG INVENTORY"
  hosts: "all"
  gather_facts: False
  serial:
    - 1
    - 2
    - 5
  connection: "network_cli"
  tasks:
<omitted>
...

This will execute the PLAY with one host, then the second iteration would run on two host, then 5 host for however many iterations remain.

Using a percentage.

---
- name: "DEBUG INVENTORY"
  hosts: "all"
  gather_facts: False
  serial:
    - "30%"
  connection: "network_cli"
  tasks:
<omitted>
...

It is also possible to mix and match the different options. The serial Ansible Documentation is excellent.

Ordered

When I use the keyword order in my play definition and use sorted, the playbook is run on the host in alphabetical order.

Note: I’m back to using the default strategy of linear.

The playbook:

---
- name: "DEBUG INVENTORY"
  hosts: "all"
  gather_facts: False
  order: "sorted"
  connection: "network_cli"
  tasks:
    - name: "10010: GET DEVICE NAMES"
      debug:
        msg: "{{ inventory_hostname }}"
    - name: "10015: GET DEVICE OS"
      debug:
        msg: "{{ ansible_network_os }}"
...

▶ ansible-playbook -i inventory.cfg pb.yml

PLAY [DEBUG INVENTORY] ******************************************************************************************************************

TASK [10010: GET DEVICE NAMES] **********************************************************************************************************
ok: [csr1] => {
    "msg": "csr1"
}
ok: [csr2] => {
    "msg": "csr2"
}
ok: [csr3] => {
    "msg": "csr3"
}
ok: [eos-leaf1] => {
    "msg": "eos-leaf1"
}
ok: [eos-leaf2] => {
    "msg": "eos-leaf2"
}
ok: [eos-spine1] => {
    "msg": "eos-spine1"
}
ok: [eos-spine2] => {
    "msg": "eos-spine2"
}
ok: [nxos-spine1] => {
    "msg": "nxos-spine1"
}
ok: [vmx1] => {
    "msg": "vmx1"
}
ok: [nxos-spine2] => {
    "msg": "nxos-spine2"
}
ok: [vmx2] => {
    "msg": "vmx2"
}
ok: [vmx3] => {
    "msg": "vmx3"
}

TASK [10015: GET DEVICE OS] *************************************************************************************************************
ok: [csr1] => {
    "msg": "ios"
}
ok: [csr2] => {
    "msg": "ios"
}
ok: [csr3] => {
    "msg": "ios"
}
ok: [eos-leaf2] => {
    "msg": "eos"
}
ok: [eos-leaf1] => {
    "msg": "eos"
}
ok: [eos-spine1] => {
    "msg": "eos"
}
ok: [eos-spine2] => {
    "msg": "eos"
}
ok: [nxos-spine2] => {
    "msg": "nxos"
}
ok: [vmx1] => {
    "msg": "junos"
}
ok: [nxos-spine1] => {
    "msg": "nxos"
}
ok: [vmx2] => {
    "msg": "junos"
}
ok: [vmx3] => {
    "msg": "junos"
}

PLAY RECAP ******************************************************************************************************************************
csr1                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
csr2                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
csr3                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-leaf1                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-leaf2                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-spine1                 : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-spine2                 : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
nxos-spine1                : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
nxos-spine2                : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vmx1                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vmx2                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vmx3                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

The playbook execution ran the host in alphabetical order. This could be useful for simple human readability or potentially useful if your companies hostname standard can be used to interpret certain device roles.

Throttle

Lastly, I’ll cover the throttle keyword. I’ve used throttle quite a bit in network automation when working with network orchestrators or other REST APIs that use rate limiting. If I have a large list of hosts that I need to gather information on from an API and that API rate limits the number of connections per minute, I can setup throttle to slow down the execution of that task within the playbook.

The playbook:

---
- name: "DEBUG INVENTORY"
  hosts: "all"
  gather_facts: False
  connection: "network_cli"
  tasks:
    - name: "10010: GET DEVICE NAMES"
      debug:
        msg: "{{ inventory_hostname }}"
    - name: "10015: GET DEVICE OS"
      throttle: 1
      debug:
        msg: "{{ ansible_network_os }}"
...

For this one, unfortunately, showing the output of the file doesn’t demonstrate the value. The way this playbook would execute would be based on the default forks of 5 on task 10010, the next task 10015 being run one host at a time (ignoring the forks).

In order to help demonstrate the time it took, I will quickly use callback to get the timer.

In my ansible.cfg under the [defaults] I will add:

callback_whitelist = profile_tasks

The result of the playbook now shows the timer.

▶ ansible-playbook -i inventory.cfg pb.yml

PLAY [DEBUG INVENTORY] ******************************************************************************************************************

TASK [10010: GET DEVICE NAMES] **********************************************************************************************************
Thursday 29 April 2021  11:20:05 -0500 (0:00:00.017)       0:00:00.017 ******** 
ok: [vmx1] => {
    "msg": "vmx1"
}
ok: [nxos-spine1] => {
    "msg": "nxos-spine1"
}
ok: [nxos-spine2] => {
    "msg": "nxos-spine2"
}
ok: [vmx3] => {
    "msg": "vmx3"
}
ok: [vmx2] => {
    "msg": "vmx2"
}
ok: [csr1] => {
    "msg": "csr1"
}
ok: [csr2] => {
    "msg": "csr2"
}
ok: [csr3] => {
    "msg": "csr3"
}
ok: [eos-leaf1] => {
    "msg": "eos-leaf1"
}
ok: [eos-leaf2] => {
    "msg": "eos-leaf2"
}
ok: [eos-spine1] => {
    "msg": "eos-spine1"
}
ok: [eos-spine2] => {
    "msg": "eos-spine2"
}

TASK [10015: GET DEVICE OS] *************************************************************************************************************
Thursday 29 April 2021  11:20:09 -0500 (0:00:03.836)       0:00:03.853 ******** 
ok: [nxos-spine1] => {
    "msg": "nxos"
}
ok: [nxos-spine2] => {
    "msg": "nxos"
}
ok: [vmx1] => {
    "msg": "junos"
}
ok: [vmx2] => {
    "msg": "junos"
}
ok: [vmx3] => {
    "msg": "junos"
}
ok: [csr1] => {
    "msg": "ios"
}
ok: [csr2] => {
    "msg": "ios"
}
ok: [csr3] => {
    "msg": "ios"
}
ok: [eos-leaf1] => {
    "msg": "eos"
}
ok: [eos-leaf2] => {
    "msg": "eos"
}
ok: [eos-spine1] => {
    "msg": "eos"
}
ok: [eos-spine2] => {
    "msg": "eos"
}

PLAY RECAP ******************************************************************************************************************************
csr1                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
csr2                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
csr3                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-leaf1                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-leaf2                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-spine1                 : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
eos-spine2                 : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
nxos-spine1                : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
nxos-spine2                : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vmx1                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vmx2                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
vmx3                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Thursday 29 April 2021  11:20:22 -0500 (0:00:13.460)       0:00:17.314 ******** 
=============================================================================== 
10015: GET DEVICE OS ------------------------------------------------------------------------------------------------------------ 13.46s
10010: GET DEVICE NAMES ---------------------------------------------------------------------------------------------------------- 3.84s

Notice task 10010 took only ~4 seconds, while task 10015 which was run in throttle mode, took drastically longer at ~13 seconds.

I hope this blog post helps to demonstrate the value of strategies and other keyword options that Ansible comes with natively. Each has valid use cases and mixing and matching the options can bring additional stability and flexibility to a playbook. Every user has different concerns with making changes to an environment, but utilizing these options can help ease the natural anxiety that comes with making changes across an entire infrastructure.

-Jeff

Nautobot 1.1.0 Key Feature Overview

2021-07-22T00:00:00+00:00

Nautobot 1.1.0 is upon us! This release comes with a group of features that will offer time savings, and a customized experience for users. This post will introduce some of those features.

Additionally, be sure to check out the Nautobot 1.1.0 Key Feature Overview YouTube video, which also covers many of these features in more detail.

Computed Fields

Computed fields allow users to create custom read-only fields from data already in the database. This new feature allows the user to specify the content type (object type) to apply the computed field to (Interface, Site, Device, etc.) and then define a Jinja2 template to create the computed field for the objects.

As soon as the computed field is defined, it will immediately apply to the specified content type. This example shows a computed field for Interface objects:

Custom Jinja2 Templates

For more complex logic schemes, users can now define a custom Jinja2 filter for computed fields and custom links. Here is an example of a custom template with non-trivial logic:

Nautobot’s documentation on including Jinja2 filters has more information on the custom Jinja2 templates and how to implement them.

Computed Fields API

The computed fields feature also comes with an API enhancement, allowing the user to retrieve computed fields programmatically using the include query parameter:

http://127.0.0.1:8000/api/dcim/interfaces/?name=Ethernet1%2F1&device=ams01-edge-01&include=computed_fields

Here is a snip of the returned computed_fields output:

"computed_fields": {
    "connection-description": "Ethernet1/1.ams01-edge-01---Ethernet1/1.ams01-edge-02",
}, 

Computed field data can also be retrieved via GraphQL queries using cpf_<computed_field_slug>:

query {
  devices(name:["ams-edge-01"]) {
    interfaces {
      cpf_connection_description
    }  
  }
}

Please see the Nautobot documentation on computed fields for more information.

Config Context Schemas

Nautobot’s config context is an existing feature that allows arbitrary data to be stored in Nautobot. At scale, however, it can be helpful and necessary to apply constraints on the data structure to ensure consistency and avoid data entry errors.

In Nautobot 1.1.0, config context JSON schemas can provide such constraints.

In the example below, the config context schema applies the following constraints:

List of items
Min items = 2
Max items = 2
String type
IPv4 format

Now, when we define a new config context, we can specify the config context schema we just created. Nautobot will not allow the config context to be created if the data violates the applied schema:

Once the config context conforms to the schema, it can be created.

Check out the Nautobot documentation on config context schemas for more info.

Saved GraphQL Queries

Nautobot 1.1.0 allows users to save GraphQL queries. This will save time and effort by eliminating the need to constantly re-create frequently-used GraphQL queries.

Users can craft the query in Nautobot’s GraphiQL interface. Once the query is properly tuned, creating a saved query is as easy as pasting the query into the Add a new GraphQL query form.

Once saved, each query will have its own main page:

From the query’s main page, users can:

Edit the query
Execute the query
Open the query in Nautobot’s GraphiQL interface
Clone the query
Delete the query

Executing Saved Queries Programmatically

To execute a stored query via the REST API, a POST request can be sent to this endpoint:

/api/extras/graphql-queries/[slug]/run/

TIP: the slug is available on the query’s main page

Visit the Nautobot documentation on saved GraphQL queries for details.

Read-Only Jobs

Jobs can now be flagged as explicitly read-only to disallow committing of changes to the database. Developers can set the new read_only Meta class attribute to True, making the Job read-only.

Such Jobs have a “Read-only” badge and do not have the Commit changes checkbox, since the Job will hard-coded to prevent any changes.

Read-only Jobs also eliminate log messages about database rollbacks, which will remove confusion for users.

Users may also feel more confident about executing read-only Jobs because there is no chance of changing any data.

Reference Nautobot’s documentation on Jobs for more info on the read_only attribute and read-only Jobs.

In Nautobot 1.1.0, plugin authors can add tabs, groups, items, and buttons in the top navigation menu. This allows developers to customize the Nautobot navigation menu for their specific users.

In the example below, the Nautobot ChatOps NavMenuGroup gets promoted to a NavMenuTab named ChatOps:

Find more info on this capability in Nautobot’s navigation menu development guide.

Under the Covers

There are a couple of features to call out that will significantly expand the options and capabilities of a Nautobot deployment.

Background Tasks Use Celery

Celery replaces RQ for background test execution. RQ is deprecated starting in Nautobot 1.1.0.

RQ support for custom tasks is still supported to allow users to migrate to Celery or temporarily run Celery and RQ concurrently while custom tasks/plugins that use the RQ worker are migrated.

Support for MySQL

Nautobot now fully supports MySQL 8.x as a back-end database. PostgreSQL is also still fully supported. The user can choose which database option to run during installation.

A direct migration from PostgreSQL to MySQL is not supported. The transition requires a fresh start.

Please reach out to us at our #nautobot channel in our Network to Code public Slack workspace if you have any questions about these features!

Thank you, and have a great day!

-Tim

Using Jinja2 Macros as Template Functions

2021-07-20T00:00:00+00:00

Jinja2 is a popular text templating engine for Python that is also used heavily with Ansible. Many network engineers are familiar with it and with leveraging Jinja2 templates for device configurations. But I’ve found one feature that’s under-utilized by network engineers: Jinja2’s macro support. Macros within Jinja2 are essentially used like functions with Python and are great for repeating configs with identical structures (think interface configurations, ASA object-groups, etc).

Real-World Situation

Let’s start with a basic real-world configuration example. I’ll start with the following IOS interface configuration for a switch:

interface FastEthernet0/1
  switchport mode access
  switchport access vlan 10
interface FastEthernet0/2
  switchport mode trunk
  switchport trunk native vlan 20

I could repeat this for multiple interfaces, but two interfaces will be enough to demonstrate. If I wanted to write an Ansible playbook to generate this configuration, using structured data YAML files for holding the configuration, one way I could build the YAML file interfaces.yml would be like this:

---
all_interfaces:
  - name: FastEthernet0/1
    vlan: 10
    mode: "access"
  - name: FastEthernet0/2
    vlan: 20
    mode: "trunk"

I could then create the Jinja2 template file switch01.j2 like this:

#jinja2: lstrip_blocks: True
{% for interface in all_interfaces %}
    interface {{ interface.name }}
      switchport mode {{ interface.mode }}
    {% if interface.mode == 'trunk' %}
      switchport trunk native vlan {{ interface.vlan }}
    {% elif interface.mode == 'access' %}
      switchport access vlan {{ interface.vlan }}
    {% endif %}
{% endfor %}

When generating the configuration text using the above template, I would get the expected output as shown above in the example IOS switch config. This is all well and good, but what happens if I want to configure a second switch?

I don’t want to copy the switch01.j2 file and rename it to switch02.j2. I also don’t want to simply reuse that file. What happens if switch01 gets replaced with an NX-OS switch, or even a switch from another vendor? I would run into issues quickly as my network expanded. This presents the use case for Jinja2 macros.

Jinja2 Macros Example

To create the interfaces Jinja2 template above and convert it into a macro, I can wrap all of the code with {% macro %}{% endmacro %}. I can then use that code as a function, import it into other templates, and even pass variables into it (just like I would with a regular Python function). For example, I will first create the template all_interfaces_template.j2:

{% macro l2_interfaces(interfaces) %}
{% for interface in interfaces %}
    interface {{ interface.name }}
      switchport mode {{ interface.mode }}
    {% if interface.mode == 'trunk' %}
      switchport trunk native vlan {{ interface.vlan }}
    {% elif interface.mode == 'access' %}
      switchport access vlan {{ interface.vlan }}
    {% endif %}
{% endfor %}
{% endmacro %}

Line 1 is {% macro l2_interfaces(interfaces) %}. The first part of the tag is macro, which simply defines this tag as a tag type of macro. The next part l2_interfaces() is used to define the name of the function. Lastly, interfaces is the name of the variable I want to pass into this macro when calling it.

If I were to write this out in Python, it would look like:

def l2_interfaces(interfaces):
    for interface in interfaces:
        print(f"interface {interface['name']}")
        if interface["mode"] == "trunk":
            print(f"  switchport trunk native vlan {interface['vlan']}")
        elif interface["mode"] == "access":
            print(f"  switchport access vlan {interface['vlan']}")

Now I have a macro I can use in other Jinja2 templates. Using the same interfaces.yml YAML file from before where the interfaces are defined, I will first create a template for switch01, with filename switch01.j2:

#jinja2: lstrip_blocks: True
{% from 'interfaces_template.j2' import l2_interfaces %}
{{ l2_interfaces(all_interfaces) }}

And that’s it! Both interfaces will be properly generated as expected into the following config (shown at the top of this blog post):

interface FastEthernet0/1
  switchport mode access
  switchport access vlan 10
interface FastEthernet0/2
  switchport mode trunk
  switchport trunk native vlan 20

There are only three lines in the template file switch01.j2, but I want to explain them in detail.

Line 1

The first line is #jinja2: lstrip_blocks: True. This helps to control extra whitespace generated by Jinja2 from the tags. More info on controlling whitespace can be found on a previous NTC blog post.

Line 2

The second line is {% from 'interfaces_template.j2' import l2_interfaces %}. You’ll notice there is no closing tag for it, like {% endfrom %}, as Jinja2 does not require closing this tag.
The part 'interfaces_template.j2' references which file to import from. If it were nested in a folder called “switches/”, it would be imported with switches/interfaces_template.j2.
The last part, import l2_interfaces, tells Jinja2 the name of the macro to import.

Line 3

The last line is {{ l2_interfaces(all_interfaces) }}, which calls the function I just imported on the line above and passes in the variable all_interfaces, which is obtained from the YAML file interfaces.yml.

Now I can easily create a template for a second switch, and expand the configuration some more, with Jinja2 template switch02.j2:

{% from 'interfaces_template.j2' import l2_interfaces %}
hostname Switch02
ip domain-name networktocode.com
!
{{ l2_interfaces(all_interfaces) }}

Which generates:

hostname Switch02
ip domain-name networktocode.com
!
interface FastEthernet0/1
  switchport mode access
  switchport access vlan 10
interface FastEthernet0/2
  switchport mode trunk
  switchport trunk native vlan 20

If I wanted to add interfaces, all I would have to do is add them to interfaces.yml, then run my Jinja2 template again! While this example is good for demonstration purposes, I would ideally expand it out further in a real-world environment, to allow different interface information on a per-switch basis.

Importing Functions, Abbreviated

I can rename the function “l2_interfaces” to “l2i” when imported, just like I can when importing in Python, by adding as l2i to the end. For example: {% from 'interfaces_template.j2' import l2_interfaces as l2i %}. This can be handy when importing macros with long names.

I would then use the function like so:

{% from 'interfaces_template.j2' import l2_interfaces as l2i %}
{{ l2i(all_interfaces) }}

Wrapping It Up

I hope you’ve found this topic helpful, and possibly learned something entirely new about Jinja2! Feel free to comment below, or reach out to us at Network to Code on our public Slack at slack.networktocode.com and ask around.

-Matt

Nautobot Relationships

2021-07-15T00:00:00+00:00

The native data model in Nautobot will suffice for the majority of use cases. However, some deployments have additional requirements between objects based on the design of their networks. Nautobot now provides the capability to define new relationships to support a more customized network data model.

For example, a VLAN may need to be defined on a per-device basis, rather than the native model of a VLAN per site. This would allow a user to define and model VLANs for a network device that are locally significant.

This series of posts will outline the new user-defined relationship feature in Nautobot and explore some use cases where this feature can be applied to help model a network design.

Relationships Primer

One of the columns in a database table will store a primary key, which will uniquely identify each object. This field is used in the object-relational mapping (ORM) when defining relationships between objects. This simply allows objects to build associations with each other.

A common example used to explain this concept uses books. A publisher has a relationship with a book as it can publish a book. Each book will have a publisher.

Additional constraints can be introduced to be more specific in the relationship definition. The next sections outline the different types of relationships that can exist between objects.

One-to-Many

A one-to-many type introduces a constraint in that a book can have only one publisher, denoted with the 1 on line beside the publisher. A publisher can have many books, denoted with N for number on the many side.

Many-to-Many

A many-to-many relationship has no constraints. A customer can buy many books and a book can be bought by many customers.

One-to-One

A one-to-one relationship has a unique constraint on both sides of the relationship. For example, an order can have only one payment and a payment can be made on one order.

Nautobot Model

Nautobot is designed as a Source of Truth application aimed at modeling modern networks. Its core data models allow users to define the intended state of a network. This is achieved by providing a native data model, with its inherent relationships between the objects.

As an example, a subset of the Nautobot core data model illustrates the relationships between some of the objects in the nautobot.ipam and nautobot.circuit applications. The database tables are represented as boxes including their fields. The lines between the tables denote an association or relationship between the tables. The diagram shows how an IPAddress can have a VRF, a Prefix can have a VRF and a VLAN, and finally a Circuit can have a Provider. At a database level, we can see that these relationships are represented as foreign keys between the database tables.

INFO: This diagram was rendered using the graph models feature in the django_extensions package.

The core data model is quite established, based on years of iterative open-source development that began with the NetBox project. However, it’s very difficult to provide a standard framework that can meet the requirements of all networks. There will always be components of the network that will be different. One of the key objectives of Nautobot is to provide maximum data model flexibility. This is where the new Relationships feature can be applied. It allows users to define their own relationships between the objects in the system.

Use Case 1: IPAddress - Circuit

The first use case will focus on modeling an IPv4-based circuit, such as an enterprise router connection to an Internet Service Provider WAN. Each circuit can have two IP addresses, one at each end of the circuit. This type of relationship is not defined in the native data model. To enable it, a new relationship (highlighted in red) can be created between an IPAddress and a Circuit.

New Relationship: IPAddress - Circuit

To create a new relationship, navigate to Extensibility -> Relationships on the Nautobot web user interface. The use case in question supports one circuit having two IPs. Any model that can have more than one object in the association is defined as a many in the relationship. Enter a custom name to identify the new relationship and set the relationship type as One to Many.

The Source type is circuit | circuit as this is the one side of the relationship and it is defined as the source. The Source Label will be used to display the other end of the relationship, on the source page. For this use case the Source Label is IP Address. The Destination fields are the reverse of the Source fields. Both source and destination filters can be left blank.

Update Circuit

Once a relationship has been created, it will be visible when configuring one of the model objects. A new Relationship panel will be available to select an existing object of the defined source/destination type.

In the example, circuit A123456789 has two IP addresses associated with it. Multiple IPs can be can be selected since IPAddress is on the many side of the new relationship.

Update IP Address

For the IP addresses, one circuit can be selected in each IP address object. The selection drop-down list will only allow one object to be configured, as Circuit is on the one side of the relationship.

Defining a new relationship allows network design use cases to be managed through the web user interface by creating new associations between existing objects. This reduces the need to manage additional layers of data by adding custom fields to objects and maintaining the custom field data synchronization through scripts or playbooks. Relationships can also be defined using the REST API.

An alternative solution for this use case might involve creating a one-to-one relationship between an IP Address and Circuit. In this case, a circuit would be associated with one IP Address. Creating two of these associations would model the circuit end-to-end.

Use Case 2: VLAN - VLAN Group

Another, perhaps more advanced, use case would be to create a relationship between models that already have a native relationship. For example, a VLAN group and VLAN have an inherent One to Many relationship, whereby a VLAN Group can have many VLANs but a VLAN can only be a member of only one VLAN group. Some networks may have a need to have VLANs in multiple VLAN groups. VLANs could be members of a management group for a site, while also being part of a group that requires access to cloud-based Operational Support Systems (OSS). This use case could be managed with custom tags or custom fields, but relationships provide a way to overlay a new association between the models and link existing objects together.

The diagram below illustrates how the new relationship (highlighted in red) would be modeled, alongside the existing relationship.

New Relationship: VLAN - VLAN Group

A Many to Many relationship is required to support this use case. Following on from the guidelines in the previous use case, with the addition of a filter to restrict the relationship to VLANs with the role of leaf, filtering is defined using JSON data format to identify the model fields and their values.

In the example, the new relationship will be applied only on VLANs that have a role of leaf. Roles can be applied to VLANs to assist with network design modeling e.g., define leaf-switch VLANs.

For demonstration purposes, a short list of VLANs shows some Leaf and Core roles to illustrate the filtering capability.

Update VLAN

Editing a VLAN allows multiple VLAN groups to be associated with a single VLAN through the newly defined relationship. A key point to note is that the new relationship is managed through the Relationships panel and not through the native Group field, which is left blank.

Update VLAN Group

For VLAN group(s) configured in the relationship, apply the new VLANs. Due to the filtering applied on the VLAN side of the relationship, only VLANs with a role of Leaf can be selected.

Management of the new VLANs is through the Relationships panel on the VLAN Group page. For this advanced use case, the native VLAN view (right-hand side) on the VLAN group page is not populated.

Selecting the VLANs hyperlink displays all of the VLANs through the relationship-association table.

Conclusion

Defining new relationships is a very powerful feature in Nautobot. It supports flexible data modeling to assist with specific use cases of network design representation in a Source of Truth application.

This guide focused on how to define the relationships in the web user interface. In a future post, we’ll look at using REST and GraphQL based APIs to interact with the new relationships in Nautobot.

-Paddy

Resources

Learning Salt with Ansible References

2021-07-13T00:00:00+00:00

Have you seen or taken the 2020 NetDevOps survey? If not, don’t worry, it is held yearly from Sept 30th to Oct 31st, so you will get your chance. To quickly recap, the survey’s intention is to understand which tools are most commonly used by Network Operators and Engineers to automate their day-to-day jobs. It is a very interesting survey, and I encourage you to read it.

If we take a look at this data, particularly the questions about tools used in automation, we will see that Ansible is the most popular one. At the same time, there are other tools that are used, but not so much—like SaltStack (aka Salt). If I would speculate why SaltStack is not so popular among Network Engineers, I would say that’s because Salt is more friendly with managing the servers and not so much the networking devices. So I thought, if I write a blog about Salt with references to Ansible it might encourage engineers to learn this tool a bit more.

NOTE
I will do my best to address situations where features available to Salt do not have direct counterparts in Ansible.

Salt Terminologies

Before you start using Salt, you need to get familiar with its terminologies. The below table summarizes the ones that I will use in this blog:

SaltStack Name	Description	Ansible Name	Description
Master	Server that commands-and-controls the minions.	Server	The machine where Ansible is installed.
Minion	Managed end-hosts where the commands are executed.	Hosts	The devices that are managed by the Ansible server.
Proxy Minion	End-hosts that cannot run the `salt-minion` service and can be controlled using proxy-minion setup. *These hosts are mostly networking or IoT devices.*	N/A	N/A
Pillars	Information (e.g., username, passwords, configuration parameters, variables, etc.) that needs to be defined and associated with one or more minion(s) in order to interact with or distribute to the end-host.	Inventory and Static variables	The Ansible Inventory and variables that need to be associated with the managed host(s).
Grains	Information (e.g., IP address, OS type, memory utilization etc.) that is obtained from the managed minion. Grains also can be statically assigned to the minions.	Fact	Information obtained from the host with the `gather_facts` or similar task operation.
Modules	Commands that are executed locally on the master or on the minion.	Modules	Commands that are executed locally on the server or on the end-host.
SLS Files	(S)a(l)t (S)tate files, also known as formulas, are the representation of the state in which the end-hosts should be. SLS files are mainly written in YAML, but other languages are also supported.	Playbook/Tasks	The file where one or more tasks that need to be executed on the end-host are defined.

Salt and Ansible Playground

Let’s take a look at my playground. These are the hosts and related information involved in this blog:

Salt

Role: Hosts Salt’s master and proxy-minion configuration.
OS: Ubuntu 18.04.3
Salt: Version 3003.1
IP: 172.29.224.100

Ansible

Role: Hosts Ansible execution environment
OS: Ubuntu 18.04.3
Ansible: Version 2.10.7
IP: 172.29.224.99

IOS-XE

Role: End host for manipulation
OS: 16.12.3
IP: 172.29.224.101

NX-OS

Role: End host for manipulation
OS: 7.0.9
IP: 172.29.224.102

Junos

Role: End host for manipulation
OS: 21.2R1.10
IP: 172.29.224.103

Salt and Ansible Installation

The Salt installation can vary based on the OS that you are using, so I would suggest that you read the installation instructions that match your OS. At the end, you should have salt-master and salt-minion installed on your machine.

The Ansible installation is a bit more straightforward; use the pip install ansible==2.10.7 command to install the same version that I use.

Salt Environment Configuration

The time has come to configure the Salt environment. Confirm that salt-master service is active. If not, start it with sudo salt-master -d command.

$ service salt-master status
● salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-07-06 23:29:24 UTC; 7min ago
>>>
Trimmed for brevity
<<<

Using your favorite text editor, open the /etc/salt/master configuration file and append the below text to it:

file_roots: # Allows minions and proxy-minions to access SLS files with salt:// prefix
  base: # Name of the environment
    - /opt/salt/base/root/ # Full path where state and related files will be stored

pillar_roots: # Instructs master to look for pillar files at the specified location(s)
  base: # Name of the environment
    - /opt/salt/base/pillar # Full path where pillar files will be stored

Next, configure the proxy file in the /etc/salt/master directory and point it to the Salt master’s IP or FQDN, like so:

NOTE
In my setup, I am running the proxy service on the same machine where the salt-master is configured, so I will use the localhost.

$ cat /etc/salt/proxy
master: localhost

Save the changes and restart the salt-master service using sudo service salt-master restart command. After the restart, you will need to create necessary folders. For that, use the sudo mkdir -p /opt/salt/base/{root,pillar} command.

$ sudo mkdir -p /opt/salt/base/{root,pillar}
$ ll /opt/salt/base/
total 16
drwxr-xr-x 4 root root 4096 Jul  7 00:17 ./
drwxr-xr-x 3 root root 4096 Jul  7 00:17 ../
drwxr-xr-x 2 root root 4096 Jul  7 00:17 pillar/
drwxr-xr-x 2 root root 4096 Jul  7 00:17 root/

Great, the salt-master is ready. Now you can configure the pillar files for the proxy-minion service. This will contain instructions on how to communicate with the networking devices. Create junos-pillar.sls file in the /opt/salt/base/pillar/ folder and populate with the following data:

NOTE
Some vendors, such as Juniper and Cisco, have their own proxy types. Cisco currently has proxy module only for NXOS platform. For IOS, you would need to use either netmiko or NAPALM. To avoid complications, we will stick with netmiko for all three platforms.

proxy:
  proxytype: netmiko # Proxy type
  device_type: juniper_junos # A setting required by the netmiko
  ip: 172.29.224.103 # IP address of the host
  username: admin
  password: admin123

Next, create the nxos-pillar.sls file in the same folder and populate with following data:

proxy:
  proxytype: netmiko
  device_type: cisco_nxos
  ip: 172.29.224.102
  username: admin
  password: admin123

Finally create the ios-pillar.sls:

proxy:
  proxytype: netmiko
  device_type: cisco_xe
  ip: 172.29.224.101
  username: admin
  password: admin123

NOTE
You can create other pillar files for the same devices and use different proxytypes. For Juniper devices you can use junos proxy and for Cisco NXOS devices you can use nxos or nxos_api proxies.

To compare with Ansible, this is similar to the inventory file where you need to define the ansible_host, ansible_user, ansible_password and ansible_network_os variables.

[base]
IOS-XE ansible_host=172.29.224.101 ansible_user=admin ansible_password=admin123 ansible_network_os=ios ansible_connection=network_cli
NX-OS ansible_host=172.29.224.102 ansible_user=admin ansible_password=admin123 ansible_network_os=nxos ansible_connection=network_cli
Junos ansible_host=172.29.224.103 ansible_user=admin ansible_password=admin123 ansible_network_os=junos ansible_connection=netconf

Okay, let’s take a quick break to understand how these files were constructed. The most important one is the proxy key. Salt will look for this key, and the information within it, to set up the the proxy-minion service. The rest of the key value pairs were defined based on the official documentation that can be found here.

Now you need to create the pillar top.sls file (main file) where each pillar file will be associated with a respective proxy-minion name. The top.sls file should be located in the /opt/salt/base/pillar/ folder and should contain the following data:

base: # The name of the environment
  'Junos': # The name of the proxy-minion. Can be any name, but it is a good idea to put the device's hostname.
    - junos-pillar # This is the name of the previously created pillar file. Note the `.sls` suffix is not specified.
  'NX-OS':
    - nxos-pillar
  'IOS-XE':
    - ios-pillar

It is time to start the proxy-minion service for each host. For that, use the following commands:

NOTE
The value passed to the --proxyid argument should match the name of the proxy-minion defined in the top.sls file.

$ sudo salt-proxy --proxyid=Junos -d
$ sudo salt-proxy --proxyid=NX-OS -d
$ sudo salt-proxy --proxyid=IOS-XE -d

Finally, if everything was done properly, you should see three unaccepted keys after you issue the sudo salt-key --list-all command. These are the requests that the proxy services made to register the proxy-minions against the master node.

$ sudo salt-key --list-all
Accepted Keys:
Denied Keys:
Unaccepted Keys:
IOS-XE
Junos
NX-OS
Rejected Keys:

To accept all keys at once, issue sudo salt-key --accept-all command. When prompted press y.

$ sudo salt-key --accept-all
The following keys are going to be accepted:
Unaccepted Keys:
IOS-XE
Junos
NX-OS
Proceed? [n/Y] y
Key for minion IOS-XE accepted.
Key for minion Junos accepted.
Key for minion NX-OS accepted.

SaltStack and Ansible Ad Hoc Commands

You made it to the fun part of the blog. Here you will start executing some ad hoc commands against the networking devices. To start, you will need to install some Python packages on both Salt and Ansible hosts. Using sudo pip install netmiko junos-eznc jxmlease yamlordereddictloader napalm command install the packages.

$ sudo pip install netmiko junos-eznc jxmlease yamlordereddictloader napalm
Collecting napalm
  Downloading napalm-3.3.1-py2.py3-none-any.whl (256 kB)
     |████████████████████████████████| 256 kB 91 kB/s 
Collecting junos-eznc
  Downloading junos_eznc-2.6.1-py2.py3-none-any.whl (195 kB)
     |████████████████████████████████| 195 kB 113 kB/s 
>>>
Trimmed for brevity
<<<

Now you can run sudo salt "*" test.ping command on the salt-master

$ sudo salt "*" test.ping
IOS-XE:
    True
Junos:
    True
NX-OS:
    True

The Ansible counterpart command will be: ansible -i inv.ini base -m ping.

Now let’s take a look at the grains that salt master gathers from the devices. The sudo salt "*" grains.items command will reveal the below information. Similarly, you can do the same thing from the Ansible host using ansible -m ios_facts -i inv.ini IOS-XE, ansible -m nxos_facts -i inv.ini NX-OS, and ansible -m junos_facts -i inv.ini Junos commands.

NOTE
Unfortunately, netmiko does not pull grain information from the devices, but hopefully in the future this gets fixed.

Junos:
    ----------
    cpuarch:
        x86_64
    cwd:
        /
    dns:
        ----------
        domain:
        ip4_nameservers:
            - 192.168.65.5
        ip6_nameservers:
        nameservers:
            - 192.168.65.5
        options:
        search:
        sortlist:
    fqdns:
    gpus:
    hwaddr_interfaces:
        ----------
    id:
        Junos
<<<
TRIMMED
>>>
NX-OS:
    ----------
    cpuarch:
        x86_64
    cwd:
        /
    dns:
        ----------
        domain:
        ip4_nameservers:
            - 192.168.65.5
        ip6_nameservers:
        nameservers:
            - 192.168.65.5
        options:
        search:
        sortlist:
    fqdns:
    gpus:
    hwaddr_interfaces:
        ----------
    id:
        NX-OS
<<<
TRIMMED
>>>
IOS-XE:
    ----------
    cpuarch:
        x86_64
    cwd:
        /
    dns:
        ----------
        domain:
        ip4_nameservers:
            - 192.168.65.5
        ip6_nameservers:
        nameservers:
            - 192.168.65.5
        options:
        search:
        sortlist:
    fqdns:
    gpus:
    hwaddr_interfaces:
        ----------
    id:
        IOS-XE
<<<
TRIMMED
>>>

Below data were gathered to demonstrate what grains will look like if different proxytypes were used:

$ sudo salt "IOS-XE" napalm_net.facts
IOS-XE:
    ----------
    comment:
    out:
        ----------
        fqdn:
            IOS-XE.local.host
        hostname:
            IOS-XE
        interface_list:
            - GigabitEthernet1
            - GigabitEthernet2
            - GigabitEthernet3
            - GigabitEthernet4
        model:
            CSR1000V
        os_version:
            Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.12.3, RELEASE SOFTWARE (fc5)
        serial_number:
            ABC123ABC123
        uptime:
            5820
        vendor:
            Cisco
    result:
        True
$ sudo salt "NX-OS" nxos.grains
NX-OS:
    ----------
    hardware:
        ----------
        Device name:
            NX-OS
        bootflash:
            4287040 kB
    plugins:
        - Core Plugin
        - Ethernet Plugin
    software:
        ----------
        BIOS:
            version 
        BIOS compile time:

        NXOS compile time:
            12/22/2019 2:00:00 [12/22/2019 14:00:37]
        NXOS image file is:
            bootflash:///nxos.9.3.3.bin
$ sudo salt "Junos" junos.facts
Junos:
    ----------
    facts:
        ----------
        2RE:
            False
        HOME:
            /var/home/admin
        RE0:
            ----------
            last_reboot_reason:
                Router rebooted after a normal shutdown.
            mastership_state:
                master
            model:
                VSRX RE
            status:
                Testing
            up_time:
                9 minutes, 20 seconds
        RE1:
            None
        RE_hw_mi:
            False
        current_re:
            - master
            - fpc0
            - node
            - fwdd
            - member
            - pfem
            - re0
            - fpc0.pic0
        domain:
            None
        fqdn:
            Junos
        hostname:
            Junos
>>>
Trimmed for brevity
<<<

You have probably noticed that for the example data, I used different modules like junos.facts or nxos.grains. So why can’t we use the vendor-specific modules? This is due to the proxytype assigned to the device. Even though the salt "Junos" junos.facts is a correct command, Salt will check whether it is associated with proxytype: netmiko and, if not, it will report that it is not available for use.

To understand the syntax of the salt shell command, I would refer to the image below. The salt allows for the commands to be executed on remote systems (minions) in parallel. The documentation for all supported salt modules can be found here.

Here are some more commands for you to try:

sudo salt "Junos" netmiko.send_command "show version"
sudo salt --out json "IOS-XE" netmiko.send_command "show ip interface brief" use_textfsm=True
sudo salt "NX-OS" netmiko.send_config config_commands="['hostname NX-OS-SALT']"

SaltStack State Files and Ansible Playbooks

Ad hoc commands are very useful for checking or obtaining information from multiple devices, but they are not practical because they can’t act upon that information the way SaltState files (formulas) can. Similarly, the Ansible Playbooks are used.

You can start by creating a new pillar file, which should contain NTP server information. The file should be placed in the /opt/salt/base/pillar/ folder, named as ntp_servers.sls and contain the following data:

ntp_servers:
  - 172.29.224.1

Next, the top.sls file needs to be updated with instructions permitting all proxy-minions to use the information contained in the ntp_servers.sls file.

$ cat /opt/salt/base/pillar/top.sls
base: # Environment name
  'Junos':    # proxy-minion name
    - junos-pillar    # pillar file
  'IOS-XE':
    - ios-pillar
  'NX-OS':
    - nxos-pillar
  '*': # All hosts will match.
    - ntp_servers

Now you need to refresh the pillar information by using sudo salt "*" saltutil.refresh_pillar command.

IOS-XE:
    True
Junos:
    True
NX-OS:
    True

Confirm that information is available to all three hosts with sudo salt "*" pillar.data or sudo salt "*" pillar.items commands:

$ sudo salt "*" pillar.data
IOS-XE:
    ----------
    ntp_servers:
        - 172.29.224.1
    proxy:
        ----------
        driver:
            ios
        host:
            172.29.224.101
        password:
            admin123
        provider:
            napalm_base
        proxytype:
            napalm
        username:
            admin
NX-OS:
    ----------
    ntp_servers:
        - 172.29.224.1
    proxy:
        ----------
        connection:
            ssh
        host:
            172.29.224.102
        key_accept:
            True
        password:
            admin123
        prompt_name:
            NX-OS
        proxytype:
            nxos
        username:
            admin
Junos:
    ----------
    ntp_servers:
        - 172.29.224.1
    proxy:
        ----------
        host:
            172.29.224.103
        password:
            admin123
        port:
            830
        proxytype:
            junos
        username:
            admin

Next, you need to create a Jinja template for NTP configuration. The IOS-XE and NX-OS devices share the same configuration syntax, so all you need to do is to check the ID value stored in grains and identify the Junos device and, based on that, construct proper configuration syntax for it. Name the file as ntp_config.set and store it in the /opt/salt/base/root/ folder. The contents of the file should look like this:

{% set ntp = pillar['ntp_servers'][0] %}
{% set device_type = grains['id'] %}
{% if device_type == "Junos" %}
set system ntp server {{ ntp }}
{% else %}
ntp server {{ ntp }}
{% endif %}

Finally, you need to create the Salt State file (aka formula) and specify what it should do. The file needs to be located in the /opt/salt/base/root/ folder and named as update_ntp.sls with the following contents:

UPDATE NTP CONFIGURATION: # The ID name of the state. Should be unique if multiple IDs are present in the formula.
  module.run: # state function to run module functions
    - name: netmiko.send_config  # Name of the module function
    - config_file: salt://ntp_config.set # Argument that needs to be passed to the module function. The salt:// will convert to `/opt/salt/base/root/` path.

You might ask: “Why did we use module.run to run netmiko.send_config?” That’s a great question! SaltStack separates modules by their purposes, and execution modules cannot be used to maintain a state on the minion. For that, the state modules exist. Since netmiko does not have a state module, the module.run allows you to run execution modules in the Salt State file.

Okay, let’s compare the state file with an Ansible playbook. This is how a playbook that would do the same thing would look:

---

- name: UPDATE NTP CONFIGURATION
  hosts: base
  gather_facts: no
  vars:
    pillar:
      ntp_servers:
        - 172.29.224.1

  tasks:
    - name: 5 - UPDATE NTP CONFIGURATION FOR JUNOS
      junos_config:
        src: ntp_config.set
      vars:
        grains:
          id: Junos
      when: "ansible_network_os == 'junos'"

    - name: 10 - UPDATE NTP CONFIGURATION FOR NX-OS
      nxos_config:
        src: ntp_config.set
      vars:
        grains:
          id: NX-OS
      when: "ansible_network_os == 'nxos'"

    - name: 15 - UPDATE NTP CONFIGURATION FOR IOS
      ios_config:
        src: ntp_config.set
      vars:
        grains:
          id: IOS-XE
      when: "ansible_network_os == 'ios'"

The salt "*" state.apply update_ntp command will apply the configuration:

$ salt "*" state.apply update_ntp
IOS:
----------
          ID: UPDATE NTP CONFIGURATION
    Function: module.run
        Name: netmiko.send_config
      Result: True
     Comment: Module function netmiko.send_config executed
     Started: 02:26:17.021615
    Duration: 858.925 ms
     Changes:
              ----------
              ret:
                  configure terminal
                  Enter configuration commands, one per line.  End with CNTL/Z.
                  IOS-XE(config)#ntp server 172.29.224.1
                  IOS-XE(config)#end
                  IOS-XE#

Summary for IOS
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 858.925 ms
NXOS:
----------
          ID: UPDATE NTP CONFIGURATION
    Function: module.run
        Name: netmiko.send_config
      Result: True
     Comment: Module function netmiko.send_config executed
     Started: 02:26:17.074178
    Duration: 5612.827 ms
     Changes:
              ----------
              ret:
                  configure terminal
                  Enter configuration commands, one per line. End with CNTL/Z.

                  NX-OS(config)# ntp server 172.29.224.1
                  NX-OS(config)# end
                  NX-OS#

Summary for NXOS
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   5.613 s
SRX:
----------
          ID: UPDATE NTP CONFIGURATION
    Function: module.run
        Name: netmiko.send_config
      Result: True
     Comment: Module function netmiko.send_config executed
     Started: 02:26:17.083703
    Duration: 11961.207 ms
     Changes:
              ----------
              ret:
                  configure
                  Entering configuration mode

                  [edit]
                  admin@Junos# set system ntp server 172.29.224.1

                  [edit]
                  admin@Junos# exit configuration-mode
                  Exiting configuration mode

                  admin@Junos>

Summary for SRX
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  11.961 s

SaltStack is a very powerful tool, and covering all of its aspects, such as reactors and syslog engines, here would be impossible. But I hope that the information presented above was enough for you to get started learning it. If you have any questions, feel free to leave a comment below, and I will do my best to answer them!

-Armen

Contributing to the Nautobot Documentation

2021-07-08T00:00:00+00:00

Useful documentation is an important component of Nautobot’s usability. To that point, this week’s blog will focus on how the NTC community can contribute new content to the docs or submit corrections for the docs when they find a flaw.

Contributing to the Nautobot project via code or documentation has many benefits:

It shows leadership
It is a public example showing your commitment to putting in your own time and effort to see that things are done right
The community will appreciate your help in making Nautobot more usable
It saves others time by correcting the docs or by adding helpful new modules to the docs
It makes a nice entry on a resumé (just sayin’ . . .)

Create an Issue

The development docs state this clearly, but it bears repeating here:

Pull requests are entertained only for accepted issues. If an issue you want to work on hasn’t been approved by a maintainer yet, it’s best to avoid risking your time and effort on a change that might not be accepted.

If you are considering submitting an addition to the docs and/or a correction, be sure to open an issue for the change prior to investing the time to create a PR.

You can open an issue here:

Select Bug Report if you have found an error in the docs
Select Feature Request if you would like to expand the docs by adding a section or otherwise add content

In the issue, be sure to describe the edits/adds/deletions you propose and describe why those changes are needed and how they are helpful.

See the Nautobot wiki page on Work Intake & Issue Management for information on what to expect after submitting an issue.

Create Your Fork

Once your issue is accepted, it’s time to set up your dev environment. You will need to set up a fork of the Nautobot repository, if you don’t have one already.

On the main page for the Nautobot repository, look in the top-right corner for the Fork button and click on it. Select which of your accounts you want to place the fork in.

Clone Your Repository

Next, go to your repository page to access the options for how to clone your repository locally. The example below shows selection of the ssh method to retrieve the repository.

Click on the Code button
Select how you want to clone the respository (SSH option shown)

TIP: Instructions to set up SSH keys in your GitHub account, which enables use of the ssh clone method, can be found here.

With your fork ready, go to the CLI (or your favorite IDE) and clone your Nautobot repository fork. The example below shows use of the ssh resource we copied in the picture above.

NOTE: This requires git to be installed on your system

% 
% git clone git@github.com:tim-fiola/nautobot.git
Cloning into 'nautobot'...
remote: Enumerating objects: 65038, done.
remote: Counting objects: 100% (330/330), done.
remote: Compressing objects: 100% (170/170), done.
remote: Total 65038 (delta 217), reused 262 (delta 160), pack-reused 64708
Receiving objects: 100% (65038/65038), 38.94 MiB | 24.31 MiB/s, done.
Resolving deltas: 100% (51959/51959), done.
% 

Once the download is complete, change to the nautobot directory. Do a git remote -v to verify your origin.

% cd nautobot 
% git remote -v                                                          
origin	git@github.com:tim-fiola/nautobot.git (fetch)
origin	git@github.com:tim-fiola/nautobot.git (push)
%                                                                       
%  

You will also want to track the official Nautobot repository, so you can pull in any changes.

To do this, use the git remote add upstream git@github.com:nautobot/nautobot.git command. This will add the Nautobot respository as the remote upstream.

% git remote add upstream git@github.com:nautobot/nautobot.git             
%                                                                            
% git remote -v                                                           
origin	git@github.com:tim-fiola/nautobot.git (fetch)
origin	git@github.com:tim-fiola/nautobot.git (push)
upstream	git@github.com:nautobot/nautobot.git (fetch)
upstream	git@github.com:nautobot/nautobot.git (push)
% 

Switch to the develop branch and then pull from the upstream’s develop branch. This will ensure your local develop branch is up to date and lessen the chances of merge conflicts.

%                                                                      
% git checkout develop                                           
Already on 'develop'
Your branch is up to date with 'origin/develop'.
%                                                           
% git pull upstream develop                                
. . . 
----- < snip > -----
. . . 
 scripts/cibuild.sh                                    |  5 +++++
 40 files changed, 626 insertions(+), 156 deletions(-)
 create mode 100644 examples/okta/README.md
 create mode 100644 examples/okta/group_sync.py
 delete mode 100644 nautobot/project-static/jquery-ui-1.12.1/package.json
%                                                           

Create Your Branch

Now create your local branch where you will perform your edits.

TIP: The command shown below will create a new branch in your local installation. This new branch will need to be pushed to your origin for tracking; this guide will show that step when the time comes.

%                                                            
% git checkout -b tim-fiola-doc-blog                         
Switched to a new branch 'tim-fiola-doc-blog'
%                                                                                                

Verify your branch is clean:

% git status                                                                                                          
On branch tim-fiola-doc-blog
nothing to commit, working tree clean
%    

In our example, we created the new branch locally and so the branch does not exist in our origin yet.

We will use git push --set-upstream origin <local-branch-name> to add the branch to our origin. Our local branch will then track this upstream branch.

% git push --set-upstream origin tim-fiola-doc-blog                              
Enumerating objects: 204, done.
Counting objects: 100% (180/180), done.
Delta compression using up to 12 threads
Compressing objects: 100% (63/63), done.
Writing objects: 100% (120/120), 22.95 KiB | 11.47 MiB/s, done.
Total 120 (delta 91), reused 83 (delta 55), pack-reused 0
remote: Resolving deltas: 100% (91/91), completed with 37 local objects.
remote: 
remote: Create a pull request for 'tim-fiola-doc-blog' on GitHub by visiting:
remote:      https://github.com/tim-fiola/nautobot/pull/new/tim-fiola-doc-blog
remote: 
To github.com:tim-fiola/nautobot.git
 * [new branch]        tim-fiola-doc-blog -> tim-fiola-doc-blog
Branch 'tim-fiola-doc-blog' set up to track remote branch 'tim-fiola-doc-blog' from 'origin'.
%   

Install Poetry

Nautobot uses Poetry, which will transparently create a virtualenv for you, automatically install all dependencies required for Nautobot to operate, and also install the nautobot-server CLI command that you will utilize to interact with Nautobot from here on out. More info on Poetry can be found in the Nautobot docs.

Use the following command to install Poetry: curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -

TIP: you may have to use | python3 - instead of | python - in that curl command, depending on your Python setup.

$ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python3 -
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/home/tim/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

You are installing 1.1.7. When using the current installer, this version does not support updating using the 'self update' command. Please use 1.2.0a1 or later.
Installing Poetry (1.1.7): Done

Poetry (1.1.7) is installed now. Great!

To get started you need Poetry's bin directory (/home/tim/.local/bin) in your `PATH`
environment variable.

Add `export PATH="/home/tim/.local/bin:$PATH"` to your shell configuration file.

Alternatively, you can call Poetry explicitly with `/home/tim/.local/bin/poetry`.

You can test that everything is set up by executing:

`poetry --version`

$

NOTE: As of the writing of this article, Poetry had just switched to a new install script (the one used above). I discovered this when I installed Poetry on my local machine using the now deprecated curl install script (curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -) and saw the deprecation message. More info on this change can be found here, on the Poetry webpage.

Use Poetry to Install Nautobot Dependencies

The repository you cloned has a nautobot directory. Open a new shell, and change to that directory.

Execute poetry install from the CLI to install dependencies.

% cd nautobot  
% poetry install                                          
Installing dependencies from lock file

Package operations: 102 installs, 0 updates, 0 removals

  • Installing six (1.16.0)
  • Installing certifi (2021.5.30)
  • Installing chardet (4.0.0)
. . . 
----- < snip > -----
. . . 
Installing the current project: nautobot (1.0.3-beta.1)
%

Serve the Docs

Your environment is now set up to allow you to edit the documentation. We highly recommend serving the documents locally while you edit. Serving the docs locally allows you to see the fully rendered documents and what your changes will look like.

First, activate Poetry’s virtual environment (venv) by running poetry shell in the nautobot directory:

% pwd                                                                                                                          (tim-fiola-doc-blog)nautobot
/Users/timothyfiola/nautobot_blog/nautobot
% poetry shell                                                                                                                 (tim-fiola-doc-blog)nautobot
Spawning shell within /Users/timothyfiola/Library/Caches/pypoetry/virtualenvs/nautobot-yNEcye0N-py3.8
Restored session: Thu Jul  1 15:39:50 MDT 2021
% . /Users/timothyfiola/Library/Caches/pypoetry/virtualenvs/nautobot-yNEcye0N-py3.8/bin/activate                               (tim-fiola-doc-blog)nautobot
(nautobot-yNEcye0N-py3.8) %   

Now, run the mkdocs serve command from within the virtual environment:

(nautobot-yNEcye0N-py3.9)  % mkdocs serve                              
INFO    -  Building documentation... 
WARNING -  Config value: 'python'. Warning: Unrecognised configuration name: python 
INFO    -  Cleaning site directory 
INFO    -  The following pages exist in the docs directory, but are not included in the "nav" configuration:
  - installation/centos.md
  - installation/selinux-troubleshooting.md
  - installation/ubuntu.md
  - models/circuits/circuit.md
  - models/circuits/circuittermination.md
. . . 
----- < snip > -----
. . . 
  - release-notes/index.md 
WARNING -  Documentation file 'additional-features/jobs.md' contains a link to '../media/admin_ui_run_permission.png' which is not found in the documentation files. 
WARNING -  Documentation file 'configuration/required-settings.md' contains a link to '.' which is not found in the documentation files. 
WARNING -  Documentation file 'core-functionality/power.md' contains a link to '../media/power_distribution.png' which is not found in the documentation files. 
INFO    -  Documentation built in 2.67 seconds 
[I 210621 14:32:43 server:335] Serving on http://127.0.0.1:8001
INFO    -  Serving on http://127.0.0.1:8001
[I 210621 14:32:43 handlers:62] Start watching changes
INFO    -  Start watching changes
[I 210621 14:32:43 handlers:64] Start detecting changes
INFO    -  Start detecting changes

Notice the INFO - Start detecting changes message in the terminal. This indicates that the server will detect changes and update the served docs as the changes are detected.

Keep this command running in the terminal. Go to http://127.0.0.1:8001 to view the docs:

Example Doc Contribution

This example will show a fix for a documentation bug for a section in the Development Environment that needs a correction.

NOTE: The Nautobot issue for this correction is #600.

A screenshot of the docs prior to the fix is below, where it mentions invoke createsuperuser. It should also mention that, in a fresh install, it’s necessary to run invoke migrate prior to invoke createsuperuser.

Locate the File(s) to Modify

To help locate the file that should be modified, notice the URL for the page we want to modify has a path that ends with development/getting-started/

https://nautobot.readthedocs.io/en/latest/development/getting-started/

Open another terminal window and go to the nautobot directory (be sure to keep the doc serve process running).

To help see where we can access the page we want to modify, check out the mkdocs.yml page in the nautobot directory:

View the mkdocs.yml file; there will be a top-tier section called nav. Within nav is a section called Development, and beneath that, Getting Started.

 % cat mkdocs.yml                                                                                                                                                      
dev_addr: '127.0.0.1:8001'
. . . 
----- < snip > -----
. . . 
nav:
    - Introduction: 'index.md'
. . . 
----- < snip > -----
. . .  
    - Development:
        - Introduction: 'development/index.md'
        - Getting Started: 'development/getting-started.md'
        - Best Practices: 'development/best-practices.md'
        - Style Guide: 'development/style-guide.md'
        - Extending Models: 'development/extending-models.md'
        - Application Registry: 'development/application-registry.md'
        - User Preferences: 'development/user-preferences.md'
        - Release Checklist: 'development/release-checklist.md'
    - Release Notes:
        - Version 1.0: 'release-notes/version-1.0.md'
 % 

This matches closely to the development/getting-started/ URL path and is likely where we’d want to get started.

Navigate to the directory docs/development:

% pwd                                                                                                                          (tim-fiola-doc-blog)nautobot
/Users/timothyfiola/nautobot_blog/nautobot
% ls -l 
total 392
-rw-r--r--   1 timothyfiola  staff     119 Jun 18 13:15 CHANGELOG.md
-rw-r--r--   1 timothyfiola  staff     120 Jun 18 13:15 CONTRIBUTING.md
-rw-r--r--   1 timothyfiola  staff   10174 Jun 18 13:15 LICENSE.txt
-rw-r--r--   1 timothyfiola  staff     267 Jun 18 13:15 NOTICE
-rw-r--r--   1 timothyfiola  staff    2475 Jun 18 13:15 README.md
drwxr-xr-x   8 timothyfiola  staff     256 Jun 18 13:15 development
drwxr-xr-x   6 timothyfiola  staff     192 Jun 18 13:15 docker
lrwxr-xr-x   1 timothyfiola  staff      13 Jun 18 13:15 docs -> nautobot/docs
drwxr-xr-x   5 timothyfiola  staff     160 Jun 18 13:18 examples
-rwxr-xr-x   1 timothyfiola  staff    4160 Jun 18 13:15 install.sh
-rw-r--r--   1 timothyfiola  staff     453 Jun 18 13:15 invoke.yml.example
-rwxr-xr-x   1 timothyfiola  staff     111 Jun 18 13:15 manage.py
-rw-r--r--   1 timothyfiola  staff    4315 Jun 18 13:18 mkdocs.yml
drwxr-xr-x  16 timothyfiola  staff     512 Jun 18 13:28 nautobot
-rw-r--r--   1 timothyfiola  staff     617 Jun 18 13:15 nautobot.code-workspace
-rw-r--r--   1 timothyfiola  staff  117557 Jun 18 13:15 poetry.lock
-rw-r--r--   1 timothyfiola  staff    3997 Jun 18 13:15 pyproject.toml
drwxr-xr-x   4 timothyfiola  staff     128 Jun 18 13:18 scripts
-rw-r--r--   1 timothyfiola  staff   17630 Jun 18 13:15 tasks.py
% cd docs/development/ 
docs/development %    

Make Your Edits

We’ll edit the nautobot/docs/development/getting-started.md file, adding a tip admonishment that notifies the user of the requirement:

Save the file then go back to your browser, where you are serving the docs locally; you should see the change:

NOTE: You will see the change rendered automatically only if you kept the process that was rendering the docs running in the terminal.

If you look back at your terminal window where you initiated the docs to be served, you’ll see this message in the log after you make the change:

[I 210621 15:09:03 handlers:92] Reload 1 waiters: /Users/timothyfiola/nautobot_blog/nautobot/docs/development/getting-started.md
INFO    -  Reload 1 waiters: /Users/timothyfiola/nautobot_blog/nautobot/docs/development/getting-started.md

This notifies you that the mkdocs server noticed a change in the repository, and so it automatically re-rendered the documentation.

Make the Commit

Add the getting-started.md file to your git staging, then make the commit for your change:

% git status          
On branch tim-fiola-doc-blog
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   nautobot/docs/development/getting-started.md

no changes added to commit (use "git add" and/or "git commit -a")
% git add nautobot/docs/development/getting-started.md                    
%                                                                         
% git status                                                                 
On branch tim-fiola-doc-blog
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	modified:   nautobot/docs/development/getting-started.md

% git commit -m 'made correction in docs'                                                                                                               (tim-fiola-doc-blog)nautobot
[tim-fiola-doc-blog 090cc0c1] made correction in docs
 1 file changed, 3 insertions(+)
% 

Push the Change

Now push your change to your origin branch:

% git push                                                               
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 12 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 592 bytes | 592.00 KiB/s, done.
Total 6 (delta 5), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (5/5), completed with 5 local objects.
To github.com:tim-fiola/nautobot.git
   a9ca3a2b..090cc0c1  tim-fiola-doc-blog -> tim-fiola-doc-blog
%   

Create a PR

The commit in the prior section updated the branch in our forked repository. Now it’s time to submit a pull request (PR).

Navigate to your forked Nautobot repository
Click on the branches link
On the branches screen, click on Yours

Next, find the branch with your changes in the list and click on the New pull request button next to the branch name.

Fill in the PR form. Be sure to link to the approved issue that your PR is fixing. You can do this by simply adding the issue number after the ### Fixes:# text in the PR form. You will know that the issue reference succeeds when the auto-link shows, as in the picture below:

When you have filled out the PR form completely, submit the PR.

What to Expect

Once you submit your PR, you can expect that there will be some conversation within it, including some suggestions from Nautobot’s core Developers. Keep your eyes open for updates to the PR so you can respond accordingly.

Once everything is in order, the PR will be staged for a merge in the next release. The Nautobot Developers, Developer Advocates, and the community will appreciate your help in making Nautobot more usable!

Thank you and have a great day!

-Tim

Understanding the Ansible Project Packaging

2021-07-06T00:00:00+00:00

Ansible released 3.x in February 2021, which continues the project’s recent announcement that the code base will be splitting. As the project continues the transition, it’s important to understand some of the new terminology and understand how it might affect your environment.

The main difference is the philosophy on installation, i.e., installation separation of core and community packages. Ansible 3.0.0 is a community package that relies on ansible-base but comes natively with a subset of collections.

On top of the project split there is also going to be a change in the Ansible core engine naming.

Ansible core engine is called ansible-base which is new for version 2.10, and ansible-core with 2.11+.

Ansible 3

As mentioned previously, the installation of Ansible 3.x is considered a community package which will install ansible-base and ansible 3.x. The package list will no longer just show ansible as it has in the past.

In a fresh virtual environment Ansible is installed.

(ansible-3)
~/ansible-testing
▶ pip install ansible
Collecting ansible
  Downloading ansible-3.2.0.tar.gz (31.3 MB)
     |████████████████████████████████| 31.3 MB 25.7 MB/s
Collecting ansible-base<2.11,>=2.10.7
  Using cached ansible-base-2.10.7.tar.gz (5.7 MB)
---- output omitted ----
Successfully installed MarkupSafe-1.1.1 PyYAML-5.4.1 ansible-3.2.0 ansible-base-2.10.7 cffi-1.14.5 cryptography-3.4.7 jinja2-2.11.3 packaging-20.9 pycparser-2.20 pyparsing-2.4.7

If we check out packages with pip freeze | grep ansible, the project split is evident. We have ansible 3.2.0 with its dependency of ansible-base pinned to specific versions >=2.10.6,<2.11.

Note: 2.11 is when ansible-base is renamed to ansible-core.

▶ pip freeze | grep ansible
ansible==3.2.0
ansible-base==2.10.7

▶ pip show ansible | grep Requires
Requires: ansible-base

Since Ansible 3.x is the community package to see the native collections, execute ansible-galaxy collection list:

(ansible-3)
~/ansible-testing
▶ ansible-galaxy collection list

# /Users/ntc/ansible-testing/ansible-3/lib/python3.9/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    1.4.1
ansible.netcommon             1.5.0
ansible.posix                 1.2.0
ansible.utils                 2.0.2
ansible.windows               1.4.0
arista.eos                    1.3.0
---- omitted ----
vyos.vyos                     1.1.1
wti.remote                    1.0.1

Ansible 4

Ansible 4.0.0 was released in May 2021. It is similar to Ansible 3.x, i.e., it will be a community package that includes a subset of collections, the main difference will be Ansible 4.x dependency will be ansible-core which is >=2.11,<2.12.

(ansible-4)
~/ansible-testing
▶ pip install ansible
Collecting ansible
  Downloading ansible-4.1.0.tar.gz (34.0 MB)
     |████████████████████████████████| 34.0 MB 14.3 MB/s 
Collecting ansible-core<2.12,>=2.11.1
  Downloading ansible-core-2.11.1.tar.gz (6.1 MB)
     |████████████████████████████████| 6.1 MB 9.5 MB/s 
---- output omitted ----

Checking PIP freeze we see ansible-core was installed.

▶ pip freeze | grep ansible
ansible==4.1.0
ansible-core==2.11.1

▶ pip show ansible | grep Requires
Requires: ansible-core

After the install, running ansible-galaxy collection list shows the collections that came natively with the community package. This is similar to Ansible 3.x.

(ansible-4)
~/ansible-testing
▶ ansible-galaxy collection list

# /Users/ntc/ansible-testing/ansible-4/lib/python3.9/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    1.5.0  
ansible.netcommon             2.1.0  
ansible.posix                 1.2.0  
ansible.utils                 2.2.0  
ansible.windows               1.6.0  
arista.eos                    2.1.2  
---- omitted ----
vyos.vyos                     2.3.0  
wti.remote                    1.0.1 

Collection Flexibility

For anyone that has worked in or works in a locked down environment, the Ansible project split offers additional flexibility that allows a user to install ansible-base(3.0) or ansible-core(4.0) and then only install the collections that are needed. For example, perhaps the environment only has Cisco IOS—it is possible to run the core Ansible and then install only the Cisco IOS collection.

To demonstrate that functionality, we will install ansible-base and then install the Cisco IOS collection.

Install Ansible Base:

(ansible-base-only)
~/ansible-testing
▶ pip install ansible-base==2.10.7

Collecting ansible-base==2.10.7
  Using cached ansible-base-2.10.7.tar.gz (5.7 MB)
---- omitted ----

Successfully installed MarkupSafe-1.1.1 PyYAML-5.4.1 ansible-base-2.10.7 cffi-1.14.5 cryptography-3.4.7 jinja2-2.11.3 packaging-20.9 pycparser-2.20 pyparsing-2.4.7

PIP shows only ansible-base is installed.

(ansible-base-only)
~/ansible-testing
▶ pip freeze | grep ansible
ansible-base==2.10.7

Checking the collection list shows no collections installed.

▶ ansible-galaxy collection list

Next, install the Cisco IOS collection.

▶ ansible-galaxy collection install cisco.ios
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Installing 'cisco.ios:2.0.1' to '/Users/ntc/.ansible/collections/ansible_collections/cisco/ios'
Downloading https://galaxy.ansible.com/download/cisco-ios-2.0.1.tar.gz to /Users/ntc/.ansible/tmp/ansible-local-333579nqv2gur/tmp_96gsuu5
cisco.ios (2.0.1) was installed successfully
Installing 'ansible.netcommon:2.0.1' to '/Users/ntc/.ansible/collections/ansible_collections/ansible/netcommon'
Downloading https://galaxy.ansible.com/download/ansible-netcommon-2.0.1.tar.gz to /Users/ntc/.ansible/tmp/ansible-local-333579nqv2gur/tmp_96gsuu5
ansible.netcommon (2.0.1) was installed successfully
Installing 'ansible.utils:2.0.2' to '/Users/ntc/.ansible/collections/ansible_collections/ansible/utils'
Downloading https://galaxy.ansible.com/download/ansible-utils-2.0.2.tar.gz to /Users/ntc/.ansible/tmp/ansible-local-333579nqv2gur/tmp_96gsuu5
ansible.utils (2.0.2) was installed successfully

When the Cisco IOS collection is installed it will execute a dependency check and install required collections. In the case of Cisco IOS, the Ansible Netcommon and Ansible Utils collections are installed.

Listing the collections shows only the 3 collections versus the 85+ in the community package.

▶ ansible-galaxy collection list

# /Users/ntc/.ansible/collections/ansible_collections
Collection            Version
--------------------- -------
ansible.netcommon     2.0.1
ansible.utils         2.0.2
cisco.ios             2.0.1

Summary

As the Ansible project continues to evolve the code base split changes the way Ansible is managed in an environment. It becomes more flexible, but also has some additional dependencies that you need to be aware of. While troubleshooting new Ansible installs, a few helpful commands include:

ansible --version

Note: The Ansible version will show the core/base version number NOT the community package version.

ansible-galaxy collection list

These commands show which collections are installed and the path to the collections respectively.

Finally, the ansible-galaxy command for collections doesn’t currently support an uninstall option. To delete a collection, use rm -rf /path/to/collection. To force an update of a collection use the --force command line argument.

-Jeff

The NTC Mag

Nautobot Plugin: Single Source of Truth (SSoT)

Overview

SSoT Dashboard

Data Source / Data Target Views

SSoT Job Result Database and Views

Installing the Plugin

Using the Plugin

Accessing the SSoT Dashboard

Running a Data Synchronization Job

Viewing Synchronization Results

Reviewing History and Job Details

Running Again

Finding, Creating, and Enabling More Jobs

Next Steps

Manipulating Data with Jinja Map and Selectattr

Selecattr

Map

Other Useful Combinations

References:

Nautobot, Beyond Network Source of Truth

High-Level Architecture

Project Structure

nautobot_plugin_recipe_filter/init.py

pyproject.toml

Creating the Custom Jinja2 Filter

recipe_filters.py

Example Config Settings

Creating a Custom Link

Guide to Create a Custom Link

Browsing to Add Custom Link Form

Filling Out the Add Custom Link Form

Using the Custom Link

Guide to Create a Region

Browsing to Add Region Form

Filling Out the Add Region Form

Guide to Create a Site

Browsing to Add Site Form

Filling Out the Add Site Form

Viewing the Recipe Recommendation

Viewing the Custom Link on the Site Page

Clicking the Link to View the Recipe

Conclusion

Ansible Execution Strategies

Strategy Basics

Other Execution Tune-ables

Strategies and Keyword Interrelationship

Free Strategy

Debug Strategy

Host Pinned

Serial

Ordered

Throttle

Nautobot 1.1.0 Key Feature Overview

Computed Fields

Custom Jinja2 Templates

Computed Fields API

Config Context Schemas

Saved GraphQL Queries

Executing Saved Queries Programmatically

Read-Only Jobs

Plugin-Defined Navigation

Under the Covers

Background Tasks Use Celery

Support for MySQL

Using Jinja2 Macros as Template Functions

Real-World Situation

Jinja2 Macros Example

Line 1

Line 2

Line 3

Importing Functions, Abbreviated

Wrapping It Up

Nautobot Relationships

Relationships Primer

One-to-Many

Many-to-Many

One-to-One

Nautobot Model

Use Case 1: IPAddress - Circuit