The NTC Mag

Why did Network to Code Fork NetBox?

2021-02-25T00:00:00+00:00

[This post was reviewed and edited by many of the people who have contributed to the Nautobot project at Network to Code.]

Over the last five years, NetBox has gained adoption as an IP Address Management (IPAM) and Data Center Infrastructure Management (DCIM) platform while also serving as a Source of Truth for documenting networks and network automation projects. In fact, Network to Code has been heavy users, supporters, and contributors of NetBox for years. While we started offering NetBox professional services in 2018 and support for NetBox in 2019, we have continued to support an ever-growing customer base that has evolving requirements around Source of Truth and more broadly around creating holistic network automation solutions. Because of this, and in order to better serve our customers and the community, Network to Code made the difficult decision to fork NetBox and drive it in new directions.

Let’s dive into the details.

Background

Network to Code has been a power user of NetBox for the past 3+ years. We’ve designed and deployed numerous solutions that revolve around and integrate with NetBox. We have used the existing models, extensibility features, worked on a myriad of private customer-specific forks, created ways to manage settings and URLs, as well as created plugins for various use cases. In short, over the course of countless projects, we have seen first-hand what works well for users and what has room for improvement.

There have been questions raised in different public forums if Network to Code has been a contributor to NetBox. The answer is yes. Let’s explore some of those contributions.

First and foremost, as part of the NetBox community, Network to Code has contributed numerous blog articles, published videos, hosted a NetBox Day, presented at conferences, enabled and participated in innumerable Slack discussions hosted on the Network to Code Slack workspace, participated in podcasts, hosted low-cost training, and more. While all of this was not hand-to-keyboard development, NTC has undoubtedly contributed thousands of hours toward NetBox in this capacity in recent years.

For contributions to the core of NetBox, NTC employees (not counting the lead maintainer of NetBox, who was employed by NTC for 18 months and spent the majority of his time actively maintaining NetBox during that time) have contributed 78 issues and 35 PRs, two of which are still being worked on. Some of these contributions have been significant in scope, such as the initial implementation of the NetBox plugin API that was released in NetBox 2.8.

When you look at code and development, there are NTC projects that also build on top of NetBox. We have built a Device Onboarding plugin, Prometheus metrics plugin, a number of other plugins that we are working to open source, as well as projects like network-importer to simplify adding data into NetBox. Each of these projects individually are the result of hundreds of hours, adding up to many months of dedicated development effort.

While we have never claimed to be an official sponsor of NetBox, we were effectively an unofficial and informal sponsor of the project. Lastly, one of the core committers on our fork had also been a maintainer of NetBox for a number of years while at Network to Code.

We have always had candid discussions on feature requests, support, and on the direction that we believed was in the best interest of the long term success of NetBox. As time went on and after numerous discussions, there became a growing divergence in our vision about what a Source of Truth for networking should look like and how to get there, and the NetBox project team suggested that we should consider forking. Late in 2020, we took a step back to think about the options.

Taking what we’ve learned deploying and integrating NetBox solutions for customers over the past few years, what we have seen and heard from the community, coupled with discussions with the NetBox project team, Network to Code came to the conclusions that it was best to fork NetBox with the vision and direction necessary for powering network automation platforms. Next, from a business and technical perspective, let’s explore the key reasons we considered and what drove us to make the decision.

We need to provide world class support for our clients.

We need to offer high-touch support models with Service Level Agreements (SLAs) we can guarantee. We need flexibility to offer Long-term Support (LTS) for customers who can’t upgrade at the pace of a fast moving open source project. In addition, we need to be able to deliver the features required to get a system into production in an enterprise environment. Some examples include simplifying the ability to integrate with Single Sign-On (SSO) services such as SAML or OIDC and adding support for additional database backends such as MySQL.

Define a strategic vision for Source of Truth that enables network automation.

We firmly believe that a Source of Truth is a fundamental requirement to enable data-driven network automation. This is where we are able to leverage our collective experience at Network to Code and add functionality that simplifies integrating with network automation solutions. While the new platform can still be used for network documentation, its key focus is on enabling network automation. After all, network automation is our core business at Network to Code. It has become apparent through our customer projects that there is a need for more flexible APIs and more seamless ways to integrate with other data sources.

Build a Network Automation App Platform.

This is one area that we think is quite unique. The goal is to leverage the rich data stored in the Source of Truth and build high-value apps that use and consume that data while interacting with other systems and network devices as needed. The best metaphor to consider is a smartphone. It will always be a “phone.” However, this “phone” is a delivery mechanism for high-value apps that we all get to use on a daily basis (should you choose to). In this example, the Source of Truth is the phone, i.e. the delivery mechanism. Our goal is to enable the Source of Truth as a platform through which high-value apps are deployed. These apps can be lightweight (creating new models and APIs) or be used to deliver solutions (pre/post change, integrate with monitoring, perform backups). We believe it should be up to the users to decide.

Community Committed

Network to Code has deep roots in the community. As mentioned earlier, we have spent thousands of hours on community work around NetBox. If you look at every other project we’ve been a part of, that number goes up significantly. From open sourcing various projects of our own to contributing to other such projects to engaging in discussions on numerous forums, community is extremely important to who we are as a company.

We know there is A LOT of work to do to gain the respect and trust of the community as we move forward with Nautobot. We are committed to fostering user and developer communities that are fully transparent. We will be hanging out in the #nautobot channel on the Network to Code Slack. Please come join and let us know what you think! You can also view the initial Nautobot roadmap.

If you have ideas, enhancements, or feature requests for Nautobot, please don’t hesitate to open an issue on GitHub.

Going Forward

Just to get us where we are with Nautobot, we have invested close to a “work year” of effort across our team just over the past few months. This effort and focus is not going to stop. While Nautobot is much more than NetBox with additional features, it’s actually not even about what you see today. This is just the first release. If you haven’t already done so, please check out the roadmap. You’ll see that our goal is to empower flexibility and extensibility while truly being a Source of Truth foundation for network automation. Our vision will see Nautobot continue to diverge from NetBox in such a way that the separation of the two will become much more distinct.

Git History

When we first released the Nautobot project on GitHub, we had made a decision, based on technical considerations, to not keep the complete NetBox commit history. It was never our intent to “erase” the NetBox commit history because that history exists within the NetBox project and we are making every attempt to be fully transparent about the forked status and attribute proper credit. We have heard your feedback that this can give the appearance of dismissing the history and community engagement that has brought NetBox to where it is today, and we apologize for that. We are grateful and we are re-adding the NetBox Git history to Nautobot. Below are the technical reasons we had in mind when we chose to reset the commit history.

First, we thought that it would make it possible to draw a clear dividing line between the development of NetBox and the development of Nautobot so there would be no ambiguity as to whether a given commit in the Git history is a shared NetBox commit that was inherited by Nautobot, or whether a commit is specific to Nautobot and distinct from NetBox. This would make it easier to see over time how the two projects have diverged.

Second, we wanted to avoid any ambiguity about version numbers. Creating a new repository from our own version 1.0 allowed us to avoid these collisions and confusion without needing to do anything messy like deleting old tags and releases from the history.

Third, we have made a large number of invasive changes related to packaging and renaming which greatly reduce the historical relevance of the history from a purely technical standpoint.

Fourth, it reduced the size of the repository substantially. A fresh clone of nautobot.git is only about 28 MB in size, versus a fresh clone of upstream netbox.git weighing in at about 185 MB.

We remain grateful to all developers of and contributors to NetBox, and hope that this provides appropriate clarity as to the reasons why we initially chose this particular approach.

Closing

The decision to fork and create something new was not a decision we took lightly at Network to Code. We are committed to creating something great and to ensuring Nautobot will become a thriving, transparent, and engaging project and community. We’re already seeing a lot of positive feedback from the community based on the direction planned for Nautobot.

We welcome you on the journey and hope you come along for the ride. We look forward to the evolution of Nautobot and seeing Nautobot used to power network automation solutions across the world.

-Jason

Introducing Schema Enforcer

2021-02-11T00:00:00+00:00

These days, most organizations heavily leverage YAML and JSON to store and organize all sorts of data. This is done in order to define variables, be provided as input for generating device configurations, define inventory, and for many other use cases. Both YAML and JSON are very popular because both languages are very flexible and are easy to use. It is relatively easy for users who have little to no experience working with structured data (as well as for very experienced programmers) to use JSON and YAML because the formats do not require users to define a schema in order to define data.

As the use of structured data increases, the flexibility provided because these languages don’t require data to adhere to a schema create complexity and risk. If a user accidentally defines the data for ntp_servers in two different structures (e.g. one is a list, and one is a dictionary), automation tooling must be written to handle the differences in inputs in some way. Often times, the automation tooling just bombs out with a cryptic message in such cases. This is because the tool consuming this data rightfully expects to have a contract with it, that the data will adhere to a clearly defined form and thus the tool can interact with the data in a standard way. It is for this reason that APIs, when updated, should never change the format in which they provide data unless there is some way to delineate the new format (e.g. an API version increment). By ensuring data is defined in a standard way, complexity and risk can be mitigated.

With structured data languages like YAML and JSON which do not inherently define a schema (contract) for the data they define, a schema definition language can be used to provide this contract, thereby mitigating complexity and risk. Schema definition languages come with their own added maintenance though as the burden of writing the logic to ensure structured data is schema valid falls on the user. The user doesn’t just need to maintain structured data and schemas, they also have to build and maintain the tooling that checks if data is schema valid. To allow users to simply write schemas and structured data and worry less about writing and maintaining the code that bolts them together, Network to Code has developed a tool called Schema Enforcer. Today we are happy to announce that we are making Schema Enforcer available to the community.

Check it out on Github!

What is Schema Enforcer

Schema Enforcer is a framework for allowing users to define schemas for their structured data and assert that the defined structured data adheres to their schemas. This structured data can (currently) come in the the form of a data file in JSON or YAML format, or an Ansible inventory. The schema definition is defined by using the JSONSchema language in YAML or JSON format.

Why use Schema Enforcer?

If you’re familiar with JSONSchema already, you may be thinking “wait, doesn’t JSONSchema do all of this?”. JSONSchema does allow you to validate that structured data adheres to a schema definition, but it requires for you to write your own code to interact with and manage the data’s adherence to defined schema. Schema Enforcer is meant to provide a wrapper which makes it easy for users to manage structured data without needing to write their own code to check their structured data for adherence to a schema. It provides the following advantages over just using JSONSchema:

Provides a framework for mapping data files to the schema definitions against which they should be checked for adherence
Provides a framework for validating that Ansible inventory adheres to a schema definition or multiple schema definitions
Prints clear log messages indicating each data object examined which is not adherent to schema, and the specific way in which these data objects are not adherent
Allows a user to define unit tests asserting that their schema definitions are written correctly (e.g. that non-adherent data fails validation in a specific way, and adherent data passes validation)
Exits with an exit code of 1 in the event that data is not adherent to schema. This makes it fit for use in a CI pipeline along-side linters and unit tests

An Example

I’ve created the following directories and files in a repository called new_example.

The directory includes

structured data (in YAML format) defining ntp servers for the host chi-beijing-rt01 inside of the file at hostvars/chi-beijing-rt01/ntp.yml
a schema definition inside of the file at schema/schemas/ntp.yml

If we examine the file at hostvars/chi-being-rt01/ntp.yml we can see the following data defined in YAML format.

# jsonschema: schemas/ntp
---
ntp_servers:
  - address: 192.2.0.1
  - address: 192.2.0.2

Note the comment # jsonschema: schemas/ntp at the top of the YAML file. This comment is used to declare the schema that the data in this file should be checked for adherence to, as well as the language being used to define the schema (JSONSchema here). Multiple schemas can be declared by comma separating them in the comment. For instance, the comment # jsonschema: schemas/ntp,schemas/syslog would declare that the data should be checked for adherence to two schema, one schema with the ID schemas/ntp and another with the id schemas/syslog. We can validate that this mapping is being inferred correctly by running the command schema-enforcer validate --show-checks

The --show-checks flag shows each data file along with a list of every schema IDs it will be checked for adherence to.

Other mechanisms for mapping data files to schemas against which they should be validated. See docs/mapping_schemas.md in the Schema Enforcer git repository. for more details.
YAML supports the addition of comments using an octothorp. JSON does not support the addition of comments. To this end, only data defined in YAML format can declare the schema to which it should adhere with a comment. Another mechanism for mapping needs to be used if your data is defined in JSON format.

If we examine the file at schema/schemas/ntp.yml we can see the following schema definition. This is written in the JSONSchema language and formatted in YAML.

---
$schema: "http://json-schema.org/draft-07/schema#"
$id: "schemas/ntp"
description: "NTP Configuration schema."
type: "object"
properties:
  ntp_servers:
    type: "array"
    items:
      type: "object"
      properties:
        name:
          type: "string"
        address:
          type: "string"
          format: "ipv4"
        vrf:
          type: "string"
      required:
          - "address"
    uniqueItems: true
additionalProperties: false
required:
  - "ntp_servers"

The schema definition above is used to ensure that:

The ntp_servers property is of type hash/dictionary (object in JSONSchema parlance)
No top level keys can be defined in the data file besides ntp_servers
It’s value is of type array/list
Each item in this array must be unique
Each element of this array/list is a dictionary with the possible keys name, address and vrf
- Of these keys, address is required, name and vrf can optionally be defined, but it is not necessary to define them.
- address must be of type “string” and it must be a valid IP address
- name must be of type “string” if it is defined
- vrf must be of type “string” if it is defined

Here is an example of the structured data being checked for adherence to the schema definition.

We can see that when schema-enforcer runs, it shows that all files containing structured data are schema valid. Also note that Schema Enforcer exits with a code of 0.

What happens if we modify the data such that the first ntp server defined has a value of the boolean true and add a syslog_servers dictionary/hash type object at the top level of the YAML file.

# jsonschema: schemas/ntp
---
ntp_servers:
  - address: true
  - address: 192.2.0.2
syslog_servers:
  - address: 192.0.5.3

We can see that two errors are flagged. The first informs us that the first element in the array which is the value of the ntp_servers top level key is a boolean and a string was expected. The second informs us that the additional top level property syslog_servers is a property that is additional to (is not specified in) the properties defined by the schema, and that additional properties are not allowed per the schema definition. Note that schema-enforcer exits with a code of 1 indicating a failure. If Schema Enforcer were to be used before structured data is ingested into automation tools as part of a pipeline, the pipeline would never have the automation tools consume the malformed data.

Validating Ansible Inventory

Schema Enforcer supports validating that variables defined in an Ansible inventory adhere to a schema definition (or multiple schema definitions).

To do this, Schema Enforcer first constructs a dictionary containing key/value pairs for each attribute defined in the inventory. It does this by flattening the varibles from the groups the host is a part of. After doing this, schema-enforcer maps which schemas it should use to validate the hosts variables in one of two ways:

By using a list of schema ids defined by the schema_enforcer_schema_ids attribute (defined at the host or group level).
By automatically mapping a schema’s top level properties to the Ansible host’s keys.

That may have been gibberish on first pass, but the examples in the following sections will hopefully make things clearer.

An Examle of Validating Ansible Variables

In the following example, we have an inventory file which defines three groups, nyc, spine, and leaf. spine and leaf are children of nyc.

[nyc:children]
spine
leaf

[spine]
spine1
spine2

[leaf]
leaf1
leaf2

The group spine.yaml has two top level keys; dns_servers and interfaces.

cat group_vars/spine.yaml
---
dns_servers:
  - address: true
  - address: "10.2.2.2"
interfaces:
  swp1:
    role: "uplink"
  swp2:
    role: "uplink"

schema_enforcer_schema_ids:
  - "schemas/dns_servers"
  - "schemas/interfaces"

Note the schema_enforcer_schema_ids variable. This declaratively tells Schema Enforcer which schemas to use when running tests to ensure that the Ansible host vars for every host in the spine group are schema valid.

Here is the interfaces schema which is declared above:

bash$ cat schema/schemas/interfaces.yml
---
$schema: "http://json-schema.org/draft-07/schema#"
$id: "schemas/interfaces"
description: "Interfaces configuration schema."
type: "object"
properties:
  interfaces:
    type: "object"
    patternProperties:
      ^swp.*$:
        properties:
          type:
            type: "string"
          description:
            type: "string"
          role:
            type: "string"

Note that the $id property is what is being declared by the schema_enforcer_schema_ids variable.

When we run the schema-enforcer ansible command with the --show-pass flag, we can see that the spine1 and spine2’s defined dns_servers attribute did not adhere to schema.

By default, schema-enforcer prints a “FAIL” message to stdout for each object in the data file which does not adhere to schema. If no objects fail to adhere to schema definitions, a single line is printed indicating that all data files are schema valid. The --show-pass flag modifies this behavior such that, in addition the the default behavior, a line is printed to stdout for every file that is schema valid indicating it passed the schema adherence check.

In looking at the group_vars/spine.yaml group above. This is because the first dns server in the list which is the value of dns_servers has a value of the boolean true.

bash$ cat schema/schemas/dns.yml
---
$schema: "http://json-schema.org/draft-07/schema#"
$id: "schemas/dns_servers"
description: "DNS Server Configuration schema."
type: "object"
properties:
  dns_servers:
    type: "array"
    items:
      type: "object"
      properties:
        name:
          type: "string"
        address:
          type: "string"
          format: "ipv4"
        vrf:
          type: "string"
        required:
            - "address"
    uniqueItems: true
required:
  - "dns_servers"

In looking at the schema for dns servers, we see that DNS servers address field must be of type string and format ipv4 (e.g. IPv4 address). Because the first element in the list of DNS servers has an address of the boolean true it is not schema valid.

Another Example of Validating Ansible Vars

Similar to the way that schema-enforcer validate --show-checks can be used to show which data files will be checked by which schema definitions, the schema-enforcer ansible --show-checks command can be used to show which Ansible hosts will be checked for adherence to which schema IDs.

From the execution of the command, we can see that 4 hosts were loaded from inventory. This is just what we expect from our earlier examination of the .ini file which defines Ansible inventory. We just saw how spine1 and spine2 were checked for adherence to both the schemas/dns_servers and schemas/interfaces schema definitions, and how the schema_enforcer_schema_ids var was configured to declare that devices belonging to the spine group should adhere to those schemas. Lets now examine the leaf group a little more closely.

cat ansible/group_vars/leaf.yml
---
dns_servers:
  - address: "10.1.1.1"
  - address: "10.2.2.2"

In the leaf.yml file, no schema_enforcer_schema_ids var is configured. There is also no individual data defined at the host level for leaf1 and leaf2 which belong to the leaf group. This brings up the question, how does schema-enforcer know to check the leaf switches for adherence to the schemas/dns_servers schema definition?

The default behavior of schema-enforcer is to map the top level property in a schema definition to vars associated with each Ansible host that have the same name.

bash$ cat schema/schemas/dns.yml
---
$schema: "http://json-schema.org/draft-07/schema#"
$id: "schemas/dns_servers"
description: "DNS Server Configuration schema."
type: "object"
properties:
  dns_servers:
    type: "array"
    items:
      type: "object"
      properties:
        name:
          type: "string"
        address:
          type: "string"
          format: "ipv4"
        vrf:
          type: "string"
        required:
            - "address"
    uniqueItems: true
required:
  - "dns_servers"

Because the property defined in the schema definition above is dns_servers, the matching Ansible host var dns_servers will be checked for adherence against it.

In fact, if we make the following changes to the leaf group var definition then run schema-enforcer --show-checks, we can see that devices belonging to the leaf group are now slated to be checked for adherence to both the schemas/dns_servers and schemas/interfaces schema definitions.

cat ansible/group_vars/leaf.yml
---
dns_servers:
  - address: "10.1.1.1"
  - address: "10.2.2.2"
interfaces:
  swp01:
    role: uplink

Using Schema Enforcer

O.K. so you’ve defined schemas for your data, now what? Here are a couple of use cases for Schema Enforcer we’ve found to be “juice worth the squeeze.”

1) Use Schema Enforcer in your CI system to validate defined structured data before merging code. Virtually all git version control systems (GitHub, GitLab…etc) allow the ability to configure tests which must pass before code can be merged from a feature branch into the code base. Schema Enforcer can be turned on along side your other tests (unit tests, linters…etc). If your data is not schema valid, the exact reason why the data is not schema valid will be printed to the output of the CI system when the tool is run and the tool will exit with a code of 1 causing the CI system to register a failure. When the CI system sees a failure, it will not allow the merge of data which is not adherent to schema.

2) Use it in a pipeline. Say you have YAML structured data which defines the configuration for network devices. You can run schema enforcer as part of a pipeline and run it before automation tooling (Ansible, Python…etc) consumes this data in order to render configurations for devices. If the data isn’t schema valid, the pipeline will fail before rendering configurations and pushing them to devices (or exploding with a stack trace that takes you 30 minutes and lots of googling to troubleshoot).

3) Run it after your tooling generates structured data and prints it to a file. In this case, Schema Enforcer can act as a sanity check to ensure that your tooling is dumping correctly structured output.

Where Are We Going Next

We plan to add the support for the following features to Schema Enforcer in the future:

Validation of Nornir inventory attributes
Business logic validation

Have a use case for Schema Enforcer? Try it out and let us know what you think! Do you want the ability to write schema definitions in YANG or have another cool idea? We are iterating on the tool and we are open to feedback!

-Phillip Simonds

Introduction to PromQL

2021-02-09T00:00:00+00:00

Time series databases and their query languages are tools with increasing popularity for a Network Automation Engineer. However, sometimes these tools may be overlooked by network operators for more “pressing” day-to-day workflow automation. Time series databases offer valuable network telemetry that will reveal important insights for network operations, such as security breaches, network outages, and slowdowns that degrade the user experience.

In this post, we will review the Prometheus Query Language (PromQL) to demonstrate the value and capabilities of processing time series. This review will offer use cases of PromQL for network engineers and data scientists.

What is Prometheus?

Prometheus is an open source systems monitoring and alerting toolkit. As you can see in the figure below, the heart of Prometheus includes a Time Series Database (TSDB) and the PromQL Engine. Exporters run locally on monitored hosts and export local metrics related to device health, such as CPU and memory utilization, and services, such as HTTP. The alert mechanism implemented with Prometheus, triggers alerts based on events and predefined thresholds. Prometheus has a web UI that we will be using in the examples of this post. In addition, the Prometheus measurements can be visualized using Grafana dashboards.

Source: Prometheus Overview

What is a TSDB?

In simple words, it is a database that stores time series. Then, what is a time series? It is a set of time-stamps and their corresponding data. A TSDB is optimized to store these time series data efficiently, measure changes, and perform calculations over time. PromQL is the language that was built to retrieve data from the Prometheus TSDB. In networking, this could mean tracking the state of an interface or bandwidth utilization over time.

Why PromQL?

There are several other TSDBs, one of the most well known is InfluxDB. Both Prometheus TSDB and InfluxDB are excellent tools for telemetry and time series data manipulation. PromQL’s popularity has been growing fast because it is a comprehensive language to consume time series data. Multiple other solutions are starting to support PromQL, such as NewRelic that recently added support for PromQL and Timescale with Promscale.

Now that we have all the prerequisite knowledge we can dive deep into the PromQL data model and dissect language queries.

Prometheus Data Model

The first part of the Prometheus data model is the metric name. A metric name is uniquely identified, and it indicates what is being measured. A metric is a dimension of a specific feature. Labels are the second part of the data model. A label is a key-value pair that differentiates sub-dimensions in a metric.

Think of a metric, ex. interface_in_octets, as an object with multiple characteristics, ex., device_role. As you can see in the figure below, each label can pick a value for this characteristic, i.e. device_role="leaf". The combination of metrics and labels return a time series identifier, i.e., a list of tuples that provide the (timestamp, value) of the object with the specific characteristic. The timestamps are given in Unix time, milliseconds precision and the values that correspond to them are floating point type.

As a Network Automation Engineer you can think of many examples of metrics, such as interface_speed, bgp_hold_time, packets_dropped, etc. All these metrics can be characterized by a variety of labels, such as device_platform, host, instance, interface_name etc.

With that data model in mind, let us next dissect a query in PromQL.

The anatomy of a query

The simplest form of a PromQL query may include just a metric. This query returns multiple single value vectors, as you can see below. All the applicable labels and value combinations that these labels can be assigned are given as a result of this simple query.

Metrics

What kind of metrics does PromQL support? There are four kinds of metrics:

Counters: these are metrics that can only increase, for example: interface counters, API call counters, etc.
Gauges: the values of these metrics can go up and down, for example: bandwidth, latency, packets dropped, etc. Gauges and counters are useful for network engineers because they can measure already existent features of a system.
Summaries: this metric is useful to data scientists and if your application includes data analytics. To use this metric you need have control of what you can measure and drill into additional details. A summary metric aggregates thousands of events to one metric. Specifically it counts observations and sums all the observed values. It can also calculate quantiles of these values. If you have an application that is being monitored, you can use the summaries for API request durations.
Histograms: this is another metric that is more useful to a data scientist than a network engineer. Histogram metrics can be defined as summaries that are “bucketized”. Specifically they count observations and place them in configurable buckets. A histogram can be used to measure response sizes on an application.

Label Filtering

Now that we know what kinds of metrics we can include in our query, let us review how we can filter the query to retrieve more specific and meaningful results. This can be done with label filtering that includes the following operations:

# equal, returns interface speed for device with name jcy-bb-01
interface_speed{device="jcy-bb-01.infra.ntc.com"}
# not equal, returns the opposite of the above query
interface_speed{device!="jcy-bb-01.infra.ntc.com"}
# regex-match, matches interface Ethernet{1, 2, 3, 4, 5, 6, 7}
interface_speed{interface=~"Ethernet1/[1-7]"}
# not regex-match, returns the opposite of the above query
interface_speed{interface!~"Ethernet1/[1-7]"}

Not only can you use the equal and not equal signs to filter your queries, but you can filter using regular expressions. To learn more about regular expressions for network engineers, check our previous blog.

Functions

One of my favorite parts of PromQL are the functions that can manipulate the time series identifiers. Below, I include an example of the function rate(), that is useful for network metrics, and the function predict_linear(), that is useful if you perform data analytics.

How fast does a counter change?

The function rate() can be used with counter metrics to demonstrate how fast a counter increases. Specifically, it calculates the per second increase for a time period. This is a useful function to the network engineer, since counters are a common metric applied in networks. For example packet counting, interface octets counting are counters and the rate() function offers useful insights on how these counters increase.

#per second increase of counter averaged over 5 mins
rate(interface_in_octets{device_role="leaf"}[5m])

The next figure will help you understand the details of how the rate() function is calculated. The interval $\Delta$t indicates the time interval during which we want to calculate the rate. The X marks indicate the per second samples that are used to calculate multiple rates per second. The rate() function averages these calculations during the interval $\Delta$t. If the counter is reset to 0, the rate() function will extrapolate the sample as can be seen with the blue X marks.

Instance vs. Range Vectors

You probably have noticed that the example of the rate() function above, uses a different type of syntax. Specifically it identifies the time series during an interval, from the example above the interval is 5 minutes ([5m]). This results to a range vector, where the time-series identifier returns the values for a given period, in this case 5 minutes. On the other hand, an instance vector returns one value, specifically the single latest value of a time series. The figures below shows the differences in the results of an instance vector versus a range vector.

#instance vector
interface_speed{device="jcy-bb-01.infra.ntc.com"}

#range vector
interface_speed{device="jcy-bb-01.infra.ntc.com"}[5m]

In the first figure, only one value per vector is returned whereas in the second, multiple values that span in the range of 5 minutes are returned for each vector. The format of these values is: value@timestamp.

Offsets

You may be wondering: all of this is great, but where is the “time” in my “time-series”? The offset part of the query can retrieve data for a specific time interval. For example:

# interface speed status for the past 24 hrs
rate(interface_in_octets{device="jcy-bb-01.infra.ntc.com"}[5m] offset 24h)

Here we combine the function rate(), that samples the interface_in_octets counter every second for five minutes, with offset that gives us historical data for the past 24 hours.

Can I predict the next 24 hours?

Of course! PromQL provides the function predict_linear(), a simple machine learning model that predicts the value of a gauge in a given amount of time in the future, by using linear regression. This function is of more interest to a data scientist that wants to create forecasting models. For example, if you want to predict the disk usage in bytes within the next hour based on historic data, you would use the following query:

#predict disk usage bytes in an hour, using the last 15 mins of data
predict_linear(demo_disk_usage_bytes{job="demo"}[15m], 3600)

Linear regression fits a linear function to a set of random data points. This is achieved by searching for all possible values for the variables a, b that define a linear function f(x)=ax+b. The line that minimizes the mean Euclidean distance of all these data points is the result of the linear regression model, as you can see in the image below:

Aggregation

PromQL queries can be highly dimensional. This means that one query can return a set of time series identifiers for all the combinations of labels, as you can see below:

#multi-dimensional query
rate(demo_api_request_duration_seconds_count{job="demo"}[5m])

What if you want to reduce the dimensions to a more meaningful result, for example the sum of all the API request durations in seconds? This would result in a single-dimension query that is the result of adding multiple instance vectors together:

#one-dimensional query, add instance vectors
sum(rate(demo_api_request_duration_seconds_count{job="demo"}[5m]))

You may choose to aggregate over specific dimensions using labels and the function by(). In the example below, we perform a sum over all instances, paths, and jobs. Note the reduction of the number of vectors returned:

# multi-dimensional query - by()
sum by(instance, path, job) (rate(demo_api_request_duration_seconds_count{job="demo"}[5m]))

We can perform the same query excluding labels using the function without():

sum without(method, status) (rate(demo_api_request_duration_seconds_count{job="demo"}[5m]))

This results to the same set of instance vectors:

Additional aggregation over dimensions can be done with the following functions:

min(): selects the minimum of all values within an aggregated group.
max(): selects the maximum of all values within an aggregated group.
avg(): calculates the average (arithmetic mean) of all values within an aggregated group.
stddev(): calculates the standard deviation of all values within an aggregated group.
stdvar(): calculates the standard variance of all values within an aggregated group.
count(): calculates the total number of series within an aggregated group.
count_values(): calculates number of elements with the same sample value.

Conclusion

Thank you for taking this journey with me, learning about the time series query language, PromQL. There are many more features to this language such as arithmetic, sorting, set functions etc. I hope that this post has given you the opportunity to understand the basics of PromQL, see the value of telemetry and TSDBs, and that it has increased your curiosity to learn more.

-Xenia

Useful Resources

PromQL Demo, official demo where you can practice PromQL queries.
Alerting with Prometheus blog by Josh VanDeraa, Network To Code.
Monitoring Websites with Telegraf and Prometheus blog by Josh VanDeraa, Network To Code.

Regex for Network Engineers

2021-01-26T00:00:00+00:00

Every IT professional will encounter Regex at some point in their career. Contrary to how it looks, Regex is not just some funny strings of random characters. You can in fact find it powering some of the world’s most critical technology infrastructure. Regex is supported out of the box in many different technologies, so learning just the basics can really accelerate your automation workflow. This post will teach you the basics and give you some real-world applications of Regex for Network Engineers.

Sounds good! But what is Regex?

Well, this is Regex:

[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}

Yeah, I know, looks kind of like hieroglyphics, right? Well, this may look like a foreign language, especially if you are just starting out, but as with any kind of new language, we need to work through the basics first.

Okay, for real this time. What is Regex?

Regex, or regular expression, is a pattern matching engine used to find or parse text and outputs for specified patterns. Regex is built into tools like Vim, grep, and even Python! We can use Regex to parse text and outputs so we can get only the data we need.

Why is it important for Network Engineers?

Well, there are patterns all around us in the networking world. A MAC address is nothing more than a pattern of some characters that can be A-F and 0-9 followed by a period or colon (depending on who you ask). And take the IPv4 address for example. If you had to teach someone how to spot an IPv4 address in a random list of characters, what would you tell them?

Well, you might say an IPv4 address is:

One to three digits followed by a period
One to three digits followed by another period
One to three digits followed by another period
One to three digits NOT followed by a period.

Is this starting to make sense?

[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}

Well, hold your horses, we’re almost there. Let’s quickly go over some basics.

Basics

Many people, myself included, use the website Regex101 when making and testing new patterns. With this we can actually see what the Regex engine is thinking when it matches our patterns.

You can see that since I typed “text” in the Regular Expression box, it will highlight only the words that match “text”. Regex101 also includes a handy Explanation and other information on the right.

Let’s go into the different syntax you’ll be using to create Regex patterns.

Character classes [ ]

First things first, we need to learn about character classes. A character class in Regex is represented by brackets [], and it is where you put all the characters like 0-9 or A-Z that you want to match.

Quantifiers { }

A quantifier in Regex is represented by curly braces {}, which are used to express how many times you want to match your characters. For example, if you want something to match one to three times, you use {1,3}. The comma , acts as a “through” in Regex.

Escape characters \

As you may have noticed, Regex uses a lot of regular everyday characters like .[]{}/ as their pattern syntax. But what if you want to match literally something like the period .? Well, any characters like that can be matched by adding an escape character before the character. In Regex this is represented by the backslash \.

So . becomes \. to match. And, yes, you can even escape character the escape character by doing a double backslash \\.

Put these skills to work!

So that is a very basic overview of Regex. So enough talking—let’s get to work! Let’s look at our IPv4 instructions again:

One to three digits followed by a period
One to three digits followed by another period
One to three digits followed by another period
One to three digits NOT followed by a period.

For digits we’re going to make a capture group for all possible digits. This would be written as [0-9].

For one to three we’re going to use our handy quantifier to express that range. This would be written as {1,3}.

Finally, the period . is a special Regex syntax, so we’ll need to use the escape character \. This would be written as \..

Okay, now let’s just repeat that—don’t forget to omit the last period .!

Look at that!

[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}

Not so scary now, is it? So let’s put this new pattern to work. The following is a simple show ip route that, of course, contains IPv4 addresses.

Let’s drop our pattern and see what happens.

SUCCESS!! Regex with our pattern can now match any IPv4 address we throw at it!

Conclusion

Thanks for reading and spending some time learning Regex with me. This, of course, is a very basic overview that just scratches the surface. We can continue to improve our pattern to make it more and more accurate at matching IPv4 addresses. For example, 999.999.999.999 would be a valid match with our pattern, but not a valid IPv4 address. This article is just meant to be a good jumping off point for you to make your own patterns.

And I hope you do feel empowered to go out and make your own Regex patterns! There are so many novel and unique ways Network Engineers are using Regex today. One of the most popular use case is parsing the CLI outputs from our network devices. There are many open sourced projects that need custom Regex patterns for the many different network devices out in the wild. We’ll go through some of those and how you can contribute to projects like NTC Templates and Cisco’s Genie Parsers in a later post.

In the meantime, can you work out a pattern to match MAC addresses? Try it out, and let me know your solution in the comments below!

Jinja2: Assemble Strategy

2021-01-12T00:00:00+00:00

Because you are reading this post, you’re likely aware of the great power of the Jinja2 templating language. You might have turned an entire data center configuration in one huge complex template, doing crazy conditionals to render some lines and skip some others. Spines or leaves devices do not make any difference to you, as long as you have a device_role variable which helps you to build the right config. Multi-vendor is not a thing for you anymore - Juniper, Cisco, Nexus (…you name them) all together in your super_master.j2. One template to rule them all! Hundreds and hundreds of lines that do the magic and make you scream every time template error while templating string or UndefinedVariable pops out! You end up digging into the code for hours, commenting blocks of lines, and trying to run it again..and again. Rolling back git commits, making the effort to remember when the last time was that the template rendered properly. What about when you needed to add a few config lines just for a specific bunch of devices or you tried to port your template and reuse it for another project?

You suddenly realize that you need a different design to make your template more scalable so you will not go crazy every time you need to work on it. Let’s explore what options we have and what the best approach might be.

Single template

Let’s use the previous case as an example. We will leverage the Ansible rendering engine. Remember, this is a templating strategy, so the below examples are also true for all those applications which rely on Jinja as template language, such as SALT, python, etc. Based on two groups of devices - spines and leaves you could build a single base.j2 such as:

{% for iface in interfaces %}
interface {{ iface['name'] }}
 {% if 'Management' in iface['name'] %}
 vrf forwarding mgmt
 {% endif %}
 {% if iface['ip'] %}
 ip address {{ iface['ip'] }}
 {% endif %}
 {% if 'loopback' not in iface['name'] %}
 mtu {{ iface['mtu'] }}
 {% endif %}
 [...]
!
{% endfor %}
{% if 'leaves' in group_names %}
 {% for vxlan in vxlans %}
interface {{ vxlan['interface'] }}
 vxlan source-interface {{ vxlan['source_update'] }}
 vxlan udp-port {{ vxlan['port'] }}
   {% for vlan_vni in vxlan['vlan_vni'] %}
     {% for vlan, vni in vlan_vni.items() %}
 vxlan vlan {{ vlan }} vni {{ vni }}
     {% endfor %}
   {% endfor %}
 [...]
   {% endfor %}
!
 {% endfor %}
{% endif %}
{% if 'spines' in group_names %}
 {% for peer in bgp['peers'] %}
 neighbor {{ peer['name'] }} peer-group
 neighbor {{ peer['name'] }} remote-as {{ peer['remote_as'] }}
 neighbor {{ peer['name'] }} maximum-routes {{ peer['attributes']['max_routes'] }} warning-only
   {% if peer['attributes']['update_source'] %}
 neighbor {{ peer['name'] }} update-source {{ peer['attributes']['update_source'] }}
   {% endif %}
   {% if peer['attributes']['ebgp_multihop'] %}
 neighbor {{ peer['name'] }} ebgp-multihop {{ peer['attributes']['ebgp_multihop'] }}
   {% endif %}
   {% if peer['attributes']['next_hop_unchanged'] %}
 neighbor {{ peer['name'] }} next-hop-unchanged
   {% endif %}
   {% if peer['attributes']['next_hop_self'] %}
 neighbor {{ peer['name'] }} next-hop-self
   {% endif %}
 [...]
 {% endfor %}
{% endif %}

Well…that was a lot of code to read just for an interface, VXLAN, and BGP configuration. Now try to imagine building a full running-config how complex it would become. Catching an error becomes tricky. Making future implementations feels like trying to move through a minefield. Let’s see how we can improve our template design.

Include strategy

Jinja comes to the rescue with include tag.

We can break up our base.j2 in small chunks of templates and move them under the appropriate folder:

├── base.j2
├── bgp
│   └── bgp.j2
├── interfaces
│   └── interfaces.j2
└── vxlan
   └── vxlan.j2

so, our base.j2 will look like this:

{% include `interface/interface.j2` %}
 
{% if 'leaves' in group_names %}
{% include `vxlan/vxlan.j2` %}
{% endif %}
 
{% if 'spines' in group_names %}
{% include `bgp/bgp.j2` %}
{% endif %}

Much better, isn’t it? Every template is now properly organized under its own folder and file (i.e. interface under interface/interface.j2) and we included them into base template. By still applying the group conditional logic, we can have the desired config for each device. As you can see, the code becomes more readable, easier to implement (if we need to update a BGP template, we will work only on bgp.j2), easier to troubleshoot, and more portable.

Great! But what if this can be further simplified? Get ready, minimalists!

Jinja assemble strategy

Still using the include tag, we can assemble our configuration, plugging in or out parts of templates and looping through a list of includes for each group of device.

Assuming we are still using Ansible, we can group devices per role and have a group_vars folder that looks like this:

├── leaves
│   └── main.yml
└── spines
   └── main.yml

Where leaves/main.yml…

assemble:
 - 'interface/interface.j2'
 - 'vxlan/vxlan.j2'

…and spines/main.yml

assemble:
 - 'interface/interface.j2'
 - 'bgp/bgp.j2'

Defining what template we want to include in our assemble process, and taking advantage of group_vars, our base.j2 will contain just a simple for loop:

{% for item in assemble %}
{% include item %}
{% endfor %}

We can now plug in or out templates by just appending or removing a new include under our lists. Our base.j2 will loop through the lists based on group_vars and assemble the final config.

That’s all. It could not be simpler than that!

By the way, if you ever had troubles with whitespaces or indentation in Jinja (…and I am sure you have!) make sure to check this out!

Federico

Using NetBox for Ansible Source of Truth

2021-01-05T00:00:00+00:00

This content was originally posted on the Ansible Blog on 2020-12-08. Here you will learn about NetBox at a high level, how it works to become a Source of Truth (SoT), and look into the use of the Ansible Content Collection, which is available on Ansible Galaxy. The goal is to show some of the capabilities that make NetBox a terrific tool and will be using NetBox as your network Source of Truth for automation!

Source of Truth

Why a Source of Truth? The Source of Truth is where you go to get the intended state of the device. There does not need to be a single Source of Truth, but you should have a single Source of Truth per data domain, often referred to as the System of Record (SoR). For example, if you have a database that maintains your physical sites that is used by teams outside of the IT domain, that should be the Source of Truth on physical sites. You can aggregate the data from the physical site Source of Truth into other data sources for automation. Just be aware that when it comes time to collect data, then it should come from that other tool.

The first step in creating a network automation framework is to identify the Source of Truth for the data, which will be used in future automations. Oftentimes for a traditional network, the device itself has been considered the SoT. Reading the configuration off of the device each time you need a configuration data point for automation is inefficient, and presumes that the device configuration is as intended, not simply left there in troubleshooting or otherwise inadvertently left. When it comes to providing data to teams outside of the network organization, exposing an API can help to speed up gathering data without having to check in with the device first.

NetBox

For a Source of Truth, one popular open source choice is NetBox. From the primary documentation site https://netbox.readthedocs.io, “NetBox is an open source web application designed to help manage and document computer networks”. NetBox is currently designed to help manage your:

DCIM (Data Center Infrastructure Management)
IPAM (IP Address Management)
Data Circuits
Connections (Network, console, and power)
Equipment racks
Virtualization
Secrets

Since NetBox is an IPAM tool, there are misconceptions at times about what NetBox is able to do. To be clear, NetBox is not:

Network monitoring
DNS server
RADIUS server
Configuration management
Facilities management

Why NetBox?

NetBox is a tool that is built on many common Python based open source tools, using Postgres for the backend database and Python Django for the back-end API and front-end UI. The API is extremely friendly as it supports CRUD (Create, Read, Update, Delete) operations and is fully documented with Swagger documentation. The NetBox Collection helps with several aspects of NetBox including an inventory plugin, lookup plugin, and several modules for updating data in NetBox.

NetBox gives a modern UI from the point of view of a network organization to help document IP addressing, while keeping the primary emphasis on network devices, system infrastructure, and virtual machines. This makes it ideal to use as your Source of Truth for automating.

NetBox itself does not do any scanning of network resources. It is intended to have humans maintain the data as this is going to be the Source of Truth. It represents what the environment should look like.

Ansible Content Collection for NetBox

You will find the Collection within the netbox-community GitHub organization. Here you find a Docker container image, device-type library, community generated NetBox reports, and source code for NetBox itself.

If you are unfamiliar with what an Ansible Content Collection is, please watch this brief YouTube video.

The Galaxy link for the Collection is at https://galaxy.ansible.com/netbox/netbox.

The NetBox Collection allows you to get started quickly in adding information into a NetBox instance. The only requirements are to supply an API key and a URL to get started. With this Collection, a base inventory, and a NetBox environment you are able to get a Source of Truth populated very quickly.

Let’s walk through the base setup to get to a place where you are starting to use the NetBox Inventory Plugin as your Ansible inventory. First is the example group_vars/all.yml file that will have the list of items to be used with the tasks.

---
site_list:
  - name: “NYC”
    time_zone: America/New_York
    status: Active
  - name: “CHI”
    time_zone: America/Chicago
    status: Active
  - name: “RTP”
    time_zone: America/New_York
    status: Active
manufacturers:  # In alphabetical order
  - Arista
  - Cisco
  - Juniper
device_types:
  - model: “ASAv”
    manufacturer: “Cisco”
    slug: “asav”
    part_number: “asav”
    Full_depth: False
  - model: “CSR1000v”
    manufacturer: “Cisco”
    slug: “csr1000v”
    part_number: “csr1000v”
    Full_depth: False
  - model: “vEOS”
    manufacturer: “Arista”
    slug: “veos”
    part_number: “veos”
    Full_depth: False
  - model: “vSRX”
    manufacturer: “Juniper”
    slug: “vsrx”
    part_number: “vsrx”
    Full_depth: False
platforms:
  - name: “ASA”
    slug: “asa”
  - name: “EOS”
    slug: “eos”
  - name: “IOS”
    slug: “ios”
  - name: “JUNOS”
    slug: “junos”

The first step is to create a site. Since NetBox models physical gear, you install equipment at a physical location. Whether that is in your own facilities or inside of a cloud, this is a site. The module for this is the netbox.netbox.netbox_site module. A task in the playbook may be:

- name: "TASK 10: SETUP SITES"
  netbox.netbox.netbox_site:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data: "{{ item }}"
  loop: “{{ site_list }}

The next two pieces are the base to add devices to NetBox. In order to create a specific device, you also need to have the device type and manufacturer in your NetBox instance. To do this there are specific modules available to create them. Platforms will help to identify what OS the device is. I recommend that you use what your automation platform is using—something like IOS, NXOS, and EOS are good choices and should match up to your ansible_network_os choices. These tasks look like the following:

- name: "TASK 20: SETUP MANUFACTURERS"
  netbox.netbox.netbox_manufacturer:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      name: "{{ manufacturer }}"
  loop: "{{ manufacturers }}"
  loop_control:
    loop_var: manufacturer

- name: "TASK 30: SETUP DEVICE TYPES"
  netbox.netbox.netbox_device_type:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      model: "{{ device_type.model }}"
      manufacturer: "{{ device_type.manufacturer }}"
      slug: "{{ device_type.slug }}"
      part_number: "{{ device_type.part_number }}"
      is_full_depth: "{{ device_type.full_depth }}"
  loop: "{{ device_types }}"
  loop_control:
    loop_var: device_type
    label: "{{ device_type['model'] }}"

- name: "TASK 40: SETUP PLATFORMS"
  netbox.netbox.netbox_platform:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      name: "{{ platform.name }}"
      slug: "{{ platform.slug }}"
  loop: "{{ platforms }}"
  loop_control:
    loop_var: platform
    label: "{{ platform['name'] }}"

At this stage you are set to add devices and device information to NetBox. The following tasks leverage the ansible_facts that Ansible automatically gathers. So for these particular device types, no additional parsing/data gathering is required outside of using Ansible to gather facts. In this example for adding a device, you will notice custom_fields. A nice extension of NetBox is that if there is not a field already defined, you can set your own fields and use them within the tool.

- name: "TASK 100: NETBOX >> ADD DEVICE TO NETBOX"
  netbox.netbox.netbox_device:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      name: "{{ inventory_hostname }}"
      device_type: "{{ ansible_facts['net_model'] }}"
      platform: IOS  # May be able to use a filter to define in future
      serial: "{{ ansible_facts['net_serialnum'] }}"
      status: Active
      device_role: "{{ inventory_hostname | get_role_from_hostname }}"
      site: “ANSIBLE_DEMO_SITE"
      custom_fields:
        code_version: "{{ ansible_facts['net_version'] }}"

- name: "TASK 110: NETBOX >> ADD INTERFACES TO NETBOX"
  netbox.netbox.netbox_device_interface:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      device: "{{ inventory_hostname }}"
      name: "{{ item.key }}"
      form_factor: "{{ item.key | get_interface_type }}"  # Define types
      mac_address: "{{ item.value.macaddress | ansible.netcommon.hwaddr }}"
    state: present
  with_dict:
    - "{{ ansible_facts['net_interfaces'] }}"

Once you have the interfaces you can add in IP address information that is included in the ansible_facts data, I show three steps. First is to add a temporary interface (TASK 200), then add the IP address (TASK 210), and finally associate the IP address to the device (TASK 220).

- name: "TASK 200: NETBOX >> Add temporary interface"
  netbox.netbox.netbox_device_interface:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      device: "{{ inventory_hostname }}"
      name: Temporary_Interface
      form_factor: Virtual
    state: present

- name: "TASK 210: NETBOX >> ADD IP ADDRESS OF ANSIBLE HOST"
  netbox.netbox.netbox_ip_address:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      family: 4
      address: "{{ ansible_host }}/24"
      status: active
      interface:
        name: Temporary_Interface
        device: "{{ inventory_hostname }}"

- name: "TASK 220: NETBOX >> ASSOCIATE IP ADDRESS TO DEVICE"
  netbox.netbox.netbox_device:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      name: "{{ inventory_hostname }}"
      device_type: "{{ ansible_facts['net_model'] }}"
      platform: IOS
      serial: "{{ ansible_facts['net_serialnum'] }}"
      status: Active
      primary_ip4: "{{ ansible_host }}/24"

Ansible Inventory Source

At this point you have NetBox populated with all of your devices that were in your static inventory. It is now time to make the move to using NetBox as the Source of Truth for your Ansible dynamic inventory plugin. This way you don’t have to keep finding all of the projects that need to get updated when you make a change to the environment. You just need to change your Source of Truth database - NetBox.

You define which inventory plugin to use with a YAML file that defines the characteristics of how to configure your intended use of the plugin. Below is an example, showing you are able to query many components of NetBox for use within your Ansible inventory. You may wish to only make an update to your access switches? Use the query_filters key to define what NetBox API searches should be executed. Take a look at the plugin documentation for updated supported parameters on GitHub or ReadTheDocs. The compose key allows you to pass in additional variables to be used by Ansible, as such the platform from above would be used with the ansible_network_os key. This is where you see the definition and what would get passed from the inventory source.

This definition also has groups created based on the device_roles that are defined in NetBox and the platforms. So you would be able to access all platforms_ios devices or platforms_eos as an example, based on the information in the Source of Truth.

---
plugin: netbox.netbox.nb_inventory
api_endpoint: http://netbox03
validate_certs: false
config_context: false
group_by:
 - device_roles
 - platforms
compose:
 ansible_network_os: platform.slug
query_filters:
 - site: "minnesota01"
 - has_primary_ip: True

Extending NetBox with Plugins

One of the more recent feature additions to NetBox itself is the ability to extend it via your own or community driven plugins. From the wiki: “Plugins are packaged Django apps that can be installed alongside NetBox to provide custom functionality not present in the core application” GitHub Link. You can find some of the featured plugins in the community at that link. Some include:

Dynamic DNS Connector
NetBox Onboarding Plugin (from Network to Code) - This will read additional information about the device and make updates to NetBox
NetBox QR Code - Generate QR Codes about the device
SSO using SAML2

There are many plugins available to the community for you to choose from—or you can write your own add ons! Search on GitHub for the topic NetBox Plugin.

Summary

NetBox and Ansible together are a great combination for your network automation needs!

NetBox is an excellent open source tool that helps make it easy to create, update, and consume as a Source of Truth. The APIs are easy to use and make updates to the DB with, even if you did not want to use the NetBox Collection available for Ansible. Having a tool that is flexible, capable, and accurate is a must for delivering automation via a Source of Truth. NetBox delivers on each of these.

This post was inspired by a presentation done in March 2020 at the Minneapolis Ansible Meetup. For additional material on this, I have many of these tasks available as a working example on GitHub. The YouTube recording of the presentation from the Ansible Meetup is available.

-Josh

Parsing XML with Python and Ansible

2020-12-21T00:00:00+00:00

Which data type is more popular, JSON or XML? I believe the overwhelming majority will say “JSON”, which is understandable, because JSON is easier to read, easier to understand, and more human friendly. Besides that, there are a lot of reading materials around JSON and how to handle such data. But what to do when only XML is supported?

The default solution for most of us today is to convert XML to JSON or Python dictionary. But this approach has some significant drawbacks because XML and JSON are not 100% compatible. XML doesn’t have the same distinction between a list and a dictionary. Depending on what you are processing you can end up with a different datastructure if you have one element returned, an interface for example, or multiple.

There is a better way to process XML data in Python by using the native construct and libraries available. This blog is intended to give you tips and tricks on how to parse XML data with Python and Ansible.

XML Data Structure

In this section, we will briefly go over the XML Data Structure and the terminology used in it, which will help us better understand the XML data.

The image below depicts a sample XML data in a tree format.

The corresponding XML data will look like this:

<rpc-reply>
    <interface-information>
        <physical-interface type='eth'>
            <name>ge-0/0/0</name>
            <admin-status>up</admin-status>
            <oper-status>up</oper-status>
            <logical-interface>
                <name>ge-0/0/0.0</name>
                <admin-status>up</admin-status>
                <oper-status>up</oper-status>
                <filter-information>
                </filter-information>
                <address-family>
                    <address-family-name>inet</address-family-name>
                    <interface-address>
                        <ifa-local>172.16.0.151/24</ifa-local>
                    </interface-address>
                </address-family>
            </logical-interface>
        </physical-interface>
    </interface-information>
</rpc-reply>

NOTE
Some XML data will have namespaces, and we will address them a bit later.

Now let’s explain the terminology depicted in the image:

root element - The element at the root of the XML tree. XML data will always have one root element. This element can also be referred to as parent element.
child element - Any element right below the root element is a child element. The root element may contain one or more child elements.
attribute - A tag for an element providing some information about it.
data - The data the element contains.

NOTE
This is not a complete overview of the XML, but rather a quick introduction to it. The complete information about the XML can be found here.

Parsing XML with Python

Python has a very sophisticated built-in library called xml.etree.ElementTree to deal with XML data. Before we jump into the Python interpreter and start parsing the data, we will need to address XML XPath and the methods available to the xml.etree.ElementTree class.

XML XPath Support

XPath uses path expressions to find element(s) and relative data in the XML document. The xml.etree.ElementTree supports the following XPath expressions¹:

Syntax	Meaning
tag	Selects all child elements with the given tag. For example, spam selects all child elements named spam, and spam/egg selects all grandchildren named egg in all children named spam. `{namespace}` selects all tags in the given namespace, `{}spam` selects tags named spam in any (or no) namespace, and `{}*` selects only tags that are not in a namespace.
*	Selects all child elements, including comments and processing instructions. For example, `*/egg` selects all grandchildren named egg.
.	Selects the current node. This is mostly useful at the beginning of the path, to indicate that it’s a relative path.
//	Selects all subelements, on all levels beneath the current element. For example, .//egg selects all egg elements in the entire tree.
..	Selects the parent element. Returns None if the path attempts to reach the ancestors of the start element (the element find was called on).
[@attrib]	Selects all elements that have the given attribute.
[@attrib=’value’]	Selects all elements for which the given attribute has the given value. The value cannot contain quotes.
[tag]	Selects all elements that have a child named tag. Only immediate children are supported.
[.=’text’]	Selects all elements whose complete text content, including descendants, equals the given `text`.
[tag=’text’]	Selects all elements that have a child named `tag` whose complete text content, including descendants, equals the given `text`.
[position]	Selects all elements that are located at the given position. The position can be either an integer (1 is the first position), the expression `last()` (for the last position), or a position relative to the last position (e.g., `last()-1)`.

xml.etree.ElementTree Methods and Attributes

These are the methods and attributes that we will be using:

NOTE
All elements of the loaded XML data will be objects of the Element Python class and will support these methods and attributes.

Methods:

iter("<element>") - recursively iterates over all the sub-tree below the root element. Requires the element name.
findall("<xpath-expression>") - finds elements using XPath expression.
find("<element>") - finds the first child element with a particular tag.
get("<attribute-tag>") - gets the element’s attribute value.

Attributes:

tag - shows the tag name of the element (rpc-reply).
attrib - shows the value of the attribute ({'type': 'eth'}).
text - shows the data assigned to the element (ge-0/0/0.0).

Now we are ready to work with the XML data. Let’s start by opening the Python interpreter and importing the xml.etree.ElementTree class as ET for brevity.

>>> import xml.etree.ElementTree as ET

We will need to load the data now. It can be done two ways. Either load the XML file with the parse() method then obtain the root element with getroot() method, or load it from string using fromstring().

NOTE
We will be using the XML data shown in the XML Data Structure section, but with more interfaces.

>>> tree = ET.parse('interface_data.xml')
>>> root = tree.getroot()

>>> root = ET.fromstring(interface_data)

To print the tag of the root element we can use the tag attribute

>>> root.tag
'rpc-reply

Let’s iterate of the child elements of the root element and print its tag names:

>>> for child in root:
...      print(child.tag)
...
interface-information

As we can see from the output, the root element has only one child element, with interface-information tag name. To see more data, we can iterate over the interface-information element and print its elements’ tag names and attributes:

>>> for child in root:
...     for element in child:
...         print(f"Element Name: {element.tag}  Element Attribute: {element.attrib}")
...
Element Name: physical-interface  Element Attribute: {'type': 'eth'}
Element Name: physical-interface  Element Attribute: {'type': 'eth'}
Element Name: physical-interface  Element Attribute: {'type': 'eth'}
Element Name: physical-interface  Element Attribute: {'type': 'eth'}
Element Name: physical-interface  Element Attribute: {'type': 'eth'}

Even though the nested iteration works, it is not optimal, since the XML tree can have several layers. Instead, we can use iter() method and provide an element name as a parameter. In this case, we will iterate over only the found elements. For example, let’s iterate over all name elements and show their contents.

>>> for element in root.iter('name'):
...     print(element.text)
...
ge-0/0/0
ge-0/0/0.0
ge-0/0/1
ge-0/0/1.0
ge-0/0/2
ge-0/0/2.0
ge-0/0/3
ge-0/0/3.0
ge-0/0/4
ge-0/0/4.0

In the same way, we can iterate over the elements in question, but with the help of the findall() method. For it, we will need to provide XPath expression as an argument. This time, we would like to print all IP addresses that are assigned to the ifa-local element:

NOTE
The period before the forward slashes indicates that the search needs to happen from current element.

>>> for element in root.findall('.//ifa-local'):
...     print(element.text)
...
172.16.0.151/24
192.168.10.1/24
10.10.10.1/24
172.31.177.1/24
192.168.0.1/24

The XPath expression to find a particular element with a specific attribute will be a little bit different. For it, we will need to use findall(.//*[@type]) syntax to search from current element against all the elements that have an attribute tag of type:

>>> for element in root.findall('.//*[@type]'):
...     print(element.tag, element.attrib)
...
physical-interface {'type': 'eth'}
physical-interface {'type': 'eth'}
physical-interface {'type': 'eth'}
physical-interface {'type': 'eth'}
physical-interface {'type': 'eth'}

Parsing XML Data ith Namespaces

XML Namespaces are used to group certain elements identified by Uniform Resource Identifier (URI) and avoid any element name conflicts in the XML document. Detailed information about XML Namespaces can be found here.

Let’s look at an XML data with name spaces.

<rpc-reply xmlns:junos="http://xml.juniper.net/junos/20.2R0/junos">
    <interface-information xmlns="http://xml.juniper.net/junos/20.2R0/junos-interface" junos:style="terse">
        <physical-interface type='eth'>
            <name>ge-0/0/0</name>
            <admin-status>up</admin-status>
            <oper-status>up</oper-status>
            <logical-interface>
                <name>ge-0/0/0.0</name>
                <admin-status>up</admin-status>
                <oper-status>up</oper-status>
                <filter-information>
                </filter-information>
                <address-family>
                    <address-family-name>inet</address-family-name>
                    <interface-address>
                        <ifa-local>172.16.0.151/24</ifa-local>
                    </interface-address>
                </address-family>
            </logical-interface>
        </physical-interface>
    </interface-information>
</rpc-reply>

If we try to find a specific element using one of the methods that we discussed before, soon we will find out that no elements are matched.

>>> tree_xmlns = ET.parse("interface_data_with_xmlns.xml")
>>> root_xmlns = tree_xmlns.getroot()
>>> for element in root_xmlns.findall('.//ifa-local'):
...     print(element.text)
...
>>>

The reason is that each element of the interface-information child element is appended with {URI} tag. To see how it looks, we can print the tag name of the root’s child-element:

NOTE
The root element does not have the URI tag appended since it has only XML name definition rather than assignment.
>> print(root_xmlns[0].tag)
{http://xml.juniper.net/junos/20.2R0/junos-interface}interface-information
>>
Now, if we change the syntax and search for the all ifa-local elements, we should get the desired result:
>> for element in root_xmlns.findall('.//{http://xml.juniper.net/junos/20.2R0/junos-interface}ifa-local'):
...     print(element.text)
...
172.16.0.151/24
192.168.10.1/24
10.10.10.1/24
172.31.177.1/24
192.168.0.1/24
Python recommends a different way to search for elements with namespace. For that, we will need to create a dictionary and map a key to the namespace URI, then, using the desired class method, provide the dictionary as a second argument. The syntax looks like this:
>> ns = dict(interface="http://xml.juniper.net/junos/20.2R0/junos-interface")
>> for element in root_xmlns.findall('.//interface:ifa-local', ns):
...     print(element.text)
...
172.16.0.151/24
192.168.10.1/24
10.10.10.1/24
172.31.177.1/24
192.168.0.1/24
In Python 3.8 and above, the namespace can be referenced with an asterisk character(*):
>> for element in root_xmlns.findall('.//{*}ifa-local'):
...     print(element.text)
...
172.16.0.151/24
192.168.10.1/24
10.10.10.1/24
172.31.177.1/24
192.168.0.1/24

Parsing XML with Ansible

Parsing XML with Ansible can be done using two methods. The first method involves using community.general.xml module which is very well documented. The second, more interesting, one using the parse_xml filter. Here we will be showing the second option.

`parse_xml` Ansible Filter

To use the filter, we will need to create a specification file. The file has two parts: the first part defines the elements that need to be extracted from the XML data. The second part defines the variables and their values that will be available in the Ansible Playbook. We will start by looking at the first part.

keys:
  result:
    value: ""
    top: interface-information/physical-interface
    items:
      name: name
      admin_status: admin-status
      oper_status: oper-status
      ifl_name: logical-interface/name
      ip_addr: .//ifa-local

Let’s break it all down:

keys - This is a predefined root level key name.
result - The name of this key can be set to anything.
value - This is a predefined key which holds a Jinja2 variable which will be mapped to a variable that will be defined in the second part of the specification file.
top - This key holds the path to the element that will be iterated over for data extraction. Note that the path starts from the child element not the root element.
items - This is a predefined key which will hold multiple items.
- name - Key that will hold the value that is assigned to the <name></name> element.
- admin_status - Key that will hold the value that is assigned to the <admin-status></admin-status> element.
- oper_status - Key that will hold the value that is assigned to the <oper-status></oper-status> element.
- ifl_name - Key that will hold the value that is assigned to the <name></name> element which is a child elment of the <logical-interface></logical-interface> element.
- ip_addr - Key that will hold the value that is assigned to the <ifa-local></ifa-local> element. Note that here we are using XPath expression to locate the desired element.

Now let’s look at the second part of the file and break it down:

vars:
  dev_intf:
    name: ""
    admin_status: ""
    oper_status: ""
    ifl_name: ""
    ip_addr: ""

vars - This is a well-known Ansible key for defining variables.
dev_intf - The root key named by us that will hold the data extracted from the XML.
- name - Variable key that will hold the value assigned to the name key defined under items.
- admin_status - Variable key that will hold the value assigned to the admin_status defined key under items.
- oper_status - Variable key that will hold the value assigned to the oper_status defined key under items.
- ifl_name - Variable key that will hold the value assigned to the ifl_name defined key under items.

The entire specs file looks like this:

keys:
  result:
    value: ""
    top: interface-information/physical-interface
    items:
      name: name
      admin_status: admin-status
      oper_status: oper-status
      ifl_name: logical-interface/name
      ip_addr: .//ifa-local
vars:
  dev_intf:
    name: ""
    admin_status: ""
    oper_status: ""
    ifl_name: ""
    ip_addr: ""

Now let’s look at the simple Ansible Playbook in which we will use the parse_xml filter.

- name: READ XML
  connection: local
  gather_facts: no
  hosts: localhost
  tasks:
    - set_fact:
        parsed_xml_data: ""
    - debug:
        var: parsed_xml_data

Finally, let’s run the Playbook against the XML data and see the output.

$ ansible-playbook pb_parse_xml.yml

PLAY [READ XML] *******************************************************************************************************

TASK [set_fact] *******************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************
ok: [localhost] => {
    "xml_data_parsed": {
        "result": [
            {
                "admin_status": "up",
                "ifl_name": "ge-0/0/0.0",
                "ip_addr": "172.16.0.151/24",
                "name": "ge-0/0/0",
                "oper_status": "up"
            },
            {
                "admin_status": "up",
                "ifl_name": "ge-0/0/1.0",
                "ip_addr": "192.168.10.1/24",
                "name": "ge-0/0/1",
                "oper_status": "up"
            },
            {
                "admin_status": "up",
                "ifl_name": "ge-0/0/2.0",
                "ip_addr": "10.10.10.1/24",
                "name": "ge-0/0/2",
                "oper_status": "up"
            },
            {
                "admin_status": "up",
                "ifl_name": "ge-0/0/3.0",
                "ip_addr": "172.31.177.1/24",
                "name": "ge-0/0/3",
                "oper_status": "up"
            },
            {
                "admin_status": "up",
                "ifl_name": "ge-0/0/4.0",
                "ip_addr": "192.168.0.1/24",
                "name": "ge-0/0/4",
                "oper_status": "up"
            }
        ]
    }
}

PLAY RECAP ************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

I hope this post will help you understand how to parse XML data natively in Python and Ansible without converting it to JSON automatically.

-Armen

Ansible Constructed Inventory

2020-12-11T00:00:00+00:00

Ansible’s inventory management system is one of the strongest features of the platform. The options for using different inventory sources make it quite extensible. Starting with a static inventory of ini or yaml files can be fine for a small deployment or proof-of-concept, but this only scales so far. To scale beyond that, you will likely need to use a dynamic inventory plugin or script. With all the plugins that are available today, it is often better to go with one designed to work with your dynamic inventory source, but keeping in mind, that plugins are limited in their capability as-is.

In the case of the NetBox inventory plugin, there are several options out of the box by which to group hosts using different data points already available within the application (docs here):

  sites  
  site  
  tenants  
  tenant  
  racks  
  rack  
  rack_group  
  rack_role  
  tags  
  tag  
  device_roles  
  role  
  device_types  
  device_type  
  manufacturers  
  manufacturer  
  platforms  
  platform  
  region  
  cluster  
  cluster_type  
  cluster_group  
  is_virtual  
  services  

These are great options for a lot of use cases, but what if there is a need to group hosts by something that is not supported natively? Well, any inventory plugin can be extended with the functionality of the Ansible constructed builtin (docs here). Custom groupings can be created, using either the groups (which is based on a boolean) or the keyed_groups (which is based on naming groups using Jinja2 logic) parameters to dynamically assign hosts into groups… that probably does not mean much right now, but this will be demonstrated shortly by using the keyed_groups option.

Simple Example

As a simple example, perhaps it is necessary to group on the network OS, which is stored in NetBox under the platform key, but a different group name is needed. It is possible to use the keyed_groups parameter to create groups with a custom prefix, network_os in this example netbox_inventory.yml file:

keyed_groups:
  - key: platform
    prefix: "network_os"
    separator: "_"

With this configuration, since all the devices in this instance are Cisco IOS, they are in a single group, named network_os_ios

brandomando@brandomando:$ ansible-inventory -i netbox_inventory.yml --graph
@all:
  |--@network_os_ios:
  |  |--LVO-RTR-01
  |  |--RVA-RTR-01
  |  |--LVO-SWI-01
  |  |--RVA-SWI-01
  |--@ungrouped:

Here you can see that the result is equivalant to 'network_os' + '_' + platform which dynamically created a group called network_os_ios.

Slightly More Complex Jinja Filter Example

This functionality can be extended with any Python code by building a custom filter. Let’s see what that looks like.

One attribute of a device model that you may want to group on is device family, but that is not one of the natively available options, since this is not a field in NetBox. A simple filter can be used to map the device_type to a device family, which can then be used to dynamically assign hosts to their associated family group.

class FilterModule(object):

    def filters(self):
        return {
            'devicetype_family': self.devicetype_family
        }

    def devicetype_family(self, device_type):

        devicetype_map = {
            'ISR1921': 'isr1k', 
            'ISR1941': 'isr1k', 
            'ISR4321': 'isr4k', 
            'ISR4331': 'isr4k', 
            'ws-c3560cx-12pc-s': 'cat3k', 
            'ws-c3650-24ps': 'cat3k', 
        }

        return devicetype_map[device_type]

In the NetBox nb_inventory plugin configuration file, we can use the keyed_groups functionality, in combination with the custom filter, to create a dynamic inventory group assignment.

keyed_groups:
  - key: device_type | devicetype_family
    prefix: "family"
    separator: "_"

By running the keyed_groups key through the devicetype_family filter, the Ansible dynamic inventory will now have custom groups that are dynamically assigned based on the device_type parameter returned by NetBox

brandomando@brandomando:$ ansible-inventory -i netbox_inventory.yml --graph
@all:
  |--@family_cat3k:
  |  |--LVO-SWI-01
  |  |--RVA-SWI-01
  |--@family_isr1k:
  |  |--LVO-RTR-01
  |--@family_isr4k:
  |  |--RVA-RTR-01
  |--@role_rtr:
  |  |--LVO-RTR-01
  |  |--RVA-RTR-01
  |--@role_swi:
  |  |--LVO-SWI-01
  |  |--RVA-SWI-01
  |--@site_lvo:
  |  |--LVO-RTR-01
  |  |--LVO-SWI-01
  |--@site_rva:
  |  |--RVA-RTR-01
  |  |--RVA-SWI-01
  |--@ungrouped:

Compose Example

In addition to being able to group by custom logic, we can also use filters within the compose section of the NetBox inventory plugin config file to set custom variables in our rendered inventory as well. Let’s use the same filter plugin in this example.

compose:
  ansible_network_os: platform.slug
  family: device_type.slug | devicetype_family

Looking at an ansible-inventory output for one of the hosts, we can see that ansible_network_os is now dynamically assigned using the platform parameter from NetBox, and the family variable is now set by our filter.

brandomando@brandomando:$ ansible-inventory -i ./lib/netbox_inventory.yml --host LVO-RTR-01 -y
ansible_host: 10.100.0.5
ansible_network_os: ios  <--- generated from "platform.slug"
custom_fields: {}
device_type: isr4331
family: isr4k            <--- generated from "device_type.slug | devicetype_family"
is_virtual: false
manufacturer: cisco
platform: ios
primary_ip4: 10.100.0.5
regions:
- las-vegas-nv
- us-west
- united-states
- north-america
role: rtr
services: []
site: lvo
tags: []

These options are all features from the Ansible builtin constructed inventory plugin, which the NetBox community netbox.netbox.nb_inventory plugin extends.

Full Example

Demonstrating all of the components together:

plugin: netbox.netbox.nb_inventory
api_endpoint: http://localhost:8000
validate_certs: True
config_context: False

group_by:
  - device_roles
device_query_filters:
  - has_primary_ip: 'true'

keyed_groups:
  - key: platform
    prefix: "network_os"
    separator: "_"
  - key: device_type | devicetype_family
    prefix: "family"
    separator: "_"

compose:
  ansible_network_os: platform.slug
  family: device_type.slug | devicetype_family

Recap

This functionality is great for both users of the plugin and the plugin writers. The plugin writer no longer has to consider every possible use case that a user might have. The user can leverage the extensibility that is provided by the Ansible plugin framework and can be used for any number of customized groupings or variable assignments within dynamic inventories.

-Brandon

Parsing Strategies - PyATS Genie Parsers

2020-12-01T00:00:00+00:00

Thank you for joining me for Part 3 of the parsing strategies blog series. This post will dive deeper into using Cisco’s PyATS Genie library for parsing. For further reference in this blog post, I’ll be referring to the Genie library just by Genie. Genie also uses Regular Expressions (RegEx) under the hood to parse the output received from a device whether it’s semi-structured data, XML, YANG, etc. This is a key difference from other parsers, but for now, let’s stick with parsing semi-structured data.

Let’s move on and dive deeper into what the show lldp neighbors parser looks like, how it works, and how we need to modify our existing playbook to use the Genie parsers.

Genie Primer

Before we get too deep into how Genie works, you can see all the available parsers. The number of parsers has increased dramatically over the last several months and is starting to include more vendors, which is great to see.

Genie uses Python classes to build two important parsing functions:

Schema class: This class defines the schema the structured output should adhere to.
Parser class: This class defines the actual parsing methods for the specific command.

One key difference between what we’ve covered so far and Genie parsers is the ability to connect to devices and grab the necessary output, which is the default behavior, but also allows users to provide the output instead, thus working like most parsers that are separate from the device interaction.

Another key difference is the ability to use other parsing strategies within Genie such as TextFSM or Template Text Parser (TTP), but for the sake of this post, we will be covering RegEx.

You can find more detailed information on how the Genie parsers work at their developer guide.

Let’s dive into our particular parser.

"""show_lldp.py
   supported commands:
     *  show lldp
     *  show lldp entry *
     *  show lldp entry [<WORD>]
     *  show lldp interface [<WORD>]
     *  show lldp neighbors
     *  show lldp neighbors detail
     *  show lldp traffic
"""
import re

from genie.metaparser import MetaParser
from genie.metaparser.util.schemaengine import Schema, \
                                         Any, \
                                         Optional, \
                                         Or, \
                                         And, \
                                         Default, \
                                         Use

# import parser utils
from genie.libs.parser.utils.common import Common

We can see this parser is declared in the show_lldp.py module and supports several variations of the show lldp commands. It then imports re, which is the built in RegEx library. The next import is the MetaParser that makes sure that the parsers’ output adheres to the defined schema. After MetaParser is imported, the schema related imports take place that helps build the actual schema we’ll see shortly. After that, the Common class that provides helper functions is imported.

The schema class provides us insight into what the output will look like. Let’s take a look at the initial definition.

class ShowLldpNeighborsSchema(MetaParser):
    """
    Schema for show lldp neighbors
    """
    schema = {
        'total_entries': int,
        'interfaces': {
            Any(): {
                'port_id': {
                    Any(): {
                        'neighbors': {
                            Any(): {
                                'hold_time': int,
                                Optional('capabilities'): list,
                            }
                        }
                    }
                }
            }
        }
    }

We can see that ShowLldpNeighborsSchema will be a subclass of the MetaParser class imported at the beginning of the file. Within the ShowLldpNeighborsSchema class, we define our schema attribute used to make sure our output adheres to the schema before returning it to the user.

The schema is a dictionary and is expecting a total_entries key with an integer value and an interfaces key, used to define a dictionary. Each interface will be a key within the interfaces dictionary and the data obtained from the output is defined in several other nested dictionaries. Each key value pair specifies the key and the type of value it must be. There are also Optional keys not required to pass schema validation.

Now that we see the schema, we can mock up what our potential output would be.

{
    "total_entries": 1,
    "interfaces": {
        "GigabitEthernet1": {
            "port_id": {
                "Gi1": {
                    "neighbors": {
                        "iosv-0": {
                            "capabilities": [
                                "R"
                            ],
                            "hold_time": 120
                        }
                    }
                }
            }
        }
    }
}

Let’s move onto the defined parser class.

class ShowLldpNeighbors(ShowLldpNeighborsSchema):
    """
    Parser for show lldp neighbors
    """
    CAPABILITY_CODES = {'R': 'router',
                        'B': 'mac_bridge',
                        'T': 'telephone',
                        'C': 'docsis_cable_device',
                        'W': 'wlan_access_point',
                        'P': 'repeater',
                        'S': 'station_only',
                        'O': 'other'}

    cli_command = ['show lldp neighbors']

We can see the ShowLldpNeighbors class is inheriting the ShowLldpNeighborsSchema class we just covered. Now there is a mapping for the short form capabilities codes returned within the output when neighbors exist, and the long form the parser wants to return to the user.

The next defined variable is cli_command. It specifies the commands executed by the parser if no output is provided.

Let’s take look at the cli method to see what will be executed when a user specifies the cli parser for show lldp neighbors command.

Each type of output will be specified as a method under the parser class. For example, if the device returns xml, there will be an xml method that will parse and return structured data that adheres to the same schema as the cli output.

Let’s explore the code in bite size chunks to show what’s happening.

    def cli(self, output=None):
        if output is None:
            cmd = self.cli_command[0]
            out = self.device.execute(cmd)
        else:
            out = output

        parsed_output = {}

We can see the cli method takes an optional argument named output, but defaults to None. The first logic determines whether the user has provided the output or whether the parser needs to execute the command against the device. The connection the parser uses is provided by the PyATS library. This means no other library is required such as netmiko or napalm to connect to the devices.

        # Total entries displayed: 4
        p1 = re.compile(r'^Total\s+entries\s+displayed:\s+(?P<entry>\d+)$')

        # Device ID           Local Intf     Hold-time  Capability      Port ID
        # router               Gi1/0/52       117        R               Gi0/0/0
        # 10.10.191.107       Gi1/0/14       155        B,T             7038.eeff.572d
        # d89e.f3ff.58fe      Gi1/0/33       3070                       d89e.f3ff.58fe
        p2 = re.compile(r'(?P<device_id>\S+)\s+(?P<interfaces>\S+)'
                        r'\s+(?P<hold_time>\d+)\s+(?P<capabilities>[A-Z,]+)?'
                        r'\s+(?P<port_id>\S+)')

After the parsed_output variable is instantiated the next step is to define the RegEx expressions used to find the valuable data within the device output. Since the output is tabulated, which means it’s defined as a table, all values we care about will be on the same line (row) for each neighbor.

The parser uses re.compile to specify the RegEx expression ahead of time for use later in the code. Typically, re.compile is used when the same expression is used multiple times.

p1 will provide the total_entries within our schema, by using the Named Capturing Groups ability within the re library. Luckily, Cisco provide great documentation within the code to tell you what each RegEx is expecting to capture. p2 defines the RegEx used to capture the neighbor related information. We can see it uses mostly \S+ which captures any non-whitespace since the output is straight forward. But we can see the capabilities named capture group is a bit more complicated. It’s expecting at least one or more capital letter or comma, and then zero or one of that RegEx expression. This may be better explained by their example if we look at the capabilities column, it shows that it can capture a single capability, two capabilities with a comma, or zero capabilities.

Now let’s look at the remaining code to see how it uses these RegEx expressions.

        for line in out.splitlines():
            line = line.strip()

            # Total entries displayed: 4
            m = p1.match(line)
            if m:
                parsed_output['total_entries'] = int(m.groupdict()['entry'])
                continue

            # Device ID           Local Intf     Hold-time  Capability      Port ID
            # router               Gi1/0/52       117        R               Gi0/0/0
            # 10.10.191.107       Gi1/0/14       155        B,T             7038.eeff.572d
            # d89e.f3ff.58fe      Gi1/0/33       3070                       d89e.f3ff.58fe
            m = p2.match(line)
            if m:
                group = m.groupdict()

                intf = Common.convert_intf_name(group['interfaces'])
                device_dict = parsed_output.setdefault('interfaces', {}). \
                                          setdefault(intf, {}). \
                                          setdefault('port_id', {}). \
                                          setdefault(group['port_id'], {}).\
                                          setdefault('neighbors', {}). \
                                          setdefault(group['device_id'], {})

                device_dict['hold_time'] = int(group['hold_time'])

                if group['capabilities']:
                    capabilities = list(map(lambda x: x.strip(), group['capabilities'].split(',')))
                    device_dict['capabilities'] = capabilities


            continue

        return parsed_output

We can see the parser performs a for loop through the output using the splitlines method to provide a list of each line within the output. It will strip any whitespace on either side of the string.

The parser will attempt to match the p1 compiled RegEx and if it captures it, will then add total_entries to the parsed_output dictionary and then continue to the next line in the output.

If the parser didn’t capture anything for p1, it will then attempt to match p2. If a match occurs, it will then the groupdict() method to return all the named capture groups and their values as a dictionary.

We can now see the parser uses the convert_intf_name method from the Common class imported at the top of the file.

You can review the code here.

Once the interface name is converted, the parser adds the interfaces dictionary to the parsed_output variable by extracting the information captured or defaulting to an empty dictionary for any non-captured data and then assigning it to the device_dict variable.

After the device_dict is specified, it adds the hold time to it.

The next step is to strip and split the capabilities into a list and add to the device_dict variable. The parser will then continue to the next line of the output.

Once all lines are parsed, it will return the parsed_output to the user as long as it passes schema validation.

I believe this can be easier to understand than something like TextFSM since it’s written in Python and Python is a popular language among network automation engineers.

Let’s move on and review the topology again.

The Topology.. Again

Below is a picture of the lab topology we’re using to validate LLDP neighbors. It’s a simple topology with three Cisco IOS routers connected together and have LLDP enabled.

Ansible Setup.. Again

We’ve already covered most of the Ansible setup in part 2, but we’ll explain the small changes we have to make to use the Genie parsers within Ansible.

Here is a look at a host var we’ve defined as a refresher since there are no changes here.

---
approved_neighbors:
  - local_intf: "Gi0/0"
    neighbor: "iosv-1"
  - local_intf: "Gi0/1"
    neighbor: "iosv-2"

Now let’s take a look at the changes in pb.validate.neighbors.yml.

---
- hosts: "ios"
  connection: "ansible.netcommon.network_cli"
  gather_facts: "no"

  tasks:
    - name: "PARSE LLDP INFO INTO STRUCTURED DATA"
      ansible.netcommon.cli_parse:
        command: "show lldp neighbors"
        parser:
          name: ansible.netcommon.pyats
        set_fact: "lldp_neighbors"

    - name: "MANIPULATE THE DATA TO BE IN THE SAME FORMAT AS TEXTFSM TO PREVENT CHANGING FINAL ASSERTION TASK"
      set_fact:
        lldp_neighbors: "{{ lldp_neighbors | convert_data }}"

    - name: "ASSERT THE CORRECT NEIGHBORS ARE SEEN"
      assert:
        that:
          - "lldp_neighbors | selectattr('local_interface', 'equalto', item['local_intf']) | map(attribute='neighbor') | first == item['neighbor']"
      loop: "{{ approved_neighbors }}"

Using the ansible.netcommon.pyats parser requires genie and pyats to be installed via pip install genie pyats.

There are a few things to dissect with the playbook. First, we changed the parser to ansible.netcommon.pyats. Second, we added another task to manipulate the data we get back from the parser, into a similar format as the second blog post in this series so we don’t have to change the last task. I did the conversion within a custom filter plugin due to the structure of the data and the ease of handling this within Python. You will see the output below once we run our playbook.

Playbook Output

Let’s go ahead and run the playbook and see what output we get.

❯ ansible-playbook pb.validate.neighbors.yml -k -vv
ansible-playbook 2.10.3
  config file = /Users/myohman/Documents/local-dev/blog-posts/ansible.cfg
  configured module search path = ['/Users/myohman/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/myohman/.virtualenvs/3.8/main/lib/python3.8/site-packages/ansible
  executable location = /Users/myohman/.virtualenvs/3.8/main/bin/ansible-playbook
  python version = 3.8.6 (default, Nov 17 2020, 18:43:06) [Clang 12.0.0 (clang-1200.0.32.27)]
Using /Users/myohman/Documents/local-dev/blog-posts/ansible.cfg as config file
SSH password:
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: pb.validate.neighbors.yml ***********************************************************************
1 plays in pb.validate.neighbors.yml

PLAY [ios] ************************************************************************************************
META: ran handlers

TASK [Parse LLDP info into structured data] ***************************************************************
task path: /Users/myohman/Documents/local-dev/blog-posts/pb.validate.neighbors.yml:9
ok: [iosv-2] => changed=false
  ansible_facts:
    lldp_neighbors:
      interfaces:
        GigabitEthernet0/0:
          port_id:
            Gi0/1:
              neighbors:
                iosv-0:
                  capabilities:
                  - R
                  hold_time: 120
        GigabitEthernet0/1:
          port_id:
            Gi0/1:
              neighbors:
                iosv-1:
                  capabilities:
                  - R
                  hold_time: 120
      total_entries: 2
  parsed:
    interfaces:
      GigabitEthernet0/0:
        port_id:
          Gi0/1:
            neighbors:
              iosv-0:
                capabilities:
                - R
                hold_time: 120
      GigabitEthernet0/1:
        port_id:
          Gi0/1:
            neighbors:
              iosv-1:
                capabilities:
                - R
                hold_time: 120
    total_entries: 2
  stdout: |-
    Capability codes:
        (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
        (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other

    Device ID           Local Intf     Hold-time  Capability      Port ID
    iosv-1              Gi0/1          120        R               Gi0/1
    iosv-0              Gi0/0          120        R               Gi0/1

    Total entries displayed: 2
  stdout_lines: <omitted>
ok: [iosv-0] => changed=false
  ansible_facts:
    lldp_neighbors:
      interfaces:
        GigabitEthernet0/0:
          port_id:
            Gi0/0:
              neighbors:
                iosv-1:
                  capabilities:
                  - R
                  hold_time: 120
        GigabitEthernet0/1:
          port_id:
            Gi0/0:
              neighbors:
                iosv-2:
                  capabilities:
                  - R
                  hold_time: 120
      total_entries: 2
  parsed:
    interfaces:
      GigabitEthernet0/0:
        port_id:
          Gi0/0:
            neighbors:
              iosv-1:
                capabilities:
                - R
                hold_time: 120
      GigabitEthernet0/1:
        port_id:
          Gi0/0:
            neighbors:
              iosv-2:
                capabilities:
                - R
                hold_time: 120
    total_entries: 2
  stdout: |-
    Capability codes:
        (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
        (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other

    Device ID           Local Intf     Hold-time  Capability      Port ID
    iosv-2              Gi0/1          120        R               Gi0/0
    iosv-1              Gi0/0          120        R               Gi0/0

    Total entries displayed: 2
  stdout_lines: <omitted>
ok: [iosv-1] => changed=false
  ansible_facts:
    lldp_neighbors:
      interfaces:
        GigabitEthernet0/0:
          port_id:
            Gi0/0:
              neighbors:
                iosv-0:
                  capabilities:
                  - R
                  hold_time: 120
        GigabitEthernet0/1:
          port_id:
            Gi0/1:
              neighbors:
                iosv-2:
                  capabilities:
                  - R
                  hold_time: 120
      total_entries: 2
  parsed:
    interfaces:
      GigabitEthernet0/0:
        port_id:
          Gi0/0:
            neighbors:
              iosv-0:
                capabilities:
                - R
                hold_time: 120
      GigabitEthernet0/1:
        port_id:
          Gi0/1:
            neighbors:
              iosv-2:
                capabilities:
                - R
                hold_time: 120
    total_entries: 2
  stdout: |-
    Capability codes:
        (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
        (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other

    Device ID           Local Intf     Hold-time  Capability      Port ID
    iosv-2              Gi0/1          120        R               Gi0/1
    iosv-0              Gi0/0          120        R               Gi0/0

    Total entries displayed: 2
  stdout_lines: <omitted>

TASK [MANIPULATE THE DATA TO BE IN STANDARD FORMAT] *******************************************************
task path: /Users/myohman/Documents/local-dev/blog-posts/pb.validate.neighbors.yml:16
ok: [iosv-0] => changed=false
  ansible_facts:
    lldp_neighbors:
    - local_interface: Gi0/1
      neighbor: iosv-2
    - local_interface: Gi0/0
      neighbor: iosv-1
ok: [iosv-1] => changed=false
  ansible_facts:
    lldp_neighbors:
    - local_interface: Gi0/1
      neighbor: iosv-2
    - local_interface: Gi0/0
      neighbor: iosv-0
ok: [iosv-2] => changed=false
  ansible_facts:
    lldp_neighbors:
    - local_interface: Gi0/1
      neighbor: iosv-1
    - local_interface: Gi0/0
      neighbor: iosv-0

TASK [Assert the correct neighbors are seen] **************************************************************
task path: /Users/myohman/Documents/local-dev/blog-posts/pb.validate.neighbors.yml:20
ok: [iosv-0] => (item={'local_intf': 'Gi0/0', 'neighbor': 'iosv-1'}) => changed=false
  ansible_loop_var: item
  item:
    local_intf: Gi0/0
    neighbor: iosv-1
  msg: All assertions passed
ok: [iosv-2] => (item={'local_intf': 'Gi0/0', 'neighbor': 'iosv-0'}) => changed=false
  ansible_loop_var: item
  item:
    local_intf: Gi0/0
    neighbor: iosv-0
  msg: All assertions passed
ok: [iosv-1] => (item={'local_intf': 'Gi0/0', 'neighbor': 'iosv-0'}) => changed=false
  ansible_loop_var: item
  item:
    local_intf: Gi0/0
    neighbor: iosv-0
  msg: All assertions passed
ok: [iosv-0] => (item={'local_intf': 'Gi0/1', 'neighbor': 'iosv-2'}) => changed=false
  ansible_loop_var: item
  item:
    local_intf: Gi0/1
    neighbor: iosv-2
  msg: All assertions passed
ok: [iosv-1] => (item={'local_intf': 'Gi0/1', 'neighbor': 'iosv-2'}) => changed=false
  ansible_loop_var: item
  item:
    local_intf: Gi0/1
    neighbor: iosv-2
  msg: All assertions passed
ok: [iosv-2] => (item={'local_intf': 'Gi0/1', 'neighbor': 'iosv-1'}) => changed=false
  ansible_loop_var: item
  item:
    local_intf: Gi0/1
    neighbor: iosv-1
  msg: All assertions passed
META: ran handlers
META: ran handlers

PLAY RECAP ************************************************************************************************
iosv-0                     : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
iosv-1                     : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
iosv-2                     : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

I ran this playbook with some verbosity to show what each task returns and the format of our parsed data.

If we take a closer look at the output of the first task, we can see under the parsed key as well as setting the fact (lldp_neighbors), that we have our structured data from running the raw output through Genie.

The second task shows the loop for each host and the item it’s using during the loop. If you look back at our playbook, we’re using both the local_intf and neighbor for our assertions from our approved_neighbors variable.

Summary

We converted the data using a custom filter plugin, but we could have easily adjusted the facts and final assertions to align with the output we receive back from Genie. It’s also valuable to show the possibility of having a single playbook using any parser to run operational assertions. If we were in production, we could make the convert_data custom filter plugin translate several different parser formats into a parser agnostic format.

For brevity, here is our tree output to show you what the folder structure looks like to use a custom filter plugin.

❯ tree
.
├── ansible.cfg
├── filter_plugins
│   └── custom_filters.py
├── group_vars
│   ├── all
│   │   └── all.yml
│   └── ios.yml
├── host_vars
│   ├── iosv-0.yml
│   ├── iosv-1.yml
│   └── iosv-2.yml
├── inventory
└── pb.validate.neighbors.yml

Finally the contents of the custom_filters.py.

# -*- coding: utf-8 -*-

from __future__ import absolute_import, division, print_function
__metaclass__ = type


import re

def convert_genie_data(data):

    intfs = []
    for k,v in data['interfaces'].items():
        intf_name = "Gi" + re.search(r'\d/\d', k).group(0)
        intf_dict = {}
        intf_dict['local_interface'] = intf_name
        neighbor_intf = list(v['port_id'].keys())[0]
        intf_dict['neighbor'] = list(v['port_id'][neighbor_intf]['neighbors'].keys())[0]
        intfs.append(intf_dict)

    return intfs


class FilterModule:
    def filters(self):
        filters = {
            'convert_data': convert_genie_data,
        }
        return filters

I hope you enjoyed this blog post and understand a little bit more about Genie parsers and how to consume them with Ansible. The next post in this series will go over Ansible Engine parsing.

-Mikhail

Getting Started Using the Kubernetes Collection in Ansible Tower

2020-11-17T00:00:00+00:00

In this post I’ll cover setting up and working with the community.kubernetes collection in Ansible Tower. I’ll describe my experience with the initial installation of the collection on the Ansible Tower controller, discuss using a Custom Credential Type in Ansible Tower for authentication to the Kubernetes API, and cover retrieving and parsing the output from the Kubernetes cluster. Finally, I’ll provide a sample playbook to create a pod in Kubernetes and query the pod status using the collection.

While the main topic and examples are focused on using the community.kubernetes collection, much of the information here is applicable to other Ansible collections or modules as well. For example, getting Tower to recognize a collection, creating and using a Custom Credential Type, and parsing output using the json_query filter are very relevant when using any Ansible module or collection.

Overview

Ansible collections were introduced recently as a way to help scale Ansible development by allowing the maintainers of the modules to manage them outside of Ansible core. As part of this transition, the core Kubernetes modules have been migrated to the community.kubernetes collection. The modules have the same functionality and syntax as the previous core modules, but will be maintained in the collection going forward. With that being the case, it is recommended to begin utilizing the collection rather than the core modules. The goal of this post is to help the reader get started with using the collection, specifically on Ansible Tower or AWX.

Setup

First, it is necessary to install the openshift Python module which is used by the community.kubernetes modules on the Ansible Tower controller. By default Ansible Tower uses a virtual environment located at /var/lib/awx/venv/, so you must first activate that virtual environment and then use pip to install the module.

[centos@ip-10-125-1-252 ~]$ source /var/lib/awx/venv/ansible/bin/activate
(ansible) [centos@ip-10-125-1-252 ~]$ pip install openshift

The above requires elevated privileges. If you are not running as root but have sudo privileges, run the command by specifying the full path to the pip utility: sudo /var/lib/awx/venv/ansible/bin/pip install openshift

The next step is installing the collection on the Ansible Tower controller and ensuring Tower is able to find the modules in the collection. I have been able to verify two methods that work on a Tower controller running Ansible 2.9.7 and Tower 3.6.3:

Option 1: Downloading the Collection On-Demand

With this option, Ansible Tower will download the collection each time the playbook is run. Create a folder in the root of your project called collections and inside that folder include a requirements.yml file with contents such as below:

---
collections:
  - name: community.kubernetes
    version: 0.9.0

The directory structure of the project would thus be:

├── collections
    └── requirements.yml

This might not be desirable due to the delay incurred when downloading the collection, in which case you can opt for Option #2 below.

Caching of collections may remediate this in future versions of Tower/AWX. See issue #7643. Thanks to NTC vet Josh VeDeraa for the tip!

Option 2: Include a copy of the collection in your source control repository

In order to avoid having to pull the collection on each playbook run, the collection can be installed as a folder in your project. This means it will be checked into your source control repository and be tracked along with your playbooks and other files. To accomplish this, first we need to create an ansible.cfg file in the root of the repository and create the following lines:

[defaults]
collections_paths = ./

Then run the ansible-galaxy command to install a copy of the collection into an ansible_collections folder inside your project. The requirements file should have the same contents as shown above in Option #1.

ansible-galaxy collection install -r requirements.yml

This will create an ansible_collections folder at the root of your repo which contains the Kubernetes collection. The directory structure in the project will then be:

├── ansible.cfg
├── ansible_collections
│   └── community
│       └── kubernetes
└── requirements.yml

Commit the newly created ansible_collections folder into the repo, and make sure to re-sync the Project in Ansible Tower.

Avoiding the “unresolved module” problem on the Tower controller

When I originally began working on this project, I began by installing the collection locally on the Tower controller using Ansible Galaxy which would typically be the way to install collections. As it turned out, installing the module locally on the controller and then trying to get Ansible Tower to find the collections is difficult or may even be impossible, at least on our version of Tower (3.6.3). For example, you might be tempted to do this on the controller:

ansible-galaxy collection install community.kubernetes

This pulls down the collection to the Tower controller, however if you attempt to launch a template using one of the collection modules you will get an error such as:

ERROR! couldn't resolve module/action 'community.kubernetes.k8s_info'. This often indicates a misspelling, missing collection, or incorrect module path.

Even after locating where Galaxy installed the collections, /home/centos/.ansible/collections on our Tower system, and then putting this path under the collections_paths key in /etc/ansible/ansible.cfg, the modules still could not be located.

It seems like this should have worked, but at least on our version of Tower it did not. The options in the previous section were discovered only after many hours of trying to make this work, so save yourself some time!

Testing that Tower can find the module

Now that we have set things up to allow Tower to successfully find the collection, we can test it with a playbook that uses the collection. The below playbook uses the k8s_info module to gather information about the Kubernetes worker nodes.

---
- name: RETRIEVE WORKER NODE DETAILS FROM KUBERNETES NODE
  hosts: localhost
  gather_facts: no

  collections: 
    - community.kubernetes
  
  tasks:
  - name: GET WORKER NODE DETAILS
    community.kubernetes.k8s_info:
      kind: Node
      namespace: default 

The playbook called pb_test_setup.yml is checked into our GitHub repository, which is set up as a Project in Ansible Tower. It is then referenced in the below Template:

If we run this playbook from the Tower controller, it will fail because we haven’t yet defined our credentials to access the control plane. A large amount of output will be displayed, but if you scroll up to the beginning of the output for the task GET WORKER NODE DETAILS, it should show the following output:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found.

This means that the module has been found, but we cannot connect to the Kubernetes control plane yet because we haven’t defined a kube-config file which provides the credentials to do so. In the next section I’ll show how to do that by defining a Custom Credential in Ansible Tower.

Handling Credentials

To access the Kubernetes control plane, the credentials are typically stored in a Kubeconfig file which is stored locally on the user workstation in the file ~/.kube/config by default. To securely store the credentials on the Ansible Tower controller, we can load the contents of the Kubeconfig file into a Credential in Ansible Tower. Tower does not currently have a native Kubernetes credential type, but it does provide the ability to create custom Credential Types. The steps below show how to create a Custom Credential Type in Ansible Tower to support storing of the Kubeconfig contents as a Credential. First a Kubernetes Credential Type is created, and then a Credential is created using the new credential type which stores the contents of the Kubeconfig file. We’ll then apply the Credential to the Job Template.

Navigate to Administration > Credential Types and add a new credential type.
Enter the following YAML in the Input Configuration:

---
fields:
  - id: kube_config
    type: string
    label: kubeconfig
    secret: true
    multiline: true
required:
  - kube_config

This defines a single input field in the credential that will be used to store the contents of the Kubeconfig file. Setting secret to true will cause the contents to be hidden in the Tower UI once saved. The label defines a caption for the field, and setting multiline to true causes a text entry field to be created which accepts multiple lines of text.

Enter the below YAML in the Injector Configuration:
```
---
env:
  K8S_AUTH_KUBECONFIG: "{{ tower.filename.kubeconfig }}"
file:
  template.kubeconfig: "{{ kube_config }}"
```
The Injector Configuration provides the ability to store the credential data input by the user on the Tower controller system in a way that allows it to be later referenced by an Ansible playbook, such as environment or extra variables. The above Injector Configuration will create a file called kubeconfig on the controller containing the content the user enters in the kube_config field of the Credential. The contents of the kubeconfig file created on the controller are then stored inside an environment variable on the Tower controller. The modules in the community.kubernetes collection by default look for the Kubeconfig in the environment variable K8S_AUTH_KUBECONFIG.

Injector Configuration is described here in the Ansible documentation

When complete the configuration should look similar to the image below:

Navigate to Credentials and add a new Credential.
In the Credential Type, select the Kubernetes custom Credential Type that was just created. In the Kubeconfig field, copy/paste the contents of your Kubeconfig file (usually ~/.kube/config). Your configuration should look similar to the following:

Once you click Save, notice that the Kubeconfig contents are encrypted

Apply the configuration to the Job Template. Navigate to Resources > Templates and under Credentials search for the Kubernetes credential we just defined. Under Credential Type select Kubernetes.

When finished, your template should look similar to this:

We are now ready to execute the playbook in Tower. If all is working correctly, we should have a successful job completion…

In the case of our setup we were using Amazon EKS for our Kubernetes cluster, so we also had to install the AWS CLI on the Tower controller, create an AWS credential in Tower containing the AWS API keys, and apply the AWS credential to the job template.

As you can see, the job was successful but it wasn’t very exciting because we aren’t yet displaying any of the information that we collected. In the next section, we’ll look at how to display and parse the output.

Displaying and Parsing Output

In order to display data, we first need to register a variable to store the output of the task and then create a second task using the debug module to display the variable. In the below playbook, I have added the register command to the initial task to store the output of the task in a variable called node_result, and added a debug task below it to display the registered variable.

  tasks:
  - name: GET WORKER NODE DETAILS
    community.kubernetes.k8s_info:
      kind: Node
      namespace: default
    register: node_result

  - name: DISPLAY OUTPUT
    debug:
      msg: "{{ node_result }}"

When the above playbook is run, the debug task will display a large amount of data about the Kubernetes worker node(s).

{
    "msg": {
        "changed": false,
        "resources": [
            {
                "metadata": {
                    "name": "ip-172-17-13-155.us-east-2.compute.internal",
                    "selfLink": "/api/v1/nodes/ip-172-17-31-155.us-east-2.compute.internal",
                    "uid": "27a57359-4019-458e-988f-9d4984e74662",
                    "resourceVersion": "275328",
                    "creationTimestamp": "2020-09-29T18:52:11Z",
                    "labels": {
                        "beta.kubernetes.io/arch": "amd64",
                        "beta.kubernetes.io/instance-type": "t3.medium",
                        "beta.kubernetes.io/os": "linux",
                        "eks.amazonaws.com/nodegroup": "eks-nodegroup",
                        "eks.amazonaws.com/nodegroup-image": "ami-0c619f57dc7e552a0",
                        "eks.amazonaws.com/sourceLaunchTemplateId": "lt-0ac1bb9bae7ecb7f6",
                        "eks.amazonaws.com/sourceLaunchTemplateVersion": "1",
                        "failure-domain.beta.kubernetes.io/region": "us-east-2",
                        "failure-domain.beta.kubernetes.io/zone": "us-east-2a",
                        "kubernetes.io/arch": "amd64",
                        "kubernetes.io/hostname": "ip-172-17-13-155.us-east-2.compute.internal",
                        "kubernetes.io/os": "linux",
                        "node.kubernetes.io/instance-type": "t3.medium",
                        "topology.kubernetes.io/region": "us-east-2",
                        "topology.kubernetes.io/zone": "us-east-2a"
                    },
                    "annotations": {
                        "node.alpha.kubernetes.io/ttl": "0",
                        "volumes.kubernetes.io/controller-managed-attach-detach": "true"
                    }
                },
                "spec": {
                    "providerID": "aws:///us-east-2a/i-0ba92f585eebfae76"
                },
                "status": {
                    "capacity": {
                        "attachable-volumes-aws-ebs": "25",
                        "cpu": "2",
                        "ephemeral-storage": "20959212Ki",
                        "hugepages-1Gi": "0",
                        "hugepages-2Mi": "0",
                        "memory": "3977908Ki",
                        "pods": "17"
                    },
                    "allocatable": {
                        "attachable-volumes-aws-ebs": "25",
                        "cpu": "1930m",
                        "ephemeral-storage": "18242267924",
                        "hugepages-1Gi": "0",
                        "hugepages-2Mi": "0",
                        "memory": "3422900Ki",
                        "pods": "17"
                    },
                    "conditions": [
                        {
                            "type": "MemoryPressure",
                            "status": "False",
                            "lastHeartbeatTime": "2020-09-30T20:24:06Z",
                            "lastTransitionTime": "2020-09-29T18:52:11Z",
                            "reason": "KubeletHasSufficientMemory",
                            "message": "kubelet has sufficient memory available"
                        },
                        {
                            "type": "DiskPressure",
                            "status": "False",
                            "lastHeartbeatTime": "2020-09-30T20:24:06Z",
                            "lastTransitionTime": "2020-09-29T18:52:11Z",
                            "reason": "KubeletHasNoDiskPressure",
                            "message": "kubelet has no disk pressure"
                        },
                        {
                            "type": "PIDPressure",
                            "status": "False",
                            "lastHeartbeatTime": "2020-09-30T20:24:06Z",
                            "lastTransitionTime": "2020-09-29T18:52:11Z",
                            "reason": "KubeletHasSufficientPID",
                            "message": "kubelet has sufficient PID available"
                        },
                        {
                            "type": "Ready",
                            "status": "True",
                            "lastHeartbeatTime": "2020-09-30T20:24:06Z",
                            "lastTransitionTime": "2020-09-29T18:52:31Z",
                            "reason": "KubeletReady",
                            "message": "kubelet is posting ready status"
                        }
                    ],
                    "addresses": [
                        {
                            "type": "InternalIP",
                            "address": "172.17.31.155"
                        },
                        {
                            "type": "ExternalIP",
                            "address": "13.14.131.143"
                        },
                        {
                            "type": "Hostname",
                            "address": "ip-172-17-31-155.us-east-2.compute.internal"
                        },
                        {
                            "type": "InternalDNS",
                            "address": "ip-172-17-31-155.us-east-2.compute.internal"
                        },
                        {
                            "type": "ExternalDNS",
                            "address": "ec2-13-14-131-143.us-east-2.compute.amazonaws.com"
                        }
                    ],
                    "daemonEndpoints": {
                        "kubeletEndpoint": {
                            "Port": 10250
                        }
                    },
                    "nodeInfo": {
                        "machineID": "ec2475ce297619b1bcfe0d56602b5284",
                        "systemUUID": "EC2475CE-2976-19B1-BCFE-0D56602B5284",
                        "bootID": "d591d21a-46de-4011-8941-12d72cb5950e",
                        "kernelVersion": "4.14.193-149.317.amzn2.x86_64",
                        "osImage": "Amazon Linux 2",
                        "containerRuntimeVersion": "docker://19.3.6",
                        "kubeletVersion": "v1.17.11-eks-cfdc40",
                        "kubeProxyVersion": "v1.17.11-eks-cfdc40",
                        "operatingSystem": "linux",
                        "architecture": "amd64"
                    },
                    "images": [
                        {
                            "names": [
                                "ceosimage/4.24.2.1f:latest"
                            ],
                            "sizeBytes": 1768555566
                        },
                        {
                            "names": [
                                "602401143452.dkr.ecr.us-east-2.amazonaws.com/amazon-k8s-cni@sha256:400ab98e321d88d57b9ffd15df51398e6c2c6c0167a25838c3e6d9637f6f5e0c",
                                "602401143452.dkr.ecr.us-east-2.amazonaws.com/amazon-k8s-cni:v1.6.3-eksbuild.1"
                            ],
                            "sizeBytes": 282945379
                        },
                        {
                            "names": [
                                "nfvpe/multus@sha256:9a43e0586a5e6cb33f09a79794d531ee2a6b97181cae12a82fcd2f2cd24ee65a",
                                "nfvpe/multus:stable"
                            ],
                            "sizeBytes": 277329369
                        },
                        {
                            "names": [
                                "602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/kube-proxy@sha256:cbb2c85cbaa3d29d244eaec6ec5a8bbf765cc651590078ae30e9d210bac0c92a",
                                "602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/kube-proxy:v1.17.9-eksbuild.1"
                            ],
                            "sizeBytes": 130676901
                        },
                        {
                            "names": [
                                "602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/coredns@sha256:476c154960a843ac498376556fe5c42baad2f3ac690806b9989862064ab547c2",
                                "602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/coredns:v1.6.6-eksbuild.1"
                            ],
                            "sizeBytes": 40859174
                        },
                        {
                            "names": [
                                "602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/pause@sha256:1cb4ab85a3480446f9243178395e6bee7350f0d71296daeb6a9fdd221e23aea6",
                                "602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/pause:3.1-eksbuild.1"
                            ],
                            "sizeBytes": 682696
                        }
                    ]
                },
                "apiVersion": "v1",
                "kind": "Node"
            }
        ],
        "ansible_facts": {
            "discovered_interpreter_python": "/usr/libexec/platform-python"
        },
        "failed": false
    },
    "_ansible_verbose_always": true,
    "_ansible_no_log": false,
    "changed": false
  }

What if we are only looking for a subset of that data? This is where a handy filter that is available in Ansible called json_query comes in. The json_query filter can be invoked by using the pipe | symbol followed by the json_query command and passing in a query as an argument. A query can be built to parse the JSON output and return specific data that we are looking for. Let’s say we wanted to return only the IP addressing assigned to the worker node. To do this, let’s add another set of tasks to the playbook:

  - name: SET VARIABLE STORING EKS WORKER NODE ADDRESS
    set_fact:
      eks_worker_address: "{{ node_result | json_query(query) }}"
    vars:
      query: "resources[].status.addresses"
  
  - name: DISPLAY EKS WORKER ADDRESS
    debug:
      msg: "{{ eks_worker_address }}"

Here we are using the set_fact module to set another variable called eks_worker_address that will contain our filtered results. The task variable query contains the query syntax that is passed to the json_query filter above. Breaking down the query, in the previous node_result variable output it can be seen that resources is a List containing many elements. The double brackets [] next to resources in the query instructs the filter to return all elements in the list. Next, under resources we have a number of dictionary keys, one of which is the status key. The .status after resources[] will filter the results to only content under the status key. Finally we have .addresses, which is a key that references another list containing all addresses on the worker node. When the task is run, the output should look similar to this:

"msg": [
    [
        {
            "type": "InternalIP",
            "address": "172.17.13.155"
        },
        {
            "type": "ExternalIP",
            "address": "3.14.130.143"
        },
        {
            "type": "Hostname",
            "address": "ip-172-17-13-155.us-east-2.compute.internal"
        },
        {
            "type": "InternalDNS",
            "address": "ip-172-17-13-155.us-east-2.compute.internal"
        },
        {
            "type": "ExternalDNS",
            "address": "ec2-3-14-130-143.us-east-2.compute.amazonaws.com"
        }
    ]
]

What if we wanted to filter this further and just return the External DNS name of our Kubernetes worker node? To accomplish this, we can filter on the type key in the list of addresses for the “ExternalDNS” value. To do this, the query in the task can be modified as shown below:

  - name: SET VARIABLE STORING EKS WORKER NODE ADDRESS
    set_fact:
      eks_worker_address: "{{ node_result | json_query(query) }}"
    vars:
      query: "resources[].status.addresses[?type=='ExternalDNS']"

Now the output is limited to just the ExternalDNS…

{
    "msg": [
        [
            {
                "type": "ExternalDNS",
                "address": "ec2-3-14-130-143.us-east-2.compute.amazonaws.com"
            }
        ]
    ],
    "_ansible_verbose_always": true,
    "_ansible_no_log": false,
    "changed": false
}

This only scratches the surface of what is possible when parsing with the json_query filter. Behind the scenes, Ansible is using a utility called JMESPATH. The query language in JMESPATH is the same syntax that is used for the query argument passed to the json_query filter. Have a look at the documentation to learn more.

Deploying Kubernetes Resources

In this final section, I’ll discuss using the community.kubernetes k8s module to initiate deployment of a Busybox pod in Kubernetes. Busybox is a lightweight utility container that is often used to validate and troubleshoot deployments. The k8s module can be used to deploy resources into Kubernetes by launching them from a Kubernetes definition file. For those familiar with using the kubectl command line utility, this is equivalent to running the command kubectl apply -f [deployment_file]. Given that we will be deploying a pod from a definition file, we must first create the definition file:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  labels:
    app: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox
    command: ['sleep','3600']
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

The above yaml should be saved as busybox.yml. Next we create an Ansible playbook called pb_deploy_busybox.yml that will use the k8s module to apply the definition file.

---
- name: DEPLOY BUSYBOX
  hosts: localhost
  gather_facts: no
  
  tasks:

  - name: DEPLOY BUSYBOX TO KUBERNETES
    community.kubernetes.k8s:
        definition: "{{ lookup('template', 'busybox.yml') | from_yaml }}"
        state: present

Commit this to the the source control repository, and synchronize the project in Ansible Tower. The Job Template in Ansible Tower should look similar to this:

Here’s the output that will be shown when the template is launched in Ansible Tower:

PLAY [DEPLOY BUSYBOX] **********************************************************
09:44:35

TASK [DEPLOY BUSYBOX TO KUBERNETES] ********************************************
09:44:35

changed: [localhost]

PLAY RECAP *********************************************************************
09:44:37

localhost                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

From the above output we can assume the pod was deployed successfully, and we could confirm from the command line using the kubectl get pod command. However, we can also use a similar output parsing strategy as was used in the previous section to gather information about running pods. Here we’ll use the k8s_info module to report on running Kubernetes pods, and filter the output using the json_query filter. Add another playbook called pb_display_pod_info.yml.

---
- name: RETRIEVE POD DETAILS FROM KUBERNETES
  hosts: localhost
  gather_facts: no
  
  collections: 
    - community.kubernetes

  tasks:
  - name: GET POD DETAILS
    community.kubernetes.k8s_info:
      kind: Pod
      namespace: default
    register: pod_result

  - name: DISPLAY OUTPUT
    debug:
      msg: "{{ pod_result }}"

This once again yields a lot of data, on which we can use json_query to filter the output to what we want to see. Let’s add a few more tasks to filter the output.

  - name: FILTER FOR POD NAME AND STATE
    set_fact:
      filtered_result: "{{ pod_result | json_query(query) }}"
    vars:
      query: "resources[].{name: status.containerStatuses[].name, status: status.containerStatuses[].state}"
 
  - name: DISPLAY FILTERED RESULTS
    debug:
      msg: "{{ filtered_result }}"

This time we are using a slightly different syntax for the query which creates a dictionary containing the values selected from the larger JSON output produced in the previous task, and that is stored in pod_result. Within the curly brackets {}, the name key is what we chose to hold the value retrieved by referencing status.containerStatuses[].name. Similarly, the status key holds the state of the container retrieved from status.containerStatuses[].state. The final output should be similar to below.

{
    "msg": [
        {
            "name": [
                "busybox"
            ],
            "status": [
                {
                    "running": {
                        "startedAt": "2020-10-01T13:44:38Z"
                    }
                }
            ]
        }
    ]

This method of filtering and storing data can be used not only for display purposes, but also to dynamically populate variables that can later be used in sub-sequent Ansible tasks if needed.

That does it for this post, I hope it provides some useful hints to get up and running quickly automating Kubernetes with Ansible Tower!

-Matt

The NTC Mag

Why did Network to Code Fork NetBox?

Background

We need to provide world class support for our clients.

Define a strategic vision for Source of Truth that enables network automation.

Build a Network Automation App Platform.

Community Committed

Going Forward

Git History

Closing

Introducing Schema Enforcer

What is Schema Enforcer

Why use Schema Enforcer?

An Example

Validating Ansible Inventory

An Examle of Validating Ansible Variables

Another Example of Validating Ansible Vars

Using Schema Enforcer

Where Are We Going Next

Introduction to PromQL

What is Prometheus?

What is a TSDB?

Why PromQL?

Prometheus Data Model

The anatomy of a query

Metrics

Label Filtering

Functions

How fast does a counter change?

Instance vs. Range Vectors

Offsets

Can I predict the next 24 hours?

Aggregation

Conclusion

Useful Resources

Regex for Network Engineers

Sounds good! But what is Regex?

Okay, for real this time. What is Regex?

Why is it important for Network Engineers?

Basics

Character classes [ ]

Quantifiers { }

Escape characters \

Put these skills to work!

Conclusion

Jinja2: Assemble Strategy

Single template

Include strategy

Jinja assemble strategy

Using NetBox for Ansible Source of Truth

Source of Truth

NetBox

Why NetBox?

Ansible Content Collection for NetBox

Ansible Inventory Source

Extending NetBox with Plugins

Summary

Parsing XML with Python and Ansible

XML Data Structure

Parsing XML with Python

XML XPath Support

xml.etree.ElementTree Methods and Attributes

Parsing XML Data ith Namespaces

Parsing XML with Ansible

parse_xml Ansible Filter

Ansible Constructed Inventory

Simple Example

Slightly More Complex Jinja Filter Example

Compose Example

Full Example

Recap

Parsing Strategies - PyATS Genie Parsers

Genie Primer

The Topology.. Again

Ansible Setup.. Again

Playbook Output

Summary

Getting Started Using the Kubernetes Collection in Ansible Tower

Overview

Setup

`parse_xml` Ansible Filter