The NTC Mag

Monitoring Websites with Telegraf and Prometheus

2020-05-28T00:00:00+00:00

In network service delivery, the network exists to have applications ride on it. Yes, even voice is considered an application when it is riding over the top of the network. We have explored in previous posts how to get telemetry data from your network devices to get an understanding of how they are performing from a device perspective. Now, in this post, I will move on to exploring how to monitor web applications and DNS using Telegraf, Prometheus, and Grafana. Often your operations teams will receive reports of websites not working for a user or you are just looking to get some more visibility into your own web services. The following method could be used to get more insight into the network and the name resolution required for those applications.

There are also several other Telegraf inputs available including ping (ICMP) and TCP tests. As of this post in May 2020 there are 181 different input plugins available to choose from. Take a look at the Telegraf plugins for more details and explore what other plugins you may be able to use to monitor your environment.

I will not be going into the setup of these tools, as this is already covered in a previous post. The previous posts in the series are:

These posts can help you get up and running when it comes to monitoring your network devices in CLI, SNMP, and gNMI.

Blackbox exporter from Prometheus is also a valid choice for this process, and I encourage you to try both the Telegraf and Blackbox exporters in your environment.

Sequence Diagram

Telegraf Setup - HTTP Response

Telegraf has the HTTP Response plugin that does exactly what we would be looking to use for gathering metrics about a HTTP response. This lets you define the list of websites that you wish to monitor, set options for proxy, response timeout, method, any data you may want to include in the body, and various responses. Take a look at the plugin documentation for more details. Here is the configuration that is going to get setup for this demonstration:

#####################################################
#
# Check on status of URLs
#
#####################################################
[[inputs.http_response]]
  urls = ["https://www.networktocode.com", "https://blog.networktocode.com", "https://www.service-now.com"]
  method = "GET"
  follow_redirects = true

#####################################################
#
# Export Information to Prometheus
#
#####################################################
[[outputs.prometheus_client]]
  listen = ":9012"
  metric_version = 2

Upon executing this, here are the relevant Prometheus metrics that we are gathering:

# HELP http_response_content_length Telegraf collected metric
# TYPE http_response_content_length untyped
http_response_content_length{method="GET",result="success",result_type="success",server="https://blog.networktocode.com",status_code="200"} 1.791348e+06
http_response_content_length{method="GET",result="success",result_type="success",server="https://www.networktocode.com",status_code="200"} 123667
http_response_content_length{method="GET",result="success",result_type="success",server="https://www.service-now.com",status_code="200"} 478636
# HELP http_response_http_response_code Telegraf collected metric
# TYPE http_response_http_response_code untyped
http_response_http_response_code{method="GET",result="success",result_type="success",server="https://blog.networktocode.com",status_code="200"} 200
http_response_http_response_code{method="GET",result="success",result_type="success",server="https://www.networktocode.com",status_code="200"} 200
http_response_http_response_code{method="GET",result="success",result_type="success",server="https://www.service-now.com",status_code="200"} 200
# HELP http_response_response_time Telegraf collected metric
# TYPE http_response_response_time untyped
http_response_response_time{method="GET",result="success",result_type="success",server="https://blog.networktocode.com",status_code="200"} 0.371015121
http_response_response_time{method="GET",result="success",result_type="success",server="https://www.networktocode.com",status_code="200"} 0.186775794
http_response_response_time{method="GET",result="success",result_type="success",server="https://www.service-now.com",status_code="200"} 0.658694795
# HELP http_response_result_code Telegraf collected metric
# TYPE http_response_result_code untyped
http_response_result_code{method="GET",result="success",result_type="success",server="https://blog.networktocode.com",status_code="200"} 0
http_response_result_code{method="GET",result="success",result_type="success",server="https://www.networktocode.com",status_code="200"} 0
http_response_result_code{method="GET",result="success",result_type="success",server="https://www.service-now.com",status_code="200"} 0

You have several pieces that come back right away including:

content_length: How long the content is
response_code: HTTP response code
response_time: How long did it take for the request to process
result_code: This is a function of Telegraf, to take an OK response to map to 0

Telegraf - DNS Check

On top of this, I want to also show how to add in a second input. We will add in a DNS query to test the name resolution of the sites as well to verify that the DNS lookup is working as expected. This could also be extended to test and verify DNS from a user perspective within your environment.

#####################################################
#
# Check on status of URLs
#
#####################################################
[[inputs.http_response]]
  urls = ["https://www.networktocode.com", "https://blog.networktocode.com", "https://www.service-now.com"]
  method = "GET"
  follow_redirects = true

[[inputs.dns_query]]
  servers = ["8.8.8.8"]
  domains = ["blog.networktocode.com", "www.networktocode.com", "www.servicenow.com"]

#####################################################
#
# Export Information to Prometheus
#
#####################################################
[[outputs.prometheus_client]]
  listen = ":9012"
  metric_version = 2

The new section is:

[[inputs.dns_query]]
  servers = ["8.8.8.8"]
  domains = ["blog.networktocode.com", "www.networktocode.com", "www.servicenow.com"]

Based on the plugin definition we are going to define to use the Google DNS resolver. And the interesting domains that we are going to verify are blog.networktocode.com, www.networktocode.com, and the popular ITSM tool ServiceNow.

Here is what gets added to the Prometheus Client output:

# HELP dns_query_query_time_ms Telegraf collected metric
# TYPE dns_query_query_time_ms untyped
dns_query_query_time_ms{domain="blog.networktocode.com",rcode="NOERROR",record_type="NS",result="success",server="8.8.8.8"} 70.950858
dns_query_query_time_ms{domain="www.networktocode.com",rcode="NOERROR",record_type="NS",result="success",server="8.8.8.8"} 48.118903
dns_query_query_time_ms{domain="www.servicenow.com",rcode="NOERROR",record_type="NS",result="success",server="8.8.8.8"} 48.552328
# HELP dns_query_rcode_value Telegraf collected metric
# TYPE dns_query_rcode_value untyped
dns_query_rcode_value{domain="blog.networktocode.com",rcode="NOERROR",record_type="NS",result="success",server="8.8.8.8"} 0
dns_query_rcode_value{domain="www.networktocode.com",rcode="NOERROR",record_type="NS",result="success",server="8.8.8.8"} 0
dns_query_rcode_value{domain="www.servicenow.com",rcode="NOERROR",record_type="NS",result="success",server="8.8.8.8"} 0
# HELP dns_query_result_code Telegraf collected metric
# TYPE dns_query_result_code untyped
dns_query_result_code{domain="blog.networktocode.com",rcode="NOERROR",record_type="NS",result="success",server="8.8.8.8"} 0
dns_query_result_code{domain="www.networktocode.com",rcode="NOERROR",record_type="NS",result="success",server="8.8.8.8"} 0
dns_query_result_code{domain="www.servicenow.com",rcode="NOERROR",record_type="NS",result="success",server="8.8.8.8"} 0

The corresponding values gathered from the dns_query input are:

dns_query_query_time_ms: Amount of time it took for the query to respond
dns_query_rcode_value: Return code value for a DNS entry
dns_query_result_code: Code defined by Telegraf for the response

Prometheus

The configuration for Prometheus at this point has a single addition to gather the statistics for each of the websites:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'telegraf website'
    scrape_interval: 10s
    static_configs:
      - targets:
        - "localhost:9012"

When you navigate to the base page to check on how Prometheus is doing with polling the data you can get a base graph. Here you see that all three sites are appearing on the graph with respect to response time:

Grafana

What does it look like to get this information into a graph on Grafana?

Grafana - Websites

To build this chart, this is a small configuration. In the Metrics section I only put the query of http_response_response_time. With the legend I set it to {{ server }} to get the website address as the table legend.

In the visualization section, the only thing that is needs to be doneis to adjust in the Left Y Axis Unit to be seconds (s) to provide the proper Y-Axis Metric.

Grafana - DNS

This is going to be another small configuration panel, similar to the previous one. In the Metrics section the corresponding query to get response time is dns_query_query_time_ms. The legend you then set to {{ domain }} to match that of what is in the query shown above.

In the visualization section, you should use the Unit of milliseconds (ms). If you copied the panel from the Website panel, don’t forget to change this. The unit of measure is in fact different and the time scale would be off.

Summary

Hopefully this post will help you gain some insight into your environment. We have been using this process internally at Network to Code already, keeping an eye on our key services that we rely on to understand if there is an individual issue or an issue with the service. Let us know your thoughts and comments! To continue the conversation, check out the #Telemetry channel inside the Network to Code Slack. Sign up at slack.networktocode.com.

-Josh

Upgrade Your Python Project With Poetry

2020-05-19T00:00:00+00:00

Dependency management and virtual environments are integral to the Python ecosystem, yet the primary tools in use today are far from ideal. Some of the primary methods are:

Dependencies management: pip by way of requirements.txt is still the de facto solution for most of us. While this approach has worked in the past, there are limitations when it comes to guaranteeing that the same project will be consistently installed.
Virtual environments: a common setup is to use virtualenv to create your virtual environment and manually activate it using source <path to venv>/activate. While this approach works, it requires the user to know which venv needs to be activated for each project and the command to execute can be lengthy.
Code packaging: (only applicable if you need to share your code), it is common to use setuptools in a setup.py file, but this solution also has some shortcomings.

If you are using any or all of the methods described above, you should take a look at Poetry to help you manage your Python project(s). Poetry’s goal is to simplify the management of Python packaging and dependencies. Amongst other things, Poetry can help:

Manage your dependencies by replacing requirements.txt
Manage your virtualenv by simplifying the creation and activation of a virtualenv for your project
Manage your Python package by replacing setup.py
Publish your application to PyPi
Turn Python functions into command line programs
Ensure package integrity

It sounds like magic and too good to be true, but there is really nothing magical happening here. Poetry is just a modern tool implementing the best practices from Python and other tools to manage a project properly. Poetry is leveraging 2 main files:

pyproject.toml: As the main configuration file for your Python project, this file can be edited manually and Poetry also helps to manage the file. The pyproject.toml file is not specific to Poetry and is meant to be the main configuration file for your Python project and all the tools surrounding it (Poetry, Black, etc.). It was introduced to the Python community in 2016 by PEP 518 to improve how to define Python packages, but its scope has increased year over year to become the default configuration file.
poetry.lock: A lock file managed by Poetry, this file should never be edited manually. With the poetry.lock file, Poetry brings a much-needed feature to Python dependencies management where we can separately maintain the list of primary dependencies, the list of development dependencies, and the exact version of the libraries that should be installed on a system. This feature is common on other languages but it has been used infrequently in Python’s setup.py or requirements.txt in the past. If you have ever generated your requirements.txt file with pip freeze > requirements.txt to ensure that you’ll always install the same version of your dependencies, you should be familiar with the problem the lock file solves. While pip freeze works most of the time, it’s not a great solution and it’s prone to version conflicts between projects, which may require manual intervention.

If you are interested in reading more about the story behind pyproject.toml, I recommend reading this blog from Brett Cannon.

Install Poetry

To install Poetry on Mac OS, Linux or Windows(bash) the recommended method is to use the below command on your system

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python

For convenience, Poetry is also available via pip but it’s not the recommended method to install it. I usually reserve that for when I need to install it within a Docker container: pip install poetry.

Manage Python dependencies and virtual environment with Poetry

Below is a simple pyproject.toml file to keep track of the dependencies for a project named mypythonproject.
This file can either be generated manually or Poetry can help you to generate it with poetry init.

[tool.poetry]
name = "mypythonproject"
version = "0.1.0"
description = "My awesome Python project"
authors = ["NTC <info@networktocode.com>"]

[tool.poetry.dependencies]
python = "^3.6"
click = "^7.1.1"

Taking a closer look at the file, the first section [tool.poetry] contains information about the project itself and the second section [tool.poetry.dependencies] defines the list of dependencies for the project, including both the Python version and the list of external dependencies that would usually be in a requirements.txt file.

The pyproject.toml file should be at the root of your project (here it’s the only file in my directory). Poetry will automatically install all the dependencies with poetry install (this replaces for pip install -r requirements.txt, python setup.py install, pip install ., or pip install -e .)

➜  mypythonproject# ll
total 8
-rw-r--r--  1 damien  staff   203B May 13 09:17 pyproject.toml
➜  mypythonproject#
➜  mypythonproject# poetry install
The currently activated Python version 2.7.16 is not supported by the project (^3.6).
Trying to find and use a compatible version.
Using python3 (3.7.7)
Creating virtualenv mypythonproject-0zMZkBqq-py3.7 in /Users/damien/Library/Caches/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  - Installing click (7.1.2)
➜  mypythonproject#

During the installation, Poetry automatically generates the poetry.lock file to track the exact version of the dependencies that have been install on my system. If the poetry.lock file was already present, it would have installed the exact version of click defined in the lock file, instead of trying to install the latest one from PyPi.

➜  mypythonproject# ll
total 16
-rw-r--r--  1 damien  staff   606B May 13 09:43 poetry.lock
-rw-r--r--  1 damien  staff   203B May 13 09:17 pyproject.toml

➜  mypythonproject# cat poetry.lock
[[package]]
category = "main"
description = "Composable command line interface toolkit"
name = "click"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
version = "7.1.2"

[metadata]
content-hash = "1876b927e070ae12d1e9090f5ea6bcdd2bb35f09269fc2182bcb9399c5e1be2a"
python-versions = "^3.6"

[metadata.files]
click = [
    {file = "click-7.1.2-py2.py3-none-any.whl", hash = "sha256:dacca89f4bfadd5de3d7489b7c8a566eee0d3676333fbb50030263894c38c0dc"},
    {file = "click-7.1.2.tar.gz", hash = "sha256:d2b5255c7c6349bc1bd1e59e08cd12acbbd63ce649f2588755783aa94dfb6b1a"},
]

Both the pyproject.toml and the poetry.lock should be tracked in source control (git). Notice the hash values in the lock file, these values ensure the package installed locally is exactly the same as intended.

Also, during poetry install, Poetry created a new virtual environment for my project because it detected that no virtual environment was already associated with the project. Poetry is able to manage multiple environments per project and provides some commands to easily manage these virtual environments.

poetry env info to list the existing env
poetry shell to activate the default virtualenv (replaces source <path to venv>/activate, or workon <project> if you use virtualenvwrapper )
poetry run to run a command within the default virtual environment without activating it

➜  mypythonproject# poetry env info
 
Virtualenv
Python:         3.7.7
Implementation: CPython
Path:           /Users/damien/Library/Caches/pypoetry/virtualenvs/mypythonproject-0zMZkBqq-py3.7
Valid:          True

System
Platform: darwin
OS:       posix
Python:   /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7

➜  mypythonproject# poetry shell
The currently activated Python version 2.7.16 is not supported by the project (^3.6).
Trying to find and use a compatible version.
Using python3 (3.7.7)
Spawning shell within /Users/damien/Library/Caches/pypoetry/virtualenvs/mypythonproject-0zMZkBqq-py3.7
➜  mypythonproject . /Users/damien/Library/Caches/pypoetry/virtualenvs/mypythonproject-0zMZkBqq-py3.7/bin/activate
(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# 

It’s possible to disable the virtual environment management in Poetry with poetry config virtualenvs.create false if you want to manage your virtual environment on your own or if you don’t want to use a virtual environment at all.

Add a new dependency to your project

Poetry provides a method to easily install and track a new dependency for your project: poetry add <python package>

In the example below, I’m adding jinja2 as a dependency to my project:

(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# poetry add jinja2
Using version ^2.11.2 for jinja2

Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 2 installs, 0 updates, 0 removals

  - Installing markupsafe (1.1.1)
  - Installing jinja2 (2.11.2)

Poetry automatically updated the pyproject.toml and the poetry.lock file in the process:

[tool.poetry]
name = "mypythonproject"
version = "0.1.0"
description = "My awesome Python project"
authors = ["NTC <info@networktocode.com>"]

[tool.poetry.dependencies]
python = "^3.6"
click = "^7.1.1"
jinja2 = "^2.11.2"

Poetry can also maintain a list of dependencies specific to your development environment. To add a new dependency to the development dependencies list you need to add the option -D: poetry add -D pytest. This will create a new section [tool.poetry.dev-dependencies] in the pyproject.toml file.

[tool.poetry.dependencies]
python = "^3.6"
click = "^7.1.1"
jinja2 = "^2.11.2"

[tool.poetry.dev-dependencies]
pytest = "^5.4.2"

Managing Python package with Poetry

As mentioned in the introduction, Poetry can also manage your Python package.
By default, Poetry will look for a directory with the name of the project and it will try to install it. In my example, since my project is named mypythonproject in the pyproject.toml, Poetry will automatically look for a directory with this name and install it.

I created a very simple file named cli.py in the directory mypythonproject

# mypythonproject/cli.py 

def main():
    print("hi there")

Here is how the project looks on my file system.

(mypythonproject-0zMZkBqq-py3.7)  ➜  mypythonproject#
.
├── mypythonproject
│   └── cli.py
├── poetry.lock
└── pyproject.toml

Running poetry install again will automatically install the delta between the pyproject.toml file and my environment, here the only delta is the library mypythonproject itself.

(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# poetry install
Installing dependencies from lock file

No dependencies to install or update

  - Installing mypythonproject (0.1.0)

Once installed, I can access my code from anywhere as long as I’m still within the same virtual environment. In the example below, I moved outside of the project directory and imported the function main() in Python with from mypythonproject.cli import main

(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# cd /
(mypythonproject-0zMZkBqq-py3.7) ➜  /
(mypythonproject-0zMZkBqq-py3.7) ➜  / python
Python 3.7.7 (default, Mar 10 2020, 15:43:33)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from mypythonproject.cli import main
>>> main()
hi there

We can also check the list of installed packages within the virtual environment with pip list:

(mypythonproject-0zMZkBqq-py3.7) ➜  / pip list | grep mypythonproject
mypythonproject       0.1.0      /Users/damien/projects/mypythonproject

If the name of your directory does not match the name of your project, you need to tell Poetry from which directory to install using packages key as part of the main [tool.poetry] section of the pyproject.toml:

[tool.poetry]
name = "mypythonproject"
version = "0.1.0"
description = "My awesome Python project"
authors = ["NTC <info@networktocode.com>"]
packages = [
    { include = "mylibraryname" },
]

Creating command line programs with Poetry

Another feature that is extremely useful in Poetry is the ability to easily turn a Python function into an executable/program that will be available in your PATH.

Building on the previous example, I can convert my function main() into a CLI tool with if __name__ == "__main__":. At this point I can execute it as a script as long as I know its exact location.

def main():
    print("hi there")

if __name__ == "__main__":
    main()

(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# python mypythonproject/cli.py
hi there

By leveraging [tool.poetry.scripts] feature, I can automatically turn my function main() into an executable, here called myawesomecli:

[tool.poetry.scripts]
myawesomecli = "mypythonproject.cli:main"

After reinstalling the library with poetry install, I now have access to a new executable myawesomecli:

(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# myawesomecli
hi there

(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# which myawesomecli
/Users/damien/Library/Caches/pypoetry/virtualenvs/mypythonproject-0zMZkBqq-py3.7/bin/myawesomecli

Conclusion

I hope this introduction to Poetry convinced you to give it a try, I know it’s hard to change our habits when it comes to tools and development environment sometimes. I wish I had tried Poetry a long time ago instead of waiting months before transitioning.
Poetry actually does even more than what we covered in this article, so I encourage you to check out the official documentation!

-Damien (@damgarros)

NetDevOps Concepts - An Introduction

2020-05-12T00:00:00+00:00

As the networking industry moves to embrace NetDevOps, there are often many terms used that network practitioners are unfamiliar with. This can lead to many people feeling unsure about what role a new technology could play in their organization, or even being afraid to ask for clarity about a discussed concept. In fact, there are a few questions we see asked time and time again.

“What is CI/CD, why would it have a pipeline, and what does that have to do with my network?”

“What does IaC or SoT mean?”

Or even, “What is NetDevOps anyway and why should I care?”

In this series of blog posts, we will delve into NetDevOps in a way that ensures you have familiarity and a basic understanding of what NetDevOps is, as well as some of the key components. We will also talk briefly about how, and why, you might apply a some of these concepts to operating your network.

What is NetDevOps?

Many fantastic blog posts and presentations discussing what NetDevOps is as a philosophy exist, and I won’t revisit all of their points here. Suffice to say that NetDevOps brings key concepts from the DevOps movement, which we’ll talk about deeply in subsequent blog posts, and applies them to operating and building networks.

With the proper application of NetDevOps concepts, you no longer have to think about your network as static, rigid, and fragile. Instead, you can start to treat it as something flexible and responsive to your desires or business needs. Some examples of these concepts include:

Continuous Integration/Continuous Deployment (CI/CD)
Infrastructure as Code (IaC)
Source of Truth (SoT)
Minimum Viable Product (MVP)
ChatOps

As for seeing these concepts in action, let us say that you need to deploy a new site to your corporate VPN. Without a NetDevOps approach to managing your network, you could spend a week fighting by hand to build the proper IKE/ISAKMP settings for the tunnel and documenting the site in your systems, applying changes, and then testing to ensure everything is functional.

However, if you are using a few key NetDevOps concepts to manage your network, your process could look more like this:

Message new vpn site to @my_netdevops_bot in your chat application (Slack/MS Teams/Webex Teams)
Fill in the site name, equipment type, and other metadata as prompted by the bot.
@my_netdevops_bot uses that information to:
- Create the site in your Source of Truth (SoT)
- Merge a Jinja template with the relevant data from your SoT
- Present you with the rendered configuration in chat application
Upon approval of the change, @my_netdevops_bot will connect to the devices, and deploy these configuration changes
@my_netdevops_bot then queries the network (or other systems such as monitoring systems) to determine if the change was successful and present you with confirmation in chat.
- If the change failed for some reason, @my_netdevops_bot can then start to troubleshoot some basic things on your behalf and present that information to you as well.

This is not a far-fetched or cutting edge example, this is just one of the many ways in which we see the application of NetDevOps concepts on a daily basis across all types of organizations.

“But…”

Usually, about this point in the conversation, there is a “But…” lurking off in the wings. Some greatest hits include, “But my network is too old” or “But my network is too unique.” My personal favorites are, “But my network is too large/too small!” as if there was a mythical Goldilocks zone where these concepts apply. The list goes on, but for almost every exception raised, there is an answer. Most of these can be addressed by two main responses.

First, no one is suggesting that you go directly from zero to the scenario described above. Embracing a NetDevOps mentality and approach to your network operations is an iterative process. Applying just a few of these concepts (and not all of them) to a single small element of your network is the best way to begin your NetDevOps journey. Even small, simple steps can pay large dividends. If you are manually generating device configurations, start to look at building a templating system for your device configurations and keeping it as code in a Git repository. This first step of only generating the configurations, leaving the engineer to actually qualify and apply them, can pave the way for many other steps in the future. Even in and of itself, simple configuration generation from a template can provide time savings and operational wins by removing human error and ensuring there is a single standard configuration.

Secondly, no network is too special to take advantage of these concepts. Is your network very large and complicated? Is literally every single component unique? Pick a single place in your network to start implementing a Source of Truth (such as NetBox). Having the data in an easily accessible API and/or GUI, and in a known standard format, will show value immediately for any other concepts you wish to apply. Does your network lack coherent standards? Start to enforce them on a single item of configuration, such as the AAA configuration, via Infrastructure as Code principals. Ensuring that the AAA configuration on all devices is a known/expected value at all times is extremely valuable from an operational and security perspective. Your compliance auditors will thank you!

Each of the above strategies relies on the concept of a Minimum Viable Product. When you start to embrace NetDevOps concepts for operating your network, do not expect to solve all of your problems in the first try. Pick one straightforward problem to solve. Build your solutions and tooling to solve that problem, then begin using it in your day-to-day work. If it is successful, look at adapting your tools to solve additional problems. As described above, perhaps you tackle only the AAA configuration first. When that is fully managed in a Source of Truth, and as code instead of raw device configuration, you can expand to managing the SNMP configuration on all your devices as well. Next, you could tackle management ACLs to your devices or NTP and DNS settings, really it’s up to what makes sense in your network. In this way you will rapidly expand the portions of your network that are being managed via NetDevOps.

Next Steps

This is the first in a series of posts on NetDevOps concepts, and I highly recommend staying tuned for the subsequent posts where we dive into some of these concepts in greater detail. We’ll have use cases and examples for each of them, including how to tie these all together into a coherent network automation strategy in the end.

If you’re impatient, you can always reach out to info@networktocode.com for more information on our services and training to help you take the next steps on your NetDevOps journey.

-Brett

Monitor Your Network With gNMI, SNMP, and Grafana

2020-05-05T00:00:00+00:00

This post, the second in a series focused on using Telegraf, Prometheus, and Grafana for Network Telemetry, will focus on transforming data and making additional graphs within Grafana. This post will cover the following topics:

Telegraf
- Gathering streaming data with gNMI, as an alternative to SNMP
- Changing data with Enum and Replacement
- Tagging Data
Prometheus
- Prometheus Query Language (PromQL)
Advancing Your Grafana Capabilities
- Variables
- Tables (BGP Table)
- Device Dashboards vs Environment Dashboards

Here is where you can find the first post in the series on how to gather data from SNMP based devices.

Purpose

The intent of this post is to demonstrate how to bring multiple telemetry gathering methods into one. In our experience, a successful telemetry & analytics stack should be able to collect data transparently from SNMP, telemetry Streaming (gNMI) and CLI/API. We covered SNMP and CLI gathering in previous posts. This post will focus on gathering telemetry data with gNMI. Beyond the collection of data, when we are collecting the same type of data from multiple sources it’s important to ensure that the data will have the format in the database. In this post, we’ll look at how Telegraf can help normalize and decorate the data before sending it to the database.

Network Topology

In the topology there is a mix of devices per the table below:

Device Name	Device Type	Telemetry Source
houston	Cisco IOS-XE	SNMP
amarillo	Cisco NXOS	SNMP
austin	Cisco IOS-XR	gNMI
el-paso	Cisco IOS-XR	gNMI
san-antonio	Cisco IOS-XR	gNMI
dallas	Cisco IOS-XR	gNMI

This blog post was created based on a Cisco-only environment, but if you’re interested in a multi-vendor approach check out @damgarros’s NANOG 77 presentation on YouTube. That video shows how to use only gNMI to collect data from Arista, Juniper, and Cisco devices in a single place. This topology used here is meant to show the collection from multiple sources (SNMP + gNMI) in one.

Application Installs Note

Software installation was covered in the previous post in this series, and I recommend taking a look at either that post for the particular installation instructions, or heading over to the product page referenced in the introduction.

Overview

Here is the sequence of events that is being addressed in this post. I am starting with Telegraf gathering and collecting gNMI data from network devices. This is being processed into Prometheus metrics that will be scraped by a Prometheus server. Then Grafana will generate graphs on the data that is gathered and processed appropriately.

gNMI Introduction

gNMI stands for gRPC (Remote Procedure Calls) Network Management Interface. gRPC is a standard developed by Google that leverages HTTP/2 for transport using Protocol Buffers. gNMI is a gRPC-based protocol to get configuration and telemetry from a network device. All messages are defined as protocol buffers that intend to keep data as small as possible in the definition to be as efficient as possible. The data is serialized into the proper format by the device and sent off. This can hold quite a bit of information and is read by the receiver. You can take a look at the gNMI reference for more detailed information.

gNMI can handle not only telemetry data that this post is about, but also is intended to transport configuration about the device as well.

So why use gNMI? gRPC is incredibly fast and efficient at transmitting data, and by extension gNMI is also fast and efficient.

gNMI Cisco Configuration

gNMI is supported by many of today’s leading network vendors. As an example for configuring a Cisco IOS-XR device here are the configuration lines needed to enable gNMI in this demo environment:

grpc
 port 50000
 no-tls

Pretty straight to the point. If you wish to create a subscription model within the Cisco IOS-XR there are some more detailed configuration options available. Take a look at Cisco’s Guide to Configure Model-driven Telemetry

Telegraf

Gathering Streaming Data With gNMI

The first step that I will be walking through is setting up Telegraf to subscribe to gNMI data. This is specifically to collect telemetry data from IOS-XR devices in this lab scenario. With gNMI, like other streaming Telemetry subscriptions, you need to tell the network device that you want to subscribe to receive the data. The device will then send the periodic updates of telemetry data to the receiver. There is a periodic “keep-alive” message sent to keep the subscription active by the subscriber.

gNMI Telegraf Configuration

Telegraf has a plugin that will take care of the subscription and the input section looks like the code below. Note that the subscription port is defined within the addresses section.

[[inputs.cisco_telemetry_gnmi]]
    addresses = ["dallas.create2020.ntc.cloud.tesuto.com:50000"]
    username = <redacted>
    password = <redacted>

    ## redial in case of failures after
    redial = "10s"
    tagexclude = ["openconfig-network-instance:/network-instances/network-instance/protocols/protocol/name"]

    [[inputs.cisco_telemetry_gnmi.subscription]]
        origin = "openconfig-interfaces"
        path = "/interfaces/interface"

        subscription_mode = "sample"
        sample_interval = "10s"

    [[inputs.cisco_telemetry_gnmi.subscription]]
        name = "bgp_neighbor"
        origin = "openconfig-network-instance"
        path = "/network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/state"

        subscription_mode = "sample"
        sample_interval = "10s"

[[outputs.prometheus_client]]
  listen = ":9011"

The configuration shows that you define the address, username, and password. This configuration also shows a redial setup in case of a failure and particular subscriptions to be excluded from the request.

There are two subscriptions that we are subscribing to in this instance:

openconfig-interfaces
openconfig-network-instance (To collect BGP neighbor state)

In each of these cases the sampling will be every 10 seconds in this demo, which means that the device will send the statistics every 10 seconds. Every 10 seconds there will be new metrics available to be scraped by Prometheus. The sample interval and Prometheus scrape interval should be the same interval.

To collect the telemetry for this demo we are once again using the Prometheus client output from Telegraf. Telegraf will collect, process, and format the data that will then be scraped by a Prometheus server. Let’s take a look at what that output looks like next.

gNMI Output - BGP

I’m only going to take a look at a few of the items in the output here. There are too many that would fill up too much real estate in your screen to make it worthwhile.

# HELP bgp_neighbor_messages_received_UPDATE Telegraf collected metric
# TYPE bgp_neighbor_messages_received_UPDATE untyped
bgp_neighbor_messages_received_UPDATE{device="dallas",identifier="BGP",name="default",neighbor_address="10.0.0.1",peer_type="EXTERNAL",role="leaf"} 9
bgp_neighbor_messages_received_UPDATE{device="dallas",identifier="BGP",name="default",neighbor_address="10.0.0.17",peer_type="EXTERNAL",role="leaf"} 0
bgp_neighbor_messages_received_UPDATE{device="dallas",identifier="BGP",name="default",neighbor_address="10.0.0.25",peer_type="EXTERNAL",role="leaf"} 9
bgp_neighbor_messages_received_UPDATE{device="dallas",identifier="BGP",name="default",neighbor_address="10.0.0.9",peer_type="EXTERNAL",role="leaf"} 9

Some items were removed to assist in readability and message delivery

The output is what you would expect. A list of the neighbors identified by the neighbor_address key in the tags. With the BGP subscription you get:

bgp_neighbor_established_transitions
bgp_neighbor_last_established
bgp_neighbor_messages_received_NOTIFICATION
bgp_neighbor_messages_received_UPDATE
bgp_neighbor_messages_sent_NOTIFICATION
bgp_neighbor_messages_sent_UPDATE
bgp_neighbor_peer_as
bgp_neighbor_peer_as
bgp_neighbor_queues_output
bgp_neighbor_session_state

gNMI Output - Interfaces

There are a lot of statistics sent back with the interface subscription. We’ll be taking a look at just one of them, interface_state_counters_in_octets, in this instance. We get a look at each interface and its associated counter in the data.

interface_state_counters_in_octets{device="dallas",name="GigabitEthernet0/0/0/0",role="leaf"} 3.2022595e+07
interface_state_counters_in_octets{device="dallas",name="GigabitEthernet0/0/0/1",role="leaf"} 3.077077e+06
interface_state_counters_in_octets{device="dallas",name="GigabitEthernet0/0/0/2",role="leaf"} 1.5683204947e+10
interface_state_counters_in_octets{device="dallas",name="GigabitEthernet0/0/0/3",role="leaf"} 1.627459e+06
interface_state_counters_in_octets{device="dallas",name="GigabitEthernet0/0/0/4",role="leaf"} 1.523158e+06
interface_state_counters_in_octets{device="dallas",name="GigabitEthernet0/0/0/5",role="leaf"} 35606
interface_state_counters_in_octets{device="dallas",name="GigabitEthernet0/0/0/6",role="leaf"} 35318
interface_state_counters_in_octets{device="dallas",name="GigabitEthernet0/0/0/7",role="leaf"} 35550
interface_state_counters_in_octets{device="dallas",name="GigabitEthernet0/0/0/8",role="leaf"} 35878
interface_state_counters_in_octets{device="dallas",name="GigabitEthernet0/0/0/9",role="leaf"} 36684
interface_state_counters_in_octets{device="dallas",name="MgmtEth0/RP0/CPU0/0",role="leaf"} 2.2033861e+07
interface_state_counters_in_octets{device="dallas",name="Null0",role="leaf"} 0
interface_state_counters_in_octets{device="dallas",name="SINT0/0/0",role="leaf"} 0

This is great information, and we have seen something similar with SNMP. Now to the transformations that Telegraf offers.

Changing data with Enum and Replacement

Telegraf has a couple of different processors available to process the data and get it into a format that is appropriate and consistent for your environment. Let’s take a look at a couple of them and how they are used in the use case here.

Telegraf - Enum

The first processor used is within the BGP data collection. When the data comes back from the subscription for a BGP session state, it comes back as a string value. It is great to be able to read the current state, but is not very helpful for a Time Series Data Base (TSDB). A TSDB is looking to get the data back represented as a number of some sort, either an integer or a float. The whole point is to measure information at a point in time.

The Telegraf process then looks like this:

To accommodate this, the use of the enum processor is put into action. The following is added to the configuration:

[[processors.enum]]
  [[processors.enum.mapping]]
    ## Name of the field to map
    field = "session_state"

    [processors.enum.mapping.value_mappings]
      IDLE = 1
      CONNECT = 2
      ACTIVE = 3
      OPENSENT = 4
      OPENCONFIRM = 5
      ESTABLISHED = 6

Within the session_state any instances of the string IDLE will be replaced with the integer 1. This is then set up to store for the long term within a TSDB. This is the same then for all of the rest of the states as well, with ESTABLISHED states stored as the integer 6. Later in Grafana this number will be reversed into the word for representation on a graph.

Telegraf - Rename

The second processor that is used in this demo is the rename processor. This rename processor has a function to replace items. Below is what is used to rename the SNMP counters that are collected for SNMP devices and moved to match the names for gNMI.

[[processors.rename]]
  [[processors.rename.replace]]
    field = "ifHCInOctets"
    dest = "state_counters_in_octets"

  [[processors.rename.replace]]
    field = "ifHCOutOctets"
    dest = "state_counters_out_octets"

This states that if looking for ifHCInOctets - replace the field with state_counters_in_octets. And the same for the outbound with ifHCOutOctets replacing with state_counters_out_octets. Once Telegraf has replaced those fields, you can use the data gathered with SNMP and that with gNMI in the same queries!

Tagging Data

Tagging data is one of the biggest favors that you can do for yourself. Tagging gives flexibility for future analysis, comparison, and graphing data points. For instance if you tag your BGP neighbors with the upstream peer provider, you will be able to easily identify the interfaces which belong to that particular peer. If you have four geographically diverse interfaces, this will allow you to quickly identify the interfaces based on the tag rather than manually deciding later at the time of graphing or alerting.

This brings us to the third Telegraf prcoessor in this post regex processor. This processor will take a regex search pattern, and complete the replacement. Something new here is that if you use the result_key option, a new field will be created and not replace what is there, resulting in a whole new field. This regex replacement will add a new tag for intf_role using server as the definition.

  [[processors.regex.tags]]
    key = "name"
    pattern = "^GigabitEthernet0\\/0\\/0\\/2$"
    replacement = "server"
    result_key = "intf_role"

Looking at just this particular replacement in the output, there are now additional tags for graphing, alerting, and general data analysis.

interface_state_admin_status{device="dallas",intf_role="server",name="GigabitEthernet0/0/0/2",role="leaf"} 1
interface_state_counters_in_broadcast_pkts{device="dallas",intf_role="server",name="GigabitEthernet0/0/0/2",role="leaf"} 8

Prometheus

Prometheus Query Language

Throughout the upcoming Grafana section you will get to see a number of PromQL (Prometheus Query Language) queries. Take a look at the Prometheus.io basics page to get full documentation of the queries that are available. It is these queries that are being executed that will be used by Grafana to populate the data in the graphs.

Grafana

Through the next several sections you will get to see how to build a dashboard using PromQL and variable substitution, among other topics to build these dashboards on a per device basis. From a device perspective dashboard, these two dashboards look different in the number of interfaces and neighbors displayed, but they are born out of the same dashboard configuration.

Variables

First, you’ll need to set up the variable device that is seen on the upper left hand corner of the dashboard. When I first started building dashboards I remember that this may be one of the most important skills when looking to level up your Grafana dashboards, as it will allow you to get significant amount of value while reducing re-work to keep adding additional devices into a panel.

Variables - Adding to Your Dashboard

To add a dashboard wide variable follow these steps:

Navigate into your dashboard
Click on the gear icon in the upper right hand navigation section
Click on Variables
Click the green New button on the right hand side

This image already had a variable added, which is the devices

Once in the new screen you will have the following image:

Here you will be defining a PromQL query to build out your device list. In the bottom section of the screen you see the heading of Preview of values. Here you will be able to observe a sample of what the query will result in for your variables.

The fields that you need to fill in include:

Field	Information Needed
Name	Name of the variable you wish to use
Type	Query
Data source	Prometheus
Refresh	When would you like to refresh the variables? Use the dropdown to select which fits your org best
Query	PromQL to get the data points
Regex	Regex search to reduce the search results

You can experiment with the rest of the fields as you see fit to get your variables defined properly.

Once you have your search pattern set, make sure to click Save on the left hand side of Grafana.

To reference the variables once they are created, you use the dollar sign ($) in front of the name for Grafana to execute that as a variable within a query. Within the Legend area the use of Jinja-like formatting of the double curly braces will identify as a variable.

Grafana Plugins

Grafana is extensible via the use of plugins. There are quite a few plugins available for Grafana and I encourage you to take a look at the plugin page to be able to search for what you may want to use on your own. There are 3 types of plugins: Panel, Data Source, and App to help extend the Grafana capabilities.

Grafana Discrete Plugin (BGP State Over Time)

The next table to take a look at is using a feature within Grafana that allows you to add plugins. I’ll look at how we were able to build out the graph. This can help to identify issues quickly in your environment by just looking at the dashboard. Take a look at this where there was a BGP neighbor that was down. It is quickly identifiable on the dashboard and that action will be needed.

The two panels in the top row are using a Grafana plugin called Discrete. This provides data values in the color that is defined within the configuraiton over time. The panel then gives you the ability to hover over to see the changes over time. You install the plugin with the grafana-cli ccommand:

grafana-cli plugins install natel-discrete-panel
sudo systemctl restart grafana

Once installed you can setup a new panel with the panel type of Discrete.

The panel will be created with the following parameters

BGP Session Status - Discrete Panel 1

Query: Prometheus data source

Key	Value
Metrics	bgp_neighbor_session_state{device=”$device”}
Legend	{{ device }} {{ neighbor_address }}
Min step
Resolution	1/1
Fromat	Time series
Instant	Unchecked

You will note that in the Metrics section the variable reference is $device, noted by the dollar sign in the device name. The Legend has two variables included in both the device and neighbor_address within the Legend. This is what gets displayed in the discrete table for each line.

Critical Interface State - Discrete Panel 2

Now because the interfaces have been assigned a label, a discrete panel can be generated to show the interface state, along with the role. For demonstration, we are naming this panel Critical Interfaces, the interfaces for Servers or Uplinks to other network devices have been labeled as with server or uplink accordingly. By querying for any role we can get thi information into the panel. The legend has the value of {{device}} {{name}} > {{intf_role}} > {{neighbor}} to get to the appropriate mappings that are to be shown. This is the resulting panel:

To get to this panel we can see the following discrete panel settings. This panel build is a little bit smaller, but gets a lot of information added into a panel!

Device Dashboards vs Environment Dashboards

This is not a pick one over the other segment, rather this is saying that both should be present in your Grafana Dashboard setup.

In this post I have gone through and shown a lot of device-specific panels. The value here is that you are able to get to a device by device-specific view very quickly, without having to create a separate page for each and every device in your environment. That said, that the panels can be expanded by the use of variables to identify individual devices.

You should also look at using an environment dashboard where you are getting specific pieces of information to match your need. Need to know what an application performance looks like that includes Network, Server, Storage, and Application performance? You can work to build out these dashboards by hand, but this will take longer to build. As you leverage tags in the gathering of telemetry into your TSDB, you will be on your way to building dashboards in an automated fashion to get the big picture very quickly.

Conclusion

Hopefully this has been helpful. Again, check out the first post in the series if you need more information on these tools generally. In the next post, I will cover how to advance your Prometheus environment with monitoring remote sites and a I’ll discuss a couple of methodologies to enable alerting within the environment.

The next post will include how to alert using this technology stack.

-Josh

The State of Network Operation Through Automation / NetDevOps Survey 2019

2020-04-28T00:00:00+00:00

Network automation has become prevalent in the network industry over the last few years and yet we have little data on the state of the market today. There is a lot of discussion about Ansible and Python but beyond that there is not a good source for those seeking to understand what tools are being used by different companies, what operations people are automating the most/least, or even how long it is taking on average to learn network automation.

The NetDevOps Survey project was started in 2016 to address these questions and more. The idea was to start a survey about the network automation industry to help bring clarity to these questions. Network automation is deeply rooted in open source, and it was decided to make the project open and collaborative, following the best practices from open source projects. The intention was to have the survey be both anonymous and vendor neutral.

When the initiative started in 2016, 20 of us came together to define the first set of questions. At the time, I was working at Juniper and Jason Edelman was on the early days of Network to Code, but we worked together collaboratively on the project.

After a few years of inactivity, the second edition of the survey was released in October 2019. This is in big part thanks to Francois Caen who pushed for it to come back, and provided the help to organize this new edition.

As we worked on updating the survey for the 2019 edition, we tried to reuse the same questions as much as possible to allow us to compare the evolution of the responses over time. We also added a completely new section to understand how organizations and individuals are transitioning into network automation. This section was suggested by the community and was a welcome addition–the insights we are getting from it have been very interesting.

Participants in the 2019 NetDevOps Survey

The 2019 edition resulted in 300 responses which is about the same number that as the first edition in 2016.

The first set of questions was designed to give a better understanding of the type of networks and environments the participants come from.

Looking at the graphs below, there is a good distribution of participants both in term of types of environment and network sizes with an average around 1000 devices. It’s interesting to note that while there is a lot of coverage (blogs/podcasts/press) around network automation, most sources are focused on data centers. 60% of the participants in this survey are also managing Campus and/or WAN networks, but the data center is still the environment mentioned by most participants (~75%). This number has declined slightly since 2016, when data centers were mentioned by 80% of participants. These numbers are in line with the migration to the cloud that we here at Network to Code have observed with our customers.

State of Network Automation Through Automation

The main section of the survey is meant to understand which day-to-day operations are currently automated and which tools are used for each use case. We put together a list of 13 of the most common operations spanning topics such as configuration management, troubleshooting, and software upgrades.

While the 3 main operations that are automated today are focused on configuration management, it is interesting to see a significant increase around compliance check and pre-post changes. At the bottom of the graph we are also seeing a noticeable increase in responses on troubleshooting and software qualification.

Configuration Management

If we look specifically at configuration management, it’s interesting to see that 60% of the participants are using Ansible and roughly the same percentage are also using some scripts at different levels of abstraction. Nornir and Saltstack are both used by ~10% of the participants, an impressive achievement for these 2 open source projects that have been mainly driven/promoted by the community. Kudos to David Barroso, Mircea Ulinic, Kirk Byers, and Dmitry Figol.

the graph is a little bit misleading because we split scripts in 2 categories this year but if we add #2 and #3 we are close to 60%.

Interestingly, on average, participants selected more than 2 responses to this question, which means that a lot of participants are using more than 2 solutions to generate and deploy configuration. This fact got me curious, so I decided to dive deeper into the responses to understand which tools people are mostly using in addition to Ansible.
In the graph below, I narrowed down the responses to only the participants that selected Ansible. It is interesting to note that 12% of them are also using Nornir and more than 60% are using some scripts in addition to Ansible. There is not enough information to truly explain the reasoning behind these results but it is something I think it would be interesting to investigate deeper in the next edition.

As a side note, there are a lot of interesting analytics that haven’t been done yet on the data, such as diving deeper into each response or exploring how certain groups of participants respond to specific questions. If you are interested in doing some analysis on your own, the database and some tools are available in GitHub.

Maturity level / Automated Changes

At Network to Code, we often refer to network automation as a journey, which takes couple of years on average. As a part of the survey, I was personally interested in understanding the current level of maturity of our industry. How fast or how slowly is the market evolving? In the graph below, we can see that 37% of the participants have been leveraging automation in a significant way for less than 1 year and another 29% have been for 1 to 2 years. These numbers will be interesting to monitor year over year.

Another way to measure the level of maturity is to look at how manual and automated changes are coexisting, or not, within an organization. Usually, in the most advanced environment, manual changes are completely forbidden. There are two questions in the survey that give some good insight on this topic:

Do you allow configuration to be manually changed in the CLI in addition to automated deployment?
Have you automated the decision to deploy a new configuration?

To the first question, 14.5% of the participants indicated that they don’t allow manual changes in addition to automated deployment. This marks a significant increase from 2016, where only 8.8% of the participants responded “No”. And 46% of the participants indicated that they have fully or partially automated the decision to deploy a new configuration.

Anomaly Detection / Telemetry & Analytics

There has been an increase in conversations and projects surrounding telemetry and analytics in the last few years. A lot of my friend and colleagues working for webscale companies have reported using or building new telemetry and analytics stacks that are becoming an integral part of automation platforms.

Interestingly, the two questions related to anomaly detection/telemetry and analytics are showing a different picture. The majority of participants are still leveraging traditional monitoring solutions based on SNMP/Syslog and leveraging mostly Up/Down signals to detect issues in the network. With only 40% of the participants leveraging flows data and 10% using end to end probes.

My personal take-away is that today, telemetry and analytics is where network automation was 3-4 years ago with a significant disconnect between the most advanced companies and traditional enterprises.
A few years ago, network automation was not even a topic for most enterprise engineers, while a handful of companies were already all-in. At the pace at which the industry is moving these days, I think telemetry and analytics will make some progress in the enterprise space in the next couple of years.

Transition to Network Automation

As mentioned earlier, based on the input of the community we added a new section to understand how both organizations and individuals are transitioning to network automation, how long is it taking, what strategies are they adopting and more.

Team / Org

The results to the question what actions did you team take to transition to network automation show that most enterprises don’t have a concrete strategy and are relying on their existing staff to learn on their own or are just sending them to training. Less than 20% of the participants mentioned hiring a dedicated resource for network automation and less than 10% mentioned working with a consulting firm to help them in their automation journey.

Individual

As individuals, most participants (81%) estimated that it took them less than 1000 hours to learn network automation and 25% even estimated less than 200 hours. The majority of participants had to invest some personal time to learn new skills, while 40% where able to learn on the job either part-time or full-time.

Overall 34% of the participants mentioned that it took them less than 1 year to make the transition and another 45% estimated the transition at 1 to 2 years.

Industry Trends

The last section of the survey focuses on trends. What topics and tools are, or are not, top of mind right now? For this section, we selected a dozen tools and another dozen topics. For each of them we asked the participants if they are:

Already using them in production (dark green)
Currently evaluating them (green)
Thinking about it (light green)
Not interested (grey)
No idea (orange)

There is a of information in the graphs below, so it is hard to cover everything but my personal takeaways are:

35% of the participants are already using a Source of Truth (SoT) in production and another 50% are either evaluating one or thinking about it. In our experience at Network to Code, a SoT (or SoT strategy) is a critical component of a network automation strategy and it often seems like the topic is not getting enough attention. It’s very encouraging to see such high level of interest in this topic.
The level of adoption for ChatOps is still relatively low, with only ~15% of the participants using them in production and almost 30% of the participants expressing no interest. At NTC, we are seeing a lot of interesting use cases that can be solved with ChatOps and we are expecting this technology to get adopted more broadly in the future.
DevOps, Infrastructure as Code (IaC), and CI/CD are getting a lot of interest and are getting used in production more and more.

On the tools side, there is even more going on. My personal takeaways are:

Git and Ansible are used in production at a massive scale–both solutions are used in production by ~70% of the participants.
Modern monitoring tools like ELK, Grafana, Prometheus & Influx are used in production by more than 30% of the participants. These numbers are encouraging but don’t necessarily align with the previous responses to the anomaly detection questions. This could be explained if both new and legacy solutions are coexisting right now and the new solutions are still mostly used for visibility but are not used yet for alerting.
Nornir and Network Verification Software (Batfish, Forward Networks, etc. ) have a disproportionate ratio of production deployment compared to the level of participants evaluating or considering them. These two technologies will be interesting to monitor in the upcoming months/years.

Evolution over Time

Another interesting way to look at these results is to examine the evolution of the responses between 2016 and 2019. I selected a few below that I found the most interesting/surprising.

Looking at Git and Ansible, it’s interesting to see that for both technologies the level of interest was already very high in 2016 but the deployment in production were significantly lower. Both have gained significant market share in the last few years.

On the other side, solutions like Chef and Puppet have followed the opposite trajectory with a significant decrease in interest and deployment in production from the participants over the last three years.

The results surrounding event driven automation are surprising because, while the level of interest was already very high in 2016, the number of deployments in production has not significantly increased between 2016 and 2019. One explanation could be that EDA requires a higher level of maturity and expertise to be properly deployed in production. Based on the previous results, with 2/3 of the participants using automation for less than 2 years, it’s likely that the market has not reached this level of maturity yet.

Last but not least, it’s interesting to visualize the progression of Infrastructure as Code, CI/CD, and NAPALM over the last few years. Increased interest in these topics confirms what we are witnessing every day with our customers.

More graphs are available in Github

NetDevOps Survey

If you’re interested in learning more about about the NetDevOps Survey project, you can find the project on Github or join the conversation in the #netdevops_survey channel in the Network to Code slack channel.

All the results are available in Github in different formats:

Raw TSV files
SQLite Database with a Python library to query it
150+ Graphs similar to the one used in this blog

The plan is to start working on the 2020 Edition around August 2020 to have it ready to accept responses by October 2020.

How to help

If you’re interested in helping with the project or providing feedback, the best way to reach us is to open an issue on GitHub or join us in Slack.

At this point one of our biggest concerns is increasing the visibility of the project. The more participants we can get for the next edition, the deeper the insights and the better the project. Being community driven, we’ve been lacking marketing support to reach a broader and more diverse audience. Anything you can do to help here would go a long way.

Thanks for reading all the way to the end and for your interest in this project. If you are interested in diving deeper, the complete results of the 2019 Edition are available online.
I am personally looking forward to reading more analysis and hearing more perspectives on these results. I’m also looking forward to the next edition.

-Damien (@damgarros)

Network Telemetry for SNMP Devices

2020-04-21T00:00:00+00:00

This is going to be the first in a multipart series where I will be taking a look at a method to get telemetry data from network devices into a modern Time Series Database (TSDB).

In this particular post I will be working through adding SNMP based device data into the Prometheus TSDB. I will be using Telegraf from InfluxData to gather the SNMP data from Cisco devices on an emulation platform. Prometheus will then scrape the data from Telegraf and store the metrics. I will then show in how to start building out graphs within Grafana.

Here is an example of a Grafana dashboard that could be made:

From @SNMPguy for Cisco Live

Gathering Data - Concepts

At this point there are many advertisements that Streaming Telemetry is a must have in this day and age for gathering network device metrics. However, there are still quite a few network devices that do not support Streaming Telemetry in networks today. If you have a large deployment of these types of devices are you out of luck if you want to use a modern TSDBs? No you are not. Gathering data into a TSDB is all about just that, gathering data. If you gather the data via Streaming Telemetry or SNMP, either way, you are gathering the data. Streaming Telemetry is generally thought of as less intensive of a process on devices and has some other benefits. So if you can gather the data with Streaming Telemetry, then you should. But if you must use SNMP, this article is here to help you out.

Gathering Data - CLI Parsing

Through this post you will see information gathered via SNMP. If you wish to look at using CLI parsing as a method to get metrics, take a look at our previous post.

Gathering Data via SNMP

This post will outline what Telegraf has to offer when it comes to gathering data. Telegraf is an application made available by InfluxData that will gather data from various places. The gathering of information is known as an input. Then you will see how to send or make the data available for TSDB - Prometheus. These are known as the outputs. You can take a look at the plugins list to see the list of plugins for Telegraf 1.14, which as of this writing (2020-04-21) is the latest version.

Telegraf has the capability to also transform, tag, and modify data as needed. Portions of that will be covered in a follow-up post.

Within the configuration files you can setup to have a single Telegraf process poll multiple devices or you can have multiple Telegraf processes or containers, with each one polling one device. In this post I will be showing how to configure a single device to be polled by Telegraf. By this nature you can have your Telegraf agents centralized or distributed as needed.

A Prometheus nuance is that Prometheus will assume a device is down if Prometheus is unable to scrape the device’s metric page. But collecting SNMP data, the collection will be of the Telegraf process, which should get tied to its ability to poll the device. So additional configuration will be needed for Prometheus alerting in respects to reading metrics from a Telegraf plugin.

The SNMP configuration is made within the Telegraf configuration. This configuration may look like the following:

[[inputs.snmp]]
  agents = ["minneapolis.ntc"]
  version = 2
  community = "SecuredSNMPString"
  interval = "60s"
  timeout = "10s"
  retries = 3

  [[inputs.snmp.field]]
    name = "hostname"
    oid = ".1.3.6.1.2.1.1.5.0"
    is_tag = true

  [[inputs.snmp.field]]
    name = "uptime"
    oid = "1.3.6.1.2.1.1.3.0"

  [[inputs.snmp.field]]
    name = "cpmCPUTotal1min"
    oid = ".1.3.6.1.4.1.9.9.109.1.1.1.1.4.7"

  #####################################################
  #
  # Gather Interface Statistics via SNMP
  #
  #####################################################

  # IF-MIB::ifTable contains counters on input and output traffic as well as errors and discards.
  [[inputs.snmp.table]]
    name = "interface"
    inherit_tags = [ "hostname" ]
    oid = "IF-MIB::ifTable"

    # Interface tag - used to identify interface in metrics database
    [[inputs.snmp.table.field]]
      name = "name"
      oid = "IF-MIB::ifDescr"
      is_tag = true

  # IF-MIB::ifXTable contains newer High Capacity (HC) counters that do not overflow as fast for a few of the ifTable counters
  [[inputs.snmp.table]]
    name = "interface"
    inherit_tags = [ "hostname" ]
    oid = "IF-MIB::ifXTable"

    # Interface tag - used to identify interface in metrics database
    [[inputs.snmp.table.field]]
      name = "name"
      oid = "IF-MIB::ifDescr"
      is_tag = true
  
  # EtherLike-MIB::dot3StatsTable contains detailed ethernet-level information about what kind of errors have been logged on an interface (such as FCS error, frame too long, etc)
  [[inputs.snmp.table]]
    name = "interface"
    inherit_tags = [ "hostname" ]
    oid = "EtherLike-MIB::dot3StatsTable"
  
    # Interface tag - used to identify interface in metrics database
    [[inputs.snmp.table.field]]
      name = "name"
      oid = "IF-MIB::ifDescr"
      is_tag = true

Note: In testing I have found that the Cisco CPU query can be different per device. I recommend testing per platform and perhaps per OS version to verify that the SNMP polling works properly. I have found that issuing the command snmpwalk -v 2c -c SecuredSNMPString minneapolis.ntc .1.3.6.1.4.1.9.9.109.1.1.1.1.4 to find the response. You can also look at some other SNMP OIDs available as well for Cisco at their doc page

Difference Between snmp.table and snmp.field

Now, we’ll briefly dig in to what each of the lines are doing here. When an SNMP field is defined, this is going to act like an snmpget on a device. The first section that we call hostname is getting the hostname of the device.

What is a Tag?

The is_tag will be used as a tag on the data that is called later. Tags are data points that will help to classify other pieces of information. This can be helpful in filtering data points, or associating data points with a particular query or other data point.

Tags will be covered in more detail in a subsequent post, but note that by leveraging tags in your templates that build the Telegraf configuration you are able to identify key components in the environment that will enhance the monitoring capabilities.

Jumping ahead, and using the Prometheus output you can see some of these tags and fields in action. snmp_ is added to the front of the name as a part of the Prometheus export. You are able see the result of the query on the right most, outside of the {}. Inside of the {} you have the various tags that are being applied.

# HELP snmp_cpmCPUTotal1min Telegraf collected metric
# TYPE snmp_cpmCPUTotal1min untyped
snmp_cpmCPUTotal1min{agent_host="minneapolis.ntc",device="minneapolis",host="225bb1fc7f4c",hostname="minneapolis.ntc",} 31
# HELP snmp_uptime Telegraf collected metric
# TYPE snmp_uptime untyped
snmp_uptime{agent_host="minneapolis.ntc",device="minneapolis",host="225bb1fc7f4c",hostname="minneapolis.ntc",} 1.2636057e+07

Exporting the SNMP Data

There are two majority leaders in my opinion in the open source TSDB market, InfluxDB and Prometheus. Both have outputs that you can leverage with Telegraf to get the data into the TSDB. I will focus on the Prometheus methodology here. By exporting data with the Prometheus output there are a couple of benefits. One, the data is able to be scraped by the Prometheus system. The second is you can get a very good visual representation of the data for troubleshooting your connections.

If you are using InfluxDB as your DB and need to troubleshoot, I find setting up a Prometheus exporter as a helpful step to be able to see what tags are being defined and what data is being gathered from an SNMP standpoint.

Output to Prometheus Configuration

The configuration for Telegraf to use the Prometheus metrics exporter is relatively short and sweet. Telegraf handles the heavy lifting once you set the configuration file.

#####################################################
#
# Export SNMP Information to Prometheus
#
#####################################################

[[outputs.prometheus_client]]
  listen = ":9012"
  metric_version = 2

Here you see that the section begins with [[outputs.prometheus_client]]. This is with no indentation within the configuration file. It sets the metric_version to 2, and then sets a port that the metrics will be exposed at on, here tcp/9012. The url is then http://<server_url/ip>:<listen_port>/metrics. Note the /metrics as defined is a best practice of Prometheus.

Let’s take a look at the output from the metrics page below. There are many more metrics that get exposed than just what is shown. This will show only the one related to octets inbound on the interface.

Prometheus Output

Within the tags you see the main metric name begins with interface_. This is added by the client exporter to assist in classification of the metric. You then see the actual metric name as collected by SNMP. Here it is appended to the end of interface_ to get the metric name.

You also see the tags that are assigned to the metric being presented. Below is a table of the tag and where it came from:

Tag	Came From
agent_host	Created by Telegraf
host	Host that is collecting the data, here the name of the Docker container
hostname	Tag defined within the input section for gathering the hostname, the input section specifies `inherit_tags` to inherit the hostname
ifName	Within the `inputs.snmp.table.field` section of the ifTable, noted by is_tag
name	The name of the interface, defined in the input section

After the tags, the Prometheus metric definition indicates that this is where the actual measurement is to be placed. The Prometheus engine will “scrape” this information from the HTTP page and then ingest the data appropriately into its DB.

# HELP interface_ifHCInOctets Telegraf collected metric
# TYPE interface_ifHCInOctets untyped
interface_ifHCInOctets{agent_host="minneapolis.ntc",host="225bb1fc7f4c",hostname="minneapolis.ntc",ifName="Gi1",name="GigabitEthernet1"} 2.4956199e+07
interface_ifHCInOctets{agent_host="minneapolis.ntc",host="225bb1fc7f4c",hostname="minneapolis.ntc",ifName="Gi7",name="GigabitEthernet7"} 0
interface_ifHCInOctets{agent_host="minneapolis.ntc",host="225bb1fc7f4c",hostname="minneapolis.ntc",ifName="Gi8",name="GigabitEthernet8",} 0
interface_ifHCInOctets{agent_host="minneapolis.ntc",host="225bb1fc7f4c",hostname="minneapolis.ntc",ifName="Nu0",name="Null0",} 0
interface_ifHCInOctets{agent_host="minneapolis.ntc",host="225bb1fc7f4c",hostname="minneapolis.ntc",ifName="Vo0",name="VoIP-Null0",} 0
interface_ifHCInOctets{agent_host="minneapolis.ntc",host="225bb1fc7f4c",hostname="minneapolis.ntc",ifName="Gi3",name="GigabitEthernet3",} 1.092917e+08
interface_ifHCInOctets{agent_host="minneapolis.ntc",host="225bb1fc7f4c",hostname="minneapolis.ntc",ifName="Gi2",name="GigabitEthernet2",} 1.477766e+06
interface_ifHCInOctets{agent_host="minneapolis.ntc",host="225bb1fc7f4c",hostname="minneapolis.ntc",ifName="Gi4",name="GigabitEthernet4",} 1.9447063e+07
interface_ifHCInOctets{agent_host="minneapolis.ntc",host="225bb1fc7f4c",hostname="minneapolis.ntc",ifName="Gi5",name="GigabitEthernet5",} 1.2468643e+07
interface_ifHCInOctets{agent_host="minneapolis.ntc",host="225bb1fc7f4c",hostname="minneapolis.ntc",ifName="Gi6",name="GigabitEthernet6",} 1.6549974e+07

Prometheus

After getting the data into a format that Prometheus can read, you need to install Prometheus. You will get a link for the long lived installation, but the best part about Prometheus is that you can get up and running by just executing the binary file.

Installation - Binary Execution

Link: Prometheus installation provides for documentation on getting Prometheus up and running on your system.

Installation - Download, Decompress, and Copy Binary to Local Folder

For this the installation will be of the 2.16.0 version that has a download link of https://github.com/prometheus/prometheus/releases/download/v2.16.0/prometheus-2.16.0.linux-amd64.tar.gz.

On a Linux host, wget is able to download the file into your local working directory.

josh@prometheus_demo:~$ wget https://github.com/prometheus/prometheus/releases/download/v2.16.0/prometheus-2.16.0.linux-amd64.tar.gz
--2020-03-14 18:20:41--  https://github.com/prometheus/prometheus/releases/download/v2.16.0/prometheus-2.16.0.linux-amd64.tar.gz
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/6838921/13326f00-4ede-11ea-98d2-3ed3a8fdfe99?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200314%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200314T182041Z&X-Amz-Expires=300&X-Amz-Signature=9d4b3578b43c357056d75698f94bf8fb3263510787046db5fe04fabd3196023a&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprometheus-2.16.0.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2020-03-14 18:20:41--  https://github-production-release-asset-2e65be.s3.amazonaws.com/6838921/13326f00-4ede-11ea-98d2-3ed3a8fdfe99?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200314%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200314T182041Z&X-Amz-Expires=300&X-Amz-Signature=9d4b3578b43c357056d75698f94bf8fb3263510787046db5fe04fabd3196023a&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprometheus-2.16.0.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.238.3
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.238.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 59608515 (57M) [application/octet-stream]
Saving to: ‘prometheus-2.16.0.linux-amd64.tar.gz’

prometheus-2.16.0.linux-amd64.tar.gz        100%[==========================================================================================>]  56.85M  23.8MB/s    in 2.4s

2020-03-14 18:20:44 (23.8 MB/s) - ‘prometheus-2.16.0.linux-amd64.tar.gz’ saved [59608515/59608515]

josh@prometheus_demo:~$ tar -xvzf prometheus-2.16.0.linux-amd64.tar.gz
prometheus-2.16.0.linux-amd64/
prometheus-2.16.0.linux-amd64/LICENSE
prometheus-2.16.0.linux-amd64/promtool
prometheus-2.16.0.linux-amd64/NOTICE
prometheus-2.16.0.linux-amd64/consoles/
prometheus-2.16.0.linux-amd64/consoles/node.html
prometheus-2.16.0.linux-amd64/consoles/index.html.example
prometheus-2.16.0.linux-amd64/consoles/prometheus-overview.html
prometheus-2.16.0.linux-amd64/consoles/node-disk.html
prometheus-2.16.0.linux-amd64/consoles/node-overview.html
prometheus-2.16.0.linux-amd64/consoles/node-cpu.html
prometheus-2.16.0.linux-amd64/consoles/prometheus.html
prometheus-2.16.0.linux-amd64/console_libraries/
prometheus-2.16.0.linux-amd64/console_libraries/menu.lib
prometheus-2.16.0.linux-amd64/console_libraries/prom.lib
prometheus-2.16.0.linux-amd64/prometheus
prometheus-2.16.0.linux-amd64/prometheus.yml
prometheus-2.16.0.linux-amd64/tsdb

cp prometheus-2.16.0.linux-amd64/prometheus .

Create a Base Configuration on Host

You can use this as a start of the configuration, it will be stored in the same local directory that you are working in. It is setting a default scrape interval for other jobs that do not have a scrape_interval set to 15s. The example will use prometheus_config.yml for the file name.

global:
  scrape_interval: "15s"

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: "5s"
    static_configs:
      - targets: ['localhost:9090']

Execution

Now that there is a configuration file ready to go, you can start the local server. This will start up without polling anything other than the local Prometheus instance.

josh@prometheus_demo:~$ ./prometheus --config.file="prometheus_config.yml"
level=info ts=2020-03-14T18:29:50.782Z caller=main.go:295 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2020-03-14T18:29:50.783Z caller=main.go:331 msg="Starting Prometheus" version="(version=2.16.0, branch=HEAD, revision=b90be6f32a33c03163d700e1452b54454ddce0ec)"
level=info ts=2020-03-14T18:29:50.783Z caller=main.go:332 build_context="(go=go1.13.8, user=root@7ea0ae865f12, date=20200213-23:50:02)"
level=info ts=2020-03-14T18:29:50.783Z caller=main.go:333 host_details="(Linux 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64 prometheus_demo (none))"
level=info ts=2020-03-14T18:29:50.783Z caller=main.go:334 fd_limits="(soft=1024, hard=1048576)"
level=info ts=2020-03-14T18:29:50.783Z caller=main.go:335 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-03-14T18:29:50.784Z caller=web.go:508 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-03-14T18:29:50.784Z caller=main.go:661 msg="Starting TSDB ..."
level=info ts=2020-03-14T18:29:50.788Z caller=head.go:577 component=tsdb msg="replaying WAL, this may take awhile"
level=info ts=2020-03-14T18:29:50.788Z caller=head.go:625 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2
level=info ts=2020-03-14T18:29:50.788Z caller=head.go:625 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2
level=info ts=2020-03-14T18:29:50.788Z caller=head.go:625 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2
level=info ts=2020-03-14T18:29:50.789Z caller=main.go:676 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-03-14T18:29:50.789Z caller=main.go:677 msg="TSDB started"
level=info ts=2020-03-14T18:29:50.790Z caller=main.go:747 msg="Loading configuration file" filename=prometheus_config.yml
level=info ts=2020-03-14T18:29:50.790Z caller=main.go:775 msg="Completed loading of configuration file" filename=prometheus_config.yml
level=info ts=2020-03-14T18:29:50.790Z caller=main.go:630 msg="Server is ready to receive web requests."

At the end you should see a message that states that the Server is ready to receive web requests.

Prometheus

With a web browser, open to the URL: http://<server_ip>:9090 or if using a local installation http://localhost:9090 which should add a redirect to /graph and bring you to a screen like this:

Once you have Prometheus loaded, you can start to use PromQL to do a few searches. The system currently only has one metric source, about itself. This is where a query to see what the process looks like can be done. In the search box enter the query scrape_duration_seconds and click Execute. A response is given back in text form that has an Element and a Value to it.

When changing to view the graph of these queries you start to see what may be possible within this time series DB.

Update and Add Network URLs to the Prometheus Config

Now the configuration will get updated to poll two hosts that have SNMP working on it. You see that the http:// and /metrics portions are removed. If not supplied these are applied by default. The prometheus_config.yml file will now look like below:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'snmp'
    scrape_interval: 60s
    static_configs:
      - targets:
        - 'jumphost.create2020.ntc.cloud.tesuto.com:9012'
        - 'jumphost.create2020.ntc.cloud.tesuto.com:9001'

Prometheus PromQL SNMP Example

After updating the Prometheus configuration and starting the Prometheus server you can now start to get SNMP data into the graph form. Now updating the PromQL to query for interface_ifHCInOctets you can start to see what the data is that Prometheus is getting from the SNMP data that Telegraf is presenting.

This is all nice, but it is hardly a system that will have a lot of graphs and be something to present to others. This is the role that Grafana will play as a graphing engine.

Grafana

Download and Install Grafana

sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_6.6.2_amd64.deb
sudo dpkg -i grafana_6.6.2_amd64.deb

josh@prometheus_demo:~$ wget https://dl.grafana.com/oss/release/grafana_6.6.2_amd64.deb
josh@prometheus_demo:~$ wget https://dl.grafana.com/oss/release/grafana_6.6.2_amd64.deb
--2020-03-15 19:21:08--  https://dl.grafana.com/oss/release/grafana_6.6.2_amd64.deb
Resolving dl.grafana.com (dl.grafana.com)... 2a04:4e42:3b::729, 151.101.250.217
Connecting to dl.grafana.com (dl.grafana.com)|2a04:4e42:3b::729|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 63232320 (60M) [application/x-debian-package]
Saving to: ‘grafana_6.6.2_amd64.deb’

grafana_6.6.2_amd64.deb                     100%[==========================================================================================>]  60.30M  19.8MB/s    in 3.1s

2020-03-15 19:21:12 (19.8 MB/s) - ‘grafana_6.6.2_amd64.deb’ saved [63232320/63232320]

josh@prometheus_demo:~$ sudo dpkg -i grafana_6.6.2_amd64.deb
Selecting previously unselected package grafana.
(Reading database ... 67127 files and directories currently installed.)
Preparing to unpack grafana_6.6.2_amd64.deb ...
Unpacking grafana (6.6.2) ...
Setting up grafana (6.6.2) ...
Adding system user `grafana' (UID 111) ...
Adding new user `grafana' (UID 111) with group `grafana' ...
Not creating home directory `/usr/share/grafana'.
### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
 sudo /bin/systemctl daemon-reload
 sudo /bin/systemctl enable grafana-server
### You can start grafana-server by executing
 sudo /bin/systemctl start grafana-server
Processing triggers for systemd (237-3ubuntu10.39) ...
Processing triggers for ureadahead (0.100.0-21) ...

Enable Grafana to Start on Boot and Start Grafana Server

 sudo /bin/systemctl daemon-reload
 sudo /bin/systemctl enable grafana-server
 sudo /bin/systemctl start grafana-server

josh@prometheus_demo:~$ sudo /bin/systemctl daemon-reload
josh@prometheus_demo:~$ sudo systemctl enable grafana-server
Synchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable grafana-server
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-server.service → /usr/lib/systemd/system/grafana-server.service.
josh@prometheus_demo:~$ sudo systemctl start grafana-server

Verify Grafana is Running

I like to verify that Grafana is in fact running by checking for the listening ports. You can do this by using the ss -lt command to get the output, and checking that there is a *:3000 entry in the output. TCP/3000 is the default port for Grafana.

josh@prometheus_demo:~$ ss -lt
State                 Recv-Q                 Send-Q                                   Local Address:Port                                     Peer Address:Port
LISTEN                0                      128                                      127.0.0.53%lo:domain                                        0.0.0.0:*
LISTEN                0                      128                                            0.0.0.0:ssh                                           0.0.0.0:*
LISTEN                0                      128                                               [::]:ssh                                              [::]:*
LISTEN                0                      128                                                  *:3000                                                *:*

Verify - Navigate to the Default Page

The default login is admin/admin. When you first log in you will be prompted for a new admin password.

Getting to the Graphing

Re-start Prometheus

Before you add in additional data sources that are needed, you need to restart the service on your Linux host.

josh@prometheus_demo:~$ ./prometheus --config.file=prometheus_config.yml

Add Data Source to Grafana

Now that you are in you need to add a data source for Grafana. In this demo you are going to see us add a localhost connection to Prometheus. Going back to the web interface on the main menu that you started into you can click on Add datasource.

In this instance of 6.6.x Grafana had Prometheus on the top of the list. Navigate to where you see Prometheus and click select.

The data source will bring you to a configuration screen.

Here make the following changes:

Field Changes from Default

Field	Setting
URL	http://localhost:9090

Once modified, click Save and Test to test and verify connectivity to the DB. If you setup a different host as the Prometheus server, then you would enter the hostname/IP address combination that corresponds to the Prometheus host.

When you get the message Data source is working you have successfully connected.

Grafana Dashboard Creation

Now navigate to the left hand navigation and select the plus icon, select Dashboard to get a new dashboard created.

You get a new panel page, and then select Add Query.

Once on the new query page we will set a search to get the Inbound utilization on an interface. Set up the query as follows:

Note that the queries used on this Grafana example are going to be of PromQL - the Prometheus Query Langague. In this graphic, the {{ifName}} is telling Grafana that ifName is the variable to lookup to add to the legend for each measurement.

If your data source for Grafana is Graphite or InfluxDB, you would use the same query language used by the database system of the data source.

To explain what each item is doing to help generate your own queries. Given the following PromQL query:

rate(interface_ifHCInOctets{hostname="houston.tesuto.internal"}[2m])*8

Rate

The rate query from Prometheus covers the rate of change. With SNMP, the number gathered for Interface utilization is an increasing number, not a rate. So the Prometheus system needs to calculate what that rate is. The [2m] indicates to calculate the per-second rate measured over the past 2 minutes.

Metric Name

The metric name in the query is interface_ifHCInOctets. This is the metric that was taken a look at earlier in the post. This is the exact measurement.

Query Tags

The tags in the search is to help filter out what is being searched upon to give the proper graph. In this instance you will only see interfaces on the device hostname houston.tesuto.internal.

Math

In the query there is a *8 at the end. This is to convert the measurement from octets as defined in the metric over to bits. An octet is 8 bits, thus the multiplication by 8.

Visualization Changes

Now we’re going to make a few more updates on the graph. Here are the changes being made on the Visualization section (2nd of four items on the left hand side of the panel configuration). Specifically, the changes being made are in the Axis subsection. You can play around with settings in the upper section to get some changes made to the graphs.

Setting	Modification
Left Y: Unit	bits/sec (under Data Rate)
Legend Values: Min	Checked
Legend Values: Avg	Checked
As Table	Checked
To Right	Checked

General Section Changes

Here is where you can set the title of the panel. Let’s change that to Houston Interface Utilization. After making the update, click on the upper left to go back to the dashboard.

The panel size can be adjusted in size by dragging the corners as you see fit to make your dashboard.

Update Dashboard Name

On the main dashboard page to change the name on the dashboard select the Save icon on the upper right. This will give you a prompt with a New Name and Folder to save the dashboard into. This allows you to add heirarchy to your dashboarding system.

Important Note – if you make changes you do need to save the changes. Grafana as of this version does not save changes after a change. It does require you to save your changes once you are done making changes.

After you save the changes you get a visual confirmation that the changes are saved and that you now have a title on dashboard!

Conclusion

Hopefully this will help on your journey! In a follow-up post I will take a look at a few more capabilities within Telegraf, Prometheus, and Grafana.

How to gather streaming data with gNMI
Telegraf Tags
Transforming data with Telegraf
Prometheus queries
Grafana Tables
Grafana Thresholds & Alerts

To continue on in the journey, take a look at Network Telemetry - Advancing Your Dashboards and monitoring websites.

-Josh

Intro to Data Structures

2020-04-14T00:00:00+00:00

From an automation standpoint, it is important to understand what data structures are, how to use them, and more importantly how to access the data within the structure.

This post won’t go into how to build data structures as that is a whole topic in and of itself, but how do we obtain the data we want from a data structure that a system provides us such as Ansible, a web API, etc? We’re going to use Python when dissecting a data structure as we can use it to tell us what type the data is or what a specific type is within the data structure.

There are two main data types that we will discuss as they’re the most common.

Let’s start with looking at what a dictionary is and how we can get data from a dictionary using the built-in Python interactive interpreter.

Dictionaries

A dictionary is referred to as a mapping in other programming languages and is made up of key value pairs. The key must be an integer or a string, where the value may be any type of object. Dictionaries are mainly used when the order does not matter, but accessing specific data that can be found by a key. Let’s take a look at a dictionary.

>>> my_dict = {
...    'test_one': 'My first key:value',
...    'test': 'My second key:value',
...    10: 'Look at my key',
...}
>>> my_dict
{'test': 'My second key:value', 'test_one': 'My first key:value', 10: 'Look at my key'}
>>> my_dict.keys()
['test', 'test_one', 10]
>>> my_dict.values()
['My second key:value', 'My first key:value', 'Look at my key']
>>> type(my_dict)
<type 'dict'>

Note how the dictionary is in a different order than we created it in, which means the keys are significant in our ability to extract the values stored within the dictionary.

Let’s take a look at how to access data within a dictionary.

>>> my_dict['test_one']
'My first key:value'
>>> my_dict[10]
'Look at my key'
>>> my_dict['test_two']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'test_two'

Notice if we try and access a key that does not exist, we get a KeyError. This error can be avoided by using the .get() method on the dictionary. This will attempt to get the key from the dictionary and, if it doesn’t exist, will return None. It also accepts an argument that it will return if the key does not exist.

>>> test = my_dict.get("test_two")
>>> test
>>> type(test)
<class 'NoneType'>
>>> test = my_dict.get("test_two", "Return this value")

>>> test
'Return this value'
>>> if my_dict.get('test_one'):
...    print('It exists!')
...
It exists!

Now that we understand how dictionaries work and how we can obtain data from a dictionary, let’s move onto lists.

Lists

A list is referred to as an array in other programming languages and is a collection of different data types that are stored in indices within the list. A list can consist of objects of any type (strings, integers, dictionaries, tuples, etc.). The order of a list is maintained in the same order the list is created and the data can be obtained by accessing the indexes of the list.

Let’s take a look at creating a list.

>>> my_list = [
...     'index one',
...     {'test': 'dictionary'},
...     [1, 2 ,3],
... ]
>>> my_list
['index one', {'test': 'dictionary'}, [1, 2, 3]]
>>> for item in my_list:
...     type(item)
...
<type 'str'>
<type 'dict'>
<type 'list'>

As you can see, the list can store different data types, and the order is in the same order that we contructed the list in. We also iterated over the list to access each index, but we can access each item by the index they’re stored at as well.

>>> my_list[0]
'index one'
>>> my_list[1]
{'test': 'dictionary'}
>>> my_list[2]
[1, 2, 3]
>>> my_list[3]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range
>>> type(my_list)
<type 'list'>

The first index of a list starts at zero and increments up by one at each index. Let’s add a new item to the list and validate the order is still intact.

>>> my_list.append(5)
>>> my_list
['index one', {'test': 'dictionary'}, [1, 2, 3], 5]
>>> len(my_list)
4
>>> my_list[3]
5

Now that we understand how lists work and how we can obtain data from a list, let’s move on and put this all together when we encounter data structures in the wild!

How to Navigate Data Structures

After the above sections, it seems like navigating and obtaining the information from a data structure is a no-brainer, but can be intimidating when you come across a more complex data structure.

facts = {
    "ansible_check_mode": False,
    "ansible_diff_mode": False,
    "ansible_facts": {
        "_facts_gathered": True,
        "discovered_interpreter_python": "/usr/bin/python",
        "net_all_ipv4_addresses": [
            "192.168.1.1",
            "10.111.41.12",
            "172.16.133.1",
            "172.16.130.1",
        ],
        "net_filesystems": [
            "bootflash:"
        ],
        "net_filesystems_info": {
            "bootflash:": {
                "spacefree_kb": 5869720.0,
                "spacetotal_kb": 7712692.0
            }
        },
        "net_gather_network_resources": [],
        "net_gather_subset": [
            "hardware",
            "default",
            "interfaces",
            "config"
        ],
        "net_hostname": "csr1000v",
        "net_image": "bootflash:packages.conf",
        "net_interfaces": {
            "GigabitEthernet1": {
                "bandwidth": 1000000,
                "description": "MANAGEMENT INTERFACE - DON'T TOUCH ME",
                "duplex": "Full",
                "ipv4": [
                    {
                        "address": "10.10.20.48",
                        "subnet": "24"
                    }
                ],
                "lineprotocol": "up",
                "macaddress": "0050.56bb.e14e",
                "mediatype": "Virtual",
                "mtu": 1500,
                "operstatus": "up",
                "type": "CSR vNIC"
            }
        }
    }
}

The above data structure is what we get from gather facts in Ansible. We’re going to deal with the data structure outside of Ansible so we can determine breakdown each type data type in the structure. This is a great example as it’s a real world data structure and has nesting that we will need to traverse to get the data we want.

Let’s start by looking at the data type of the initial structure and then see how we can get the ansible_check_mode data.

>>> type(facts)
<class 'dict'>
>>> facts.get('ansible_check_mode')
False

As you can see, the initial data structure is a dictionary and since ansible_check_mode is in this initial dictionary it makes it a key. We can get the value of ansible_check_mode by using the .get() method.

What if we want to loop over all the IP addresses within net_all_ipv4_addresses? Let’s see how we can do that.

>>> type(facts['ansible_facts'])
<class 'dict'>
>>> facts['ansible_facts'].keys()
dict_keys(['_facts_gathered', 'discovered_interpreter_python', 'net_all_ipv4_addresses', 'net_filesystems', 'net_filesystems_info', 'net_gather_network_resources', 'net_gather_subset', 'net_hostname', 'net_image', 'net_interfaces'])
>>> type(facts['ansible_facts']['net_all_ipv4_addresses'])
<class 'list'>
>>> for ip in facts['ansible_facts']['net_all_ipv4_addresses']:
...     print(ip)
...
192.168.1.1
10.111.41.12
172.16.133.1
172.16.130.1

As we can see above, net_all_ipv4_addresses is a key within the ansible_facts dictionary. We have to navigate through two nested dictionaries to get to the list of IPv4 addresses we want to print out.

Let’s move on and obtain the IP address and subnet mask on GigabitEthernet1.

>>> type(facts['ansible_facts']['net_interfaces'])
<class 'dict'>
>>> type(facts['ansible_facts']['net_interfaces']['GigabitEthernet1'])
<class 'dict'>
>>> type(facts['ansible_facts']['net_interfaces']['GigabitEthernet1']['ipv4'])
<class 'list'>
>>> len(facts['ansible_facts']['net_interfaces']['GigabitEthernet1']['ipv4'])
1
>>> type(facts['ansible_facts']['net_interfaces']['GigabitEthernet1']['ipv4'][0])
<class 'dict'>
>>> gi1_subnet = facts['ansible_facts']['net_interfaces']['GigabitEthernet1']['ipv4'][0]['subnet']
>>> gi1_address = facts['ansible_facts']['net_interfaces']['GigabitEthernet1']['ipv4'][0]['address']
>>> f"{gi1_address}/{gi1_subnet}"
'10.10.20.48/24'

This is definitely a more complex data structure to traverse to get the data we need. We’ll walk through how to traverse this data structure.

We can see that net_interfaces is a dictionary so we’ll use a key to traverse to the next level. The data we’re interested in is in the GigabitEthernet1 key. We see that is also a dictionary so we understand to get to the next level in the hierarchy, we will use another key which is ipv4. The ipv4 data is stored within a list and the length of the list is one, which means we can access it at index zero. The data within index zero is a dictionary which means we now need to access the address and subnet via their respective keys.

We store the address and the subnet in their own variables that tell the story of how we got to the data we want.

dictionary[dictionary][dictionary][dictionary][dictionary][list][dictionary]

Now that you understand how each data type works, you can tackle any complex data structure you encounter.

Let’s take a look at another example and keep flexing this muscle memory. Let’s determine how much space has been used on bootflash: by subtracting the spacefree_kb value from the spacetotal_kb value.

>>> type(facts['ansible_facts'])
<class 'dict'>
>>> type(facts['ansible_facts']['net_filesystems_info'])
<class 'dict'>
>>> type(facts['ansible_facts']['net_filesystems_info']['bootflash:'])
<class 'dict'>
>>> space_free = facts['ansible_facts']['net_filesystems_info']['bootflash:']['spacefree_kb']
>>> space_free
5869720.0
>>> space_total = facts['ansible_facts']['net_filesystems_info']['bootflash:']['spacetotal_kb']
>>> space_total
7712692.0
>>> space_used = space_total - space_free
>>> space_used
1842972.0

As you can see, we had to navigate through four dictionaries including the intitial structure, but we didn’t have to navigate through any lists this time to get the data we wanted.

Remember that these complex data structures can be intimidating, but breaking down each data type within the structure helps us deconstruct them into smaller chunks to navigate and process until we get to the data we want.

-Mikhail

Office Manager Appreciation Day

2020-04-10T00:00:00+00:00

Pet appreciation day is April 11! But at a company with a large remote workforce, our pets aren’t just companions – around here we lovingly refer to them as our “office managers.” Whether they are demanding treats or need us to take them for a walk, pets are a key part of many home office set ups and are great excuse to take a quick break during a busy day (see our roundup of work from home tips for more on the importance of breaking during the work day). In honor of pet appreciation day, we’d like to highlight the key roles and responsibilities some of our four-legged friends have taken on around the home office:

Dixie

Department:

New York Pet Detective - Nap Division

Responsibilities:

Enforcing nap time regularly, ensuring “fair” distribution of carrots.

Mars, Anya

Department:

Mars: International Affairs - Russia
Anya: Side-eye Extraordinaire & Doge W/ Opinion

Responsibilities:

Enforcing nap time regularly, ensuring “fair” distribution of carrots. Mars: Zdravstvuyte comrades, after many hours journey from the Russian motherland and long lines for immigration I have joined my fellow Houston Office Manager Anya in keeping two-legged keeper in line. He needs constant reminding that my miska or bowl is not > 75% full and Anya must also be reminded that I am now the ranking member of NTC Houston!!

Anya: My “roommate” recently had another hooman come live in MY house and had the nerve to bring a cat! I guess it’s not too bad, the cat is too busy looking for something called “Moose & Squirrel”. Anywho, this is my house and it is my duty to stand back and judge everything hoomans do when I’m not busy making memes. #SendSnackooosssss

Birdie, Norman, and Scout

Department:

Snacks

Responsibilities:

Birdie: Organizes all office outings (with a strong preference towards long walks).
Norman: Tracks food metrics and provide daily meal reminders.
Scout: Prefers to keep a high-level view and ensure that Norman and Birdie handle all relevant office tasks.

Riley, Mya

Department:

Riley: Property Management
Mya: Union Representation

Responsibilities:

Riley: Primary intruder detection. Hiding under benches, behind chairs, between toilets and tubs. Eating random objects. Quieter than a cat until a door is knocked upon or open, then chief barker.
Mya: Strict enforcement of mealtimes and overall breaks. Bathroom breaks. Lunch cleanup breaks. Break breaks. Resident whiner and prima donna. Steals more food than is necessary.

-The NTC Team

Exploring Python’s args and kwargs in a Networking Context

2020-04-07T00:00:00+00:00

At a certain point in time, while working with Python, you might have seen code that uses *args and **kwargs as parameters in functions. This feature brings excellent flexibility when developing Python code. I’m going to go over some of the basics of packing arguments into a function, showing how to pass a variable amount of positional arguments and key-value pairs into a function. Feel free to follow along if you have access to a Python interpreter.

Note: Print functions are used for demonstration. These functions don’t create a connection to a device. The goal is to focus on *args and **kwargs.

Using args

The current function, device_connection, has four different required positional parameters and are printed out inside the function.

def device_connection(ip, device_type, username, password):
    print("ip: ", ip)
    print("device_type: ", device_type)
    print("username :", username)
    print("password: ", password)

This next step is calling the device_connection function and it’s passing string arguments at the exact number of parameters in the function. As expected, when calling the function, the data is printed.

>>>
>>> device_connection("10.10.10.2", "cisco_ios", "cisco", "cisco123")
ip:  10.10.10.2
device_type:  cisco_ios
username : cisco
password:  cisco123
>>>

As mentioned before, these are positional parameters and if the order in which they are called is incorrect, then the output is printed inaccurately. In this case, the device connection is not established if the function was built correctly for connectivity. The next example shows what it looks like when calling the function incorrectly.

>>>
>>> device_connection("cisco_ios", "10.10.10.2", "cisco", "cisco123")
ip:  cisco_ios
device_type:  10.10.10.2
username : cisco
password:  cisco123
>>>

Notice how in the above output, the IP value is now populated with the device type string while the device_type is populated with 10.10.10.2.

Another caveat could be if another argument is defined, because the connection requires a different SSH port to establish the connection. If added when calling the function, it fails because the current function was only built to take 4 arguments. Take a look at the example below. The new argument with the port number is added to the function.

>>>
>>> device_connection("cisco_ios", "10.10.10.2", "cisco", "cisco123", "8022")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: device_connection() takes 4 positional arguments but 5 were given
>>>

The next section is going to show how to solve some of these issues.

Using *args

The basics of packing with *-operator are compressing multiple values into an agument/parameter. The first line inside the function is using the type function inside a print statement to display the data type returned from the *args variable. The next few lines have print statements accessing each element of the tuple, which are accessed similar to a list by the index number.

Note: The asterisk (*) symbol is needed to pack the data into a tuple, or it fails and views the parameter as a single argument when the function call is initiated.

def device_connection(*args):
    print(type(args))
    print("tuple: ", args)
    print("ip: ", args[0])
    print("device_type: ", args[1])
    print("username: ", args[2])
    print("password: ", args[3])

The example below calls the device_connection function, except for this time, the parameter used when building the function is *args.

>>>
>>> device_connection("10.10.10.2", "cisco_ios", "cisco", "cisco123")
<class 'tuple'>
tuple:  ('10.10.10.2', 'cisco_ios', 'cisco', 'cisco123')
ip:  10.10.10.2
device_type:  cisco_ios
username:  cisco
password:  cisco123
>>>

Notice how the function output above printed each argument added to function call and it was “compressed” into the args variable.

Another way of accessing data from a tuple is to use a for loop like the example below.

def device_connection(*args):
    print(type(args))
    print("tuple: ", args)
    for arg in args:
        print(arg)

Data can also be assigned to variables and used as input when calling the function.

ip = "10.10.10.2"
device_type = "cisco_ios"
username = "cisco"
password = "cisco123"

Check out the output when the function is called.

>>>
>>> device_connection(ip, device_type, username, password)
<class 'tuple'>
tuple:  ('10.10.10.2', 'cisco_ios', 'cisco', 'cisco123')
10.10.10.2
cisco_ios
cisco
cisco123
>>>

Notice how each value was accessed using a for loop when called inside the function without having to provide the index.

Another great thing about using *args when creating a function is that it can take multiple inputs without having to specify it when building the function. So if another value needs to be processed, then it can be passed as another argument when calling the function.

Note: The * asterisk is doing all the “magic” when packing all the arguments; args can be any arbitrary variable.

In this step a new variable called port is created with a value of 22.

port = 22

The example below shows the output when calling the device_connection when passing all the same previous variables plus the new variable.

>>>
>>> device_connection(ip, device_type, username, password, port)
<class 'tuple'>
tuple:  ('10.10.10.2', 'cisco_ios', 'cisco', 'cisco123', 22)
10.10.10.2
cisco_ios
cisco
cisco123
22
>>>

Using **kwargs

Previously when passing parameters with no *-operator, it was by just passing a variable argument. If the same number of arguments aren’t passed when calling the function, it returns with an error. When using the single *-operator multiple arguments could be passed as options and the data is packed into a tuple.

Another way of using function parameters is to use the ** double asterisk operator and any arbitrary variable when packing key:value pairs as a parameter into a dictionary.

The function below is using **kwargs with a variable. Inside the function, it prints the data type and the data that is passed when calling the function.

def device_connection(**kwargs):
    print(type(kwargs))
    print(kwargs)

The example below shows the output when calling the function with two keyword arguments.

>>>
>>> device_connection(device_type="ios", ip="10.10.10.2")
<class 'dict'>
{'device_type': 'ios', 'ip': '10.10.10.2'}
>>>

Like mentioned before, the (**) asterisk operator is doing the work to pack the data into a variable. Users don’t have to use the variables named args and kwargs. They could use an arbitrary variable such as **connection_data instead of **kwargs, as shown in the example below.

def device_connection(**connection_data):
    print(type(connection_data))
    print(connection_data)

The example below shows the output when calling the function using **connection_data.

>>>
>>> device_connection(device_type="ios", ip="10.10.10.2")
<class 'dict'>
{'device_type': 'ios', 'ip': '10.10.10.2'}
>>>

This next example calls the function again, but with more keyword arguments.

>>>
>>> device_connection(device_type="ios", ip="10.10.10.2", username="cisco", password="cisc123")
<class 'dict'>
{'device_type': 'ios', 'ip': '10.10.10.2', 'username': 'cisco', 'password': 'cisc123'}
>>>

Notice how the data type output is a dictionary and the input arguments are key:value pairs.

Something else to point out is that *args can also be used with **kwargs together in a single function. Using both *args and **kwargs together gives the function some more options to choose from in terms of how it can consume the data.

The example below is using a variable containing a dictionary of device data.

csr1 = {
    'device_type': 'ios',
    'ip':   '10.10.10.2',
    'username': 'cisco',
    'password': 'cisco123',
    'port': 8022,
    'secret': 'secret'}

The function below is built to take in both types of parameters.

def device_connection(*args, **kwargs):
    print("*args: ", type(args))
    print("**kwargs: ", type(kwargs))
    print("args[0]: ", args[0])
    print("args: ", args)
    print("kwargs: ", kwargs)

When calling the function, notice how the first argument is the variable. The second and third arguments are keyword arguments.

The first output prints out a tuple for *args, the second output is dict for **kwargs, the third output gets access to the dictionary by accessing index 0 in the tuple and the last two outputs print out the data “packed” in args and kwargs.

>>>
>>> device_connection(csr1, device_type="ios", host="csr1")
*args:  <class 'tuple'>
**kwargs:  <class 'dict'>
args[0]:  {'device_type': 'ios', 'ip': '10.10.10.2', 'username': 'cisco', 'password': 'cisco123', 'port': 8022, 'secret': 'secret'}
args:  ({'device_type': 'ios', 'ip': '10.10.10.2', 'username': 'cisco', 'password': 'cisco123', 'port': 8022, 'secret': 'secret'},)
kwargs:  {'device_type': 'ios', 'host': 'csr1'}
>>>

Summary

The examples shown in this blog use networking data to explore Python parameters in a function using *(for tuples) and **(for dictionaries). There are different ways of building Python functions, and they don’t always need to use * and ** operators for every function built. Sometimes functions can be basic and only need specific data. But when a function takes in variable amount or unknown amount of data, then * and ** operators can come in handy to build more flexible functions.

-Hector

How to Monitor Your VPN Infrastructure with Netmiko, NTC-Templates, and a Time Series Database

2020-03-30T00:00:00+00:00

With many people being asked to work from home, we have heard several customers looking to enhance the visibiility and monitoring of their VPN infrastructure. In this post I will show you how you can quickly collect information from your Cisco ASA firewall leveraging Netmiko, NTC-Templates (TextFSM), combined with Telegraf/Prometheus/Grafana. The approach would work on other network devices, not just Cisco ASAs. Considering the recent demand for ASA information, we will use this as an example.

Here is what the data flow will look like:

Users will connect to the ASA for remote access VPN services
Python: Collects information from the device via CLI and gets structured data by using a NEW template to parse the CLI output. This is presented via stdout in Influx data format
Telegraf: Generic collector that has multiple plugins to ingest data and that can send data to many databases out there.
- INPUT: Execute the Python script every 60s and read the results from stdout.
- OUTPUT: Expose the data over HTTP in a format compatible with Prometheus
Prometheus: Time Series DataBase (TSDB). Collects the data from Telegraf over HTTP, stores it, and exposes an API to query the data
Grafana: Solution to build dashboards, natively support Prometheus to query data.

An alternative to creating this Python script, you could have looked at using the Telegraf SNMP plugin as well. An SNMP query would be quicker than using SSH and getting data if you want basic counts. In this you will see that you can get custom metrics into a monitoring solution without having to use only SNMP.

Execution of Python

If executing just the Python script without being executed by Telegraf, this is what you would see:

$ python3 asa_anyconnect_to_telegraf.py --host 10.250.0.63
asa connected_users=1i,anyconnect_licenses=2i

This data will then get transformed by Telegraf into an output that is usable by Prometheus. It is possible to remove the requirement for Telegraf and have Python create the Prometheus Metrics. We wanted to keep the Python execution as simple as possible. To use the prometheus_client library check out their Github page.

Python Script

In this post we have the following components being used:

Python:
- Netmiko to SSH into an ASA, gather command output, and leverage the corresponding NTC Template
- NTC Template which is a TextFSM template for parsing raw text output into structured data
Telegraf: Takes the output of the Python script as an input and translates it to Prometheus metrics as an output

Python Requirements

The Python script below will have the following requirements set up before hand:

ENV Variables for authentication into the ASA
- ASA_USER: Username to log into the ASA
- ASA_PASSWORD: Password to log into the ASA
- ASA_SECRET (Optional): Enable password for the ASA, if left undefined will pick up the ASA_PASSWORD variable
Required Python Packages:
- Netmiko: For SSH and parsing
- Click: For argument handling - to get the hostname/IP address of the ASA
Github Repository for NTC-Templates setup in one of two ways:
- Cloned to user home directory cd ~ and git clone https://github.com/networktocode/ntc-templates.git
- NET_TEXTFSM Env variable set NET_TEXTFSM=/path/to/ntc-templates/templates/

The specific template is the newer template for cisco asa show vpn-sessiondb anyconnect introduced March 18, 2020

Python Code

There are two functions used in this quick script:

"""
(c) 2020 Network to Code
Licensed under the Apache License, Version 2.0 (the "License").
You may not use this file except in compliance with the License.
You may obtain a copy of the License at
http: // www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Python application to gather metrics from Cisco ASA firewall and export them as a metric for Telegraf
"""
from itertools import count
import os
import sys
import re
import click
from netmiko import ConnectHandler

def print_influx_metrics(data):
    """
    The print_influx_metrics function takes the data collected in a dictionary format and prints out
    each of the necesary components on a single line, which matches the Influx data format.

    Args:
        data (dictionary): Dictionary of the results to print out for influx
    """
    data_string = ""
    cnt = count()
    for measure, value in data.items():
        if next(cnt) > 0:
            data_string += ","
        data_string += f"{measure}={value}i"

    print(f"asa {data_string}")

    return True


def get_anyconnect_license_count(version_output):
    """
    Searches through the `show version` output to find all instances of the license and gets the
    output into integers to get a license count.

    Since there could be multiple ASAs in a cluster or HA pair, it is necessary to gather multiple data
    points for the license count that the ASAs are licensed for. This function uses regex to find all of
    the instances and returns the total count based on the `show version` command output.  

    Args:
        version_output (String): Output from Cisco ASA `show version`
    """
    pattern = r"AnyConnect\s+Premium\s+Peers\s+:\s+(\d+)"
    re_list = re.findall(pattern, version_output)

    total_licenses = 0
    for license_count in re_list:
        total_licenses += int(license_count)

    return total_licenses


# Add parsers for output of data types
@click.command()
@click.option("--host", required=True, help="Required - Host to connect to")
def main(host):
    """
    Main code execution
    """
    # Get ASA connection Information
    try:
        username = os.environ["ASA_USER"]
        password = os.environ["ASA_PASSWORD"]
        secret = os.getenv("ASA_SECRET", os.environ["ASA_PASSWORD"])
    except KeyError:
        print("Unable to find Username or Password in environment variables")
        print("Please verify that ASA_USER and ASA_PASSWORD are set")
        sys.exit(1)

    # Setup connection information and connect to host
    cisco_asa_device = {
        "host": host,
        "username": username,
        "password": password,
        "secret": secret,
        "device_type": "cisco_asa",
    }
    net_conn = ConnectHandler(**cisco_asa_device)

    # Get command output for data collection
    command = "show vpn-sessiondb anyconnect"
    command_output = net_conn.send_command(command, use_textfsm=True)

    # Check for no connected users
    if "INFO: There are presently no active sessions" in command_output:
        command_output = []

    # Get output of "show version"
    version_output = net_conn.send_command("show version")

    # Set data variable for output to Influx format
    data = {"connected_users": len(command_output), "anyconnect_licenses": get_anyconnect_license_count(version_output)}

    # Print out the metrics to standard out to be picked up by Telegraf
    print_influx_metrics(data)


if __name__ == "__main__":
    main()

Telegraf

Now that the data is being output via the stdout of the script, you will need to have an application read this data and transform it. This could be done in other ways as well, but Telegraf has this function built in already.

Telegraf will be setup to execute the Python script every minute. Then the output will be transformed by defining the output.

Telegraf Configuration

The configuration for this example is as follows:

# Globally set tags that shuld be set to meaningful tags for searching inside of a TSDB
[agent]
hostname = "demo"

[global_tags]
  device = "10.250.0.63"
  region = "midwest"

[[inputs.exec]]
  ## Interval is how often the execution should occur, here every 1 min (60 seconds)
  interval = "60s"
  # Commands to be executed in list format
  # To execute against multiple hosts, add multiple entries within the commands
  commands = [
      "python3 asa_anyconnect_to_telegraf.py --host 10.250.0.63"
  ]

  ## Timeout for each command to complete.
  # Tests in lab environment next to the device with local authentication has been 6 seconds
  timeout = "15s"

  ## Measurement name suffix (for separating different commands)
  name_suffix = "_parsed"

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## More about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "influx"

# Output to Prometheus Metrics format
# Define the listen port for which TCP port the web server will be listening on. Metrics will be
# available at "http://localhost:9222/metrics" in this instance.
# There are two versions of metrics and if `metric_version` is omitted then version 1 is used
[[outputs.prometheus_client]]
  listen = ":9222"
  metric_version = 2

Telegraf Output Example

Here is what the metrics will look like when exposed, without the default Telegraf information metrics.

# HELP asa_parsed_anyconnect_licenses Telegraf collected metric
# TYPE asa_parsed_anyconnect_licenses untyped
asa_parsed_anyconnect_licenses{device="10.250.0.63",host="demo",region="midwest"} 2
# HELP asa_parsed_connected_users Telegraf collected metric
# TYPE asa_parsed_connected_users untyped
asa_parsed_connected_users{device="10.250.0.63",host="demo",region="midwest"} 1

There are two metrics of anyconnect_licenses and connected_users that will get scraped. There are a total of 2 Anyconnect licenses available on this firewall with a single user connected. This can now get scraped by the Prometheus configuration and give insight to your ASA Anyconnect environment.

Prometheus Installation

There are several of options for installing a Prometheus TSDB (Time Series DataBase)including:

Precompiled binaries for Windows, Mac, and Linux
Docker images
Building from source

To get more details on installation options take a look at the Prometheus Github page.

Once installed you can navigate to the Prometheus API query page by going to http://<prometheus_host>:9090. You will then be presented with a search bar. This is where you can start a query for your metric that you wish to graph, such as start typing asa. Prometheus will help with an autocomplete set of options in the search bar. Once you have selected what you wish to query you can select Execute. This will give you a value at this point. To see what the query looks like over time you can select Graph next to the world console to give you a graph over time. Grafana will then use the same query language to add a graph.

Once up and running, you add your Telegraf host to the scraping configuration and Prometheus will start to scrape the HTML page provided and add the associated metrics into its TSDB.

A good video tutorial for getting started with Prometheus queries with network equipment can be found on YouTube from NANOG 77

Grafana Installation

Grafana is the dashboarding component of choice in the open source community. Grafana is able to use several sources to create graphs including modern TSDBs of InfluxDB and Prometheus. With the latest release, Grafana can even use Google Sheets as a datasource.

As you get going with Grafana there are pre-built dashboards available for download.

TYou will want to download Grafana to get started. There are several installation methods available on their download page including options for:

Linux
Windows
Mac
Docker
ARM (Raspberry Pi)

The Prometheus website has an article that is helpful for getting started with your own Prometheus dashboards.

If the video just above this link does not show up, you can see the video on the Network to Code YouTube.

In the next post you can see how to monitor websites and DNS queries will include how to alert using this technology stack.

-Josh